At first glance many of your requirements seem doable, but upon further thought it’s really very difficult to come up with reasonable explanations for them. That said, nothing is impossible, it’s just the explanations might not exist within the realm of classical genetics. I’ll go through each of your requirements and explain why simply having extra chromosomes or dominant alleles won’t be enough to explain the phenomenon. I’ll also give some theoretical considerations to non-classical genetic explanations.
Female and male offspring of the same parents may have different
races, e.g. human mother + vampire father = vampire daughter, human
son.
I’m assuming that you mean that offspring of some biracial matings have a chance of being the race of either of their parents. Let’s explore how this could happen. Say there is a single genomic feature that makes people vampires. It could be an extra chromosome, or an allele on a chromosome. We’ll call it V. V acts in a dominant fashion so anyone with a copy of V will become a vampire, and anyone without a copy will be a human. If a Vampire is homozygous for V, meaning they have 2 copies of it, all of that vampires haploid gametes will also contain a copy of V. This would mean that all of a vampire’s children would have a copy of V, and therefore be vampires. Not what you want. Let’s say instead the Vampire parent is heterozygous for V, so maybe they only have one copy of the V chromosome, or one copy of the dominant allele. In this case only half of the vampire’s offspring will receive a copy of V. So half will be human and half will be vampire. That’s perfect right? The problem with this is that if a vampire is only heterozygous for the V trait, then vampires won’t “breed true”. When two vampires mate there will be a 25% chance that they have a normal human child! We can’t have that! And to make matters worse, another 25% of the vampire’s couple’s spawn will receive 2 copies of V and then be homozygous for V. When those children mate with humans they will always have vampire offspring. I’ll spare the reader the trouble of going through all the other possibilities, of making V recessive or codominant or multigenic. Classical genetics isn’t going to work for us.
The central problem is that when two organisms produce offspring, half of the progeny’s genome comes from one parent, and half from another. We can fiddle with things to make what we want happen to this first generation, but when they go to reproduce, they have an equal likelihood of passing on their father’s chromosomes as they do their mother’s. We need to make some exceptions to these rules.
Perhaps you’ve heard in the news about a new technology under development called “gene drive”? The idea is that you can make an allele of a gene that is capable of “infecting” or copying itself onto another allele. So when an organism is heterozygous for a gene drive allele, the gene drive allele modifies the other allele to become a gene drive allele as well. The organisms is then homozygous at that allele. In this way the technology would allow the rapid spread of a certain allele in a wild population of organisms. It’s good for our purpose because it lets us get rid of those pesky heterozygotes. This sort of system actually already exists in nature in a couple forms. Homing endonucleases and transposons both have this sort of functionaility. Gene conversion is another way this could happen. So imagine our vampire allele V is a gene drive allele. All vampires will be homozygous for the allele and when they mate with humans all of the offspring will be heterozygous, the V allele will overwrite the human allele and all of the offspring will be vampires. Now, imagine the human counterpart to V, allele H, is also a gene drive allele. When a heterozygous embryo is created from a vampire-human cross it will contain both H and V. V will try to overwrite H and H will try to overwrite V. Say the chance of V or H winning is 50/50 and now our vampire human cross is capable of producing both vampires and humans, both of which will be homozygous.
A female of one race can be impregnated by a male of another race, but
it fails with switched races or sexes, e.g. vampire mother + human
father = no offspring.
This is definitely the easiest of your requests. Although it still is difficult to achieve with classical genetics there are some well-documented ways in which this could occur. The simplest is some sort of non-genetic barrier to reproduction. An example might be if human sperm is incompatible with the hostile environment of a female vampire’s reproductive system. Or if the human sperm was unable to recognize the zona pellucida of a vampire’s ovum. There are many ways in which fertilization could be blocked in one mating but still work in another. If you aren’t keen on non-genetic barriers, another option is imprinting. Imprinted genes are epigenetically silenced depending on their parent of origin. Humans have a handful of imprinted genes where expression from allele from the mother or father is silenced. If a human male silenced an important gene that a vampire male did not, then a male-human female-vampire coupling might result in a serious deficiency in that gene, which could make the embryo non-viable.
The race of certain offspring may be determined by the race of the
parent with the same sex, e.g. dwarf mother + gnome father = dwarf
daughter | gnome son.
This one isn’t easy either. The immediate solution that comes to mind is “Let’s put it on the sex chromosome!” and it is very wrong. Say the gnome Y chromosome contains an allele D that makes the sons who receive this Y be gnomes. Now, whenever a gnome mates with another race his sons will be gnomes and his daughters will be the race of the mother. That’s great right? The problem is, how do you ever get female gnomes?
You might think maybe we can explain this with imprinting, but imprinting only gives us a parent-of-origin effect. We need to correlate the parent-of-origin with the sex of the child, and that means putting imprinted alleles on the sex chromosomes, but I still don’t really see how it could work.
Some races are restricted to a single gender and rely on other races
for reproduction, e.g. sphinx mother + centaur/minotaur father =
sphinx daughter.
This is your second easiest requirement. I think the best explanation for the sphinx is that the features that make a sphinx a sphinx are in the mitochondrial genome. The mitochondria are inherited only from the mother through the egg and so they get around our issue of hybrids. A sphinx egg doesn’t care about the sex chromosome in whatever sperm it receives, it just makes a female sphinx.
Some hybrid races are infertile when mating with each other, but can
reproduce with members of other races (or at least one sex can), e.g.
human + elf = half-elf, human + dwarf = half-dwarf, both are
considered halflings, halfling + halfling = no offspring, but halfling
+ human = human (or maybe halfling again).
Oh man, this is hard. Having hybrids be infertile is easy, a lot of real ones are anyways. Having them be infertile with each other, but fertile with their parent species does not make a whole lot of sense. There could again be non-genetic barriers. Say female halfling reproductive systems don’t develop quite right, but the male reproductive system works fine. A halfling-halfling pairing wouldn’t produce offspring because the female would be entirely infertile. The halfling male on the other hand could still reproduce with either of its parent races. I have to warn you though that the result of a halfling-purebreed coupling is going to be a three-quarterling and may not be what you want. If you really want to get into the complex, hard science of this sort of stuff you should also take into account Haldane’s rule.
I’ve written a lot and I haven’t gone into as much detail as I would like to explaining everything. If you have any questions or need clarification I can do my best to explain in the comments, or through chat if you’d like.
