So a species has three phenotypes. If any two of the phenotypes produce offspring, it will display the phenotype of either of its parents. There is no possibly of displaying the third phenotype even if one paternal grandparent and one maternal grandparent displayed it.
How could this work under Mendelian genetics?
You're describing a monoploid (one set of chromosomes) species. With only one set of genes, there's nowhere for a non expressed allele to "hide". It's weird, since we're used to talking about diploids (or things like bees where there are both haploid and diploid individuals), but it doesn't violate any deep laws of physics, chemistry, or biology.
It won't "work" with Mendelian genetics, but that's because Mendel was describing diploids (two sets of chromosomes). It's a bit like saying grass doesn't act like wood - neither is impossible, they just have different attributes, and are described by different models.
If I had to hand-wave the cellular biology -
Pure genetics? No. Mathematically impossible. There's always a path for a recessive to show up. Suppose all three are represented by genes X, Y, and Z. You can be pure (XX, YY, ZZ) or a blend. Let's assume Z beats Y and Y beats X for phenotype expression.
Ok. Now you have two parents. Dad is XY, so he is Y. Mom is XZ, so she is Z. The problem case is when Kid is born as XX.
So... since we can't be pure, let's throw some biology in. Dad generates sperm. It is a mix of X and Y. His body could filter his own sperm to only release sperm matching his phenotype. Some sort of viral barrier kills off the X sperm. Evolutionarily, this is plausible since it results in kids that Dad is more willing to claim as his own. Mom's body could do the same for her eggs. Now Kid can only be a YZ.
You could also have Dad's sperm have some sort of chemical coating. His sperm triggers Mom's body to produce a protein matching his phenotype. Mom's body would produce a protein matching her phenotype. When egg and sperm meet, the zygote can grow if it has the right kind of protein environment. But if Kid is the throwback recessive, its protein cannot be found so the zygote starves and Mom's body self-aborts.
There are probably other variations. But you'll have to bring in something beyond raw genetics.
Depending on how limited the phenotype you're looking at is, you might make the scenario work if you can keep your desired phenotype manageable to a sex-linked trait. It will only work with traits limited enough that they can be sex-linked, since those genes are only a portion of their overall genetics - but I suppose a limited gene on that chromosome could activate, or work with other genes to create the phenotype you're looking for.
Most genes are duplicated, there are two copies (one from each parent) - this is a backup and safety net, but it will also allow recessive genes to hide, which you're trying to avoid. Sex-linked genes can avoid this redundancy, to some degree - matched genes (and therefore recessives) are only possible with females (XX), while males are defined by their unmatched genes (XY). In either case, since the genes are split by parent, and there's no room for redundancy, you can get a sort of generational cutoff, where genes are either expressed or not present, just where that lack of overlap happens
So, looking at your setup, something attached to the Y gene will always breed true to sons, and never to daughters. Something attached to the X gene can't be passed on by sons if they themselves don't show the trait - if it doesn't express itself, they don't have it. A daughter cannot pass on any portion of a trait from her father's Y gene, and a son can't pass on anything from his father's X gene. Y linked traits, therefore, would either be present or not, full stop - no recessives. X linked can be recessive, though only in daughters, so you might need to think about it a little more.
You might make it work by simply have two or three Y-linked genes (either one, two, three, or one, two, neither), for your three phenotypes, and leave it at that. Having two x-linked traits might almost work with a little tweaking, since boys would have one, the other, or neither (three states), girls could have one or the other, with having neither for the third state - you would have to assume the traits are dominant, and that those inheriting both are simply non-viable and don't survive, or else that will get tricky. Pairings between parents with different traits will simply result in no daughters, as well, but it might be possible after all that.
Of course, this is starting from earth humans - if you're building a people, you might make something more workable by tweaking the genetics. Any gene which not going to be matched can eliminate recessives - perhaps your species doesn't need redundant X-chromosomes, allowing for XO to be female (eliminating the possibility of recessives again, they will be expressed or not). Or perhaps there's a different set of genes which goes through a haploid cycle, for whatever reason. Plants, I think, alternate haploid and diploid generations - something like that will eliminate the possibility of recessives, whatever is there will be expressed or it was never present.
I'm pretty sure this isn't possible. Mendelian inheritance states that traits from BOTH parents are passed to the offspring, even if some (recessive) traits aren't expressed in the offspring. Therefore each parent has some of the grandparents' traits, and they may be passed along to the offspring.
(I've been trying to think through alternatives or workarounds to this, but the grandparents' genes always wind up in the grandchildren. I'll be very interested if someone has a solution to your question.)
The trait in question, if not the whole genome, needs to have only one copy present. It would work like the mamalien Y chromosome in not having a pair. But both sexes would have different non-pairing forms. When the complete fertilized cell is formed, one of the parent’s versions is kept and the other is thrown away.
