One species has two genders, like humans, the other has three(essentially male, female, and one without breeding capabilities that first appears as a genetic quirk but quickly becomes more prominent until it is equal to the other two). It is in an advanced society with two different regions, one for each species, on opposite sides of a small planet.
Would a society with two different species that can interbreed be viable?
Sure it's possible, by a number of processes the most obvious being hybridisation. The process by which two distinct species combine to form mostly infertile offspring (Vis-a-vis mule)- but just occasionally they are viable.
There is evidence of hybridisation between modern humans and other species of the genus Homo. In 2010, the Neanderthal genome project showed that 1–4% of DNA from all people living today, apart from most Sub-Saharan Africans, is of Neanderthal heritage.
But you can't expect them to be a perfect halfway house:
Hybrids are not always intermediates between their parents....but can show hybrid vigour, often growing larger or taller than either parent.
Another intriguing way would be through chimerism:
A woman was found to have blood containing two different blood types. Apparently this resulted from her twin brother's cells living in her body.
However it may be more obvious than this as it could occur in your creature's skin, giving a mottled blend of the two skin types you mention above by mosaicism.
Culturally, would either society accept a crossbreed? See Nazi Eugenics for an idea of what can happen. Would the prevailing cultural atitudes drive these unique creatures to the fringes, desperate, and in hiding - would they have a political voice? Allies in the general populations? Would some be driven to violence? Only you can decide.
Here's further reading to fuel your immagination:
In biology a species is defined as
"a group of living organisms consisting of similar individuals capable of exchanging genes or interbreeding" or "the largest group of organisms in which any two individuals of the appropriate sexes or mating types can produce fertile offspring, typically by sexual reproduction."
Therefore, if they can interbreed, they are by definition basically not different species but only two very different looking races of the same species i.e. variants of the same species. This does not mean that they cannot have completely different physical appearances or capabilities; as an example consider how different a chihuahua is compared to a St. Bernhard, but they are basically both dogs and can interbreed.
Short answer: Sure. Why not?
Longer answer: Horses exist. Donkeys exist. Horses and Donkeys can interbreed. Do donkeys become more horselike over time? No. Why? The offspring of an inter species breeding are typically infertile, and so can’t pass on the traits of either species. They can mess with breeding patterns by ‘stealing’ potential mates, but otherwise they have no further genetic impact.
If the two species can interbreed and produce fertile offspring then Alex2006’s answer applies: they are not two distinct species. You can get fuzzy on the whys and wherefores of when you choose to ‘split’ the species, but that’s mostly an arbitrary ‘because I said so’.
As for the social implications of whether two sapient species can coexist peacefully without the word ‘genocide’ getting used a lot: we can’t say much. But from a biological perspective if you simply have interspecies children be infertile then your problem completely goes away.
It depends on too many variables.
As alex2006 pointed out, by definition you only have one species. Let's call their makeup as XY and XX for the group 1, and xy and xx for group 2. Equal symbols mean female phenotype, different symbols mean male phenotype.
The "third gender" could arise as a variant of Klinefelter's syndrome where xxy "males" do not have any impairment at all, except for sterility, very low sexual drive and androgynous appearance.
A XY male and a xx female may interbreed and have Xx female or xY male children. Possibly xxY children too.
What happens to Xxy, XXy and xxY children? It all depends on the exact nature and interaction of the sexual chromosomes.
- the combination is immediately fatal: the foetus does not develop and you don't even get a pregnancy. To all intents and purposes, "Group B" only breeds true. An intermixing of the groups will then lead to a diminishing of the third gender group, as less of them will be born.
- the combination is later fatal: the foetus is born dead or it even damages the mother, or makes it sterile or selectively sterile (for example by a mechanism akin to mother-foetus immune incompatibility); the xx mother will from now on only be fertile with pure xy males. This will easily lead to resentment and conflict.
- the combination is harmful but not fatal: interbreeding has a high chance of children being born handicapped, or worse. Either group may blame this on the other.
- nothing much happens: the two groups can interbreed and the "third gender" demography might change, or not. The other physical characteristics will naturally blend, and there's no biological reason for racism to develop (this still doesn't mean it couldn't).
There are several other possibilities with the same, as well as wildly different, outcomes. Just for kicks, imagine that XXy children are born with some marked advantage (longer lifespan, higher intelligence...) and decide they're the new Ubermensch?
I understand "viable" in your question as "capable of long-term existence" with the biggest threat being permanent hybridization and ultimately unification into one species.
Hooded crows and carrion crows are related closely enough to interbreed. Their ranges overlap, so the two species meet. For years they were thought to be subspecies of one species and hybridization to occur widely. However, it turned out it's not so. They're classified as distinct species now. What's most important for you, they can crossbreed, but they choose not to. Hybridization is rare.
That's pretty much your world. The two groups were one species very long time ago. They were separated (eg by continental drift) and drifted apart, genetically. Now (thanks to technology?) they can meet again.
IMHO your hybrids should be at disadvantage. In our world, hybrids usually are at disadvantage, sometimes completely sterile, sometimes just having slightly more trouble breeding. A hybrid can have an advantage personally, eg. combine physical strength on one side with intelligence of another. But, for the species as a whole it's reproduction that matters the most. Even small reproductive disadvantage is enough for purebloods to always outnumber hybrids, pushing them into outskirts and maintaining distinction between two societies. That's enough to make your society maintain status quo indefinitely, what I interpret as "viable".