See also the white-throated sparrow that had a gene mutation that has, effectively, created four genders. There are still the standard "male" and "female" sexes, but a separate gene also drives the nesting and young rearing behaviors.
Offspring are genetically viable with off-pair pairings. However, due to the behavior drivers, the offspring don't survive as the pair needs a matched set of opposites: aggressive and territorial vs. nest-watching. Two agressive types don't sit on the eggs. Two nest-watching types fail to fend off predators.
See this article for more details that I might've forgotten.
As a result, the behavior gene is starting to drive phenotypical differentiation, so you get α-males and β-males looking different so that they can be identified by their counterparts: the β-females and α-females respectively, which also have phenotypical expressions for the same reason. In a few million years it wouldn't be unexpected for α/α or β/β pairs to be genetically non-viable (in the same way that male/male and female/female pairs can't reproduce) in the way you're looking for.