EDIT: I unexpectedly had a possible solution to this come to me, and posted it as answer here. Please look it up and give feedback on it.
I'm trying to come up with a plausible sex determination system that has the characteristics listed below. It's intended to be general-use, i.e. the subject species could be a modified version of Homo sapiens, humanoid aliens, a nonhuman animal, etc.; for convenience, though, I'll be assuming that the species is human(oid).
- There are five sexes: female, male, and three kinds of simultaneous hermaphrodites (berm, merm and ferm, collectively known as herms).
- Sex determination involves three sex chromosomes: X (tied to females and ferms), Y (tied to males and merms), and Z (tied to herms in general).
- All of the hermaphrodite sexes are fairly androgynous in facial features and fertile in both the male and female organs; berms have the most androgynous-looking, and their physical development follows a pattern that is intermediate between males and females (e.g. on average they're between both in height), while merms and ferms parallel males and females in their dimorphism (e.g. merms develop heavier jaws, greater height, a "V-shaped" torso, and find it much easier to achieve muscle hypertrophy).
- All possible pairings between the five sexes can produce a child of any sex.
- The sex system is stable and is in no danger of having any of the sexes being outcompeted within the foreseeable future on an evolutionary timescale.
I've looked into the various existing systems, and couldn't find anything that appeared to be sufficient for my needs. For example, an XYZ system didn't work, even if I employed a second pair of sex chromosomes (like the white-throated sparrow appears to be slowly evolving to), because every version resulted in either males/females or herms being unable to produce the other.