Phemogian are a seven-sexed species that passes gametes in a chain. Each sex can inseminate one other sex and be inseminated by another sex. Each phemogian has three parents, with the last parent in the chain completing the process.
What would be a possible sex determination system that could explain this?
The breeding happen in stages between specific sets of the species
The genetic structure of this species is very complicated, even so complicated that the base building blocks are not possible to build by only two of their kind. You only have one birthing sex while the other sexes are intermediate stages. To fit that only 3 sexes are required I would suggest a pyramid scheme.
So you have sexes A,B,C,X,Y,Z and F where F is the only species given actual birth. Sex A only has sex with sex C, sex C has sex with F, and so on. Sex C and Z are intermediate stages who will not give birth, only F gives birth.
So what happens when to sexes come together. So if sex A has sex with sex C, (sorry lot of sexs :D) sex A will pass its genetic code to C. C will combine the genetic code into a new gametes type including both the genetics of A and C. Subsequently if C and F procreate it will pass on the genetic code of A as well as its own. Thus sex F will birth a child with the genetic codes of A, C and F included. Do to the complex nature of the genetic code if different sexes try to procreate (even if it is physically possible) the genetics won't match and not gametes will be created to pass on.
NOTE: now I structured it as a pyramid but you could also choose for different system. A circle construction comes to mind where each member of the sex needs to have the gametes of both adjacent sexes to be able to create offspring. In this fashion all sexes could birth, or you can have four males and three females.
It all comes down to the genetic make up being to complex for two individuals to make up.
Caste system
The creature has a caste system, much like ants. Castes help with specialization of tasks, benefiting the species as a whole. Different from ants though they have grown in different configurations from coupling between individuals, instead of all males with one or a few queen like entities. The specialization doesn't need to be spectacular.
Other differences are that they are more like the birds of Darwin. Instead of 7 sexes, they are essentially several species closely related to each other. Each in the chain can procreate with the ones closest to it's own genes, but not further away.
Next is why they want to procreate with three species. Two species procreating is advantageous, as it allows for quicker evolution than species that procreate singularly. In addition, the handicap of requiring both the energy to procreate and the requirement of a mate filters out creatures that aren't fit enough in their environment to get both. The requirement of 3 mates can push this further, also reducing genetic defects that can happen naturally or a result of inbreeding. That means with 3 mates you can procreate closer to your family tree with less worry. If you're asking how procreation between three creatures came to be I wouldn't know, but if we can make the jump from mono procreation to a dual procreation, the jump from 2 to 3 individuals in procreation seems a much smaller one.
Some creatures like snails are hermaphrodites, having both functional sex organs. Your species can accept genetic material from another, combining it, then passing it on to a third party, who combines it as well. That way all are present can add their genetic material to the egg and it can be grown by one of the individuals.
In the end, thanks to environmental genetics and the makeup of the genetic donors, the resulting creature can grow into one of the species, possibly rejecting some other material altogether.
Seven sexes
I'm having trouble with the seven sexes part. Although I've heard that some fungi have over a thousand sexes, I never have seen any elaboration of how it works and why they are seen as different sexes. The above can approximate different sexes in the same species quite closely and might be an acceptable answer for you.
Linear-Feedback Shift Register (aka LFSR)
Numbers 3 and 7 may seem a little strange in this context, but, let's not declare an idea impossible before thinking about it for five minutes*. There actually is a simple way to arrange triplets of items into a seven-step cycle:
That's a queue of three bits, supporting a "shift" operation in which the rightmost bit is discarded and a new bit is pushed in from the left. The new bit depends on two specific bits of the previous state: If they were different the new value will be 1, and vice versa. This produces a sequence of length 7 before the triplets start to repeat (unless you start from "000", which cycles immediately). The complete sequence is left as exercise to the reader (hope the reader will not find that I messed it up).
The "bits" could be chromozomes, each of them being either X or Y (just different symbols for 0 and 1). Or maybe DNA strands with a polarity, direction? The LFSR thing, then, is their way to scramble genetic information in order to create some novelty in the offspring. Terestrial organisms do have such mechanism, which ensures that chromozomes are not just cloned from parents to children, albeit in a new combination; Instead, parts of a chromozome are mixed together, and with help of some smart evolutional engineering they remain viable. In case of phemogians, the "rightmost two" chromozomes are blended to form a new one, whose polarity/XY-ness is determined by the combination (I guess the three chromozomes in a triploid cell do not have an equal status, so it can be defined which ones are the rightmost). This "meiosis" needs to happen three times in order to build three new chromozomes, with some handwavy molecular compatibility involving the "leftmost chromozome" to enforce the LFSR rule.
It also probably means that the child sex is fully determined by the combination of parents; So, a Sil, a Trem, and a Kiim can only spawn Jeks.
Some remaining questions: Why would evolution bother with this? Seems tricky, but anyway, evolution came up with 13 or 17-year reproduction cycles, so it may have some affection for prime numbers...
What about the "000" combination? They may be non-viable (boring). They may bear some Down syndrome. Perhaps they existed once as healthy members of the species, but have separated into a new species once the LFSR law kicked in (it prevents them from interbreeding with the others). They may be the phemogian version of non-binary gender.
*And who the hell said the thing with five minutes?
Update: Got in drawing mood today (and calculating):
Let's call 0 and 1 the possible sexual genes. Then, one individual can be 001011, and produces 00, 10, and 11 gametes.
When the first individual inseminates the second one, the gametes recombine and fuse, becoming tetraploid, but it's not enough for an individual: enter the third individual, whose diploid gametes recombine and fuse with the other four, completing the set.
The individual's sex will be determined by its 6 sexual genes, grouped in three: A = 000, B = 001, C = 010, D = 011, E = 100, F = 101, G = 110. 7 sexes. Handwave that H = 111 is non-viable. So, each individual's sex is encoded into two letters: BE, FA, CC, and so on.
Add a dominance rule to establish what letter(s) dominate others, to determine the sexual fenotype.
This should be enough for any 3 individuals to procreate, and still determine the sex of the progeny.
Divergent genital morphology.
Genital shapes evolve quickly and can be engines for speciation.
Divergence in genital morphology may contribute to mechanical reproductive isolation in a millipede
Species-rich taxa usually display significant divergence in traits related to reproduction . This pattern has led to much debate over whether sexual selection may underlie rapid divergence in sexual structures and, therefore, be an overlooked “engine of speciation” . Genitalia are arguably the most variable of all sexual structures, with striking differences apparent among taxa, including between closely related species . As with other sexual traits, it is thought that intense directional sexual selection drives the rapid evolution of divergent genitalia among isolated populations , potentially leading to an increased frequency of speciation. The classic hypothesis for genital evolution posits that genitalia function as “lock-and-key” structures, which are subject to stabilizing selection, enforcing mate recognition and species isolation...
Your creatures are triploid as you say, with haploid gametes. They are hermaphrodites with structures capable of emitting and receiving spermlike gametes as well as a structure capable of allowing maturation of a completed 3-gamete zygote into an egg or embryo.
The restrictions on possible mating partners are simple matter of genital morphology according to a "lock and key" system. The described interactions for each sex are the only ones possible given the genital morphology of the participants.
This would mean that artificial inseminations and progeny could be possible between 3 partners for which their genitals would make it biologically impossible. That is the case for some spiders which are different species by virtue of different genital morphologies. Hybrids can be produced via artificial insemination.
