I've seen several people independently come up with the idea of three or more sexes where one sex is responsible for brooding the young (pregnancy, pouch, feeding, something along those lines) but doesn't actually contribute any DNA. Can you suggest some an evolutionary mechanisms that would support the development of a "brooder" sex that provides caregiving to young without passing on their genes?
Kin selection. They aren't passing on their own genes directly, but they are helping pass on the genes of their close relatives, which is almost as good. This would make the most sense in the context of eusocial creatures, like bees or ants. In situations where lots of individuals are sterile, and it makes sense because they support closely-related breeders, it would certainly work at least as well to distribute the effort of actually gestating the young as well. Perhaps if nakee mole rats (eusocial mammals) had evolved from marsupials instead, this is exactly what we'd see.
They aren't actually the same species. Rather, some normal disexual species parasitizes a second species to raise its young (like cuckoo birds, or parasitic wasps), which is culturally interpreted as a third gender due to their importance in reproduction and limited historical knowledge of genetics. This could easily evolve from straight parasitism to a mutually beneficial arrangement if the brood species is effectively domesticated, so their parasites actually take care of them and improve their own 0chances of reproducing.
A hybrid species complex. Option 2 actually requires 4 genders (one for each underlying species, so the brooders can reproduce themselves), unless the host species is either parthenogenic or hermaphroditic, but this could be eliminated with a suitable hybridization complex, resulting in one variant which is either exclusively male or exclusively female. To provide brooders, you would want an exclusively female variant. The brooders would pass on their genes either by genetic cloning which can only be triggered, for ancestral reasons, by copulation even though the donor DNA is discarded, or they can actually reproduce sexually with males as well as hosting embryos produced by non-brooding females.
Helpers at the nest is a term used in behavioural ecology and evolutionary biology to describe a social structure in which juveniles and sexually mature adolescents of either one or both sexes remain in association with their parents and help them raise subsequent broods or litters, instead of dispersing and beginning to reproduce themselves.
In your scenario, a breeding pair will produce males, females, and one or more "brooder sex" individuals. The brooder sex individuals will function like the immature jays in the scenario above - helping to feed and defend their brothers and sisters. A brooder from the prior generation would remain with its parents as these immature jays do.
Immature "nest helpers" contribute to their own genetic fitness by improving the survival rate of their little brother and sisters, who carry the same genes as the nest helper. So too the brooder sex - by improving care to the young the brooder improves the genetic fitness of these young, and so its own genetic fitness.
One could make a case that postmenopausal females of our own species are exactly a "brooder sex" as you describe. I am a big fan of Sarah Blaffer Hrdy and her social anthropology studies about this. A postmenopausal female is not going to have more children of her own, but her genes are represented in her children and grandchildren. She improves her own fitness by keeping them alive. The presence of capable, nonreproductive individuals with a stake in the survival of babies and children improves the genetic fitness of those children.
The same reasons & processes that cause pack animals to babysit siblings, nieces & nephews (cf african wild dogs & wolf packs where there's only one alpha female that reproduces) & resulted in non-reproductive drones in insects (ants, termites & bees etc).
They share DNA with at least one of the sexually reproductive pair.
That's the evolutionary route to & reason for this, no other will work.
That was the answer to your question / the rest is just musings on the theme.
The first offspring of a fertile female might be non-reproductive daughters that can carry further offspring so that (as she doesn't have to bring them to full term herself) their mother-queen can churn out offspring faster.
Maybe they reproduce like marsupials making it simple to transfer young to a non-fertile daughters pouch or perhaps the mother transfers the fertilised egg to their womb with an ovipositor.
Some ant species kidnap workers in egg or larval form from other ants nests & these kidnapped ants become workers for the kidnappers nest (taking advantage of their victims instincts) so the same could apply here.
In an intelligent species that originally evolved this way brooders might hire themselves out.
Note : If the 'brooder' doesn't share DNA with at least one of the genetic parents it's a parasitic reproductive strategy like a cuckoos' & you couldn't by any stretch of the imagination call those brooders a third sex, they're most likely a different species with their own reproductive process that's been hijacked to serve the others.
A small pedantic niggle about nomenclature : If it's not contributing DNA it's not really a third sex biologically speaking, insect drones aren't a third sex, they're infertile females, though such drones could be infertile males in a fantasy species (as evidenced by sea horses & some other species).