Estrous Cycle
In the animal kingdom we have the "estrous cycle" (when females can get pregnant) and the "rutting period" (male mating season). So here's my thinking:
Elves are almost like 4 separate species, represented by 4 different estrous cycles. The "species" is revealed by the hair color. This fertility trait is a very dominant trait carried from the mother and coupled tightly with hair color. So blond elven women are only fertile during a time of the year that means they will always bear their children in summer, and their children will be blond and (if female) have the same estrous cycle as the mother.
I kinda like this idea because I could see it developing as a form of population control. Elves are long lived, so birthing rates are a big deal and overpopulation could be a serious problem so nature's solution (perhaps aided by the elves themselves via some careful breeding over thousands of years) resulted in this pattern where only one-quarter of the female population is ever fertile all at once, and even then only during limited parts of the year. (e.g., assuming an earth-like cycle, summer babies are born in July, with a 9 month gestation period, meaning the blond women are only fertile in November. That's it. Just November.)
Being fertile year round lead to overpopulation. Being fertile only part of the year makes things too brittle -- "oh population is good now, no need for mating season this year. ... Oh no a disaster happened, we need to re-engage population growth but we have to wait a full year now". Having 4 mating seasons with 25% of the women being fertile per season let the elves have good population control but also better flexibility than once a year.
Or if you want it based off the men then do the same thing but, er, "rutting period", with the dominant gene coming from the men. But I think it makes way more sense to do an "estrous cycle" because while one man can make many females pregnant, this estrous cycles is a hard limit on population growth rates, so you have something that is (I think) genetically feasible and also has a bit of world building sense.