A different kind of founder effect - disease:
There are a multitude of diseases in the world, and viruses can cause some pretty unique ones. In conditions with poor healthcare, a minor virus no one ever noticed can become a huge public health problem. Polio was a significant virus mostly because changes in health standards meant people started catching it during a time in life where its side effects were most severe.
If ONE of your survivors had a virus that, in poor health conditions and sanitation, could infect the majority of your survivors and circulate in the population, you have a vector to cause any disability you want. Let's say your virus infected small infants during eye development and caused a defect in cones. No one noticed the virus before because it was rare, and good healthcare meant children rarely caught it. In an adult, it would be like a cold. Well-meaning survivors giving vital supplies to fellow survivors unwittingly have caused a new epidemic.
Your population wouldn't probably even understand why so many people were colorblind - the disease might not even cause any symptoms detectable in a baby. The great thing about this is that you can have it be as mild or severe as you want - some people might only have partial effects, others miss only certain colors, and a few rare people would have destroyed foveas and serious vision problems. The details are up to you, and wouldn't need to run in families (although they could, there can be sensitivities, or the disease can be hard to transmit without close contact).
The virus could also be neurological, affecting the parts of the brain that interpret color. For this, the virus could even affect adults - one day you're fine, then you catch a cold, and a week later you have red-green color blindness.
If you want to be poetic, make it a prion passed in tears - the sorrows of your survivors caused its spread.