Short answer
The factions not using desert camouflage are tritanopes.
Background
Maybe the planetary inhabitants using the non-camouflage colors aren't trichromatic species but instead miss the blue cones in their retinae. They either may not have evolved multichromatic vision, or alternatively, they lost their blue cone somehow. If they didn't evolve trichromatic vision (as did your camouflaged faction) they may be less advanced evolutionary spoken in terms of color vision. Note that their evolution then has followed a different evolutionary path than homo sapiens, given that we obtained trichromacy through a doubling of the low-frequency cone due to a dual red/green cone type. Hence, they may have followed a different evolutionary pathway unknownst here on earth. We earthlings diagnose blue-cone deficiency as tritanopia; a recessive autosomal trait in people lacking blue cones.
To expand on tritanopia, i.e., trichromatic species that loose their ability to detect blue wavelengths, miss cones with the short-wavelength (high-frequency) blue opsin. It is a rare autosomal recessive trait leading to blue color blindness. These people are definitely not 'color blind' (bad terminology), as they still have color perception, but it is different as their colors are perceived as follows (source: Colour Blind Awareness):
People with reduced blue sensitivity have difficulty identifying differences between blue and yellow, violet and red and blue and green. To these people the world appears as generally red, pink, black, white, grey and turquoise.
Also see Fig. 1. below for a visualization of tritanope vision.
Note that their ability to distinguish yellows is due to the opponent form of color vision; we as trichromats perceive colors through a red-green and yellow-blue opponent axes (known as the Hering theory of color vision). Stripping away blue cones disrupts the yellow-blue axis, making the perception of yellows also difficult as a consequence. The obvious problem is the other factions still use blue colors in your scenario, which should be adapted then in your story line.
The most common form of color blindness on earth is red/green color blindness. This form of dichromatic vision is shared among many other primates. Our trichromacy is thought to have been favored in evolution to discern ripe fruits (yellow/reds) from their unripe counterparts (green).
As another alternative, the factions not using desert camouflage may have monochromatic vision (black-and-whites only), but that would not make sense, as apparently they still apply specific colors, i.e. red, blue and green in your story abstract.


Fig. 1. Left: trichromatic vision. Right: tritanope vision. source: Colour Blind Awareness