For people to live in a desert of various ores that includes heavy metals, respiratory protection would be essential, as would be an oasis protected from dust. Any airborne heavy metal dust getting into the inhabitants lungs, or into their digestive system via contamination of water or food would lead to a short and miserable life.
Airborne dust can be dealt with using respirators, which needn't be too complex - a few layers of finely woven cloth or a container of a bundle of fibres like cotton can trap dust effectively. The dust masks need to seal fairly well, so don't expect your desert dwellers to have beards - people who use respirators on a regular basis for work can gain significant exposure via beard-induced poor sealing around the mouth and nose.
The problem with dust is that it gets on the food and in the water too. Water can be used to wash foodstuffs, which would probably be cooked in closed containers to keep out dust, but keeping water clean is a problem. Many toxic metal ores can dissolve in water, so if there were underground aquifers that were toxin-free, they could be sheltered to keep out dust for use as a primary water supply. Alternatively, if the only water available was contaminated, distillation could be used to remove contaminants. However, this would take energy in the form of fuel (or magic).
The inhabitants would have to know which ores were toxic so that they could avoid eating any plants growing in them, since such plants would not be able to be simply washed clean, and any animals from these areas would likely be toxic too.