The Crystal Desert is a Sahara-like expanse of sand, dotted with numerous large clusters of crystal, and containing numerous monolithic sandstones. What would we expect Thra geologists to determine are the processes that led to its formation?
It might be salt!
The most likely explanation is one that occurs frequently on Earth already. Here, massive salt deposits are formed when oceans are temporarily connected to inland basins, flooding them with seawater and massive quantities of salt. Later, that tenuous connection is severed and the salty water is left to evaporate, leaving behind crystals of all shapes and sizes. XKCD's What-If article on a similar topic is also required reading.
Salt is a highly likely candidate, but is a little underwhelming coming from the realm of science fiction or fantasy where such crystals are large enough to build houses in. The main disruptive force when creating crystal deserts out of salt is that the water is often turbulent, stirred up by wind and tides. Crystals grow best in hypersaturated, tranquil waters, so regular disruption will place a maximum probable size on the crystals you'll get from a desert formed like this. Many of the pictures of Dead Sea salt, for example, are slightly deceptive - these crystals are rarely found more than an inch or two in size.
Photo: Ievgen Sosnytski / Shutterstock
To maximize salt crystal size, one tactic might be to use an incredibly deep basin that's later uplifted. Crystals at the bottom of this basin would grow slowly but steadily, with less disruption from the surface and the added bonus of the extra-salty brine that sinks to the bottom
It might be glass!
Fulgurite, better known as lightning glass, is an alternative option. This wouldn't be crystal exactly, as glass is technically an amorphous solid, but it'll look much the same to travelers. When lightning strikes silica-rich landscapes like beaches or deserts, it melts the silica and produces glass. Once the nearby non-lightning-ed sand is eroded away, you're left with the glass itself sticking above the surface.
Photo: The Mineralogical Research Company
Alternatively, meteor impacts also produce glass if they strike deserts or other silica-rich areas. This aptly named impactite is often larger and more striated than fulgurite, but much rarer and harder to establish on a world with pre-existing life. Credit goes to this Worldbuilding question for this idea
Again, this is probably too small-scale for what you're hoping for. Fulgurite specimens rarely get above a few feet in size, despite what the internet would like you to believe.
One of the joys in working with Worldbuilding is that we can play with the environmental variables a little bit. Here, we'll need really big lightning bolts. I don't know enough of the physics involved in what exactly causes lightning, but neither does anyone else.
From NOAA's website:
The creation of lightning is a complicated process. We generally know what conditions are needed to produce lightning, but there is still debate about exactly how a cloud builds up electrical charges, and how lightning forms. Precipitation and convection theories both attempt to explain the electrical structure within clouds.
Precipitation theorists suppose that different sized raindrops, hail, and graupel get their positive or negative charge as they collide, with the heavier particles carrying negative charge to the lower part of the cloud.
Convection theorists believe that updrafts transport positive charges found near the ground upward through the cloud while downdrafts carry negative charges downward.
If even NOAA scientists don't know exactly what's happening, it's simple enough in a fictional story to assert that your world has enormous lightning storms that can fuse huge areas of the desert into glass. The bigger the lightning, the larger the area fused, although you'll start getting gaseous silica above 2,200 oC. Then again, lightning strikes are getting bigger on Earth too so maybe all we'll have to do is wait and find out.
Alternatively, you could have a desert-wide area bombarded with meteorites during something like a junior version of the Late Heavy Bombardment but this one has the difficulty of 1) probably wiping out all life on the planet and 2) each successive meteor strike would likely shatter any pre-existing glass, turning it back into sand.
It might, in fact, be crystal!
This is probably what you wanted all along, right? Beautiful images of Naica's crystal cave have been circulating the internet since its discovery in 2000, and the crystals seem to be exactly what you're looking for. "Luminous beams of gypsum bigger than telephone poles, nearly 12 m long and 1 m wide, jutting in all directions out of the brown limestone walls, floors, and ceiling."
The Naica cave is truly a marvel. Those crystals are estimated to have been growing for literally millions of years in the perfect conditions. The connected Cave of Swords has more abundant, but less sizable crystals because the conditions were slightly different, and many places have no crystals at all.
This type of crystal formation is certainly the most fantastic, in the literal sense of the word. However, the conditions need to be absolutely perfect, and maintaining those over an area the size of the Sahara is nigh-impossible. Given also the time required, this is a less-feasible solution - or at least less likely. Then again, it's your story! There's a lot to be said for doing what works best, and if a few geologists complain it won't be the end of the world.
This isn't an exact match, but a similar situation could do it.
Essentially, water disolves the minerals from surrounding mountains into a central depression. While there is enough seasonal rainfall to gather this mineral-enriched water to a central area, during the dry seasons much of the water evaporates and leaves the concentrated minerals to grow crystal formations.
These formations break down into smaller and smaller grains, eventually forming the sand which the wind forms into dunes.