cultural unity:
You are worried that the empire will lack transport and resources to allow it to function as a unified whole and have large-scale production. I think the key factor you need to have is cultural unity. Your empire will have a culture and society centered around the desert. THAT will be the factor that holds your empire together.
First, let's establish that it is the non-'high'-desert areas in which your populace lives. It is the minimum amounts of shepherding, agriculture, and such that allow people to live in sustainable numbers. It doesn't NEED to be easy - in fact, overcoming challenges via organization and logistics is often what moves a culture from a kingdom to an empire.
Your empire needs unifying factors that keep otherwise tough individualist desert nomads working together with soft bureaucrats and farmers. Religion is an excellent motivator, and if you have a faith emphasizing each group's critical role in society, this can tie it all together.
Your empire needs a source of wealth. Trade is the logical one, as the desert can have critical trade routes going through it. Gold hidden in the desert (mines) is also a good source of wealth. Gold can lubricate trade, and they go well together.
So a culturally unified desert society with a central faith and wealth through trade, where the neighbors are tolerated but considered outsiders, would be difficult to invade (centered as it is around a desert) and resistant to civil war. If survival depended on the logistics of the empire, people would work hard to maintain internal cohesion to preserve the whole.