You can't even go as far as to remove vegetation.
You said fantasy-ish, so I assume you mean that human life on the planet is not supported from space. If it is, we can allow some more "barren-ness", but without space support your humans need:
Oxygen is produced by plants or algae. Since you don't want big bodies of water on your planet, you will need plants. Also, your humans will need water to drink, so there need to be fresh water sources. And they will need food. On a barren, desolate, desert world, you can't grow crops. So they need to hunt. So there need to be animals. Animals need to eat something. Animals can't all eat other animals, so there need to be plants.
And finally you need shelter. Shelter can be constructed out of many things, but without wood and in a barren world, if you don't want to live in caves, you are going to have a very bad time finding any shelter. No plants and no water means nothing to hold the ground together, so the whole freaking planet would be covered by dust and sand. It would be super hard to find clay or a cave entrance that is not buried under sand in the first place.
So I guess you need to tone it down a bit. Give your planet a sizable ocean - made out of salty, toxic, algae-ridden dirty water that kills you quicker than not drinking at all. That produces oxygen. Make some animals live at the shores, and some nasty creatures in the depths. Make all of them poisonous.
Then give your planet some plants. At least SOME. Some kind of grass, maybe even a few bushes and a few small trees. It will contrast the barrenness of your world better, and make a more believable environment. Give your planet some animals. Make most of them dangerous, poisonous or just super unfriendly, but make it in a way a lone traveller in the desert might just find those 3 scorpions that you can eat (if you know how to prepare them), make him find a few little trees to find a poor shelter for the day.
If your world is just a giant brick of rock and sand with an atmosphere that is for some strange reason breathable, it is not habitable by any standards. Mars sure isn't, and what you seem to want reminds me a lot of it.