Let me provide some insight to my magic system. 75-95% of people have magic powers. (Depending on where you are.) These powers manifest themselves any time from birth to the end of puberty. These magical powers fall under seven broad categories. Air, water, earth, plasma (fire/lightning), plants, light, and animals. Animal magic is tricky, because living things resist magic. But, many animal mages can influence animals, and some can control dead animals (this is called necromancy and is a big societal no-no.) The strength and scope of magic varies from individual to individual. But, these are the most common in order. Earth, water, air, plants, animals, plasma, and light.
These magical capabilities are like the show Avatar, the last airbender, in how controlling magic works. However, in my book magic is much less powerful. For instance, an earth Mage may only be able to influence as much dirt as they could lift, and lifting this dirt would take about as much energy as doing it by hand, except you don't have to touch it, and it could be a perfect brick shape.
NOTE - People who control magic have a magic organ instead of an appendix. This makes magic, or the lack thereof, inheritable.
So, if a person has no magic by the time they're an adult, it is legal to take them as a slave. If you can use your magic to grab a jug from across the room, why have a slave to go get it? How would people's limited abilities affect the influence of slavery? For example, if you can tear weeds out of the ground with magic, or influence them to die, why have slaves to cultivate your fields? Would this make it unlikely for regular people to have slaves?
I think that wealthy people would definitely have household servants, but what about work? With the level of magic people have, would it be more feasible to pay people with magic or feed and house non-magic slaves? (They're not like us normal people, so they don't need as good of housing or food.) What types of areas would have more slavery? For example, would there be a lot of slaves in agricultural areas, and fewer in large cities? (I would also assume that some good people would have slaves just so that they get treated better than they otherwise would have.)
This question is kind of long, and I apologize for that, let me know if there is anything I can clean up.
Edit - Pulling weeds would actually be easier with magic, because you can force the plants to move however you desire. And doing things is easier with magic. It will make you tired, and you can only do it so much, but it would take less time. You also don't need to actually do anything other than exert your will over what you're controlling. You could pull weeds sitting in a lawn chair under an umbrella with a glass of lemonade (most people don't do this, because they don't have lemons and concentration is important). Also, magic allows you to do things you couldn't do by hand, such as making wind, or shooting lasers. I was describing the magical abilities of most people, not all. More powerful people would be able to move large amounts of material, or use magic for a prolonged period of time.
Edit 2 - People can exert more power than they're capable of doing physically; magic is like a muscle. (Or karate.) If you work hard at it, you can increase your power and or range. But most people don't have teachers to help them learn or grow their power. However, most people don't have enough ability to even warrant their magical instruction. People who do often end up as things like guilders, or other specialized professions. Range is normally along the lines of ten yards, but it only makes a little difference where in your sphere of influence you are using your power.