Poverty.
The end of the industrial age resulted in most labor being surplus; the work you could do with your brain or your hands wasn't worth the resource diversion to feed you compared to just having an AI with a teleprecense robot or bioengineered meat-puppet do the work.
So now we have legions of the poor. Their only hope for sustenance lies from the charity of those with resources, or providing services to people with resources that robots/meat puppets/AI cannot.
As technology is incredibly powerful and planets are fragile, the halves are afraid the despair of the halve nots will destroy what they have. So ubiquitous surveillance blocking them from getting access to non-controlled technology is deployed. Controlled technology is managed by AIs loyal to the noble cast to prevent it from being useful against the noble caste.
On the other hand, your ability to support peasants (or provide peasants with enough resources to self-support) is a status indicator among the nobility. And the more resources you can get them to provide you with (even though it is mostly worthless) the better you are at it. So you have peasants using restricted amounts of technology to work the land. The entire peasant system is a game the nobles play with the lives of the peasants, and viewed as more humane than genocide, because "what else are peasants good for?"
The peasant class is not allowed to fly, or use projectile weapons. A castle on the ground is an effective defence against whatever they can do.
Gentry classes also exist, playing more complex games. Knights who basically fight and die for the amusement of the nobles, courtiers who amuse the nobles, etc.
There are nobles who have disconnected from the world, but such beings tend to fade into irrelevance; either they leave planets, or they build small fortifications/rockets (protected by treaty, which their AI constantly monitor) where they while away their days playing with virtual worlds. Some don't have much in the way of peasants yet still interact with the world sometimes; these wizards are quirky and dangerous to deal with.
The nobles and their games are quite serious; but nobles who don't obey the rule of only permitting peasants controlled technology are subject to summary punishment by their own AIs.
A good chunk of Magic (maybe all of it) is then controlled technology that acts in ways that don't really make sense, because it isn't supposed to. It can do far more than it does; its restrictions are arbitrary.
Quite possibly even the nobles are just playing a game arranged by post-human intelligences, hence the stability of their pacts, using trivial amounts of resources from the post-human perspective. They think they are rulers of the universe, but are just peasants with a few more toys.