Greed & political corruption
they are prone to adversarial inputs, backdoors, and poisoning
All these robots still have human masters, which means the robots' activities are all directed by their human masters in pursuit of the humans' interests. So, greed and corruption, which will necessarily be rampant in the society you envision, will inevitably lead to some kind of destructive instability.
In a society where each person has their needs met and is treated fairly, there is little foothold for bad actors -- they can't bribe a person because nobody needs the bribe and won't take that risk when there's really nothing to gain, and they can't blackmail people because nobody has any hidden crimes or embarrassing secrets to use as leverage. And it's difficult to persuade a happy, healthy person to participate in your evil cabal on its merits.
Your society is not like that. In your society, the vast majority of people are not getting their most basic needs met. In your society, nobody believes they receive fair treatment -- even the super-wealthy recognize their supra-legal lifestyles as such. The losers of your society includes a vast, uncountable number of worthy people who rightly deserve a lot more and know it, in addition to the regular set of people who wrongly believe they ought to be kings. Every one of these people is a potential source of or willing accomplice in an attempt to overthrow the crooked ruling class.
Very importantly, because their basic needs aren't being met, many of them will have good reason to resort to crime at one time or another. Many of them will have embarrassing but non-criminal secrets that they can't afford to have revealed because, in this devil-take-the-hindmost society, each person's grasp on basic subsistence is extremely tenuous, and not only will employers have unchecked discretion to fire any employee for any reason, other people who don't have jobs will have powerful incentives to smear someone in hopes that they'll lose their job and create an opening.
In the real world, Nancy Pelosi once said that the United States needs a strong Republican party. People were puzzled by this (because she's a Democrat), but she was right, and the reason is this: when a person or organization is weak or dying, it does not simply curl up in a ball and wait to expire. It becomes desperate to survive, and in its desperation it will cross any line, break any law, sign any deal with the devil to stay alive. And in doing so, it gives the devil a foothold from which to do further, greater evil. And any idiot must understand that this is precisely why the devil made the offer in the first place.
What will probably happen in your society is one of two things:
- Some unscrupulous princeling will have the bright idea to exploit the robots' backdoors, turning them all into his private army overnight so he can establish a monarchy. His success is not guaranteed, but it will be an absolute bloodbath, in part because he will necessarily be one of those "après moi, le déluge" guys who doesn't care if half the population must be butchered to create a throne for him to sit on.
- Somebody who is sympathetic to the downtrodden will exploit the robots' weaknesses. Their goal will depend on their sympathies. If they come from the lower classes, they'll put the robots to violent use: if this person has actual mental discipline, they'll have a sound strategy to overthrow the rich by force, but if this person is ruled by their passions, they will probably settle for murdering the upper class. If this person comes from the upper classes, they will avoid using violence against the tyrants, and will attempt some ineffectual half-measure such as sabotaging all the robots so they temporarily can't work, or giving them all a "hilarious" hip accent (because everyone who matters knows that pwning the rich is an acceptable substitute for deposing them). The violent course might very well work. The pansy thing will just alert the rulers to their vulnerabilities, and they will promptly improve robot security and change the laws so they can seize all the robots owned by the non-rich.