Asking for a mathematical proof or theorem that AI is undesirable is oxymoronic. AI is inherently mathematical, so the task you assign is essentially to mathematically disprove the utility of mathematics.
The decision for or against AI would not be made mathematically, but ethically, I submit. Ethics and mathematics are intrinsically incompatible. That which is optimal in mathematics and that which is optimal in ethics are provably not the same, but the proof is not a mathematical proof, it is not even a logical proof. It is an moral/ethical proof.
See for instance
Logic and ethics are too often regarded as separate, if not somehow in
opposition to each other. But many great logicians, including
Aristotle, Ockham, Bolzano, De Morgan, and Russell, were capable of
incisive contributions to ethics and of heroic actions grounded in
ethical insight. Likewise many exemplary moralists, including
Socrates, Plato, Kant, Mill, Gandhi, and Martin Luther King, showed by
their teachings and actions a deep commitment to objectivity, the
ethical value that motivates logic and is served by logic. This
article explores the role of logic in ethics and the role of ethics in
logic. It is important to investigate the hypothesis that the ethics
of the future must accord logic a more central and explicit role.
Connections between ethics and irrational subjectivities must be
severed; human dignity and mutual respect can be based to a greater
extent on the universal desire for objective knowledge. Likewise it is
important to investigate the hypothesis that the logic of the future
must accord ethics a more central and explicit role. Logical
principles are important because they serve ethical goals. Logic is
peculiarly and essentially a human pursuit; the alleged disconnections
between logic and human involvement must be refuted. The caricature of
logic as a meaningless game of symbol manipulation and the caricature
of ethics as a rationalization of blind emotion must both be exposed.
Logic and ethics are in fact inseparable and each is served by
explicit recognition of its involvement with the other. [emphasis mine]
AI would be outlawed precisely because mathematics completely lacks any ethical restrictions, not because it violates any mathematical principles or that there is some mathematical theorem that proves it faulty or lacking. In fact, it is precisely because it follows only mathematical principles and not ethical principles that it would be banned. Mathematics (actuarial decisions, for instance) simply assign a number value to human life, and throw this number into a left-side right-side equation.
In ethics, on the other hand, there is no left-side/right side equation. it is all about an overall pattern, and balance within the pattern.
Take a self-driving car, for instance. An unavoidable accident, the outcome of which is inevitably either a dead child pedestrian or a dead passenger in the car. A mathematical/actuarial decision and an ethical decision are worlds apart. It is precisely because a purely mathematical decision can lead to a blatantly unethical decision that a purely mathematical/logical AI system would be outlawed.
But not based on mathematical principles, rather based on ethical principles.
Asimov's positronic brain', and the Three laws of Robotics, clearly welded ethics, logic, and mathematics together, and many of his robotics novels investigated the interplay between them. In fact, if I recall properly, AI was specifically banned from the Earth in many of his novels, castigated to the realms of off-Earth scenarios. It was ethical considerations, in point of fact, that necessitated the Three Laws be intractably embedded in all AI devices, completely over-riding any purely mathematical decisions.