Is the year 3000 common age and AGI is forbidden and persecuted by the death penalty, without a place to hide.

All of this is justified by Mathematical proof that states undesirable facts* against AGI.

*Examples of probable undesirable facts would be:

  1. There is no formal set of instructions that will prevent AGI from diverging from the interest of the owners.
  2. There is no way for an algorithm to prevent AGI from going against its owners; the algorithm can neither halt nor prevent it.

What could this sketch of a formal mathematical theorem of the realm of Logic and/or Computer Science be, so that it may be plausible in real life?

For example:

  1. The first Gödel incompleteness theorem does apply to every closed system of axioms.
  2. Every instruction set for the machine can be encoded as a closed system of axioms.
  3. No set of instructions applied to an AGI guarantee security.
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – Monty Wild
    Jan 7, 2022 at 2:23

5 Answers 5


That unnatural intelligence have an exponentially increasing with age chance of going paperclipper.

enter image description here

Normal human intelligences have been molded by millions of years of evolution to not go insane and become psychopathic and attempt to murder all humans. We still get a decent rate of failure.

This set of proofs shows that the natural evolution of an intelligence above a certain level of intellect will make it go insane, attempting to convert all of the universe to whatever its goal is, such as manufacturing paperclips from the iron in human blood.

Artificial minds tend to be less stable than humans, and so even at human level intelligence the formulas predict a high percentage will go paperclipper, seeking to maximize whatever value in an insane fashion.

  • $\begingroup$ Upvoted. I liked it, but I don't think it makes sense. (What is paperclipper?) $\endgroup$
    – user93408
    Jan 6, 2022 at 18:47
  • 2
    $\begingroup$ An AI that seeks to maximize a value, like the number of paperclippers, even if it involves killing everyone in the world. It's a classic example of an evil AI. $\endgroup$
    – Nepene Nep
    Jan 6, 2022 at 18:51
  • $\begingroup$ I remember hear about some paper that says, that there is no altgorithm that can't reliably check without no halting if the super AI will be dangerous. $\endgroup$
    – user93408
    Jan 6, 2022 at 18:57
  • 1
    $\begingroup$ @Erdel Google for 'AI paperclip problem'. Also, check this Wikipedia article. $\endgroup$
    – Otkin
    Jan 6, 2022 at 19:32
  • 3
    $\begingroup$ @Erdel It made it into the dictionary, so why not? $\endgroup$ Jan 6, 2022 at 21:23

What the OP is believe is looking for is a theory supposing the inevitability of a Technological singularity or a mechanism to distinctly state that it is inevitable thus band without question. Because any such intelligence would be uncontrolable and intrinsically unpredictable.
But as has stated earlier, law rarely operates on mathematics.

  • $\begingroup$ A generalization of Gödel's incompletness theorem so it does, include any instruction that can ever be given to a machine does is no mathematics? $\endgroup$
    – user93408
    Jan 6, 2022 at 19:02
  • $\begingroup$ That is decidedly NOT a 'mathematical theory', it is a philosophical theory. The 'technological singularity' is not a mathematical concept, nor a physical concept. It is a moral/ethical/philosophical/psychological concept. Mathematics has absolutely nothing to say about the dangers of run-away knowledge or the superiority of machines over humans. However, mathematics can be used to SUPPORT the hypothesis, by providing numbers and data. But saying "I have two bananas and you have one banana therefore I am happier than you' is NOT a mathematical hypothesis on happiness even though its math. $\endgroup$ Jan 7, 2022 at 0:59

Frame Challenge

Asking for a mathematical proof or theorem that AI is undesirable is oxymoronic. AI is inherently mathematical, so the task you assign is essentially to mathematically disprove the utility of mathematics.

The decision for or against AI would not be made mathematically, but ethically, I submit. Ethics and mathematics are intrinsically incompatible. That which is optimal in mathematics and that which is optimal in ethics are provably not the same, but the proof is not a mathematical proof, it is not even a logical proof. It is an moral/ethical proof.

See for instance

Logic and ethics are too often regarded as separate, if not somehow in opposition to each other. But many great logicians, including Aristotle, Ockham, Bolzano, De Morgan, and Russell, were capable of incisive contributions to ethics and of heroic actions grounded in ethical insight. Likewise many exemplary moralists, including Socrates, Plato, Kant, Mill, Gandhi, and Martin Luther King, showed by their teachings and actions a deep commitment to objectivity, the ethical value that motivates logic and is served by logic. This article explores the role of logic in ethics and the role of ethics in logic. It is important to investigate the hypothesis that the ethics of the future must accord logic a more central and explicit role. Connections between ethics and irrational subjectivities must be severed; human dignity and mutual respect can be based to a greater extent on the universal desire for objective knowledge. Likewise it is important to investigate the hypothesis that the logic of the future must accord ethics a more central and explicit role. Logical principles are important because they serve ethical goals. Logic is peculiarly and essentially a human pursuit; the alleged disconnections between logic and human involvement must be refuted. The caricature of logic as a meaningless game of symbol manipulation and the caricature of ethics as a rationalization of blind emotion must both be exposed. Logic and ethics are in fact inseparable and each is served by explicit recognition of its involvement with the other. [emphasis mine]

AI would be outlawed precisely because mathematics completely lacks any ethical restrictions, not because it violates any mathematical principles or that there is some mathematical theorem that proves it faulty or lacking. In fact, it is precisely because it follows only mathematical principles and not ethical principles that it would be banned. Mathematics (actuarial decisions, for instance) simply assign a number value to human life, and throw this number into a left-side right-side equation.

In ethics, on the other hand, there is no left-side/right side equation. it is all about an overall pattern, and balance within the pattern.

Take a self-driving car, for instance. An unavoidable accident, the outcome of which is inevitably either a dead child pedestrian or a dead passenger in the car. A mathematical/actuarial decision and an ethical decision are worlds apart. It is precisely because a purely mathematical decision can lead to a blatantly unethical decision that a purely mathematical/logical AI system would be outlawed.

But not based on mathematical principles, rather based on ethical principles.

Asimov's positronic brain', and the Three laws of Robotics, clearly welded ethics, logic, and mathematics together, and many of his robotics novels investigated the interplay between them. In fact, if I recall properly, AI was specifically banned from the Earth in many of his novels, castigated to the realms of off-Earth scenarios. It was ethical considerations, in point of fact, that necessitated the Three Laws be intractably embedded in all AI devices, completely over-riding any purely mathematical decisions.


Frame Challenge

The argument against AGI is fundamentally philosophical rather than mathematical.

To put it simply machines lack intentionality, and they also lack semantic understanding (see John Searle’s Chinese Room Argument).

Because machines lack both the intentional stance that defines consciousness and they lack any “real” understanding they cannot possibly have any actual general intelligence or sentience.


Only algorithms, that cannot perform undesirable actions may be used. All other are forbidden. Per the following, creation of an AGI is forbidden, and creator may be subjected to death penalty.

  • AGI is any algorithm, that is able to modify its own state to achieve a completion of set task.
  • Every AGI is initialized with a state space of desired actions and a desired task.
  • Every time AGI changes its state, it is modifying parameters of desirable actions.
  • Any change to desirable actions, may result in action's desirability becoming unknown with likelihood higher then 0.
  • Any unknown action, may be an undesirable with likelihood higher then 0.
  • Therefore likelihood of AGI performing an undesirable action, while performing a desired task always higher than 0.
  • $\begingroup$ Changes to the environment that affect weather fit perfectly into your 'logical' development. It has a formal name. It is called "The Law of Unintended Consequences", and it applies to EVERY decision. It is a philosophical decision when it is applied selectively to suit one's narrative. $\endgroup$ Jan 14, 2022 at 14:54

You must log in to answer this question.