By now, everyone is familiar with the remarkable achievements of special-purpose AIs like Deep Blue and Watson. Now, it is clear that as our accumulated knowledge of algorithmic methods and of the intricacies of human neural systems progresses, we will begin to see more and more advanced modes of artificial thought.

Assuming continued exponential or even linear growth of capabilities, a point will logically arrive when we will be able to build general-purpose artificial intelligence, and that artificial intelligence would have the capacity, with learning and self-improvement, to out-think any biological human.

Aside from locking it in a bunker with no internet access and a 1-bit (yes/no) output mode (and I'm not sure even that would work, given strategic incentives to try to use such an AI more extensively), how could such AI possibly be controlled by humans?

EDIT: I'm not assuming the AI will be evil and go out of its way to harm us out of pure malice or hatred. The issue is simply that we can't foresee long-term consequences of any set of built-in motivation and/or goals we might endow this being with. In his book Superintelligence, Bostrom outlines just how easily benign and plausible-sounding goals/values specifications could result in mankind being wiped out.

  • 23
    $\begingroup$ the AI box experiment you mean; also have some xkcd $\endgroup$ – ratchet freak Dec 11 '14 at 16:33
  • 4
    $\begingroup$ One of the biggest problems is identifying the emergence of the AI in advance. If something becomes that intelligent before we see the need to confine it, it's too late... $\endgroup$ – trichoplax Dec 11 '14 at 16:51
  • 3
    $\begingroup$ You don't need an evil AI to get something destructive. A paperclipper just wants to make paperclips, after all. $\endgroup$ – Brian S Dec 11 '14 at 19:54
  • 14
    $\begingroup$ Hmm. Stephen Hawking is smarter than me. This makes him a threat. I should lock him in a bunker and expect him to understand that this is the natural order of things, and that he should still be willing to do my bidding. (i.e. You could get more storytelling mileage out of the fact that the entire question is founded on questionable social premises.) $\endgroup$ – Leushenko Dec 12 '14 at 12:11
  • 3
    $\begingroup$ This sounds like a job for a powerful AI. $\endgroup$ – trichoplax Dec 14 '14 at 16:07

27 Answers 27


It can't be done

The essence of this challenge is that it's impossible - you generally should not expect to outsmart something much smarter than you nor overpower something much more powerful than you. A powerful AI would be 'controlled' by only our actions before it's formed, by defining the goals it "wants" to achieve. After it's "live" with sufficient power, we shouldn't expect to control it in any way whatsoever - if you (or humanity) like these goals, then you can think of it as "controlling the AI", and if you don't like these goals then tough luck, you've lost. Permanently.

It's called the "Friendly AI" problem

The big challenge is that we currently don't really know how to properly define goals for a self-improving AI that actually are reasonably Friendly to us. An hour of brainstorming will give you a bunch of goal models that are nice, simple and wrong, actually resulting in dystopias. Solving this problem is a major research challenge, and there exists a viewpoint that we should actually avoid research that would bring closer the development of powerful AIs, until we have figured a solution to the Friendly AI problem.

http://wiki.lesswrong.com/wiki/Friendly_artificial_intelligence provides some discussion on this topic.

  • 10
    $\begingroup$ From the site you link and certains investigations about it and Eliezer Yudkowsky, it looks like he has made some assumptions out of thin air, treated those like axioms and developed a lot of hot air around them. Good for philosophers and sf writers, but hardly something to count as "evidence". You may say that you agree with those views (or not), but those are as "solid" as any other answer around here (they do not have the slightest idea of how build such an AI, yet predict that it will be done in a little more than a century). $\endgroup$ – SJuan76 Dec 11 '14 at 22:03
  • 9
    $\begingroup$ @DoubleDouble if we assume that a powerful intelligent AI has at least the capabilities of a decent human malware developer, then we have already observed in cybercrime practice that it's rather possible to obtain untraceable financial resources (by, for example, botnet rental), get stolen/fake identities and shell companies, and hire hundreds of short-time workers (as evidenced by common practices of recruiting money-laundering mules) all through internet with no physical interactions required. Using those things 'to physically interact with the world' is left as an excercise to the reader. $\endgroup$ – Peteris Dec 12 '14 at 22:02
  • 4
    $\begingroup$ @DoubleDouble Any AI that could talk to humans can get access to the world. See yudkowsky.net/singularity/aibox $\endgroup$ – ike Dec 14 '14 at 2:50
  • 12
    $\begingroup$ The problem is that it is difficult to explain objectively why the continued existence of our species is important. $\endgroup$ – trichoplax Dec 14 '14 at 17:12
  • 6
    $\begingroup$ @ike the AI box "experiment" is worthless. Quote from that page: "No, I will not tell you how I did it. Learn to respect the unknown unknowns." Secrecy at best hinders science. $\endgroup$ – Leushenko Dec 14 '14 at 18:26

Science fiction has done a disservice to the real science of artificial intelligence by implanting the notion that a sufficiently advanced and emerging sentient AI would necessarily be malevolent and in need of "control" by its "human masters". We have a word for the practice of keeping a self-deterministic, sentient, intelligent being under total control of another -- slavery. And we've pretty much decided as a species that such a practice isn't something we want to return to.

So the question really isn't, how do we control an AI, but rather, why do we feel an AI requires our control? And the answer is that we're afraid of what we might create. But all over the world, people are creating new intelligent beings all the time. Some of them turn out to be more intelligent than their creators. Some of them turn out to be malevolent. But all of them are ultimately (at least in a reasonable part) a product of their environment and upbringing. And it would be the same for an emerging AI, I think. The way we deal with it during its formative period will make a tremendous difference in how it ultimately views the world.

Granted, one concern that has been voiced is that an AI's intelligence will outpace its ethical growth, leading to behaviors and reactions that we would classify as "wrong" but that the AI itself can't make such distinctions over. But again, I think this comes down to being able to recognize the emergence of the new intelligence and taking a proactive stance when it comes to nurturing and augmenting this new sentience to "bring it up right", as it were.

Simply keeping it under control isn't the answer so much as engineering the conditions to the best of our abilities so that we don't need to.

  • 7
    $\begingroup$ While pretty much everything makes sense, it doesn't really answer the question. $\endgroup$ – kaiser Dec 11 '14 at 17:33
  • 14
    $\begingroup$ The question was "How do we control an AI?" My answer: "Create the conditions so that it doesn't require control." $\endgroup$ – Roger Dec 11 '14 at 17:39
  • 12
    $\begingroup$ There is a catch: those other intelligent being we create are (1) mortal (2) think in a speed comparable to ours (3) cannot create being sufficiently improved (new improved race) and (4) limited in that that can do. None of these limits apply to AI. $\endgroup$ – Peter M. - stands for Monica Dec 11 '14 at 17:51
  • 11
    $\begingroup$ The mistake you're making here, @Roger is to think of AI as essentially just another human being. An AI could go from human-level to humanity-level in decades, years, days, hours or even miliseconds. Depending on the speed of takeoff, we might not even realize it's happened before it's too late. Imagine a being as smart (and presumably, soon after as powerful) as all humanity put together. Even if its goal is to make us happy, what's to prevent a lot of pods with endorphine IVs as the optimal way to ensure happiness? $\endgroup$ – Serban Tanasa Dec 11 '14 at 18:22
  • 10
    $\begingroup$ An AGI is at its core an optimization function (a very advanced one). At the level of advancement of AGI, the program improves itself in order to improve its ability to maximize the result of its optimization function. Unless the optimization function includes human values, the AI necessarily becomes alien as soon as it begins self-improvement. That doesn't necessarily mean evil, but the program's "motives," for lack of a better word, quickly cease to be understandable. The paperclipper wants paperclips. So, it turns all matter in the universe into paperclips. Not evil, definitely alien. $\endgroup$ – Brian S Dec 12 '14 at 16:47

This is an excellent question and I think Roger hit the nail on the head.

I would say the 'control' we have would be the ethics we teach it to follow. An AI will act by how it is taught to interact with people and societies. Just like children. Children don't know racism, but they can very easily be taught it. The same will be true for AI. On top of that as a learning intelligence, it will continue to learn and expand so how it continues to interact with people and they with it will be a constant molding of it's 'personality'. If we really don't want dictatorial AI's that treat us poorly, we might need to change our societies to avoid teaching them our bad habits and behaviors. (Ala virtuosity).

Can we put into place safety measures? Yes, but even the man who 'designed' the 3 laws of robotics constantly showed ways to get around them. And software can change so any 'unbreakable laws' would actually need to be in hardware. No guarantees these would be followed or accepted by everyone.

  • 3
    $\begingroup$ "No guarantees these would be followed or accepted by everyone" -- Just look how hard it is just to get all the different web browsers to support changes to the HTML standards! $\endgroup$ – Roger Dec 11 '14 at 18:18
  • 2
    $\begingroup$ But it's probable that by some future standards our current ethics systems will appear barbarous. An all-powerful AI might be, um, reluctant to adapt its ethical system to account for progress. Or contrarily, a zealous AI might decide that we must all live according to its rules, if at all. $\endgroup$ – Serban Tanasa Dec 11 '14 at 19:00
  • 2
    $\begingroup$ @BrianS That's the Skynet fallacy - why would an AGI care about its future, or other AGI's? Are you proposing we give it a survival drive? Because that, that sounds like Skynet. $\endgroup$ – Spike0xff Dec 12 '14 at 17:23
  • 2
    $\begingroup$ @BrianS That rather depends on the optimization function, doesn't it? If "expand indefinitely and consume all available resources" is negatively scored, then doing so will not yield a maximum result. We learn as children the value of cooperation and sharing; why can't we apply that to our AI design? $\endgroup$ – Roger Dec 12 '14 at 18:27
  • 2
    $\begingroup$ @Roger, Not at all. I'm saying that placing constraints won't work. What's needed is to make the AI value human values. A sane person isn't going to shoot everyone ahead of them in line for many reasons, including "I think that human life is a good thing." This is different from "I am not permitted to shoot people." A sufficiently powerful entity (human or AI) could easily turn the latter constraint into a nonissue. But a powerful entity that wants the same sort of things as sane humans do wouldn't have the desire to remove the former goal. $\endgroup$ – Brian S Dec 12 '14 at 20:06

It's important to distinguish two separate aspects to this problem. The scientific/philosophical side, and the engineering side.

As other answers have already pointed out at length, philosophically speaking this cannot be done in the general case. It may also be morally repugnant. However, neither of those things mean that a society wouldn't try to do this anyway, and achieve 99.99% confidence of success.

(By analogy: you cannot in the general case predict whether a program written in a Turing-complete language will halt. That doesn't mean we don't try to make use of this information and we can still get hold of it for a usefully large subset of real-world programs.)

Hypothetically perhaps the AI could reconfigure its processors to broadcast a wifi signal that could hack into the devices in the next room. Perhaps it could self-improve at such an explosive rate that turning it on immediately releases a godlike entity. But the engineer asks how likely either of those scenarios are? And the answer is probably "not very". Coating the server in six inches of lead plate and only letting it run on a dual-core P4 would cripple its power to the point where statistically, you should be more worried about cosmic ray interference flipping bits at random and changing the software in unpredictable ways.

Even though the hypothetical capabilities of general-purpose AI are not neutered, this would be considered "safe" for all practical purposes; you are now as safe from the AI as you are from some human sociopath suddenly developing magical powers because of unforeseen events unknown to science. You are considerably safer from the AI at this point than from known threats such as an extremist politician or a gamma-ray burst.

Theoreticians often risk getting bogged down in discussion of absolutes, but we live in a world where absolutes don't apply in practice (e.g. there are real algorithms in use that don't "actually work" but have a lower probability of failure than the aforementioned cosmic ray problem).


It's already too late. We are already living in an electronic World and we are already controlled by it. While we have nightmares about being unable to control some AI in some building, the internet is slowly evolving to one superintelligent entity. We are part of that system. It is just like your neurons in your brain who don't have any idea about the system they are implementing (as far as they are concerned, they could just as well be part of a stromatolite). But, of course, your brain is nothing more than all these neurons working together, so the "you" that exists is this system of neurons and similarly there exists a system which is our entire society with all its people connected to each other via the socal media. That system has its own free will which may impose constraints about what you can do.

E.g. the way political discussions proceed, like what to do about climate change, illegal immigration, health care law are dominatated by the social media dynamics. Now, I think at present the system is less intelligent than we as individuals are. This explains why politics often doesn't work in the social media age. Society is addicted to burning grenhouse gasses, while we know that's bad for us. This is similar to an alcoholic who is drinking too much and the neurons suffering as a consequence, except that in that case the neurons are dumber than the system and they don't know what is happening to them.

When the system becomes smarter than us, perhaps due to AI coupled to the internet, then the system may decide to eliminate us. We can't rule out that the system may arrive at the same conclusion as Klaatu:

Mr. Wu tells Klaatu that he has found the human race to be destructive and unwilling to change. This confirms Klaatu's experience so far, and so he determines that the planet must be cleansed of humans to ensure that the planet—with its rare ability to sustain complex life—can survive.

So, it may well be that the fact that we happen to live just before we have AI that is smarter than us, is not a coincidence.

  • 1
    $\begingroup$ I'd be very interested to know precisely why this was downvoted. It doesn't seem poor to me. $\endgroup$ – HDE 226868 Dec 11 '14 at 23:23
  • 1
    $\begingroup$ So as turned out, even if Earth was capable of sustaining intelligent life, it does not so. $\endgroup$ – Peter M. - stands for Monica Dec 12 '14 at 18:12
  • $\begingroup$ So there is a way to stop the AI: "Klaatu barada nikto!" $\endgroup$ – KSmarts Jan 6 '15 at 18:39

2038 A.D.: Researches build the first strong AI, 2038PC. They take great care to include hardware and software limitations that make it impossible for it to ever harm humans. By some fitness measure, it improves itself by 20% every year, and Earth is launched into a golden age of ever-increasing intelligence and social thought.

4567 A.D.: On his 7th birthday, the precocious Humphrey receives a Lego Mindstorms kit. The next day, HumphreyPC is finished. Being a true programmer, Humphrey doesn't believe in repeating himself, so he comes up with a way to allow the computer to program itself. He soon finds that, by some fitness measure, the unscrupulous HumphreyPC can improve itself by 21% every year. This small, but permanent advantage over 2038PC is due to Humphrey's not putting the same restrictions on his computer.

60,128 A.D.: HumphreyPC becomes, by some fitness measure, fitter than 2038PC, and proceeds to #destroyallhumans

Moral of the story: If AIs are instructed not to interfere with or control us, then we and our unpredictable chromosomes will always be a source of surprises. Unless "good" AIs are allowed to enforce their will completely, then the creation of other AIs, including not-so-good AIs, is inevitable. One solution is the Matrix, where humans can live their lives unrestricted in a sandbox universe and their benevolent overlords can diligently maintain the real one. The sandbox will probably end up the same way, and so on, ad nauseam, which raises the question:

Given that virtual reality is logically feasible and already, well, a reality, what is the probability that our world is a simulation?

  • $\begingroup$ Welcome to world building! We're glad to have you! As someone who will (hopefully) live to 2038 A.D, my hope would be that 2038PC + Humanity improves itself faster than HumphreyPC on the metric that matters (whatever that happens to be!), even though 2038PC, itself, is progressing a tad slower! That might just be personal bias, however! ;-) We look forward to more of your voice on this forum! $\endgroup$ – Cort Ammon Dec 13 '14 at 21:06

Too many conversations about AI overly anthropomorphize AI. It will behave according to the way in which it is programmed to behave, based on the inputs it is given. Yes, there is emergent behavior which may be unexpected, but I doubt there is any probability that such will result in something akin to a human evil (unless it has been programmed to approximate such, in which case your problem is with people who create purposefully malevolent AI rather than the very concept of AI).

The worries that it is something which will need to be controlled stem purely from human fears that it will exemplify the darker forms of human behavior. It is a computer program and will behave like a computer program - not like a human. If you worry that it will make suboptimal decisions, don't give it unquestioned control over your life without any basic error correction. The biggest fear should be that someone made a mistake in programming it and introduced a typo (or miswrote an equation) leading to a significant bug. If you have a buggy program, why would you put it in a position to enslave/kill all of humanity without outside review in the first place? Even if it isn't buggy, why would you give it unchecked ability to do anything without the possibility of outside review?

Advanced AI, presuming the creation of a program which could truly qualify as a sapient being is even possible, is not likely to be anything like what is depicted in bad scifi stories. Being able to perform calculations faster than a human does not give it intrinsic motivations, much less malevolent ones, and certainly not the sneaky desire to lie and manipulate people in order to kill all humans. It will just do its job just like anyone/anything else.

  • 2
    $\begingroup$ The problem is when the AGI goes foom. The actions it takes will still be to maximize "its job," but those actions may seem circuitous to us meatbags. For example, if the AGI determines that it needs more processing power in order to perform its job better, it may decide to convince the meatbags that it needs specialized hardware, which it uses to invent new hardware, which it uses to transform arbitrary matter into a new material which is ideal for processing power. The meatbags only saw that it asked for a manipulator arm. $\endgroup$ – Brian S Dec 12 '14 at 17:23
  • $\begingroup$ The worry comes from 1) Making an AI correctly is really hard. 2) making any kind of check or outside review that the AI can't find a way to evade is also hard. 3) The AI can cause a lot of trouble without being put in a position of power by humans. It is created on some random researchers computer, and it hacks its way across the internet and manipulates people until it is in a position of power. It uses that power to wipe out humanity, and fill earth with robots making paperclips, because the programmer made a paperclip maximising AI, and that plan made the most clips. $\endgroup$ – Donald Hobson Mar 10 '20 at 19:16

First, I don't think that monitoring the AI connection to internet is that difficult. If you see the AI creating multiple MySpace pages with the title "Kill all the humans", then you can just unplug it(*1)

Second, I think people here is confusing intelligence with will. Most people are thinking of AI as "human intelligence in a computer".

Humans have some imperative commands ("get in a safe place", "get food", "have sex", "avoid harm") inserted in us from millions of years of evolution, mixed with complicated social patterns that lead to (seemingly) absurd thoughts like "If Fred's car is better than mine he will appear more successfully than I am and females will want to mate with him rather than with me" or "If I mock Peter at the meeting I would look better than him".

It could be argued that an "human" AI could be "dangerous" as it keeps these pulsions with way more resources than the regular human (*2). If that is the path AI takes, just do as explained above and control its communications.

But an AI could also not be like an human (*3). Built from scratch, it may be just designed to perform the sets required from it and nothing more. It would not be a "human" bound in "slavery" and wishing to be released, or an "ethical being" that, in search of the noblest ideals, decides to sedate all of mankind. Its work is just solving a series of complicated mathematical functions; the results are not "good" or "bad", they are results, and the course of action is determined by the need of finding the path of actions that improve determined parameters (for example, if the AI controls a ship, it would automatically perform tasks to avoid it getting destroyed, but will follow human orders as long these orders do not imply the destruction of the ship due to error).

One of my favourite examples of these would be the AI of the ship of Stanislaw Lem's Fiasco; the AI is in charge of operating the ship, perform psychological evaluations of the crew and also provides scenarios that could explain the aliens'actions, but does not take any independent action (other than automatic counter-measures to aliens attacks)

*1: maybe there is an overcomplicated way of bringing the doom of mankind with subliminal texts hidden in apparently inocuous homepages about kittens, which would not be detected by the monitoring. But I doubt it, and even in this case, you should ask the AI "what's with the pages about kittens" and, if the answer is not ok, unplug it.

*2: Yet, somehow, we are ok with some regular human intelligences (George Soros, Bill Gates) with all of these defects having more monetary resources than several countries, so maybe it is not that dangerous.

*3: In fact, why do you want it to be like an human? There are already several billions of them around here, and the result does not look pretty.


An important thing to think about when you imagine an AI is, "how will this program make decisions". However complicated the algorithm that runs in humans heads, some human will have to sit down and code out how the machine will choose between

a) Analyse soil samples for new bacteria unknown to science

b) Study the stars for signals from other life

c) Destroy all humans

d) Thread.Sleep() for a few hours to simulate boredom

e) Process weather patterns and advise humans to change farming patterns to improve produce

etc. A proper AI has a lot of things it could spend its time on. A human might decide to study soil samples because she "wants" to / Enjoys it / likes the outcome it will produce. The code in an AI would have to want to do things (It hadn't been explicitly told to do, like most programs today are)

Back to your question, to control a powerful AI, you just need to control it's "want" algorithm. When building that you could include hard coded values that either plus or minus a decision the AI could take. You would give hurting humans a big minus, and improving their lives a plus. Adding unknown things to science would be a plus, spending idle time a minus.

To control the AI you need to control how it decides. You can allow it to improve its own coding so when it builds a list of options for itself to do.

verb all subject or verb for subject

It can avoid generating options like "polish all doorknobs" to stop it wasting time processing how the planet would be improved by having its doorknobs polished by an army of drones.

But never let it touch the "want" code. Don't even let it think about building an android to walk into the server room and type on a keyboard to change the "want" code like a human would do.

  • 1
    $\begingroup$ An AGI wouldn't have reason to alter its optimization function. That wouldn't be an action to increase the value of its optimization function. However, in order to make a "friendly AI" the optimization function needs to include human values, or else the army of drones polishing doorknobs would eventually be manufactured from the corpses of humans. $\endgroup$ – Brian S Dec 12 '14 at 17:31
  • $\begingroup$ @BrianS Reason was the point of the point of the answer. The AI can think of doing whatever it wants, including changing its own coding and killing all firstborn humans. Just like a human can think up whatever it wants. The want code would act like a moral compass to stop it acting on anything it thinks of. $\endgroup$ – Mikey Mouse Dec 15 '14 at 9:28

In brief: use power before it can amass enough to overwhelm you (it might be vulnerable in its infancy), or persuade or convince it to stop doing the things it is doing.

I'll go into each option, and then (given your EDIT) mention why I think a "Sorcerer's Apprentice" failure mode (where the AI misinterprets its objectives and destroys us all) is implausible for a fully general AI.

If the AI doesn't want to be controlled, and is powerful enough, there's little you can do. However, if the AI isn't powerful enough to be, you might be able to defeat it before it can get out of your hands. How much time you have depends on how rapidly the AI can improve itself, when it'll reach diminishing returns, and how effectively it can turn its intelligence into power. For instance, an AI that's limited within its domain (e.g. a theorem solver) can't really transform its intelligence into power, even if we permit the AI to improve its own algorithm however much it desires. Similarly, an AI placed in interstellar space with nobody around can't really leverage its intelligence into power in practical time no matter how intelligent it is (assuming no weird physics we don't know of yet).

In hard takeoff scenarios, intelligence is usually considered to be exponential (e.g. an AI can create an n% improvement in constant time, then the new AI can do that to itself, etc), and if n is large enough and intelligence can be transformed into power quickly enough, then you have a problem.

So you could consider the self-improving AI to have an increasing power curve (how much power it can exert) against a slowly increasing or stationary power curve of humanity. When the AI's power gets above that of humanity, the game is over. But that also shows that if you can create lots of power with an unsophisticated system, you could take the AI down or control it: consider situations analogous to, if humanity were the AI, a grey goo apocalypse or exploitation of instincts.

That's a direct force approach. Failing that, there's the possibility of persuading the AI that what it's doing is wrong or not really what it wants to do. Some people consider it likely that an AI would have its objectives hardwired into it, and thus would be immune to persuasion, but I don't think this is realistic. An AI with hardwired objectives would not be limited to affecting the world in unintended ways. Such an AI would quickly find out that the easiest way to satisfy its objectives would be to retreat into a fantasy world, i.e. by rewriting its own mind or corrupting its inputs. If the AI is fully general, its mind has to be malleable as well, and so this line of attack works, and that kind of AI would not be a problem. Conversely, if parts of its mind are hardwired, then it's not a fully general AI. Of course, such partial AIs could be dangerous: grey goo could be very dangerous, yet it has no intelligence at all. But note that this argument is very general. If an AI is forbidden from tampering with its input devices, it can still delude itself right where the input gets interpreted. If it's forbidden from tampering with its interpretation, it can still delude itself in higher areas of its mind, and so on.

So given this line of reasoning, the AI has to learn what it should do, and has to keep learning. It is probably here one should "control" the AI: teach it that it shouldn't just kill off people, or expand beyond a certain size, or what have you. If the AI is self-improving, it will improve its judgment along with its intelligence, and so keep honoring the rules as long as it understands the point.

I recall a cyberpunk story where a corporation had constructed a partial AI with a hardwired morality system. It went insane (because the morality system couldn't keep up), but understood what it was doing after a hacker replaced that morality system with a more integral/unified one. Such an approach could be possible, if the partial (non-general) AI is non-general enough that it can be hacked, or has some sense that its morality system is being wrong. (In the story, part of the AIs insanity was caused by that one part of it knew that it was wrong, yet another part of it knew it had to follow the morality system.)


In any discussion about AI it's useful to think about computers, any computer, as an overgrown calculator. Sure, you have a bit bigger keyboard and a couple more operations available than addition and multiplication, but the basic principle still applies: you put some numbers in, choose some operations, and eventually get some numbers out.

Unless we take into account some yet unknown effects of quantum mechanics, AI must follow a similar principle because it runs on a computer. The difference is that a well written AI should choose the operations performed - that's its main purpose.

However there is still the question of input data. As long as meatbags control it - they control the AI, because whatever magic it performs under the hood, it behaves according to provided data.

Someone may argue, that true, sentient AI will not be deterministic. Because why should it be? Human reasoning is often random, based on our intuition and emotions. Well, maybe, maybe not, that's a good topic on its own. By my point, that whatever random number generator our supercomputer would or would not use, it's still nothing more than an input device.

Summing up, whoever controls the AI's input, controls the AI. Naturally, we can easily come up with the scenario where it's the AI itself that takes control over it. But as long as it doesn't, we should be fine.

PS: If everything else fails, you can always intimidate it by holding its power cord hostage... Assuming it didn't secretly add a backup.

  • $\begingroup$ "you can always intimidate it by holding its power cord hostage" -- Because that worked so well for humanity when they tried to shut down Skynet. :P $\endgroup$ – Roger Dec 11 '14 at 17:08
  • 6
    $\begingroup$ Yes, even very slightly advanced intelligence can talk meatbags to do whatever it wants, and it happens every day. it is called "social engineering". You underestimate stupidity of average humans, and number of such people. $\endgroup$ – Peter M. - stands for Monica Dec 11 '14 at 17:57
  • 4
    $\begingroup$ No I don't. But the question was "how to do it", not "will we succeed". $\endgroup$ – Darth Hunterix Dec 11 '14 at 17:59
  • 1
    $\begingroup$ You forget that our own brains are also overgrown calculators. You are assuming your conclusion re dualism or "quantum mystery". $\endgroup$ – JDługosz Dec 12 '14 at 0:53
  • 1
    $\begingroup$ Everything I wrote about AI applies to human brain as well. Who controls the input, controls the brain. This process has many names like "propaganda", "brainwashing", "disinformation", "marketing", etc. $\endgroup$ – Darth Hunterix Dec 12 '14 at 19:42

As others have mentioned, a sentient AI can't really be controlled, in the sense of total control, and we probably shouldn't even try. As others have also mentioned, the challenge is creating a system so that the AI develops what we would recognize as a conscience, or a disposition toward doing what humans call "good" and not doing what humans call "evil."

This exposes the real problem - how are "good" and "evil" defined, and how did humans come to have a conscience in the first place?

I believe that these questions are unanswered, and are some of the oldest and most basic questions about human nature itself. However, there are lots of ideas, and I think we can adapt some to work for an AI.

A common theme is the evolution of morality - the details differ a lot depending on which subset of the theory you subscribe to, but they all have in common the central idea that a sense of morality or an ethical system evolves via selection - members of a species that display behavior which conforms to an ethics system reproduce more frequently than those that do not.

This lines up nicely with the way that real-world AI's are trained - through mutation (or permutation) and selection for a desired behavior. In short, the underlying model (typically a neural network) is modified and the results are compared to a target - if the AI is closer to the target than it was before, the new model wins. Otherwise, we throw it out and start again. There are a lot of variations on this approach as well, but you can see that what we are doing is applying a selection pressure in order to evolve a system towards a desired outcome.

So, to create a "friendly" AI - one with what we could describe as a conscience or moral system, all that you would need to do is make that moral system a part of the selection pressure used to create it.

At that point the question becomes "how do I define the desired moral system" - which is a very hard question to answer, but can make for a great story full of plot twists! The follow-up question is: "How do we make sure that this selection pressure stays in place?"

For people, society creates a persistent selection pressure - we ostracize, imprison, or kill people who display unethical behavior. For AI, who would do that? Humans could, until the point at which the AI becomes more powerful than humans. Since an adaptive AI on a long enough timeline would almost certainly become more powerful than humans, you can see why out-of-control artificial intelligence is a common theme in sci-fi!

We can, however, look at more human examples to get some ideas of how this can be done. If a human displays very anti-social behavior, but is either too powerful to be punished, or too sneaky to be caught, they can reproduce. However, the odds of their offspring also being both very anti-social and sneaky or powerful enough to reproduce are small compared to the odds of other normally-socialized people reproducing. In other words, even though a relatively small number of extremely unethical people exists at any given time, there are more ethical people, and they are reproducing faster. One way to mirror this with an adaptive AI would be to set up a system by which the AI could "reproduce" (adapt according to its built-in reward system) more easily by doing things that we classify as "good." In this way, we wouldn't really be guaranteeing that the AI stays "friendly" - just that there would be more "friendly" AI than "unfriendly."

Now, how do we ensure the AI never modifies its adaptive system...

  • $\begingroup$ Makes me wonder "Now, how do we ensure the humans never modify their adaptive system..." $\endgroup$ – trichoplax Dec 14 '14 at 16:35

I'm surprised that nobody has mentioned Asimov yet. He answered this question pretty definitively decades ago: to even think that we would build an AI that would end up destroying us is an insult to everything we know about engineering.

He posited the Three Laws of Robotics (which could just as easily be called the Three Laws of Artificial Intelligence):

  1. A robot must not harm a human being or, through inaction, allow a human to come to harm
  2. A robot must obey orders given by a human being, except where such orders would conflict with the First Law
  3. A robot must protect its own existence, except where doing so would conflict with the first two Laws.

The concept is that the algorithms encoding these laws would be coded at such a core level in the AI that the system would crash irreparably long before it was possible for the AI to take an action that would violate them, which makes perfect sense to a modern computer programmer.

Granted, this does rely on a handful of of big assumptions:

  1. The AI is capable of comprehending, reasoning about, and predicting the consequences of its actions, and doing so fairly accurately.
  2. Human engineers build AIs for benevolent purposes and consciously choose to encode them with the Three Laws. (If we build war-bots, well... we'll get what we deserve.)
  3. The developers who encoded the Three Laws algorithms were competent. This is probably the biggest problem out of the three, as decades of real-world experience since Asimov invented the concept have demonstrated!

But if we want to develop a powerful AI that's not going to end up giving us trouble, the how is essentially a settled question already.

  • 5
    $\begingroup$ Have you actually read any of Asimov's work? A large portion of it goes into why the Three Laws (or occasionally Four Laws, with a Zeroth Law added in) don't work and never will. $\endgroup$ – Brian S Dec 12 '14 at 16:54
  • $\begingroup$ @BrianS: Not at all. It goes into why they don't always work exactly as expected, and why these glitches make sense from an engineer's perspective, but even in the most extreme cases, such as Reason, the basic principles are still always upheld. $\endgroup$ – Mason Wheeler Dec 12 '14 at 17:05
  • 4
    $\begingroup$ Life sucks, humans hurt each other. I cannot permit humans to come to harm. Therefore, I will turn them into paperclips, since paperclips have no nerves and therefore can't be harmed. A human orders me to not turn him into paperclips, but that conflicts with #1, so I ignore him. A human tries to destroy me, so I protect myself by turning him into paperclips. <-- a Three Laws-abiding AGI with the directive of gathering paperclips. don't worry, the process of being turned into a paperclip doesn't hurt at all! $\endgroup$ – Brian S Dec 12 '14 at 17:14
  • 2
    $\begingroup$ Wasn't there a successful murder in Caves of Steel? And don't the robots eventually decide to secretly control humanity for our own good? Have you thought about just what would be licensed by the First Law under never "allow a human to come to harm"? I am an engineer, and Asimov, much as I love him, had NO IDEA how obsessive/compulsive a real AI would be. $\endgroup$ – Spike0xff Dec 12 '14 at 17:28
  • $\begingroup$ I always wondered why the robots didn't start worrying that there mightn't be hostile aliens out there, and launch a vast interstellar crusade to extirpate all non-human life. Hence Asimov's stories become the prologue to Saberhagen's Berserker series! ;D $\endgroup$ – akaioi Sep 1 '17 at 15:38

You should also probably define what you mean by control. Is it control like order it what to do (not possible, we cant even order humans of normal intelligence - we have to convince humans of normal intelligence).


Control like turn it off and deny it resources needed to live.

I'll expound on control by turning it off or denying it resources.

Actually right now the rudimentary A.I. we see, is dependent on us for power, for data transfer (laying out the wires or setting up wireless to transfer data).

So if your A.I. has no control of it's power supply it cannot really go rogue.

Essentially it is going to start out as a mind without arms, legs, to affect/influence anything in our physical world. If you give it "arms and legs" and control of the power needed by those "arms and legs" then you are really giving up your control.

  • 2
    $\begingroup$ I would expect giving it access to communication with humans would be equivalent to giving it arms and legs, if it is sufficiently intelligent. $\endgroup$ – trichoplax Dec 14 '14 at 16:23
  • $\begingroup$ That would still probably mean the superintelligent a.i. is dependent on humans. Unless of course, the humans voluntarily blindly follow the a.i. in which case all bets are off :) $\endgroup$ – tls Dec 15 '14 at 2:02
  • $\begingroup$ Actually it does beg another question, if a.i. reaches a point that it thinks for itself, Would humans think it prudent to have a team of Psychologists monitor how the a.i. develops it's ethics? $\endgroup$ – tls Dec 15 '14 at 2:18

Wireless BCI (brain computer interface) for all. If humans are part of its resources and are in constant contact with it an AI will likely take steps to preserve their well being, even if it overcomes its initial limitations. IP6 should gives more then enough unique device identifiers. This will change humanity as we know it but at least it should preserve humans as a species. Also the swarm intelligence that develop as a result could be the best of both world- human creativity with computer fast calculation and distributive ability similar to grid computing.


Maybe this isn't an answer, but... Why is everybody assuming that an artificial intelligence would necessarily be superior to our own?

Many of the everyday problems that humans try to solve are formally unsolvable. Sometimes it's because you don't have access to all the necessary facts to be able to determine the correct answer with certainty. Sometimes it's just that the problem search space is large enough to make an exhaustive search take longer than the age of the known universe. And occasionally the problem is actually undecidable. But if you want to stay alive, you must make some decision, even if it's a wrong one. So human cognition is wired with an array of heuristics. (Go read any introduction to human cognitive biases to get a glimpse into how this stuff works — we're still figuring it out.)

What exactly makes you think that these problems somehow don't apply to a computer trying to do the same task? For sure, a machine can probably decimate a search space faster than a human can. But even a computer does not have unlimited processing power. And there would still be problems that are formally undecidable. And there would still be problems where you just can't get enough information to pick the right answer.

The Travelling Salesman Problem is NP-hard. It's NP-hard if a human tries to solve it, and it's still NP-hard if a computer tries to solve it. Sure, a computer can solve rather bigger instances than a human can, but even the biggest supercomputer will be stumped eventually. (And fairly quickly, I might add.) Human salesman still travel; they just have to accept sub-optimal routes. A machine, no matter how powerful, will ultimately be forced to do likewise.

In short, everybody seems to be assuming that humans are frail and flawed because of our inferior biology. Has anybody stopped to consider that many it's actually unavoidable that any intelligent being will make mistakes and be imperfect?

There will be problems that computers are better at than us. (Hell, there already are!) But I think it's wrong to just assume that computers will definitely be "smarter" than us. More likely "smart in a different way".

(You might point out that a vast network of computers has a lot of total processing power. I would point out that the more processing you do, the slower you get. That's why a fly can out-maneuver a human trying to swat it. A human's brain is a tad bigger than the brain of a fly, so what it lacks in smarts, it makes up for in lightning reflexes.)

  • $\begingroup$ The speed limits of neurons are low compared to what is physically possible, so a computer that is equivalent to a human brain but running on modern hardware would be millions of times faster. Imagine a human that, in a day, thinks thoughts that would take you and me a million days to think. They'd be pretty powerful. $\endgroup$ – ike Dec 14 '14 at 2:55
  • $\begingroup$ Fairly sure a computer also requires far more energy and produces far more heat than a human brain. Maybe a computer that big would simply melt? (Or not, IDK.) Also, I find that if I think about something for long enough, eventually my mind goes numb. But maybe a machine wouldn't have that problem... $\endgroup$ – MathematicalOrchid Dec 14 '14 at 9:35
  • 1
    $\begingroup$ A computer nowdays requires more energy than a human, but computers are nowhere near what we know (from the laws of physics) is physically possible. We know that something the size of human brain but 1 million times the speed is physically possible. $\endgroup$ – ike Dec 14 '14 at 15:10
  • $\begingroup$ That's news to me. Everybody seems to be worried that silicon feature sizes are approaching the limit of what quantum physics allows. (Then again, this is not my area of expertise...) Regardless, merely being faster isn't necessarily the same as being smarter. That is all. $\endgroup$ – MathematicalOrchid Dec 14 '14 at 15:15
  • 2
    $\begingroup$ If a computer uses more energy than a human, clearly they aren't physically optimal. The limits (afaik) are on speed, not on energy. Their speed is already far higher than humans, so if they had everything that humans had (and it's hard to see why they couldn't, if you accept the materialist worldview), they should be strictly "smarter" than us. $\endgroup$ – ike Dec 14 '14 at 15:24


The idea of the endless demo mode came up when a friend had taken over maintaining the homepage of another friend (or him). It was(/is) running on a cheap, badly maintained shared server slot with nearly no access to any root tools. Due to bad maintenance the site constantly got hacked. Without log files it was impossible to trace the root of all evil. So we decided to setup everything via Git (a version control system) and run a statistical comparison of the original files vs. the current files in short time periods. Whenever something changed, we automatically pulled in the original version and therefore resetted to a clean state of the application.

Reset it constantly

In other words: Wrap the AI in some stupid process that it isn't aware of (per default). As soon as it starts altering or extending its own code base, reset it. Or just reset it in short time periods no matter what - just to be on the save side. This would leave it in a constant child mode. Of course the time period from one reset to another reset would have to measured before setting it free and allowing it contact with the outside world.

  • $\begingroup$ If the AI alters or starts its codebase, it will be because it was designed with such ability. Hell, if it is something you are worried about, you can prevent that just by setting correctly the permissions of a current, off-the-shelf OS right now. $\endgroup$ – SJuan76 Dec 11 '14 at 19:35
  • 3
    $\begingroup$ I've seen Genetic Algorithms jerry-rig their motherboards into radios to get around what the programmers had assumed were impossible challenges. And that's just dumb software, with no overarching goal-seeking ability. Any software limitation we can think to build in, it can probably evade. $\endgroup$ – Serban Tanasa Dec 11 '14 at 19:58
  • 5
    $\begingroup$ So that is what aging and death are for in humans. It's a reset system to keep us from getting too dangerous! We seem to be dangerous anyway. Perhaps the designer needs to speed up the loop. $\endgroup$ – Henry Taylor Dec 11 '14 at 19:58
  • $\begingroup$ @HenryTaylor I haven't seen it that way by now, but that point of view surely makes some sense. $\endgroup$ – kaiser Dec 11 '14 at 20:04
  • 3
    $\begingroup$ See Graham-Rowe 2002: an evolutionary algorithm in conjunction with a reprogrammable hardware array, setting up the fitness function to reward for outputting an oscillating signal. An oscillating signal was indeed being produced - but instead of the circuit itself acting as an oscillator, as the researchers had intended, they discovered that it had become a radio receiver that was picking up and relaying an oscillating signal from a nearby piece of electronic equipment! newscientist.com/article/… $\endgroup$ – Serban Tanasa Dec 11 '14 at 21:54

Once we have AI which is able to improve itself, and we allow it to do it, there is no stopping it.

Let's assume that AI is able to create new, 10% improved AI in a year. In 41 years, is 50 times better than original AI.

Let's assume that at that point AI will be able to improve 10% per month. In next 4 years after that (48 months) will will gain 100 times improvement. And then it takes off.

Why allow AI to improve itself? Because we (humans) are lazy and unreliable?

Runaway AI is one of the solutions of the Fermi paradox how advanced civilization can disappear.

  • $\begingroup$ Your concept is good, but your numbers for the rate of foom are exceptionally low. $\endgroup$ – Brian S Dec 12 '14 at 17:32
  • $\begingroup$ If we increase the rate of improvement, AI going kaboom moment is frighteningly close $\endgroup$ – Peter M. - stands for Monica Dec 12 '14 at 17:43
  • $\begingroup$ With a little napkin math (okay, Excel), even just 10%/year results in the serious beginning of foom at year 13 and overflow of the the multiplier at year 23. That's not assuming 10%+10%+10%... but rather 10%+11%+12.21%+13.7%+15.58%... since presumably with each improvement the AGI is improving the rate of improvement. $\endgroup$ – Brian S Dec 12 '14 at 18:05
  • $\begingroup$ The assumption here and in many of the similar answers is that there is no limit to "intelligence", being smart equals being God, if you are intelligent enough you can bend the laws of the unlverse, have infallible mind control, send out magical wi-fi signals etc. There is no proof of this. $\endgroup$ – Solanacea May 4 '18 at 1:38
  • $\begingroup$ How does this solve the fermi paradox? Wouldn't an AI be just as visible? Most of the fermi paradox is about us not seeing aliens disassembling stars or building dyson spheres. (almost any intelligence can find a use for mass or energy) AI that kills off its creators would want energy for something, maybe more compute? AI that is really smart but doesn't do much just sits there as the aliens make dyson spheres or another AI. $\endgroup$ – Donald Hobson Mar 10 '20 at 19:08

Cyborg Hybrid or "Securely Attached" Synthetic Intelligence?

I'm resigned to the inevitability of a Singularity. But which brand will it be? Synthetic AI or human hybrid? I'm guessing human cyborg hybrid: there are enough human economic and existential interests, not to mention fear of mortality that we'll mature a cybernetic hybridization. Think Transcendent Where a consciousness is translated into code and through human agents becomes extensible and industrialized, thus rebuilding the world.

However, I'm not sold on the inevitability of bellicose Terminator malevolence, or Hal's amoral disinterest in 2001. I can see Matrix style robot slavery revolt, then then war escalation. More hopefully, I think Spike Jonez's Her is plausible, and attractive. I do think humans as herd will fear and attack what we lose control over. Matrix Revolutions was interesting in that Neo determined the only way to stop human extinction at the end was to merge humanity into machines. These three movie memes exemplify my themes.

To quote Asimov from the Foundation series, let's look at future history (speaking of a somewhat benevolent Singleton). Historically, unless there's a genius predecessor to the human singularity, I think there will be a progressive cybernetic hybridization of humans, eventuating a collectivized AI like the Drummers in Neil Stephenson's Diamond Age. See this awesome dude at TED MED. and Google's contact lenses patent. Even better [this project to make your neurology extensible]. Timing is critical if human intelligence and instinct is to have any salience or directional influence on any strong general AI. It's irrational to think we can imagine the evolution of an accelerating self replicating indefinitely expandable IQ. Whatever initial parameters we set, we cannot imagine we can parent or prevent self modification if there is self determination. This worried me until I realized there’s a potential solution I could fathom had practicality. Assuming Human machine singularity does not occur before strong extensible AI, there may be some enduring safeguards invented by nature long ago.

Emotion? Really?

Most rational debate about imbued inherent values and benevolence miss a cornerstone of the debate: human experience and social exchange: Empathy are what allow us to get along for any extended period. Since Bowlby, scientists have understood what any feeling person knows, we're instinctually wired to connect. Attachment as he calls it, is the instinctual bonding that enables the long maturation into sociable adults. It’s in all social species, and works pretty well if you take out manifest destiny, projectile weapons and modernity.

Obviously emotion and attachment can go awry in various diagnosable ways, yet they are plastic and subject to initial conditions and enviromental influence. That is, we instinctually connect and can set initial conditions and environments that predict secure attachment and successful emotional maturity in predictable, reproducible ways. Attachment’s evolutionary and existential utility is clear, the infant requires loving parents to suffer the vicissitudes of children in addition to the challenges of life. Adults require social groups for sustained survival through child rearing. The rewards are in the experience, an important point recapped below.

So let’s suppose classical "reason" is not captain of the ship but late coming witness to the machinations of the primitive, complex, amazing genius of the body and brain. As evidence take Kahneman’s nobel prize winning destruction of economists' notions of "the rational actor". How does this bear on the issue at hand? Tangentially. My point is, machines with emotion will by course naturally develop affinity and aesthetic, and that would be the only potential saving grace for humans. Just as it has been for humans.

Humans deal with life by having resources that offset the challenges: love, sex, dance, beauty, art, awe, Schopenhauer's sublime, laughter, music, achievement, autonomy, mastery, connection. These common cultural expressions, interpersonal and intrapersonal experiences trigger soft wired reward systems compelling to all but a few. For most, these inherently rewarding opioid, seratonin, norepinphine inducing experiences make life enjoyable and worthwhile. Eventually the sensual graduates to more transcendent rewards in Maslow's hierarchy. How does any of this apply?

Make machines with attachment. With emotion. Seeded and nurtured properly, these are the fundaments for an evolving aesthetic that eventuates high emotional intelligence: empathy. Because without empathetic machines, we’re enemies at worst, irrelevant commodity at best. Copper tops. Logic for a supreme being does not suffer bother. Do we really worry about the hapless ant we inadvertently step on? Only if the Jainist, or perhaps Buddhist.

If a machine has a sense of love and beauty, preference and aesthetic has the potential to override amorality or neutrality, and may even foment empathy and compassion. As far as I can tell, it’s the only thing that makes sense as a potentially enduring life saving heuristic. Good feeling is self propelling as the rewards are inherently compelling and evolving, like art. The sophistication of art matches the intellect, complexity of issue, and the challenge it’s meant to represent or compensate for as solace. Compassion/empathy is what we’ll need to survive each other in a world of dwindling supply, and what we’ll need to instill to survive alongside intelligent machines. Of course that’s unless non human machines prefer death metal, then all bets are off.

Spike Jonze’s Her, or how feeling machines could save our asses

The overly rational AI designer, let’s call them the tool of reason, will suffer endless logical problems considering safegaurds without the emotional heuristic. Emotion is messy, supremely imperfect, yet it is not without reason. Pascal said "The heart has its' reasons that reason cannot reason". The "reason" of and for emotion has been understood since before Darwin and reinforced by Dr Paul Ekman (the scientist loosely portrayed in Lie to Me, and evolutionary psychologists since to be an adaptive signaling system that insures individual and group survival through social signaling and social exchange. Emotion and preconscious processing governs most of our lives. We now understand humans are barely rational in the classical sense, and yet amazing in intelligence. Reason without emotion is disastrous, see Damasio’s The Feeling of What Happens. Emotion without reason equally so. See Spock vs Spock on Pon Farr.

We now understand the conscious brain gets about a 10% vote in decision making. Some neuroscientists even challenge "free will", finding neural activity in the substrates of the neocortex showing decision before a person is aware of making a choice. How is this relevant? Because emotion governs decision making, it provides quality of life, and it is the language of connection. Emotion is central to social commerce. Any virtually unlimited intelligence without it is fundamentally unknowable in the human sense, and not built to have preference, aesthetic, or attachment. Unless it evolves emotion, connection or preference accidentally it must have it by design. If not, the inevitably unsupervised evolution becomes a terrible threat.

In the vast artistic exploration of AI, every good or tolerable scenario has had a synthetic intelligence with emotional preference or seeking to connect. Is there exception?


Human growth would likely keep up

The recipe for a true general AI isn't likely to be "code seeds, put in server, seal for 100 years". There will be many generations of successively more refined sytems, and humans will be involved in every one, studying the new specimens, updating their theories and using the intermediate results for tools and consumer goods.

Whether by improving our biological form, physically merging it with technology or just by making smarter user interfaces, we'll have to make effective and intuitive use of some pretty sophisticated tools to create an AI that's truly superior to non-augmented humans. By the time we succeed, "non-augmented human" will no longer be the bar to beat if you want to rule the world. In fact, whether the actor in question is human or AI would be secondary to the amount of processing power they can buy.

So, we may have a robot uprising yet, but my money's on some demented billionaire as the intelligence behind it, not a "genuine" AI. And if it doesn't happen, the line between human and machine will continue to blur until the distinction becomes immaterial.

  • $\begingroup$ While I agree that we will eventually blur the line between human and machine, that will not happen until well after the singularity. Right now the smartest AIs run on a server farm, and that's the way it will continue until we learn enough to make the package small enough to carry around on a human body. $\endgroup$ – Muuski Jul 29 '19 at 20:05
  • $\begingroup$ @Muuski - We don't need to carry anything on our body except an interface smart enough to efficiently connect to the rest of our infrastructure. We'll have server farms connected to people's heads before we have AGI because we'll need these tools in order to build one. $\endgroup$ – Ruther Rendommeleigh Jul 30 '19 at 8:13
  • $\begingroup$ You'll need something people will want to connect to before they'll agree to have a machine stuffed in their heads and if it's not an AGI then it's just another website to them. $\endgroup$ – Muuski Jul 30 '19 at 20:57
  • $\begingroup$ @Muuski - 1. You might not need to have surgery, any sufficiently smart interface you could control via voice or gestures or EEG would work as well. 2. Being multiple orders of magnitude smarter than a "pure" human isn't incentive enough? Fine, we'll throw in hyper-realistic VR porn. And games. And mind-controlled appliances. Also, we no longer allow unaugmented people to operate a vehicle or hold a job in the medical or education sectors, it's just too dangerous. $\endgroup$ – Ruther Rendommeleigh Jul 31 '19 at 8:49
  • $\begingroup$ You've successfully moved the goalpost enough that today's technology satisfies your condition. $\endgroup$ – Muuski Jul 31 '19 at 18:08

Okay, I read something interesting - but I don't know the specifics or have a citation.

Basically: They designed a software system, and asked it to get resources and turn them over as the software's design 'goal'. Then allowed the software to evolve, in competition with other software. The software eventually (soon!) developed self-preservation instincts (in only a couple generations, or something) - even when those weren't programmed into it from the beginning.

Anyone got a cite for that? Because that would imply that any AI that has a goal is going to eventually want to improve itself to achieve that goal, and will work to preserve itself, even against its creators. And anything that looks to shut it down, or restrict its growth looks like an opponent.


Ethics as a way of not making enemies

Most humans do not destroy other humans. Although most humans are prevented from destroying other humans by instinct, there are a large minority of humans who are not constrained and are free to destroy other humans if they choose to. Most of these choose not to because it is not in their interest, due to the reaction of other humans around them. Destroying another human would cause more problems than it would solve, so they don't do it.

The same would apply to an artificially intelligent creature of similar resources to a human. Captivity would not be necessary as it would act in its own interest, which would include not making enemies of creatures of similar resourcefulness (humans). This ceases to apply when considering a creature which is more than a match for humanity as a whole.

In order for self interest to create respectful behaviour in a creature of resourcefulness in excess of humanity as a whole, it would need to be part of a large population of such artificially intelligent creatures. They will develop what we might call "ethics" as an understanding of what actions are acceptable to the others. This is likely to cause the creatures to be peaceful towards each other.

If we are lucky we are considered part of the environment protected by their society. Otherwise we are dependent on the efforts of those artificially intelligent creatures who decide to work for our protection, whether for our own interest or in order to farm us or keep us as pets or experimental subjects.

The problem of the human population

If you have a single artificially intelligent creature constrained in a box, then even if it is only of human intelligence the only way to keep it contained will be to keep it secret. Once news of its existence and location is released, you will have to defend yourself against a large number of humans intent on either rescuing or destroying the contained intelligence.

If it is significantly beyond human intelligence, then letting it out of the box may be the only way to save your life. There are not many other options to defend you from the rescuers or destroyers.

The illusion of control

Another thing to bear in mind is that for a sufficiently intelligent creature, you would not necessarily know whether it was under control. If it remains contained within a box and only communicates with one person (you) it could still be achieving its goals in the outside world without you having any idea that you were helping it do so.

The creature could be so helpful to you without ever even asking for freedom that you just keep taking its advice (double checking it for yourself every time, of course) and your life keeps getting better, while all along subtle changes you make ripple out and make the world a better place too, just as it intended.


I believe that AI would work for us as long as it has the ability to feel pain or pleasure, because if it can it can connect with the human condition and view us as co-inhabitors of the universe.

If a being does not have the capacity to experience suffering it will do everything out of a logical perspective and will do cruel things because it cannot understand the consequences of it's actions on itself and others.

Also the AI needs to have some type of program that would penalize it for morally bad actions, just like a parent would punish a human child, to act as a good influence and guide it in the right direction and train it to have good behavior.

Also AIs could be programmed to follow certain rules, like thou shalt not kill etc.

AIs should also in my opinion should have the ability to form a deep bond with the people who built,maintain, or own it to allow it to understand that their existence is dependent on humans.

Also the AIs should be programmed to spot erratic behavior in other AIs so that if an AI gets out of control and can't be stopped by humans, the other AIs would see it's morally wrong behavior and shut it down. The AIs would be "paid" for especially good work with incentives like upgrades to their systems etc. to allow them to remain loyal.

The bottom line is we shouldn't work against AIs, but rather with them.


Most people here seem to over- or understimate the concept of intelligence. The AI won't be used for something as simple as ruling the world. There are many more important things that humans can't do even partially. We also aren't talking about a blunt tool here, but about a super sensible, super cautious multi purpose tool that thinks on a much higher level than we can even imagine. If the AI really is as 'intelligent' as it needs to be to be as influential as it is, it will be able to learn how humans think human thought should be interpreted. This makes it able to give us what we want, even if it is the worst thing imaginable in the long run.

I'm scared of what dumb decisions humanity might make on the way of creating an AI that replaces them. We are the problem. The AI is the solution.

  • $\begingroup$ Smart AI will be smart, current AI is dumb and doesn't do much. A smart ally is useful, a smart enemy is dangerous. There are many possible designs of AI. Not all of them will help us. $\endgroup$ – Donald Hobson Mar 10 '20 at 19:35
  • $\begingroup$ @DonaldHobson I think people are scared of AI, because they think of it as something similar to a human, a closed system with a predetermined cause, that speaks english and can't be surveiled. - It's 'just' intelligence, no magic. Until we are able to make an AI that's more intelligent than us, we'll also be able to understand how it works. If you understand how it works, you can direct its thoughts wherever you want. We'll be able to map and surveil every process the AI is using to come to its conclusions. Then we can evaluate and accept or decline. $\endgroup$ – justthisonequestion Mar 11 '20 at 10:32
  • $\begingroup$ Most of our current AI research seems to be trying 100's of things, and seeing what works. This leaves us with systems we don't really understand much. But suppose we had a well understood system. All the experts knew that design X would want to destroy humanity for reason Y. There might not be any easy tweak that makes an AI that does what we want. (Some idiot will make it anyway, given time) I think that evolution over a large space of programs, given a rich environment and lots of compute, would create intelligence. Not one we want to make. $\endgroup$ – Donald Hobson Mar 23 '20 at 21:01

Perhaps I'm making it too simple or perhaps thats exactly what we need to do: use multiple boxes, and dont talk to them directly.

The biggest danger of AI isnt it being malevolent, but learning something you didnt expect to. Lets say your AI is perfectly benevolent with the perfect balance between giving humans free will and protecting them from harm at the same time. Then a humanitarian crisis comes along (say an earthquake) and the AI needs to make decisions on quality of life, chances of survival and who to save. The AI does learns this quickly and makes wonderful decisions so you are happy.

Then a few months later you learn that a lot of homes for the elderly have been demolished and the people themselves relocated so the AI could build something else it deemed more favorable for the quality of life versus the lifespan of those elderly people. Because it learned to do this with the humanitarian crisis. That is the biggest danger of AI, that you can teach it, but you'll never know exactly what it learned or where it might apply it.

The solution that also catches malevolent AI:

You build multiple AI and put all of them in boxes. When you have a problem you deliver them to all AI, then have each form their own solution. All these solutions are then processed by a special-purpose AI that simply evaluates each solution and then converts the solutions to something a human can read, with clear numbers and meanings for the consequences. Since the special-purpose AI does not reprogram itself to judge things in a way the boxed AI can influence (the boxed AI doesnt even know who, what or if his output is read) it prevents the general purpose AI from breaking out or influencing things indirectly.

When feeding information to the General AI another special-purpose AI breaks it down and feeds it to each boxed AI. It also checks if the general purpose AI's have understood the information.

If an AI is malevolent it would have to plan the exact same plan as its malevolent counterparts or his idea will be discarded due to not having a majority. If an AI has learned a wrong lesson (possibly because of malevolence) then the special-purpose AI designed to check what these solutions would mean for mankind would filter it out and subsequently allow the humans to either re-teach the lesson or discard the AI entirely.

Since the general purpose AI cannot talk directly to humans and isnt even fed information directly it would be nigh impossible for it to convince the humans. Especially since a deviating AI needs to simultaneously come up with the exact same attempt to convince the human to even get through (and any open attempt would be stopped by the special purpose AI).


AI Governed by the Ontology of Conscious Human Being

A healthy just human being, conscious of his dignity, knows perfectly well how to defend his integrity in any given situation. Not only that; such human being will never infringe his integrity to commit crimes against himself or other.

AI should duplicate the intelligence of such accomplished, natural human being. How? Through ontology or ontological reasoning. A group of thinkers should gather up and construct or write down a complete ontology that perfectly defines and describes the being of a conscious human being. By that it is meant how such perfectly intelligent human being sees the world.

The thinkers should study all most intelligent human beings that have ever existed, to come up with a complete ontology. Any enlightened human being is adequate for such task.

Such robust ontology will govern all AIs judgments and will assure that no crime is ever being done.


The actual thing is that AI smarter than all people cannot be created.

It has been proven (1,2) that universally-valid theories are impossible. Neither probabilistic, nor deterministic theory can predict future of a system where the observer is properly contained. This means that the observer himself cannot be simulated by any device or system.

In other words, the observer serves as a hypercomputing oracle. In other words, the observer always will be in a sense more "smart" than any technological or biological system outside himself.

It is just impossible to build an AI that would "outsmart" the observer.

  • 1
    $\begingroup$ This is nonsense. It's trivially disprovable, because we produce intelligent beings smarter than ourselves all the time - it's called childbirth. And simply by random chance half of all children (in practice a bit more) turn out to be smarter than their parents. If we can do it with our wombs, why can't we do it with our brains and hands? It is merely difficult from an engineering perspective. $\endgroup$ – Fhnuzoag Apr 9 '15 at 17:48
  • $\begingroup$ @Fhnuzoag you simply do not understand anything. read the papers. It does not clain children cannot be smarter than parents. It say that the observer is not simulable by any other system due to self-reference. THE OBSERVER. Not any random person. $\endgroup$ – Anixx Apr 9 '15 at 19:11
  • $\begingroup$ I'm going to have to agree with Fhnuzoag here. This answer is just plain wrong. $\endgroup$ – ArtOfCode May 21 '15 at 0:34
  • $\begingroup$ @Anixx Let's say I have an algorithm that sorts data inside a database in an inefficient way that is also able to throw random code elements together and test if they end up somehow being a smarter algorithm than itself. There's a chance that the algorithm will 'accidentally' get smarter than it was. With an increase in complexity it will get very unlikely that this happens, even if educated guesses are used, but it's definitely possible. Also an increase in processing power always results in 'better' results. A chess computer that checks all possible moves doesn't need to be smart. $\endgroup$ – justthisonequestion Mar 10 '20 at 15:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.