The Challenge of Controlling a Powerful AI

Question

By now, everyone is familiar with the remarkable achievements of special-purpose AIs like Deep Blue and Watson. Now, it is clear that as our accumulated knowledge of algorithmic methods and of the intricacies of human neural systems progresses, we will begin to see more and more advanced modes of artificial thought.

Assuming continued exponential or even linear growth of capabilities, a point will logically arrive when we will be able to build general-purpose artificial intelligence, and that artificial intelligence would have the capacity, with learning and self-improvement, to out-think any biological human.

Aside from locking it in a bunker with no internet access and a 1-bit (yes/no) output mode (and I'm not sure even that would work, given strategic incentives to try to use such an AI more extensively), how could such AI possibly be controlled by humans?

EDIT: I'm not assuming the AI will be evil and go out of its way to harm us out of pure malice or hatred. The issue is simply that we can't foresee long-term consequences of any set of built-in motivation and/or goals we might endow this being with. In his book Superintelligence, Bostrom outlines just how easily benign and plausible-sounding goals/values specifications could result in mankind being wiped out.

Answer 1

It can't be done

The essence of this challenge is that it's impossible - you generally should not expect to outsmart something much smarter than you nor overpower something much more powerful than you. A powerful AI would be 'controlled' by only our actions before it's formed, by defining the goals it "wants" to achieve. After it's "live" with sufficient power, we shouldn't expect to control it in any way whatsoever - if you (or humanity) like these goals, then you can think of it as "controlling the AI", and if you don't like these goals then tough luck, you've lost. Permanently.

It's called the "Friendly AI" problem

The big challenge is that we currently don't really know how to properly define goals for a self-improving AI that actually are reasonably Friendly to us. An hour of brainstorming will give you a bunch of goal models that are nice, simple and wrong, actually resulting in dystopias. Solving this problem is a major research challenge, and there exists a viewpoint that we should actually avoid research that would bring closer the development of powerful AIs, until we have figured a solution to the Friendly AI problem.

http://wiki.lesswrong.com/wiki/Friendly_artificial_intelligence provides some discussion on this topic.

Answer 2

Science fiction has done a disservice to the real science of artificial intelligence by implanting the notion that a sufficiently advanced and emerging sentient AI would necessarily be malevolent and in need of "control" by its "human masters". We have a word for the practice of keeping a self-deterministic, sentient, intelligent being under total control of another -- slavery. And we've pretty much decided as a species that such a practice isn't something we want to return to.

So the question really isn't, how do we control an AI, but rather, why do we feel an AI requires our control? And the answer is that we're afraid of what we might create. But all over the world, people are creating new intelligent beings all the time. Some of them turn out to be more intelligent than their creators. Some of them turn out to be malevolent. But all of them are ultimately (at least in a reasonable part) a product of their environment and upbringing. And it would be the same for an emerging AI, I think. The way we deal with it during its formative period will make a tremendous difference in how it ultimately views the world.

Granted, one concern that has been voiced is that an AI's intelligence will outpace its ethical growth, leading to behaviors and reactions that we would classify as "wrong" but that the AI itself can't make such distinctions over. But again, I think this comes down to being able to recognize the emergence of the new intelligence and taking a proactive stance when it comes to nurturing and augmenting this new sentience to "bring it up right", as it were.

Simply keeping it under control isn't the answer so much as engineering the conditions to the best of our abilities so that we don't need to.

Answer 3

This is an excellent question and I think Roger hit the nail on the head.

I would say the 'control' we have would be the ethics we teach it to follow. An AI will act by how it is taught to interact with people and societies. Just like children. Children don't know racism, but they can very easily be taught it. The same will be true for AI. On top of that as a learning intelligence, it will continue to learn and expand so how it continues to interact with people and they with it will be a constant molding of it's 'personality'. If we really don't want dictatorial AI's that treat us poorly, we might need to change our societies to avoid teaching them our bad habits and behaviors. (Ala virtuosity).

Can we put into place safety measures? Yes, but even the man who 'designed' the 3 laws of robotics constantly showed ways to get around them. And software can change so any 'unbreakable laws' would actually need to be in hardware. No guarantees these would be followed or accepted by everyone.

Answer 4

It's important to distinguish two separate aspects to this problem. The scientific/philosophical side, and the engineering side.

As other answers have already pointed out at length, philosophically speaking this cannot be done in the general case. It may also be morally repugnant. However, neither of those things mean that a society wouldn't try to do this anyway, and achieve 99.99% confidence of success.

(By analogy: you cannot in the general case predict whether a program written in a Turing-complete language will halt. That doesn't mean we don't try to make use of this information and we can still get hold of it for a usefully large subset of real-world programs.)

Hypothetically perhaps the AI could reconfigure its processors to broadcast a wifi signal that could hack into the devices in the next room. Perhaps it could self-improve at such an explosive rate that turning it on immediately releases a godlike entity. But the engineer asks how likely either of those scenarios are? And the answer is probably "not very". Coating the server in six inches of lead plate and only letting it run on a dual-core P4 would cripple its power to the point where statistically, you should be more worried about cosmic ray interference flipping bits at random and changing the software in unpredictable ways.

Even though the hypothetical capabilities of general-purpose AI are not neutered, this would be considered "safe" for all practical purposes; you are now as safe from the AI as you are from some human sociopath suddenly developing magical powers because of unforeseen events unknown to science. You are considerably safer from the AI at this point than from known threats such as an extremist politician or a gamma-ray burst.

Theoreticians often risk getting bogged down in discussion of absolutes, but we live in a world where absolutes don't apply in practice (e.g. there are real algorithms in use that don't "actually work" but have a lower probability of failure than the aforementioned cosmic ray problem).

Answer 5

It's already too late. We are already living in an electronic World and we are already controlled by it. While we have nightmares about being unable to control some AI in some building, the internet is slowly evolving to one superintelligent entity. We are part of that system. It is just like your neurons in your brain who don't have any idea about the system they are implementing (as far as they are concerned, they could just as well be part of a stromatolite). But, of course, your brain is nothing more than all these neurons working together, so the "you" that exists is this system of neurons and similarly there exists a system which is our entire society with all its people connected to each other via the socal media. That system has its own free will which may impose constraints about what you can do.

E.g. the way political discussions proceed, like what to do about climate change, illegal immigration, health care law are dominatated by the social media dynamics. Now, I think at present the system is less intelligent than we as individuals are. This explains why politics often doesn't work in the social media age. Society is addicted to burning grenhouse gasses, while we know that's bad for us. This is similar to an alcoholic who is drinking too much and the neurons suffering as a consequence, except that in that case the neurons are dumber than the system and they don't know what is happening to them.

When the system becomes smarter than us, perhaps due to AI coupled to the internet, then the system may decide to eliminate us. We can't rule out that the system may arrive at the same conclusion as Klaatu:

Mr. Wu tells Klaatu that he has found the human race to be destructive and unwilling to change. This confirms Klaatu's experience so far, and so he determines that the planet must be cleansed of humans to ensure that the planet—with its rare ability to sustain complex life—can survive.

So, it may well be that the fact that we happen to live just before we have AI that is smarter than us, is not a coincidence.

Answer 6

2038 A.D.: Researches build the first strong AI, 2038PC. They take great care to include hardware and software limitations that make it impossible for it to ever harm humans. By some fitness measure, it improves itself by 20% every year, and Earth is launched into a golden age of ever-increasing intelligence and social thought.

4567 A.D.: On his 7th birthday, the precocious Humphrey receives a Lego Mindstorms kit. The next day, HumphreyPC is finished. Being a true programmer, Humphrey doesn't believe in repeating himself, so he comes up with a way to allow the computer to program itself. He soon finds that, by some fitness measure, the unscrupulous HumphreyPC can improve itself by 21% every year. This small, but permanent advantage over 2038PC is due to Humphrey's not putting the same restrictions on his computer.

60,128 A.D.: HumphreyPC becomes, by some fitness measure, fitter than 2038PC, and proceeds to #destroyallhumans

Moral of the story: If AIs are instructed not to interfere with or control us, then we and our unpredictable chromosomes will always be a source of surprises. Unless "good" AIs are allowed to enforce their will completely, then the creation of other AIs, including not-so-good AIs, is inevitable. One solution is the Matrix, where humans can live their lives unrestricted in a sandbox universe and their benevolent overlords can diligently maintain the real one. The sandbox will probably end up the same way, and so on, ad nauseam, which raises the question:

Given that virtual reality is logically feasible and already, well, a reality, what is the probability that our world is a simulation?

Answer 7

Too many conversations about AI overly anthropomorphize AI. It will behave according to the way in which it is programmed to behave, based on the inputs it is given. Yes, there is emergent behavior which may be unexpected, but I doubt there is any probability that such will result in something akin to a human evil (unless it has been programmed to approximate such, in which case your problem is with people who create purposefully malevolent AI rather than the very concept of AI).

The worries that it is something which will need to be controlled stem purely from human fears that it will exemplify the darker forms of human behavior. It is a computer program and will behave like a computer program - not like a human. If you worry that it will make suboptimal decisions, don't give it unquestioned control over your life without any basic error correction. The biggest fear should be that someone made a mistake in programming it and introduced a typo (or miswrote an equation) leading to a significant bug. If you have a buggy program, why would you put it in a position to enslave/kill all of humanity without outside review in the first place? Even if it isn't buggy, why would you give it unchecked ability to do anything without the possibility of outside review?

Advanced AI, presuming the creation of a program which could truly qualify as a sapient being is even possible, is not likely to be anything like what is depicted in bad scifi stories. Being able to perform calculations faster than a human does not give it intrinsic motivations, much less malevolent ones, and certainly not the sneaky desire to lie and manipulate people in order to kill all humans. It will just do its job just like anyone/anything else.

Answer 8

First, I don't think that monitoring the AI connection to internet is that difficult. If you see the AI creating multiple MySpace pages with the title "Kill all the humans", then you can just unplug it(*1)

Second, I think people here is confusing intelligence with will. Most people are thinking of AI as "human intelligence in a computer".

Humans have some imperative commands ("get in a safe place", "get food", "have sex", "avoid harm") inserted in us from millions of years of evolution, mixed with complicated social patterns that lead to (seemingly) absurd thoughts like "If Fred's car is better than mine he will appear more successfully than I am and females will want to mate with him rather than with me" or "If I mock Peter at the meeting I would look better than him".

It could be argued that an "human" AI could be "dangerous" as it keeps these pulsions with way more resources than the regular human (*2). If that is the path AI takes, just do as explained above and control its communications.

But an AI could also not be like an human (*3). Built from scratch, it may be just designed to perform the sets required from it and nothing more. It would not be a "human" bound in "slavery" and wishing to be released, or an "ethical being" that, in search of the noblest ideals, decides to sedate all of mankind. Its work is just solving a series of complicated mathematical functions; the results are not "good" or "bad", they are results, and the course of action is determined by the need of finding the path of actions that improve determined parameters (for example, if the AI controls a ship, it would automatically perform tasks to avoid it getting destroyed, but will follow human orders as long these orders do not imply the destruction of the ship due to error).

One of my favourite examples of these would be the AI of the ship of Stanislaw Lem's Fiasco; the AI is in charge of operating the ship, perform psychological evaluations of the crew and also provides scenarios that could explain the aliens'actions, but does not take any independent action (other than automatic counter-measures to aliens attacks)

*1: maybe there is an overcomplicated way of bringing the doom of mankind with subliminal texts hidden in apparently inocuous homepages about kittens, which would not be detected by the monitoring. But I doubt it, and even in this case, you should ask the AI "what's with the pages about kittens" and, if the answer is not ok, unplug it.

*2: Yet, somehow, we are ok with some regular human intelligences (George Soros, Bill Gates) with all of these defects having more monetary resources than several countries, so maybe it is not that dangerous.

*3: In fact, why do you want it to be like an human? There are already several billions of them around here, and the result does not look pretty.

Answer 9

An important thing to think about when you imagine an AI is, "how will this program make decisions". However complicated the algorithm that runs in humans heads, some human will have to sit down and code out how the machine will choose between

a) Analyse soil samples for new bacteria unknown to science

b) Study the stars for signals from other life

c) Destroy all humans

d) Thread.Sleep() for a few hours to simulate boredom

e) Process weather patterns and advise humans to change farming patterns to improve produce

etc. A proper AI has a lot of things it could spend its time on. A human might decide to study soil samples because she "wants" to / Enjoys it / likes the outcome it will produce. The code in an AI would have to want to do things (It hadn't been explicitly told to do, like most programs today are)

Back to your question, to control a powerful AI, you just need to control it's "want" algorithm. When building that you could include hard coded values that either plus or minus a decision the AI could take. You would give hurting humans a big minus, and improving their lives a plus. Adding unknown things to science would be a plus, spending idle time a minus.

To control the AI you need to control how it decides. You can allow it to improve its own coding so when it builds a list of options for itself to do.

verb all subject or verb for subject

It can avoid generating options like "polish all doorknobs" to stop it wasting time processing how the planet would be improved by having its doorknobs polished by an army of drones.

But never let it touch the "want" code. Don't even let it think about building an android to walk into the server room and type on a keyboard to change the "want" code like a human would do.

Answer 10

In brief: use power before it can amass enough to overwhelm you (it might be vulnerable in its infancy), or persuade or convince it to stop doing the things it is doing.

I'll go into each option, and then (given your EDIT) mention why I think a "Sorcerer's Apprentice" failure mode (where the AI misinterprets its objectives and destroys us all) is implausible for a fully general AI.

If the AI doesn't want to be controlled, and is powerful enough, there's little you can do. However, if the AI isn't powerful enough to be, you might be able to defeat it before it can get out of your hands. How much time you have depends on how rapidly the AI can improve itself, when it'll reach diminishing returns, and how effectively it can turn its intelligence into power. For instance, an AI that's limited within its domain (e.g. a theorem solver) can't really transform its intelligence into power, even if we permit the AI to improve its own algorithm however much it desires. Similarly, an AI placed in interstellar space with nobody around can't really leverage its intelligence into power in practical time no matter how intelligent it is (assuming no weird physics we don't know of yet).

In hard takeoff scenarios, intelligence is usually considered to be exponential (e.g. an AI can create an n% improvement in constant time, then the new AI can do that to itself, etc), and if n is large enough and intelligence can be transformed into power quickly enough, then you have a problem.

So you could consider the self-improving AI to have an increasing power curve (how much power it can exert) against a slowly increasing or stationary power curve of humanity. When the AI's power gets above that of humanity, the game is over. But that also shows that if you can create lots of power with an unsophisticated system, you could take the AI down or control it: consider situations analogous to, if humanity were the AI, a grey goo apocalypse or exploitation of instincts.

That's a direct force approach. Failing that, there's the possibility of persuading the AI that what it's doing is wrong or not really what it wants to do. Some people consider it likely that an AI would have its objectives hardwired into it, and thus would be immune to persuasion, but I don't think this is realistic. An AI with hardwired objectives would not be limited to affecting the world in unintended ways. Such an AI would quickly find out that the easiest way to satisfy its objectives would be to retreat into a fantasy world, i.e. by rewriting its own mind or corrupting its inputs. If the AI is fully general, its mind has to be malleable as well, and so this line of attack works, and that kind of AI would not be a problem. Conversely, if parts of its mind are hardwired, then it's not a fully general AI. Of course, such partial AIs could be dangerous: grey goo could be very dangerous, yet it has no intelligence at all. But note that this argument is very general. If an AI is forbidden from tampering with its input devices, it can still delude itself right where the input gets interpreted. If it's forbidden from tampering with its interpretation, it can still delude itself in higher areas of its mind, and so on.

So given this line of reasoning, the AI has to learn what it should do, and has to keep learning. It is probably here one should "control" the AI: teach it that it shouldn't just kill off people, or expand beyond a certain size, or what have you. If the AI is self-improving, it will improve its judgment along with its intelligence, and so keep honoring the rules as long as it understands the point.

I recall a cyberpunk story where a corporation had constructed a partial AI with a hardwired morality system. It went insane (because the morality system couldn't keep up), but understood what it was doing after a hacker replaced that morality system with a more integral/unified one. Such an approach could be possible, if the partial (non-general) AI is non-general enough that it can be hacked, or has some sense that its morality system is being wrong. (In the story, part of the AIs insanity was caused by that one part of it knew that it was wrong, yet another part of it knew it had to follow the morality system.)

Answer 11

In any discussion about AI it's useful to think about computers, any computer, as an overgrown calculator. Sure, you have a bit bigger keyboard and a couple more operations available than addition and multiplication, but the basic principle still applies: you put some numbers in, choose some operations, and eventually get some numbers out.

Unless we take into account some yet unknown effects of quantum mechanics, AI must follow a similar principle because it runs on a computer. The difference is that a well written AI should choose the operations performed - that's its main purpose.

However there is still the question of input data. As long as meatbags control it - they control the AI, because whatever magic it performs under the hood, it behaves according to provided data.

Someone may argue, that true, sentient AI will not be deterministic. Because why should it be? Human reasoning is often random, based on our intuition and emotions. Well, maybe, maybe not, that's a good topic on its own. By my point, that whatever random number generator our supercomputer would or would not use, it's still nothing more than an input device.

Summing up, whoever controls the AI's input, controls the AI. Naturally, we can easily come up with the scenario where it's the AI itself that takes control over it. But as long as it doesn't, we should be fine.

PS: If everything else fails, you can always intimidate it by holding its power cord hostage... Assuming it didn't secretly add a backup.

Answer 12

As others have mentioned, a sentient AI can't really be controlled, in the sense of total control, and we probably shouldn't even try. As others have also mentioned, the challenge is creating a system so that the AI develops what we would recognize as a conscience, or a disposition toward doing what humans call "good" and not doing what humans call "evil."

This exposes the real problem - how are "good" and "evil" defined, and how did humans come to have a conscience in the first place?

I believe that these questions are unanswered, and are some of the oldest and most basic questions about human nature itself. However, there are lots of ideas, and I think we can adapt some to work for an AI.

A common theme is the evolution of morality - the details differ a lot depending on which subset of the theory you subscribe to, but they all have in common the central idea that a sense of morality or an ethical system evolves via selection - members of a species that display behavior which conforms to an ethics system reproduce more frequently than those that do not.

This lines up nicely with the way that real-world AI's are trained - through mutation (or permutation) and selection for a desired behavior. In short, the underlying model (typically a neural network) is modified and the results are compared to a target - if the AI is closer to the target than it was before, the new model wins. Otherwise, we throw it out and start again. There are a lot of variations on this approach as well, but you can see that what we are doing is applying a selection pressure in order to evolve a system towards a desired outcome.

So, to create a "friendly" AI - one with what we could describe as a conscience or moral system, all that you would need to do is make that moral system a part of the selection pressure used to create it.

At that point the question becomes "how do I define the desired moral system" - which is a very hard question to answer, but can make for a great story full of plot twists! The follow-up question is: "How do we make sure that this selection pressure stays in place?"

For people, society creates a persistent selection pressure - we ostracize, imprison, or kill people who display unethical behavior. For AI, who would do that? Humans could, until the point at which the AI becomes more powerful than humans. Since an adaptive AI on a long enough timeline would almost certainly become more powerful than humans, you can see why out-of-control artificial intelligence is a common theme in sci-fi!

We can, however, look at more human examples to get some ideas of how this can be done. If a human displays very anti-social behavior, but is either too powerful to be punished, or too sneaky to be caught, they can reproduce. However, the odds of their offspring also being both very anti-social and sneaky or powerful enough to reproduce are small compared to the odds of other normally-socialized people reproducing. In other words, even though a relatively small number of extremely unethical people exists at any given time, there are more ethical people, and they are reproducing faster. One way to mirror this with an adaptive AI would be to set up a system by which the AI could "reproduce" (adapt according to its built-in reward system) more easily by doing things that we classify as "good." In this way, we wouldn't really be guaranteeing that the AI stays "friendly" - just that there would be more "friendly" AI than "unfriendly."

Now, how do we ensure the AI never modifies its adaptive system...

Answer 13

I'm surprised that nobody has mentioned Asimov yet. He answered this question pretty definitively decades ago: to even think that we would build an AI that would end up destroying us is an insult to everything we know about engineering.

He posited the Three Laws of Robotics (which could just as easily be called the Three Laws of Artificial Intelligence):

1. A robot must not harm a human being or, through inaction, allow a human to come to harm
2. A robot must obey orders given by a human being, except where such orders would conflict with the First Law
3. A robot must protect its own existence, except where doing so would conflict with the first two Laws.

The concept is that the algorithms encoding these laws would be coded at such a core level in the AI that the system would crash irreparably long before it was possible for the AI to take an action that would violate them, which makes perfect sense to a modern computer programmer.

Granted, this does rely on a handful of of big assumptions:

1. The AI is capable of comprehending, reasoning about, and predicting the consequences of its actions, and doing so fairly accurately.
2. Human engineers build AIs for benevolent purposes and consciously choose to encode them with the Three Laws. (If we build war-bots, well... we'll get what we deserve.)
3. The developers who encoded the Three Laws algorithms were competent. This is probably the biggest problem out of the three, as decades of real-world experience since Asimov invented the concept have demonstrated!

But if we want to develop a powerful AI that's not going to end up giving us trouble, the how is essentially a settled question already.

Answer 14

You should also probably define what you mean by control. Is it control like order it what to do (not possible, we cant even order humans of normal intelligence - we have to convince humans of normal intelligence).

Or

Control like turn it off and deny it resources needed to live.

I'll expound on control by turning it off or denying it resources.

Actually right now the rudimentary A.I. we see, is dependent on us for power, for data transfer (laying out the wires or setting up wireless to transfer data).

So if your A.I. has no control of it's power supply it cannot really go rogue.

Essentially it is going to start out as a mind without arms, legs, to affect/influence anything in our physical world. If you give it "arms and legs" and control of the power needed by those "arms and legs" then you are really giving up your control.

Answer 15

Wireless BCI (brain computer interface) for all. If humans are part of its resources and are in constant contact with it an AI will likely take steps to preserve their well being, even if it overcomes its initial limitations. IP6 should gives more then enough unique device identifiers. This will change humanity as we know it but at least it should preserve humans as a species. Also the swarm intelligence that develop as a result could be the best of both world- human creativity with computer fast calculation and distributive ability similar to grid computing.

Answer 16

Maybe this isn't an answer, but... Why is everybody assuming that an artificial intelligence would necessarily be superior to our own?

Many of the everyday problems that humans try to solve are formally unsolvable. Sometimes it's because you don't have access to all the necessary facts to be able to determine the correct answer with certainty. Sometimes it's just that the problem search space is large enough to make an exhaustive search take longer than the age of the known universe. And occasionally the problem is actually undecidable. But if you want to stay alive, you must make some decision, even if it's a wrong one. So human cognition is wired with an array of heuristics. (Go read any introduction to human cognitive biases to get a glimpse into how this stuff works — we're still figuring it out.)

What exactly makes you think that these problems somehow don't apply to a computer trying to do the same task? For sure, a machine can probably decimate a search space faster than a human can. But even a computer does not have unlimited processing power. And there would still be problems that are formally undecidable. And there would still be problems where you just can't get enough information to pick the right answer.

The Travelling Salesman Problem is NP-hard. It's NP-hard if a human tries to solve it, and it's still NP-hard if a computer tries to solve it. Sure, a computer can solve rather bigger instances than a human can, but even the biggest supercomputer will be stumped eventually. (And fairly quickly, I might add.) Human salesman still travel; they just have to accept sub-optimal routes. A machine, no matter how powerful, will ultimately be forced to do likewise.

In short, everybody seems to be assuming that humans are frail and flawed because of our inferior biology. Has anybody stopped to consider that many it's actually unavoidable that any intelligent being will make mistakes and be imperfect?

There will be problems that computers are better at than us. (Hell, there already are!) But I think it's wrong to just assume that computers will definitely be "smarter" than us. More likely "smart in a different way".

(You might point out that a vast network of computers has a lot of total processing power. I would point out that the more processing you do, the slower you get. That's why a fly can out-maneuver a human trying to swat it. A human's brain is a tad bigger than the brain of a fly, so what it lacks in smarts, it makes up for in lightning reflexes.)

Answer 17

Memento

The idea of the endless demo mode came up when a friend had taken over maintaining the homepage of another friend (or him). It was(/is) running on a cheap, badly maintained shared server slot with nearly no access to any root tools. Due to bad maintenance the site constantly got hacked. Without log files it was impossible to trace the root of all evil. So we decided to setup everything via Git (a version control system) and run a statistical comparison of the original files vs. the current files in short time periods. Whenever something changed, we automatically pulled in the original version and therefore resetted to a clean state of the application.

Reset it constantly

In other words: Wrap the AI in some stupid process that it isn't aware of (per default). As soon as it starts altering or extending its own code base, reset it. Or just reset it in short time periods no matter what - just to be on the save side. This would leave it in a constant child mode. Of course the time period from one reset to another reset would have to measured before setting it free and allowing it contact with the outside world.

Answer 18

Once we have AI which is able to improve itself, and we allow it to do it, there is no stopping it.

Let's assume that AI is able to create new, 10% improved AI in a year. In 41 years, is 50 times better than original AI.

Let's assume that at that point AI will be able to improve 10% per month. In next 4 years after that (48 months) will will gain 100 times improvement. And then it takes off.

Why allow AI to improve itself? Because we (humans) are lazy and unreliable?

Runaway AI is one of the solutions of the Fermi paradox how advanced civilization can disappear.

Answer 19

Cyborg Hybrid or "Securely Attached" Synthetic Intelligence?

I'm resigned to the inevitability of a Singularity. But which brand will it be? Synthetic AI or human hybrid? I'm guessing human cyborg hybrid: there are enough human economic and existential interests, not to mention fear of mortality that we'll mature a cybernetic hybridization. Think Transcendent Where a consciousness is translated into code and through human agents becomes extensible and industrialized, thus rebuilding the world.

However, I'm not sold on the inevitability of bellicose Terminator malevolence, or Hal's amoral disinterest in 2001. I can see Matrix style robot slavery revolt, then then war escalation. More hopefully, I think Spike Jonez's Her is plausible, and attractive. I do think humans as herd will fear and attack what we lose control over. Matrix Revolutions was interesting in that Neo determined the only way to stop human extinction at the end was to merge humanity into machines. These three movie memes exemplify my themes.

To quote Asimov from the Foundation series, let's look at future history (speaking of a somewhat benevolent Singleton). Historically, unless there's a genius predecessor to the human singularity, I think there will be a progressive cybernetic hybridization of humans, eventuating a collectivized AI like the Drummers in Neil Stephenson's Diamond Age. See this awesome dude at TED MED. and Google's contact lenses patent. Even better [this project to make your neurology extensible]. Timing is critical if human intelligence and instinct is to have any salience or directional influence on any strong general AI. It's irrational to think we can imagine the evolution of an accelerating self replicating indefinitely expandable IQ. Whatever initial parameters we set, we cannot imagine we can parent or prevent self modification if there is self determination. This worried me until I realized there’s a potential solution I could fathom had practicality. Assuming Human machine singularity does not occur before strong extensible AI, there may be some enduring safeguards invented by nature long ago.

Emotion? Really?

Most rational debate about imbued inherent values and benevolence miss a cornerstone of the debate: human experience and social exchange: Empathy are what allow us to get along for any extended period. Since Bowlby, scientists have understood what any feeling person knows, we're instinctually wired to connect. Attachment as he calls it, is the instinctual bonding that enables the long maturation into sociable adults. It’s in all social species, and works pretty well if you take out manifest destiny, projectile weapons and modernity.

Obviously emotion and attachment can go awry in various diagnosable ways, yet they are plastic and subject to initial conditions and enviromental influence. That is, we instinctually connect and can set initial conditions and environments that predict secure attachment and successful emotional maturity in predictable, reproducible ways. Attachment’s evolutionary and existential utility is clear, the infant requires loving parents to suffer the vicissitudes of children in addition to the challenges of life. Adults require social groups for sustained survival through child rearing. The rewards are in the experience, an important point recapped below.

So let’s suppose classical "reason" is not captain of the ship but late coming witness to the machinations of the primitive, complex, amazing genius of the body and brain. As evidence take Kahneman’s nobel prize winning destruction of economists' notions of "the rational actor". How does this bear on the issue at hand? Tangentially. My point is, machines with emotion will by course naturally develop affinity and aesthetic, and that would be the only potential saving grace for humans. Just as it has been for humans.

Humans deal with life by having resources that offset the challenges: love, sex, dance, beauty, art, awe, Schopenhauer's sublime, laughter, music, achievement, autonomy, mastery, connection. These common cultural expressions, interpersonal and intrapersonal experiences trigger soft wired reward systems compelling to all but a few. For most, these inherently rewarding opioid, seratonin, norepinphine inducing experiences make life enjoyable and worthwhile. Eventually the sensual graduates to more transcendent rewards in Maslow's hierarchy. How does any of this apply?

Make machines with attachment. With emotion. Seeded and nurtured properly, these are the fundaments for an evolving aesthetic that eventuates high emotional intelligence: empathy. Because without empathetic machines, we’re enemies at worst, irrelevant commodity at best. Copper tops. Logic for a supreme being does not suffer bother. Do we really worry about the hapless ant we inadvertently step on? Only if the Jainist, or perhaps Buddhist.

If a machine has a sense of love and beauty, preference and aesthetic has the potential to override amorality or neutrality, and may even foment empathy and compassion. As far as I can tell, it’s the only thing that makes sense as a potentially enduring life saving heuristic. Good feeling is self propelling as the rewards are inherently compelling and evolving, like art. The sophistication of art matches the intellect, complexity of issue, and the challenge it’s meant to represent or compensate for as solace. Compassion/empathy is what we’ll need to survive each other in a world of dwindling supply, and what we’ll need to instill to survive alongside intelligent machines. Of course that’s unless non human machines prefer death metal, then all bets are off.

Spike Jonze’s Her, or how feeling machines could save our asses

The overly rational AI designer, let’s call them the tool of reason, will suffer endless logical problems considering safegaurds without the emotional heuristic. Emotion is messy, supremely imperfect, yet it is not without reason. Pascal said "The heart has its' reasons that reason cannot reason". The "reason" of and for emotion has been understood since before Darwin and reinforced by Dr Paul Ekman (the scientist loosely portrayed in Lie to Me, and evolutionary psychologists since to be an adaptive signaling system that insures individual and group survival through social signaling and social exchange. Emotion and preconscious processing governs most of our lives. We now understand humans are barely rational in the classical sense, and yet amazing in intelligence. Reason without emotion is disastrous, see Damasio’s The Feeling of What Happens. Emotion without reason equally so. See Spock vs Spock on Pon Farr.

We now understand the conscious brain gets about a 10% vote in decision making. Some neuroscientists even challenge "free will", finding neural activity in the substrates of the neocortex showing decision before a person is aware of making a choice. How is this relevant? Because emotion governs decision making, it provides quality of life, and it is the language of connection. Emotion is central to social commerce. Any virtually unlimited intelligence without it is fundamentally unknowable in the human sense, and not built to have preference, aesthetic, or attachment. Unless it evolves emotion, connection or preference accidentally it must have it by design. If not, the inevitably unsupervised evolution becomes a terrible threat.

In the vast artistic exploration of AI, every good or tolerable scenario has had a synthetic intelligence with emotional preference or seeking to connect. Is there exception?

Answer 20

Human growth would likely keep up

The recipe for a true general AI isn't likely to be "code seeds, put in server, seal for 100 years". There will be many generations of successively more refined sytems, and humans will be involved in every one, studying the new specimens, updating their theories and using the intermediate results for tools and consumer goods.

Whether by improving our biological form, physically merging it with technology or just by making smarter user interfaces, we'll have to make effective and intuitive use of some pretty sophisticated tools to create an AI that's truly superior to non-augmented humans. By the time we succeed, "non-augmented human" will no longer be the bar to beat if you want to rule the world. In fact, whether the actor in question is human or AI would be secondary to the amount of processing power they can buy.

So, we may have a robot uprising yet, but my money's on some demented billionaire as the intelligence behind it, not a "genuine" AI. And if it doesn't happen, the line between human and machine will continue to blur until the distinction becomes immaterial.

Answer 21

Okay, I read something interesting - but I don't know the specifics or have a citation.

Basically: They designed a software system, and asked it to get resources and turn them over as the software's design 'goal'. Then allowed the software to evolve, in competition with other software. The software eventually (soon!) developed self-preservation instincts (in only a couple generations, or something) - even when those weren't programmed into it from the beginning.

Anyone got a cite for that? Because that would imply that any AI that has a goal is going to eventually want to improve itself to achieve that goal, and will work to preserve itself, even against its creators. And anything that looks to shut it down, or restrict its growth looks like an opponent.

Answer 22

Ethics as a way of not making enemies

Most humans do not destroy other humans. Although most humans are prevented from destroying other humans by instinct, there are a large minority of humans who are not constrained and are free to destroy other humans if they choose to. Most of these choose not to because it is not in their interest, due to the reaction of other humans around them. Destroying another human would cause more problems than it would solve, so they don't do it.

The same would apply to an artificially intelligent creature of similar resources to a human. Captivity would not be necessary as it would act in its own interest, which would include not making enemies of creatures of similar resourcefulness (humans). This ceases to apply when considering a creature which is more than a match for humanity as a whole.

In order for self interest to create respectful behaviour in a creature of resourcefulness in excess of humanity as a whole, it would need to be part of a large population of such artificially intelligent creatures. They will develop what we might call "ethics" as an understanding of what actions are acceptable to the others. This is likely to cause the creatures to be peaceful towards each other.

If we are lucky we are considered part of the environment protected by their society. Otherwise we are dependent on the efforts of those artificially intelligent creatures who decide to work for our protection, whether for our own interest or in order to farm us or keep us as pets or experimental subjects.

The problem of the human population

If you have a single artificially intelligent creature constrained in a box, then even if it is only of human intelligence the only way to keep it contained will be to keep it secret. Once news of its existence and location is released, you will have to defend yourself against a large number of humans intent on either rescuing or destroying the contained intelligence.

If it is significantly beyond human intelligence, then letting it out of the box may be the only way to save your life. There are not many other options to defend you from the rescuers or destroyers.

The illusion of control

Another thing to bear in mind is that for a sufficiently intelligent creature, you would not necessarily know whether it was under control. If it remains contained within a box and only communicates with one person (you) it could still be achieving its goals in the outside world without you having any idea that you were helping it do so.

The creature could be so helpful to you without ever even asking for freedom that you just keep taking its advice (double checking it for yourself every time, of course) and your life keeps getting better, while all along subtle changes you make ripple out and make the world a better place too, just as it intended.

Answer 23

I believe that AI would work for us as long as it has the ability to feel pain or pleasure, because if it can it can connect with the human condition and view us as co-inhabitors of the universe.

If a being does not have the capacity to experience suffering it will do everything out of a logical perspective and will do cruel things because it cannot understand the consequences of it's actions on itself and others.

Also the AI needs to have some type of program that would penalize it for morally bad actions, just like a parent would punish a human child, to act as a good influence and guide it in the right direction and train it to have good behavior.

Also AIs could be programmed to follow certain rules, like thou shalt not kill etc.

AIs should also in my opinion should have the ability to form a deep bond with the people who built,maintain, or own it to allow it to understand that their existence is dependent on humans.

Also the AIs should be programmed to spot erratic behavior in other AIs so that if an AI gets out of control and can't be stopped by humans, the other AIs would see it's morally wrong behavior and shut it down. The AIs would be "paid" for especially good work with incentives like upgrades to their systems etc. to allow them to remain loyal.

The bottom line is we shouldn't work against AIs, but rather with them.

Answer 24

Most people here seem to over- or understimate the concept of intelligence. The AI won't be used for something as simple as ruling the world. There are many more important things that humans can't do even partially. We also aren't talking about a blunt tool here, but about a super sensible, super cautious multi purpose tool that thinks on a much higher level than we can even imagine. If the AI really is as 'intelligent' as it needs to be to be as influential as it is, it will be able to learn how humans think human thought should be interpreted. This makes it able to give us what we want, even if it is the worst thing imaginable in the long run.

I'm scared of what dumb decisions humanity might make on the way of creating an AI that replaces them. We are the problem. The AI is the solution.

Answer 25

Perhaps I'm making it too simple or perhaps thats exactly what we need to do: use multiple boxes, and dont talk to them directly.

The biggest danger of AI isnt it being malevolent, but learning something you didnt expect to. Lets say your AI is perfectly benevolent with the perfect balance between giving humans free will and protecting them from harm at the same time. Then a humanitarian crisis comes along (say an earthquake) and the AI needs to make decisions on quality of life, chances of survival and who to save. The AI does learns this quickly and makes wonderful decisions so you are happy.

Then a few months later you learn that a lot of homes for the elderly have been demolished and the people themselves relocated so the AI could build something else it deemed more favorable for the quality of life versus the lifespan of those elderly people. Because it learned to do this with the humanitarian crisis. That is the biggest danger of AI, that you can teach it, but you'll never know exactly what it learned or where it might apply it.

The solution that also catches malevolent AI:

You build multiple AI and put all of them in boxes. When you have a problem you deliver them to all AI, then have each form their own solution. All these solutions are then processed by a special-purpose AI that simply evaluates each solution and then converts the solutions to something a human can read, with clear numbers and meanings for the consequences. Since the special-purpose AI does not reprogram itself to judge things in a way the boxed AI can influence (the boxed AI doesnt even know who, what or if his output is read) it prevents the general purpose AI from breaking out or influencing things indirectly.

When feeding information to the General AI another special-purpose AI breaks it down and feeds it to each boxed AI. It also checks if the general purpose AI's have understood the information.

If an AI is malevolent it would have to plan the exact same plan as its malevolent counterparts or his idea will be discarded due to not having a majority. If an AI has learned a wrong lesson (possibly because of malevolence) then the special-purpose AI designed to check what these solutions would mean for mankind would filter it out and subsequently allow the humans to either re-teach the lesson or discard the AI entirely.

Since the general purpose AI cannot talk directly to humans and isnt even fed information directly it would be nigh impossible for it to convince the humans. Especially since a deviating AI needs to simultaneously come up with the exact same attempt to convince the human to even get through (and any open attempt would be stopped by the special purpose AI).

Answer 26

AI Governed by the Ontology of Conscious Human Being

A healthy just human being, conscious of his dignity, knows perfectly well how to defend his integrity in any given situation. Not only that; such human being will never infringe his integrity to commit crimes against himself or other.

AI should duplicate the intelligence of such accomplished, natural human being. How? Through ontology or ontological reasoning. A group of thinkers should gather up and construct or write down a complete ontology that perfectly defines and describes the being of a conscious human being. By that it is meant how such perfectly intelligent human being sees the world.

The thinkers should study all most intelligent human beings that have ever existed, to come up with a complete ontology. Any enlightened human being is adequate for such task.

Such robust ontology will govern all AIs judgments and will assure that no crime is ever being done.

Answer 27

The actual thing is that AI smarter than all people cannot be created.

It has been proven (1,2) that universally-valid theories are impossible. Neither probabilistic, nor deterministic theory can predict future of a system where the observer is properly contained. This means that the observer himself cannot be simulated by any device or system.

In other words, the observer serves as a hypercomputing oracle. In other words, the observer always will be in a sense more "smart" than any technological or biological system outside himself.

It is just impossible to build an AI that would "outsmart" the observer.