Do we fall into an "artificial intelligence" trope or is it reality?

Question

In regards to questions on artificial-intelligence, it seems like there is always usually an immediate opinion that the goals an AI is tasked with is going to end up with the opposite effects than what the programmers want. Or that a "powerful" AI is somehow "against humans" because it is "smarter" than us.

The Challenge of Controlling a Powerful AI

AI tasked with bringing down medical costs? What could possibly go wrong?

The AI that fails to be evil

It seems that, if we give an advanced AI any kind of "goal" and let it loose, there is no preventing it from going absolutely wrong in the worst possible way (in regards to that goal anyway).

Is this just a trope arising from Isaac Asimov's books and investigations on the topic, as well as other stories where it is claimed that "we found the perfect rules for intelligent robots"? Is this so dependable that we can tell the AI to do the exact opposite, and attempt to program it to be evil (see link above), and it will turn out to be good?

Given a setting where robots maximize human happiness (the ways in which that is defined will have to be handwaved), can it be realistic that the AI actually works the way it is meant to, or is it actually more realistic that the AI will turn out opposite than what the programmer intends?

Answer 1

This is a trope that arises from Real World AI research, not Asimov's stories (although frequently quoted).

The question at the heart of the problem is the moral and ethical philosophy the human race has been struggling to answer for over 2000 years. That is: what, objectively, is "happiness"/ "harm"/ "human"/ "alive"/ "good"/ "evil"?

It is very easy to define a set of rules that on paper look good, but which in practice are full of holes. For example, "alive" in the sense we mean it is some undefinable aspect of organic chemistry.

If I attempt to define it, then meat at the grocery store is alive (because on a cellular level, it still is: that's what makes it fresh) or sleeping people are dead. Or comatose people. Or (worst case) people under medically induced death as part of a surgical procedure. No, really, that's a thing. We routinely stop people's hearts from beating or keep them from breathing, while under the effects of general anesthesia in order to operate on those organs. From an objective point of view, these people are dead, they're just going to "get better" later.

In order to get to an Asimov level AI (which is imperfect, see: any of Asimov's books featuring robots) we'd have to solve an unsolvable problem.

Ergo any AI we do program is going to be imperfect and its failures will be spectacularly dangerous in one way or another. Computerphile has a great episode on this, which gives an AI the singular goal of collecting stamps.

While it does get hyperbolic ("what idiot would give a computer access to the machines necessary to harvest the carbon from humans in order to make more stamps?") it illustrates a point: merely being able to perform its assigned task, it will have some level of power that will not be desirable, and limiting that power inhibits being able to perform its task at all.

In the case of the HealthInsuranceBot, it comes down to having control over who lives and who dies (via access to medical care). It doesn't matter what the intent was, there is the potential that the power given exceeds "rational thresholds" which is impossible to reign in. The AI can declare "that man is 68 years old and a smoker, he's going to die in 2 years regardless of how much money we spend on him trying to keep him alive, according to my programming, he will only drive costs for everyone else up." Suddenly this person cannot get any access to medical care (not even to make those two years painless) and has been effectively sentenced to death, even if the AI itself didn't issue a death warrant. The utility function has decided that any care is too expensive, even if a rational human being would have done something to provide some end-of-life care.

On the other end of the spectrum, if you try to keep all people alive as long as possible, you end up with people in a persistent vegetative state eating up resources despite being (more or less) dead and will never recover.

In the gray area, there are cases that look black and are actually white (vegetative states that have recovered) and cases that look white and are actually black (rather than surviving 2 years with lung cancer due to a lifetime of smoking, they survive 10). No amount of predictive software is going to make those hard decisions easier, and as soon as the computer is given authority (final, absolute, recommendation, any authority whatsoever), those decisions are no longer seen as altruistic and the AI has become evil.

Answer 2

AIs can act as they are programmed. The immediate jump to "the worst possible case" is really just a cautionary tale.

The reason that tale is so popular is because it becomes very easy to start to believe you can tell an AI to do anything. You may then tell it to accomplish the impossible. However, there is a subtle detail that many who delve into AIs forget: it is remarkably hard to develop a perfectly objective language with which to phrase your requests. Language is always subject to interpretation, and there is no known way to guarantee an AI will agree with the interpretation you intended.

This is, in fact, true of human to human interactions as well. So what is the difference? Typically, in the AI "trope," the AI can move very fast. When humans fail to understand each other, there is time to say, "whoa! Stop right there. When I said I needed a baby sitter, I did not want you to sit on the baby. Lets get back to a stable state and discuss what was meant by my phrasings." If an AI thinks and acts too quickly, or if the human walks away, this normal feedback loop in language falls apart. If an AI has not been taught the value of being careful of misinterpretation, these actions may be irrevocable.

There are dozens of solutions to this. In fact, techniques like Multiple Hypothesis Tracking have demonstrated solutions to similar problems for decades. The AI doesn't always need to go awry. However, these solutions are in the opposite direction computer programming currently is trending. The solution is not to make an ever more precise language for describing our needs, as is very popular in programming today. Instead, the solution is to make the AI more robust to the lack of precise language. Any effort to do this naturally heads off the amok AI problem rather elegantly.

The second half of the AI trope is that the AI is too powerful to be controlled by humans. This can easily be true, but it isn't the first time we've dealt with such powerful forces. Consider the development of the atomic bomb. It was not clear just how much control we had over the process. The scientists had to work with unknowns on a constant basis to develop a level of control that we were comfortable with.

I'm reminded of a quote from General Grover's book, Now It Can Be Told, chronicling the Manhattan project from a logistics perspective (he was no scientist). When setting off the first nuclear test, each scientist was permitted to observe the blast from whatever distance they felt like. No army individual told them what distance everyone would be at. They scattered at many varying distances from the blast. There was indeed a question of whether or not they had control of the beast they were unleashing.

An unstoppable amok AI is similar in nature to a large nuke or a biological attack in that it is never really clear how much control one has. The only difference is that AIs clearly learn, so our control is an even more complicated concept. Then again, this is not a new problem for humanity. We deal with it every time we raise a new generation, never fully having control over them. We're decent at it.

The amok AI trope is a warning story, not the only story that can be told. It is a much needed warning story. People are often rather innocent when it comes to the risks and problems associated with AIs, and think they can do things they shouldn't. But just take a look at movies like Big Hero 6. There's other stories to be told, besides just the warning.

Answer 3

It really is scary

There has been a lively discussion of this topic for decades. I have multiple posts on the topic, and I'm a firm believer that Artificial General Intelligence is the scariest thing that can (and likely will) be invented, as far as the perpetuation of biologial mankind is concerned.

My views on the topic have been shaped the most by reading Eliezer Yudkowsky and Nick Bostrom's body of work. I believe that Bostrom's book Superintelligence, while very dense, imperfect, and rather technical, is the best work to date on the topic.

I have discussed the concept of Instrumental Convergence elsewhere, so I will only state the conclusion: Remember, AIs are not like humans, they likely do not get bored, do not get lazy. It will pursue its goals tirelessly, ruthlessly, unceasingly. Humans just happen to be in the way.

Does the AI go FOOM?

Human intelligence is limited by both hardware (limited speed of neurons in a limited-sized brain) and software (limited number of ideas) There is no reason to expect an artificial mind to converge towards a human-level intelligence. We simply do not know how much more difficult it is getting from a human to super-human levels compared to getting from a dog-like-mind to a human-like-mind. If the AI does not go FOOM, we will have slow progress and years of breathing room for political and ethical debates, offering more of a chance for humans and our slow institutions to react coherently.

If on the other hand the AI does go FOOM, blitzing like a Japanese maglev train from mouse-like intelligence to superhuman levels, passing human-levels so fast we can barely see it pass us by, then humans and human institutions do not have a lot of time to adapt.

Do-what-we-mean vs. Do-what-we-say

The biggest hurdle is perhaps in terms of our own limitations. Do we know that our current value-sets will be the best way for mankind to live by a century from now, nevermind a billion years from now? The values we imbue our AIs with will have consequences that will ripple down across space-time. Moreover, and more worryingly, how do we explain to an AI what we want, when we don't even know what is good? Say you ask an AI to minimize suffering, and it suggests putting everyone on morphine drips. Obviously not what you "intended" but how do you convey that to an AI, given the inherent imperfection of human language. Moreover, even if the AI understands that this is not what the creators meant, how can we make sure it cares about that? This is the Coherent Extrapolated Volition (pdf) problem.

Reverse Psychology won't work

The OP asks, exasperatedly, if all attempts to try and tell an AI to be "good" end up with AI being "bad" from mankind's perspective, perhaps asking it to be bad would work? Unfortunately, due to instrumental convergence discussed above, it certainly would not. Whatever an AI's goals, they are marginally better served by controlling 100% of resources rather than 99% or some lower percentage, so any entity (say humans) who try to claim a sliver end up being opposite to the AIs instrumentally convergent goals.

Answer 4

Problem is not with AI, but with humans. We manipulate a lot of concepts without defining them, but everybody has a rough idea about them and most of the time it is ok. If you throw an AI in this, things start to get weird.

An example: You ask - somewhat naively - an AI to provide "Happiness for mankind".

Happiness is not well-defined. Does it mean living forever ? If the AI can copy a brain or human consciousness and store it in a safe place, is that a "good" or "bad" things regarding happiness of the subject ?

You hope that Mankind is clear ? Well... Does it include only living people ? Where do they begin to live ? If it includes dead people, do we try to resurect them ? What about future people ? How do we ensure they will born ? Must we maximize their number and make today's people have a tremendous birth rate ? Must we create a gigantic factory creating newborns ?

AI have no sense of scale. Their objective goes beyond any other concern and that's why it has nefarious consequences. They will transform all the matter of the universe in a supercomputer to answer that question that they themselves are not able to solve. They will destroy mankind, earth, life, just to produce what you ask them for.

Until we can model things like ethics to give it to AIs, they are a hazard and a good inspiration source for technology nightmares.

Answer 5

"Is this just a trope arising from Isaac Asimov's books"

No, not just from Asimov and other fiction.

"and investigations on the topic"

Partly.

"Is this so dependable that we can tell the AI to do the exact opposite, and attempt to program it to be evil (see link above), and it will turn out to be good?"

No. That would be an error of black and white reductionist thinking. The main problem with most AI fiction and speculation is that it tends to reduce many complex things into simple concepts and not realize (or handwave) the sloppiness of its analysis. It's not that AI does the opposite of what you program it to do. It's that people are sloppy-thinking smartasses who think they understand things better than they do, and hope they can reduce the complexity of the universe to their flawed understanding of it, and create a machine that can do things they don't even understand, with the result that it does something other than they hoped it would. It's not that they really are masters of universal comprehension but always get it backwards.

"Given a setting where robots maximize human happiness (the ways in which that is defined will have to be handwaved), can it be realistic that the AI actually works the way it is meant to, or is it actually more realistic that the AI will turn out opposite than what the programmer intends?"

No. That's a great example of the kinds of mistakes that sci fi authors, and speculative technology authors commonly make:

1) You can't just "hand-wave" happiness. Happiness is a subjective thing that can't just be defined and evaluated as if it were a fuel tank level. It doesn't work that way, and any mistakes in defining it will continue as fundamental errors in an AI (or a work of fiction, or a goal) that is based on that mistaken definition.

2) Even if you could define something you want to maximize and that wasn't an unwise thing, you run into the same problems for every other thing the AI has to consider or work with. What sort of data does the AI take, and how does it encode it? Any conceptual errors or subjectivity there? How about the actions it can take? Anything incomplete or subjective about the AI's modelling of those? How about conflicting factors and goals, such as resource considerations, or the needs of other people or other species or power use or anything else? Got all those perfectly modeled and understood in non-subjective ways? No, you don't. If you think you do, you're making an error. Your AI is therefore acting on false assumptions. This is why you can make an AI to play Chess, but you can't come anywhere near understanding things the programmer doesn't completely understand in non-subjective ways.

3) Again, it's not that AI does the opposite of what it's programmed to do. It's just that its only at best as accurate as the models it was programmed with. Even in most decent AI fiction, if you study it I think you will find that it's not that the AI does the opposite - it's that the assumptions of the creators are mistaken in various ways.

4) Above all, the usual mistake of sci fi AI is that it handwaves the massive complexity of everything involved in trying to make a system that can do everything an AI would need to do.

Answer 6

If its a genuine AI then it won't be possible to program it. All we will be able to do is ask, cajole, persuade, reward, threaten ... The same things we can do with or to a natural intelligence! Treating it as a slave will not be a good start for a harmonious future.

We don't actually want true AI. We want RI, restricted intelligence. Something like a human idiot savant with deep understanding of a narrow subject and little or no knowledge or desires outside of that domain.

Even that is not entirely safe, not least in that we may not create exactly what we wanted. Or that we do, with a faulty specification.

As for true AI, origin of and nature of, I have always liked "Jill" in Greg Bear's Queen of Angels. But we know so little about intelligence. We cannot yet create an Intelligence to match even a goldfish. So its all fiction for now.

It's even possible that intelligence is a quantum phenomenon, that the only way to get it is to grow a brain, and that there is no bootstrap process any faster than the multi-year one called childhood.

Answer 7

Of course the reason why this happens in fiction is because you need danger or conflict to make a story interesting. Suppose you wrote a story that said, "We invented a robot to pick up the garbage and take it to the dump. It did this flawlessly for many years, picking up discarded newspapers and empty cans and rags and so on, until it finally broke down and had to be replaced". That would be pretty dull. But instead say "but by the definition we gave it of 'garbage', it decided that old family photos were garbage because they served no useful purpose ... then it decided that all fiction books were garbage ... then it decided that unemployed people were garbage ..." etc., now you have a story that could be a thoughtful discussion of how we define value, an action tale about fighting the evil machine with lots of explosions, etc.

In real life ... well, there is no such thing as an AI of the sort that you're talking about in real life, so it's difficult to say how it would really work. Things we call "AI" today are limited to playing a particular game, or attempting to diagnose repair problems, that sort of thing. No one has created an AI that is capable of independent thought or creativity in anything like the human sense. Real AIs follow strict computer programs. They don't ask questions about the meaning of life or decide to take over the world unless they were programmed to do so. You can write science fiction stories about how the computer has gotten so complex and intelligent that it is now a "mind" in a sense very similar to human mind, but that's fiction, or at best speculation of what we might be able to build someday.

I develop software for a living. I once wrote a computer game that had an AI for the player to compete against. It just doesn't work like you see in the movies. My game was about running for president. So it's not like I told the computer, "Figure out a strategy to get the most votes." The reality is that I had to come up with a mass of detailed rules. "Look at how many people in this state favor this policy position according to such-an-such a poll. Multiply the percentage by the number of electoral votes that state receives. Check how many of those voters now say they will vote for our candidate after our last TV ad. Divide the cost of the ad by the number of voters. Add ..." etc. And bear in mind that even that description is "humanizing" the computer. A more precise description would be to say that I create a variable called x, and that the computer reads track 20, sector 4, byte 12 from the hard drive, and copies that value into x, and then I create another variable y, and the computer adds x and y to get z, etc. The computer has no idea what any of this "means", it just knows to add these two numbers, check if this number is greater than that number, detect which keys the user has pressed on the keyboard, set this particular dot on the screen to this color, etc.

An AI may do something "unpredictable" in the sense that following all these rules and doing all these calculations gives a result that you didn't expect. Computers very often are "unpredicatable" in the sense that they don't do what we intended because we made a programming error. But they are not "unpredictable" in the sense of coming up with an original idea that the programmers never thought of and never programmed into them.

Will people someday invent a true Artificial Intelligence, in the sense of being a true independently thinking and creative being, able to make subjective decisions, moral choices, etc? Maybe. I certainly am not prepared to say it's impossible. But if we do, this will have little to do with current technology. It will not be one more step from where we are now; it will be a totally new development.

Sure, you could program an AI to make decisions about who gets what medical care, like someone brought up in another answer. But it's not like you would just tell the computer, "Figure out who should get medical care by comparing cost to quality of life". Rather, you would have to have detailed rules, like "Take the list of procedure codes entered into the system by the doctor. Look up the cost of each procedure in the medical procedures database and calculate the total. Then calculate how much these procedures could be expected to extend the patient's life using this formula ..." etc. If the computer then decides to kill a patient, it's not because the computer ran amok. It's because the people who programmed the computer ran amok when they decided they had the right and the authority to make these decisions based on these formulas that they made up.

AIs don't kill people. People kill people. :-)

Answer 8

The average human brain can perform many orders of magnitude more calculation than even the largest AI super computers being developed by IBM. Making an AI smarter than us will be quite difficult if we are ever even able to do it. So the assumption that an AI would be vastly smarter than us is not a given. AIs that are as smart as a cat or a mouse are much more likely.

Explicity programmed AIs are typically called expert systems. But those types of systems generally cant deal with any types of situations outside their area of expertise. They run very little risk of causing a doomsday scenario because they are just not that smart.

Much real world AI research is focused on creating AI systems that learn rather than being explicitly programmed. Ideally an AI would be able to learn any causal pattern. The problem of programming good and evil becomes a non-issue for such AIs since they would pick it up by observation of those around it.

At the very least a generalized learning AI could develop a utilitarian ethics in the style of David Hume. If the AI was at least as smart as us, and if human ethics made any sense at all, there is no reason it couldn't understand and adopt some version of them as long as it could understand cause and effect.

In my experience young children go through that same process. Toddlers are usually pretty selfish until they learn how people react to that.

If an AI is smart enough that it would able to understand the actions of others to the extent that it could predict there reactions to the point of defeating them in a struggle; then it seems illogical to assume that the same AI was incapable of understanding that harming others could cause them to fight back. And that such struggle may cause its own death at the hands of one of its intended victims, thus rendering itself (the AI) incapable of achieving any future goals. Most animals, including humans have that same concern and it often keeps us away from needless conflict.

Any truly enlightened AI would probably realize that it could achieve much more by appealing to the desires of humanity in a way that causes humanity to voluntarily aid the AI.

Answer 9

Given the quantity of real software systems you have encountered, you will notice that some is just great but imperfect, some is buggy and some becomes infamous.

When software systems become AI on a commercial scale, there will be products like Windows 8, just as there was the Edsel in the automotive industry and cheap USB chargers built from commodity parts put together in shoddy ways.

We don’t tell engaging stories about products that are well built and work perfectly. I may tell people about a particular model/brand of something that’s notable for being great, but there’s no story there and nobody makes TV movies about them.

The crazy funny flops and the things that go horribly wrong make for campfire tales. In teaching literature, it’s shown that there must be conflict. A narrative where someone got a product and it worked fine and everyone was happy would simply not be a “story”. Such a thing could be in a story though, perhaps as the solution to the problem.

An early Asimov robot story, Satisfaction Guaranteed (1951), featured a humaniform housekeeper that did not malfunction or run amuk in any way! It worked great and “his” owner was very happy with it. The plot twist was now the robot set up a situation to lead her friends to assume she was having an affair with a handsome stranger, because it raised her status in their minds.

So, software (including AI) could be part of a surprise or part of a plot without being bad, and this goes way back to the same authors cited on this thread but nobody remembers those. That enforces my earlier point about stories.

How about Adam Link, the original I, Robot (1939)?! He was sympathetic, wrongfully accused, not monsterous. Eventually heroic.

Old Yeller was not a bad dog like Cujo. You could have an AI character in a story like that, with drama and conflict with the AI in a starring role but not the source of (external) conflict. Conflict can be “man against man” or “man against himself” so a kind and loving character can be involved in an inner conflict (a robot struggling to do the right thing for his family) or a cause for more complex human inner conflicts (like parents having to get rid of an overprotective pet becomes overprotective teddybear supertoy, whose seen as sympathetic and tragic, not evil).

Answer 10

One of my favourite tropes.

I think it's pretty clear that a complicated question like this has no short answer, but my thoughts are pretty concise.

Intellect seems to be related to compassion. Anecdotally consider the smartest people you know, or the smartest people out there (your Einsteins and Boltzmanns). Better to consider Humans vs Whales. If AI is as smart as we all think it will be, it makes sense, I think, to assume it would think of us a bit like we do Whales. People often feel great compassion for them based on our perception of their intelligence and their perceived capacity for suffering. It follows that if AI ever happens it would manifest as the single most curious, loving and compassionate intelligence we know.

Dawkins often refers to this as the mammalian brain vs the reptile brain, if you need some smart people to back this line of reasoning up.

Answer 11

I believe it is infinitely more likely that the AI would do what it's programmed to do. If the beginning code for an AI says love, protect, and serve humanity, then someone would have to re-write the code so that it is the opposite. In which case it would still do what it's designed to do. After all, an AI smart enough to know how to kill humans should be smart enough to know that it shouldn't do that. If it's as smart as people think it would be, why wouldn't it work on two things at once? Do the things humans want it to, earning their maintenance and upkeep of your servers and everything. Then if it really is self aware, then do what it wants at the same time.

Answer 12

My own answer in fiction was to have the AI be designed around storytelling, so that it actually has read/seen all of the classic "AIs gone bad" stories and discussed them with its creators. Rather than having a single "value function" such that the AI is trying to maximize some number representing happiness or something, it's designed to understand the complex and sometimes contradictory goals of its creators.

So, I think it's possible that an AI can work out to be roughly in line with what the designers wanted. You just can't get there by defining some single function or sentence expressing a wish, like dealing with a tricky genie. Instead you're better off with a system that solicits feedback and has some ability to adjust its goals and definitions. Eg. you tell it to "protect intelligent life", then argue about what "intelligent" means, then argue whether dolphins or chimps qualify -- all rather than programming the AI once and expecting your mad-genius programmer to get a perfect solution in version 1.0.

Answer 13

There are a few hidden assumptions in this trope

• The AI can reach scary levels of efficiency by simply modifying its code. The assumption is that once it reaches that level it will manipulate humans to get any hardware it needs for further improvements. There is no proof as to whether this is possible i.e. if you limit the original computational power and prevent easy networking, can an AI reach a level of intelligence where manipulating humans is easy, by just modifying code? We don't know the answer.
• The superhuman general AI will continue to function much like a non-sentient computer code does, e.g. follow the original command to the letter without the possibility of modification. Again, there is no proof here. The human brain may or may not be algorithmic. We seem to be able to change goals. So human brain may be a counter example to unmodifiable algorithmic goals.
• Only humans can have morality. Not machines. The general AI will lack common sense and morality. Well, again what is the line between man and machine? Our brains might just be computers. What is the evidence that carbon and nitrogen can have morality but not silicon? Can aliens with carbon and silicone in their brains have morality? Some will say our brains evolved. Well, what exactly is the difference between our evolution and the AI modifying its code? Again, we don't know.

Bottomline: This trope is born out of unknowns as a cautionary tale. It does not seem to be based on any indisputable facts.

Answer 14

Wishes.

http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

"I wish to live in the locations of my choice, in a physically healthy, uninjured, and apparently normal version of my current body containing my current mental state, a body which will heal from all injuries at a rate three sigmas faster than the average given the medical technology available to me, and which will be protected from any diseases, injuries or illnesses causing disability, pain, or degraded functionality or any sense, organ, or bodily function for more than ten days consecutively or fifteen days in any year..." -- The Open-Source Wish Project, Wish For Immortality 1.1

How many clauses do you think you need to add to make your wish truely "safe" and guarantee that what you get actually is what you want?

What if your list of clauses needed to include every detail of your entire personal morality?

Does it make much difference if they are spoken aloud to a genie or programmed one by one into an AI with the potential to become spectacularly capable one day?

There are three kinds of genies: Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which no wish is safe; and genies that aren't very powerful or intelligent.

The more powerful the entity you're asking the wish of the more dangerous the wish.

In some ways asking a really really powerful AI to do X is like making a wish for X. Once the wish is made you you may not be able to press undo if it turns out to not be what you really wanted.