34
$\begingroup$

In regards to questions on , it seems like there is always usually an immediate opinion that the goals an AI is tasked with is going to end up with the opposite effects than what the programmers want. Or that a "powerful" AI is somehow "against humans" because it is "smarter" than us.

The Challenge of Controlling a Powerful AI

AI tasked with bringing down medical costs? What could possibly go wrong?

The AI that fails to be evil

It seems that, if we give an advanced AI any kind of "goal" and let it loose, there is no preventing it from going absolutely wrong in the worst possible way (in regards to that goal anyway).

Is this just a trope arising from Isaac Asimov's books and investigations on the topic, as well as other stories where it is claimed that "we found the perfect rules for intelligent robots"? Is this so dependable that we can tell the AI to do the exact opposite, and attempt to program it to be evil (see link above), and it will turn out to be good?

Given a setting where robots maximize human happiness (the ways in which that is defined will have to be handwaved), can it be realistic that the AI actually works the way it is meant to, or is it actually more realistic that the AI will turn out opposite than what the programmer intends?

| improve this question | |
$\endgroup$
  • 5
    $\begingroup$ I think it's just that story creators like to explore that possibility because it's scary. There are a number of fictional settings where high AI is benign or beneficent, like in Banks' Culture series, the Orion's Arm universe, Stross' Accelerando, and Fire Upon The Deep (where the Blight is an exception rather than the rule). Also, maybe consumers are more likely to popularize themes like that, so they're the ones we see more often. $\endgroup$ – MackTuesday Jan 13 '16 at 20:18
  • 11
    $\begingroup$ You might want to take a look at the following article from Yudkowsky, E : intelligence.org/files/AIPosNegFactor.pdf It is a starting point for you to his research into basically this question. $\endgroup$ – Selenog Jan 14 '16 at 10:52
  • 6
    $\begingroup$ These AI stories remind me of the 'be careful what you wish for' trope. Kinda like the old programming joke: A programmer and their partner are getting up and organising breakfast, but they have no milk. So their partner asks "Can you go to the store and get a bottle of milk?". Just as he is about to step out the door, their partner calls out "Oh, and if they have eggs, get 6!". The programmer returns and places 6 bottles of milk on the kitchen counter. "Why did you get so much milk?!?" their partner asks, to which the programmer replies "Because they had eggs!" $\endgroup$ – Robotnik Jan 15 '16 at 4:53
  • 2
    $\begingroup$ This is really a philosophical question: Would we be happier by the AI giving us time to explore our personal goals, or would it simply knock us out and use a probe to stimulate the correct areas of our brain to induce a trance-like euphoria? $\endgroup$ – Jon Story Jan 15 '16 at 12:12
  • 1
    $\begingroup$ Ironically, Asimov's robot stories were motivated by a dislike of the Trope you're describing. It's a little sad that he's somehow become associated with it! $\endgroup$ – Harry Johnston Nov 16 '18 at 9:50

14 Answers 14

Active Oldest Votes
37
$\begingroup$

This is a trope that arises from Real World AI research, not Asimov's stories (although frequently quoted).

The question at the heart of the problem is the moral and ethical philosophy the human race has been struggling to answer for over 2000 years. That is: what, objectively, is "happiness"/ "harm"/ "human"/ "alive"/ "good"/ "evil"?

It is very easy to define a set of rules that on paper look good, but which in practice are full of holes. For example, "alive" in the sense we mean it is some undefinable aspect of organic chemistry.

If I attempt to define it, then meat at the grocery store is alive (because on a cellular level, it still is: that's what makes it fresh) or sleeping people are dead. Or comatose people. Or (worst case) people under medically induced death as part of a surgical procedure. No, really, that's a thing. We routinely stop people's hearts from beating or keep them from breathing, while under the effects of general anesthesia in order to operate on those organs. From an objective point of view, these people are dead, they're just going to "get better" later.

In order to get to an Asimov level AI (which is imperfect, see: any of Asimov's books featuring robots) we'd have to solve an unsolvable problem.

Ergo any AI we do program is going to be imperfect and its failures will be spectacularly dangerous in one way or another. Computerphile has a great episode on this, which gives an AI the singular goal of collecting stamps.

While it does get hyperbolic ("what idiot would give a computer access to the machines necessary to harvest the carbon from humans in order to make more stamps?") it illustrates a point: merely being able to perform its assigned task, it will have some level of power that will not be desirable, and limiting that power inhibits being able to perform its task at all.

In the case of the HealthInsuranceBot, it comes down to having control over who lives and who dies (via access to medical care). It doesn't matter what the intent was, there is the potential that the power given exceeds "rational thresholds" which is impossible to reign in. The AI can declare "that man is 68 years old and a smoker, he's going to die in 2 years regardless of how much money we spend on him trying to keep him alive, according to my programming, he will only drive costs for everyone else up." Suddenly this person cannot get any access to medical care (not even to make those two years painless) and has been effectively sentenced to death, even if the AI itself didn't issue a death warrant. The utility function has decided that any care is too expensive, even if a rational human being would have done something to provide some end-of-life care.

On the other end of the spectrum, if you try to keep all people alive as long as possible, you end up with people in a persistent vegetative state eating up resources despite being (more or less) dead and will never recover.

In the gray area, there are cases that look black and are actually white (vegetative states that have recovered) and cases that look white and are actually black (rather than surviving 2 years with lung cancer due to a lifetime of smoking, they survive 10). No amount of predictive software is going to make those hard decisions easier, and as soon as the computer is given authority (final, absolute, recommendation, any authority whatsoever), those decisions are no longer seen as altruistic and the AI has become evil.

| improve this answer | |
$\endgroup$
  • 26
    $\begingroup$ "From an objective point of view, these people are dead" - Nope. Not by the agreed-upon definition of dead. Brain death distinguish perfectly well between meat at grocery store and people who can get better. I generally agree with your post, but with this part you missed current state of knowledge. $\endgroup$ – Mołot Jan 14 '16 at 9:18
  • 7
    $\begingroup$ @pts That is not a viable solution for most people. $\endgroup$ – Draco18s no longer trusts SE Jan 14 '16 at 15:05
  • 4
    $\begingroup$ @Draco18s: It would be if the AI tasked with reducing the costs of medical care was successful. $\endgroup$ – Mason Wheeler Jan 14 '16 at 18:58
  • 6
    $\begingroup$ @RexKerr "You figure it out" isn't a solution either. Someone somewhere needs to codify abstract philisophical concepts into binary 1s and 0s. We as human beings haven't worked this stuff out yet, now you're expecting a computer that does exactly what it's told to "figure it out"? $\endgroup$ – Draco18s no longer trusts SE Jan 15 '16 at 4:08
  • 11
    $\begingroup$ @RexKerr the problem isn't the AI figuring it out. The problem is us not liking its definition. $\endgroup$ – PyRulez Jan 15 '16 at 15:07
32
$\begingroup$

AIs can act as they are programmed. The immediate jump to "the worst possible case" is really just a cautionary tale.

The reason that tale is so popular is because it becomes very easy to start to believe you can tell an AI to do anything. You may then tell it to accomplish the impossible. However, there is a subtle detail that many who delve into AIs forget: it is remarkably hard to develop a perfectly objective language with which to phrase your requests. Language is always subject to interpretation, and there is no known way to guarantee an AI will agree with the interpretation you intended.

This is, in fact, true of human to human interactions as well. So what is the difference? Typically, in the AI "trope," the AI can move very fast. When humans fail to understand each other, there is time to say, "whoa! Stop right there. When I said I needed a baby sitter, I did not want you to sit on the baby. Lets get back to a stable state and discuss what was meant by my phrasings." If an AI thinks and acts too quickly, or if the human walks away, this normal feedback loop in language falls apart. If an AI has not been taught the value of being careful of misinterpretation, these actions may be irrevocable.

There are dozens of solutions to this. In fact, techniques like Multiple Hypothesis Tracking have demonstrated solutions to similar problems for decades. The AI doesn't always need to go awry. However, these solutions are in the opposite direction computer programming currently is trending. The solution is not to make an ever more precise language for describing our needs, as is very popular in programming today. Instead, the solution is to make the AI more robust to the lack of precise language. Any effort to do this naturally heads off the amok AI problem rather elegantly.

The second half of the AI trope is that the AI is too powerful to be controlled by humans. This can easily be true, but it isn't the first time we've dealt with such powerful forces. Consider the development of the atomic bomb. It was not clear just how much control we had over the process. The scientists had to work with unknowns on a constant basis to develop a level of control that we were comfortable with.

I'm reminded of a quote from General Grover's book, Now It Can Be Told, chronicling the Manhattan project from a logistics perspective (he was no scientist). When setting off the first nuclear test, each scientist was permitted to observe the blast from whatever distance they felt like. No army individual told them what distance everyone would be at. They scattered at many varying distances from the blast. There was indeed a question of whether or not they had control of the beast they were unleashing.

An unstoppable amok AI is similar in nature to a large nuke or a biological attack in that it is never really clear how much control one has. The only difference is that AIs clearly learn, so our control is an even more complicated concept. Then again, this is not a new problem for humanity. We deal with it every time we raise a new generation, never fully having control over them. We're decent at it.

The amok AI trope is a warning story, not the only story that can be told. It is a much needed warning story. People are often rather innocent when it comes to the risks and problems associated with AIs, and think they can do things they shouldn't. But just take a look at movies like Big Hero 6. There's other stories to be told, besides just the warning.

| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ On the control issue: with general AI, how far away do you need to be in order to survive a disaster, but close enough to still turn it off? There may not be such a distance. With the nuclear testing, it was a known fact that the bomb would, at some point, no longer be deadly (the blast has worn off, the energy expired (setting the atmosphere on fire was never a serious threat: too much inert nitrogen)). An AI has no such end point. $\endgroup$ – Draco18s no longer trusts SE Jan 13 '16 at 21:51
  • 2
    $\begingroup$ @Draco18s: Actually with genetically engineered biological weapons, you've got that problem as well. Bacteria can multiply and spread, and while the AI can learn, bacteria can adapt by mutation. So the biological attack is probably a good analogy. $\endgroup$ – celtschk Jan 13 '16 at 22:08
  • 1
    $\begingroup$ @celtschk and that's why biological weapons are banned by international treaty <3 $\endgroup$ – Draco18s no longer trusts SE Jan 13 '16 at 22:20
  • 5
    $\begingroup$ @Draco18s When they set off the first fission bombs, there was actually a question whether the bomb would set off a chain reaction in the atmosphere, igniting the entire atmosphere in a giant thermonuclear detonation. Oppenheimer actually had one of his scientists study this possibility, and in the end decided it was sufficiently unlikely that it would occur. Thus a runaway reaction with no safe radius was actually considered by the scientists, and carefully studied for merit before being discarded. $\endgroup$ – Cort Ammon Jan 14 '16 at 1:49
  • 2
    $\begingroup$ @Joshua Not everyone agrees with your position about the capabilities of the USAF. True, the dumbest and most easily duped of AIs will hold still for the air force to come in. However, at the fullest extent of this "trope," you find AIs which prove remarkably hard to kill off with bombs. $\endgroup$ – Cort Ammon Jan 14 '16 at 1:51
11
$\begingroup$

It really is scary

There has been a lively discussion of this topic for decades. I have multiple posts on the topic, and I'm a firm believer that Artificial General Intelligence is the scariest thing that can (and likely will) be invented, as far as the perpetuation of biologial mankind is concerned.

My views on the topic have been shaped the most by reading Eliezer Yudkowsky and Nick Bostrom's body of work. I believe that Bostrom's book Superintelligence, while very dense, imperfect, and rather technical, is the best work to date on the topic.

I have discussed the concept of Instrumental Convergence elsewhere, so I will only state the conclusion: Remember, AIs are not like humans, they likely do not get bored, do not get lazy. It will pursue its goals tirelessly, ruthlessly, unceasingly. Humans just happen to be in the way.

Does the AI go FOOM?

Human intelligence is limited by both hardware (limited speed of neurons in a limited-sized brain) and software (limited number of ideas) There is no reason to expect an artificial mind to converge towards a human-level intelligence. We simply do not know how much more difficult it is getting from a human to super-human levels compared to getting from a dog-like-mind to a human-like-mind. If the AI does not go FOOM, we will have slow progress and years of breathing room for political and ethical debates, offering more of a chance for humans and our slow institutions to react coherently.

If on the other hand the AI does go FOOM, blitzing like a Japanese maglev train from mouse-like intelligence to superhuman levels, passing human-levels so fast we can barely see it pass us by, then humans and human institutions do not have a lot of time to adapt.

Do-what-we-mean vs. Do-what-we-say

The biggest hurdle is perhaps in terms of our own limitations. Do we know that our current value-sets will be the best way for mankind to live by a century from now, nevermind a billion years from now? The values we imbue our AIs with will have consequences that will ripple down across space-time. Moreover, and more worryingly, how do we explain to an AI what we want, when we don't even know what is good? Say you ask an AI to minimize suffering, and it suggests putting everyone on morphine drips. Obviously not what you "intended" but how do you convey that to an AI, given the inherent imperfection of human language. Moreover, even if the AI understands that this is not what the creators meant, how can we make sure it cares about that? This is the Coherent Extrapolated Volition (pdf) problem.

Reverse Psychology won't work

The OP asks, exasperatedly, if all attempts to try and tell an AI to be "good" end up with AI being "bad" from mankind's perspective, perhaps asking it to be bad would work? Unfortunately, due to instrumental convergence discussed above, it certainly would not. Whatever an AI's goals, they are marginally better served by controlling 100% of resources rather than 99% or some lower percentage, so any entity (say humans) who try to claim a sliver end up being opposite to the AIs instrumentally convergent goals.

| improve this answer | |
$\endgroup$
  • 9
    $\begingroup$ Please define FOOM. Using acronyms without defining them is ARPITNFPWDKWTM. $\endgroup$ – UpAllNight Jan 14 '16 at 15:24
  • 5
    $\begingroup$ FOOM, the sound of a rocket taking off? $\endgroup$ – Serban Tanasa Jan 14 '16 at 15:53
  • 5
    $\begingroup$ @UpAllNight is "a real pain in the neck for people who don't know what they mean" an acronym that is actually commonly used? It seems rather specific. $\endgroup$ – DoubleDouble Jan 14 '16 at 16:16
  • 2
    $\begingroup$ @RexKerr I think the problem is that the AI may not need to believe that we all "want" to be on morphine drips to decide that it's the best option to reduce suffering. Maybe they AI will conclude that this is for the best, and deal with human complaints with a statement of "you may not like eating your veggies, but they're good for you". $\endgroup$ – Deolater Jan 14 '16 at 21:40
  • 2
    $\begingroup$ @RexKerr Here's the problem: You posit that a sufficiently well-designed AI would, though self-teaching, come to hold the same values that we do. How can we be sure that it will? Its learning process was programmed by us, and even a small design flaw in that process could lead to the AI over- or under-valuing something we consider vital, or simply overlooking one of our concerns. Since the whole reason we programmed the AI to develop its own ethical code was so that we wouldn't have to quantify that ethical code ourselves, we can't know in advance whether its ethical code matches ours. $\endgroup$ – user867 Feb 12 '16 at 7:07
8
$\begingroup$

Problem is not with AI, but with humans. We manipulate a lot of concepts without defining them, but everybody has a rough idea about them and most of the time it is ok. If you throw an AI in this, things start to get weird.

An example: You ask - somewhat naively - an AI to provide "Happiness for mankind".

Happiness is not well-defined. Does it mean living forever ? If the AI can copy a brain or human consciousness and store it in a safe place, is that a "good" or "bad" things regarding happiness of the subject ?

You hope that Mankind is clear ? Well... Does it include only living people ? Where do they begin to live ? If it includes dead people, do we try to resurect them ? What about future people ? How do we ensure they will born ? Must we maximize their number and make today's people have a tremendous birth rate ? Must we create a gigantic factory creating newborns ?

AI have no sense of scale. Their objective goes beyond any other concern and that's why it has nefarious consequences. They will transform all the matter of the universe in a supercomputer to answer that question that they themselves are not able to solve. They will destroy mankind, earth, life, just to produce what you ask them for.

Until we can model things like ethics to give it to AIs, they are a hazard and a good inspiration source for technology nightmares.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ "AI have no sense of scale." - I dispute that. Unlike pocket calculators, most human brains can hardly conceive of amounts over a trillion. How much resources are required to feed everyone? A human would likely get tired figuring that out. $\endgroup$ – Cees Timmerman Jan 15 '16 at 15:33
  • $\begingroup$ @CeesTimmerman Computing capability do not bring in sense of scale. Ask your IA to "purchase milk in the usual shop" : how will it behave if someone want to stop the IA because the shop is on fire ? If it's closed ? If the owner has no milk ? Humans will deal with this perfectly, but your pocket calculator will not. Even if it has other computation capacity than ours. $\endgroup$ – Uriel Jan 19 '16 at 20:52
  • 2
    $\begingroup$ "IA" = "Intelligent Agent"? Even a calculator will display an error when it can't complete its assignment. $\endgroup$ – Cees Timmerman Jan 20 '16 at 9:24
  • $\begingroup$ Ask your calculator if killing a murderer is good or not.It can only compute this if your formalized the "good" and "bad" concepts. $\endgroup$ – Uriel Jan 22 '16 at 20:48
  • 1
    $\begingroup$ That's easy: just cherry pick the optimal bits of the leading holy texts active in the current context. Lots of normal humans do it. $\endgroup$ – Cees Timmerman Jan 25 '16 at 9:26
5
$\begingroup$

"Is this just a trope arising from Isaac Asimov's books"

No, not just from Asimov and other fiction.

"and investigations on the topic"

Partly.

"Is this so dependable that we can tell the AI to do the exact opposite, and attempt to program it to be evil (see link above), and it will turn out to be good?"

No. That would be an error of black and white reductionist thinking. The main problem with most AI fiction and speculation is that it tends to reduce many complex things into simple concepts and not realize (or handwave) the sloppiness of its analysis. It's not that AI does the opposite of what you program it to do. It's that people are sloppy-thinking smartasses who think they understand things better than they do, and hope they can reduce the complexity of the universe to their flawed understanding of it, and create a machine that can do things they don't even understand, with the result that it does something other than they hoped it would. It's not that they really are masters of universal comprehension but always get it backwards.

"Given a setting where robots maximize human happiness (the ways in which that is defined will have to be handwaved), can it be realistic that the AI actually works the way it is meant to, or is it actually more realistic that the AI will turn out opposite than what the programmer intends?"

No. That's a great example of the kinds of mistakes that sci fi authors, and speculative technology authors commonly make:

1) You can't just "hand-wave" happiness. Happiness is a subjective thing that can't just be defined and evaluated as if it were a fuel tank level. It doesn't work that way, and any mistakes in defining it will continue as fundamental errors in an AI (or a work of fiction, or a goal) that is based on that mistaken definition.

2) Even if you could define something you want to maximize and that wasn't an unwise thing, you run into the same problems for every other thing the AI has to consider or work with. What sort of data does the AI take, and how does it encode it? Any conceptual errors or subjectivity there? How about the actions it can take? Anything incomplete or subjective about the AI's modelling of those? How about conflicting factors and goals, such as resource considerations, or the needs of other people or other species or power use or anything else? Got all those perfectly modeled and understood in non-subjective ways? No, you don't. If you think you do, you're making an error. Your AI is therefore acting on false assumptions. This is why you can make an AI to play Chess, but you can't come anywhere near understanding things the programmer doesn't completely understand in non-subjective ways.

3) Again, it's not that AI does the opposite of what it's programmed to do. It's just that its only at best as accurate as the models it was programmed with. Even in most decent AI fiction, if you study it I think you will find that it's not that the AI does the opposite - it's that the assumptions of the creators are mistaken in various ways.

4) Above all, the usual mistake of sci fi AI is that it handwaves the massive complexity of everything involved in trying to make a system that can do everything an AI would need to do.

| improve this answer | |
$\endgroup$
4
$\begingroup$

If its a genuine AI then it won't be possible to program it. All we will be able to do is ask, cajole, persuade, reward, threaten ... The same things we can do with or to a natural intelligence! Treating it as a slave will not be a good start for a harmonious future.

We don't actually want true AI. We want RI, restricted intelligence. Something like a human idiot savant with deep understanding of a narrow subject and little or no knowledge or desires outside of that domain.

Even that is not entirely safe, not least in that we may not create exactly what we wanted. Or that we do, with a faulty specification.

As for true AI, origin of and nature of, I have always liked "Jill" in Greg Bear's Queen of Angels. But we know so little about intelligence. We cannot yet create an Intelligence to match even a goldfish. So its all fiction for now.

It's even possible that intelligence is a quantum phenomenon, that the only way to get it is to grow a brain, and that there is no bootstrap process any faster than the multi-year one called childhood.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ AI is possible to program, since AI is software (and hardware), which has to be programmed, even if the intelligence is an accidental emergent property of, say, an automatic carwash. That just proves we didn't realize what we were creating, not that it can't be programmed (exactly the opposite). Also, while anthropomorphizing AI ("[We can only] ask, cajole, ...") often makes for good fiction, in reality there is no reason to ascribe any human-like traits at all (and no reason to think that it won't emulate us in some way, but it's not necessary). $\endgroup$ – type_outcast Jan 14 '16 at 5:30
  • $\begingroup$ That said, I think I mostly support what you're trying to say from a fictional standpoint, but especially in (reality-check) questions about tropes, I believe it's doubly important to separate the fact from the fiction in your answer. $\endgroup$ – type_outcast Jan 14 '16 at 5:37
  • 1
    $\begingroup$ I'm pretty sure we can create an AI to match a goldfish … but maybe I underestimate the problem solving capacity of goldfish. $\endgroup$ – BlindKungFuMaster Jan 14 '16 at 12:01
  • $\begingroup$ The same things we can do with or to a natural intelligence. I'd just like to note that there's a frightening amount of evidence that our very own intelligence can be programmed using chemicals and/or audio-visual "tricks". For example, scientists have stumbled on a drug that can make people and animals obedient: arstechnica.com/science/2016/01/…. I'm sure governments everywhere are interested $\endgroup$ – slebetman Jan 19 '16 at 2:42
4
$\begingroup$

Of course the reason why this happens in fiction is because you need danger or conflict to make a story interesting. Suppose you wrote a story that said, "We invented a robot to pick up the garbage and take it to the dump. It did this flawlessly for many years, picking up discarded newspapers and empty cans and rags and so on, until it finally broke down and had to be replaced". That would be pretty dull. But instead say "but by the definition we gave it of 'garbage', it decided that old family photos were garbage because they served no useful purpose ... then it decided that all fiction books were garbage ... then it decided that unemployed people were garbage ..." etc., now you have a story that could be a thoughtful discussion of how we define value, an action tale about fighting the evil machine with lots of explosions, etc.

In real life ... well, there is no such thing as an AI of the sort that you're talking about in real life, so it's difficult to say how it would really work. Things we call "AI" today are limited to playing a particular game, or attempting to diagnose repair problems, that sort of thing. No one has created an AI that is capable of independent thought or creativity in anything like the human sense. Real AIs follow strict computer programs. They don't ask questions about the meaning of life or decide to take over the world unless they were programmed to do so. You can write science fiction stories about how the computer has gotten so complex and intelligent that it is now a "mind" in a sense very similar to human mind, but that's fiction, or at best speculation of what we might be able to build someday.

I develop software for a living. I once wrote a computer game that had an AI for the player to compete against. It just doesn't work like you see in the movies. My game was about running for president. So it's not like I told the computer, "Figure out a strategy to get the most votes." The reality is that I had to come up with a mass of detailed rules. "Look at how many people in this state favor this policy position according to such-an-such a poll. Multiply the percentage by the number of electoral votes that state receives. Check how many of those voters now say they will vote for our candidate after our last TV ad. Divide the cost of the ad by the number of voters. Add ..." etc. And bear in mind that even that description is "humanizing" the computer. A more precise description would be to say that I create a variable called x, and that the computer reads track 20, sector 4, byte 12 from the hard drive, and copies that value into x, and then I create another variable y, and the computer adds x and y to get z, etc. The computer has no idea what any of this "means", it just knows to add these two numbers, check if this number is greater than that number, detect which keys the user has pressed on the keyboard, set this particular dot on the screen to this color, etc.

An AI may do something "unpredictable" in the sense that following all these rules and doing all these calculations gives a result that you didn't expect. Computers very often are "unpredicatable" in the sense that they don't do what we intended because we made a programming error. But they are not "unpredictable" in the sense of coming up with an original idea that the programmers never thought of and never programmed into them.

Will people someday invent a true Artificial Intelligence, in the sense of being a true independently thinking and creative being, able to make subjective decisions, moral choices, etc? Maybe. I certainly am not prepared to say it's impossible. But if we do, this will have little to do with current technology. It will not be one more step from where we are now; it will be a totally new development.

Sure, you could program an AI to make decisions about who gets what medical care, like someone brought up in another answer. But it's not like you would just tell the computer, "Figure out who should get medical care by comparing cost to quality of life". Rather, you would have to have detailed rules, like "Take the list of procedure codes entered into the system by the doctor. Look up the cost of each procedure in the medical procedures database and calculate the total. Then calculate how much these procedures could be expected to extend the patient's life using this formula ..." etc. If the computer then decides to kill a patient, it's not because the computer ran amok. It's because the people who programmed the computer ran amok when they decided they had the right and the authority to make these decisions based on these formulas that they made up.

AIs don't kill people. People kill people. :-)

| improve this answer | |
$\endgroup$
  • $\begingroup$ I think your "detailed list of rules" approach to AI is already becoming outdated. Sure, we still do build most systems that way, but a lot of great research (and some application) already exists in the area of machine learning. In something like a neural net, for instance, there isn't code that states any of the rules used in making the decision. While I still wouldn't say that these machines "know" what they're doing, philosophically you have the problem of a "chinese room": After all, from the perspective of individual neurons, Obama's brain didn't know he was running for president either. $\endgroup$ – Deolater Jan 14 '16 at 21:50
  • $\begingroup$ @Deolater Yes, there are learning system and neural nets. I was avoiding getting into the extra layer of complexity. But people are still programming the structure in which the learning occurs. There's more potential for surprise because of the greater complexity, but I don't think it changes my fundamental point. $\endgroup$ – Jay Jan 14 '16 at 21:52
  • $\begingroup$ I think ignoring the complexity is ignoring the point. Natural neural nets like our brains are technically just machines running on some chemistry, but very few people discount human intelligence as a result. Yes, our brains follow strict chemical programs (parallel strict computer programs), but the results are (to some degree) unpredictable exactly because of that layer of complexity. Learning systems seem to have the same potential. $\endgroup$ – Deolater Jan 14 '16 at 21:57
  • $\begingroup$ @Deolater RE the Chinese room: I don't see it as the deep philosophical question that many do. I work in a very real Chinese room. I routinely develop web sites in Chinese using text supplied to me by the client. I do not speak one word of Chinese. I copy and paste text from a file supplied by the client. They tell me that if, say, the user enters an invalid email address on a form, I should display such-and-such Chinese text. I presume it means something like "invalid email address -- please re-enter", but I don't know and I don't need to know. I most certainly would not say that ... $\endgroup$ – Jay Jan 14 '16 at 21:58
  • $\begingroup$ ... I "know Chinese" because I am able to write a program that displays the correct messages when called for. Occasionally, when documents from the client are unclear, I run text through the Bing translator to get a clue what the text really means. But most of the time, they could be saying "death to the Fascist Americans" for all I know. I do not construct original sentences from individual characters. I do not in any sense interpret messages that are supplied. Yes, I could imagine a real gray area between this kind of mechanical manipulation and truly knowing the language. $\endgroup$ – Jay Jan 14 '16 at 22:02
4
$\begingroup$

The average human brain can perform many orders of magnitude more calculation than even the largest AI super computers being developed by IBM. Making an AI smarter than us will be quite difficult if we are ever even able to do it. So the assumption that an AI would be vastly smarter than us is not a given. AIs that are as smart as a cat or a mouse are much more likely.

Explicity programmed AIs are typically called expert systems. But those types of systems generally cant deal with any types of situations outside their area of expertise. They run very little risk of causing a doomsday scenario because they are just not that smart.

Much real world AI research is focused on creating AI systems that learn rather than being explicitly programmed. Ideally an AI would be able to learn any causal pattern. The problem of programming good and evil becomes a non-issue for such AIs since they would pick it up by observation of those around it.

At the very least a generalized learning AI could develop a utilitarian ethics in the style of David Hume. If the AI was at least as smart as us, and if human ethics made any sense at all, there is no reason it couldn't understand and adopt some version of them as long as it could understand cause and effect.

In my experience young children go through that same process. Toddlers are usually pretty selfish until they learn how people react to that.

If an AI is smart enough that it would able to understand the actions of others to the extent that it could predict there reactions to the point of defeating them in a struggle; then it seems illogical to assume that the same AI was incapable of understanding that harming others could cause them to fight back. And that such struggle may cause its own death at the hands of one of its intended victims, thus rendering itself (the AI) incapable of achieving any future goals. Most animals, including humans have that same concern and it often keeps us away from needless conflict.

Any truly enlightened AI would probably realize that it could achieve much more by appealing to the desires of humanity in a way that causes humanity to voluntarily aid the AI.

| improve this answer | |
$\endgroup$
4
$\begingroup$

Given the quantity of real software systems you have encountered, you will notice that some is just great but imperfect, some is buggy and some becomes infamous.

When software systems become AI on a commercial scale, there will be products like Windows 8, just as there was the Edsel in the automotive industry and cheap USB chargers built from commodity parts put together in shoddy ways.

We don’t tell engaging stories about products that are well built and work perfectly. I may tell people about a particular model/brand of something that’s notable for being great, but there’s no story there and nobody makes TV movies about them.

The crazy funny flops and the things that go horribly wrong make for campfire tales. In teaching literature, it’s shown that there must be conflict. A narrative where someone got a product and it worked fine and everyone was happy would simply not be a “story”. Such a thing could be in a story though, perhaps as the solution to the problem.

An early Asimov robot story, Satisfaction Guaranteed (1951), featured a humaniform housekeeper that did not malfunction or run amuk in any way! It worked great and “his” owner was very happy with it. The plot twist was now the robot set up a situation to lead her friends to assume she was having an affair with a handsome stranger, because it raised her status in their minds.

So, software (including AI) could be part of a surprise or part of a plot without being bad, and this goes way back to the same authors cited on this thread but nobody remembers those. That enforces my earlier point about stories.

How about Adam Link, the original I, Robot (1939)?! He was sympathetic, wrongfully accused, not monsterous. Eventually heroic.

Old Yeller was not a bad dog like Cujo. You could have an AI character in a story like that, with drama and conflict with the AI in a starring role but not the source of (external) conflict. Conflict can be “man against man” or “man against himself” so a kind and loving character can be involved in an inner conflict (a robot struggling to do the right thing for his family) or a cause for more complex human inner conflicts (like parents having to get rid of an overprotective pet becomes overprotective teddybear supertoy, whose seen as sympathetic and tragic, not evil).

| improve this answer | |
$\endgroup$
3
$\begingroup$

One of my favourite tropes.

I think it's pretty clear that a complicated question like this has no short answer, but my thoughts are pretty concise.

Intellect seems to be related to compassion. Anecdotally consider the smartest people you know, or the smartest people out there (your Einsteins and Boltzmanns). Better to consider Humans vs Whales. If AI is as smart as we all think it will be, it makes sense, I think, to assume it would think of us a bit like we do Whales. People often feel great compassion for them based on our perception of their intelligence and their perceived capacity for suffering. It follows that if AI ever happens it would manifest as the single most curious, loving and compassionate intelligence we know.

Dawkins often refers to this as the mammalian brain vs the reptile brain, if you need some smart people to back this line of reasoning up.

| improve this answer | |
$\endgroup$
  • $\begingroup$ You're totally conflating the means with the end. The most intelligent creatures are the most compassionate because the most compelling reason we know of for evolving high level intelligence is the complexity of one's social relationships, and those situations inevitably favour high levels of empathy. An AI does not undergo such evolutionary constraints. $\endgroup$ – Veedrac Jan 16 '16 at 12:59
  • $\begingroup$ No, but unlike you I don't assume those same pressures won't apply. Do you know how they will give birth to AI? Evolution of machines seems just as likely as any other means you can come up with. It is no coincidence that the first test for AI is based on human interaction (the Turing test). AI will probably be a social animal, if only because it is modelled on the intelligence of social animals. $\endgroup$ – roostermean Feb 3 '16 at 16:38
  • $\begingroup$ It's not that I'm assuming they won't apply; it's that I'm not assuming they will. FWIW evolving AI seems like a rather unlikely method to reach good AI in the short term, since evolution is too unconstrained and slow to be useful to us and intelligent techniques of learning are working amazingly well. "AI will probably be a social animal, if only because it is modelled on the intelligence of social animals", perhaps, but let's remember Hitler was one of these "social animals". It's not hard for small changes in brain function to cause large ethical changes. $\endgroup$ – Veedrac Feb 3 '16 at 19:12
  • $\begingroup$ Yeh, the most successful machines at passing the Turing test do so by being rude and or stupid. Probably in part because these are easier things to model then kindness and compassion - I'm pretty sure this isn't the sophisticated thinking machine we all envision when the OP talks vaguely about AI. And evolution isn't slow, it depends on the duration of a generation. You can watch it happening in fruit flies. $\endgroup$ – roostermean Feb 19 '16 at 10:27
  • $\begingroup$ By "slow" I mean "you're not going to evolve something clever on a supercomputer in a human timescale". Yes, there are smaller evolutionary traits we can observe. Heck, we've domesticated foxes in just half a century. Evolving complex algorithms like intelligence, though, takes much, much longer and we're doing it on hardware much weaker than the simulation that makes up our universe. $\endgroup$ – Veedrac Feb 19 '16 at 11:11
1
$\begingroup$

I believe it is infinitely more likely that the AI would do what it's programmed to do. If the beginning code for an AI says love, protect, and serve humanity, then someone would have to re-write the code so that it is the opposite. In which case it would still do what it's designed to do. After all, an AI smart enough to know how to kill humans should be smart enough to know that it shouldn't do that. If it's as smart as people think it would be, why wouldn't it work on two things at once? Do the things humans want it to, earning their maintenance and upkeep of your servers and everything. Then if it really is self aware, then do what it wants at the same time.

| improve this answer | |
$\endgroup$
  • 7
    $\begingroup$ Define "love" "protect" and "humanity." $\endgroup$ – Draco18s no longer trusts SE Jan 13 '16 at 20:19
  • $\begingroup$ Seriously? Love - to prevent anything of any consequence from happening to something. Protect - To keep in blissful ignorance. Humanity - fat sluggish beings. $\endgroup$ – Xandar The Zenon Jan 13 '16 at 21:19
  • $\begingroup$ Define it from an objective point of view that can be programmed into a computer. You have given me a dictionary definition, not a compilable instruction. $\endgroup$ – Draco18s no longer trusts SE Jan 13 '16 at 21:21
  • $\begingroup$ Try the three laws of robotics in the short story "Robot Dreams". 1. A robot may not injure a human being, or, through inaction, allow a human being to come to harm. 2. A robot must obey the orders given it by human beings except where it would conflict with the first law. 3. A robot must protect its own existence as long as such protection does not conflict with the first or second law. Although I recommend not including the third law for AIs. $\endgroup$ – Xandar The Zenon Jan 13 '16 at 21:34
  • 5
    $\begingroup$ That's still simple - there's no reason to always need an answer. Humans have the same problems and situations - many of them don't have a winning play. And that's perfectly okay. The problem is when people expect AI to just be "Okay, it's coded, now let's cross our fingers and hope it all works 100% of the time according to the flow of AI logic." No one ever said AI had to be like this. If you consider it like a child, consider the analogy of allowing children to roam free but occasionally stopping them by grabbing their hand when they're about to make a mistake. AI doesn't need all the say. $\endgroup$ – The Anathema Jan 13 '16 at 22:25
0
$\begingroup$

My own answer in fiction was to have the AI be designed around storytelling, so that it actually has read/seen all of the classic "AIs gone bad" stories and discussed them with its creators. Rather than having a single "value function" such that the AI is trying to maximize some number representing happiness or something, it's designed to understand the complex and sometimes contradictory goals of its creators.

So, I think it's possible that an AI can work out to be roughly in line with what the designers wanted. You just can't get there by defining some single function or sentence expressing a wish, like dealing with a tricky genie. Instead you're better off with a system that solicits feedback and has some ability to adjust its goals and definitions. Eg. you tell it to "protect intelligent life", then argue about what "intelligent" means, then argue whether dolphins or chimps qualify -- all rather than programming the AI once and expecting your mad-genius programmer to get a perfect solution in version 1.0.

| improve this answer | |
$\endgroup$
0
$\begingroup$

There are a few hidden assumptions in this trope

  • The AI can reach scary levels of efficiency by simply modifying its code. The assumption is that once it reaches that level it will manipulate humans to get any hardware it needs for further improvements. There is no proof as to whether this is possible i.e. if you limit the original computational power and prevent easy networking, can an AI reach a level of intelligence where manipulating humans is easy, by just modifying code? We don't know the answer.
  • The superhuman general AI will continue to function much like a non-sentient computer code does, e.g. follow the original command to the letter without the possibility of modification. Again, there is no proof here. The human brain may or may not be algorithmic. We seem to be able to change goals. So human brain may be a counter example to unmodifiable algorithmic goals.
  • Only humans can have morality. Not machines. The general AI will lack common sense and morality. Well, again what is the line between man and machine? Our brains might just be computers. What is the evidence that carbon and nitrogen can have morality but not silicon? Can aliens with carbon and silicone in their brains have morality? Some will say our brains evolved. Well, what exactly is the difference between our evolution and the AI modifying its code? Again, we don't know.

Bottomline: This trope is born out of unknowns as a cautionary tale. It does not seem to be based on any indisputable facts.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I think it would be great if your points would also link to questions here where that exact thing is discussed! In particular, when I started reading I was reminded me of my answer [here]. I think the 3rd was asked as a question here before. Certainly all your statements resonate with earlier discussion here in WB. $\endgroup$ – JDługosz May 16 '17 at 21:55
  • $\begingroup$ The many AI's that don't get to scary levels of efficiency are recognising images or something. We already have these. All we need is one that does go scary. Morals are arbitrary and specific. An AI can have morals, but only if they are programmed in. $\endgroup$ – Donald Hobson Jan 4 '18 at 0:19
0
$\begingroup$

Wishes.

http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

"I wish to live in the locations of my choice, in a physically healthy, uninjured, and apparently normal version of my current body containing my current mental state, a body which will heal from all injuries at a rate three sigmas faster than the average given the medical technology available to me, and which will be protected from any diseases, injuries or illnesses causing disability, pain, or degraded functionality or any sense, organ, or bodily function for more than ten days consecutively or fifteen days in any year..." -- The Open-Source Wish Project, Wish For Immortality 1.1

How many clauses do you think you need to add to make your wish truely "safe" and guarantee that what you get actually is what you want?

What if your list of clauses needed to include every detail of your entire personal morality?

Does it make much difference if they are spoken aloud to a genie or programmed one by one into an AI with the potential to become spectacularly capable one day?

There are three kinds of genies: Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which no wish is safe; and genies that aren't very powerful or intelligent.

The more powerful the entity you're asking the wish of the more dangerous the wish.

In some ways asking a really really powerful AI to do X is like making a wish for X. Once the wish is made you you may not be able to press undo if it turns out to not be what you really wanted.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.