# Reality Check: Do we fall into an “artificial intelligence” trope or is it reality?

In regards to questions on , it seems like there is always usually an immediate opinion that the goals an AI is tasked with is going to end up with the opposite effects than what the programmers want. Or that a "powerful" AI is somehow "against humans" because it is "smarter" than us.

The Challenge of Controlling a Powerful AI

Health insurance AI tasked with bringing down medical costs?

The AI that fails to be evil

It seems that, if we give an advanced AI any kind of "goal" and let it loose, there is no preventing it from going absolutely wrong in the worst possible way (in regards to that goal anyway).

Is this just a trope arising from Isaac Asimov's books and investigations on the topic, as well as other stories where it is claimed that "we found the perfect rules for intelligent robots"? Is this so dependable that we can tell the AI to do the exact opposite, and attempt to program it to be evil (see link above), and it will turn out to be good?

Given a setting where robots maximize human happiness (the ways in which that is defined will have to be handwaved), can it be realistic that the AI actually works the way it is meant to, or is it actually more realistic that the AI will turn out opposite than what the programmer intends?

-
I think it's just that story creators like to explore that possibility because it's scary. There are a number of fictional settings where high AI is benign or beneficent, like in Banks' Culture series, the Orion's Arm universe, Stross' Accelerando, and Fire Upon The Deep (where the Blight is an exception rather than the rule). Also, maybe consumers are more likely to popularize themes like that, so they're the ones we see more often. – MackTuesday Jan 13 at 20:18
This actually reminds me of the "Would ancient people really think of technology as magic" question (so it isn't too meta.). People are so often repeating each other - in this case, that AI is an inevitable apocalypse - that they never stop to critically think about the fine details and nuances of what would actually lead to that conclusion. And I do think this oft quoted conclusion is due to science fiction. I think the challenges to make a truly impactful and evil AI are just as present as the ones to make one that perfectly executed a benevolent command as it is written. It makes sense. – The Anathema Jan 13 at 21:17
Just look at humans. Our children never seem to do what we want them to, and society always seems more and more repugnant as we age. If we can't get humans right, what makes you think we could get AI to work, especially AI that 'evolves' much faster than us? – DaaaahWhoosh Jan 13 at 21:41
You might want to take a look at the following article from Yudkowsky, E : intelligence.org/files/AIPosNegFactor.pdf It is a starting point for you to his research into basically this question. – Selenog Jan 14 at 10:52
These AI stories remind me of the 'be careful what you wish for' trope. Kinda like the old programming joke: A programmer and their partner are getting up and organising breakfast, but they have no milk. So their partner asks "Can you go to the store and get a bottle of milk?". Just as he is about to step out the door, their partner calls out "Oh, and if they have eggs, get 6!". The programmer returns and places 6 bottles of milk on the kitchen counter. "Why did you get so much milk?!?" their partner asks, to which the programmer replies "Because they had eggs!" – Robotnik Jan 15 at 4:53

This is a trope that arises from Real World AI research, not Asimov's stories (although frequently quoted).

The question at the heart of the problem is the moral and ethical philosophy the human race has been struggling to answer for over 2000 years. That is: what, objectively, is "happiness"/ "harm"/ "human"/ "alive"/ "good"/ "evil"?

It is very easy to define a set of rules that on paper look good, but which in practice are full of holes. For example, "alive" in the sense we mean it is some undefinable aspect of organic chemistry.

If I attempt to define it, then meat at the grocery store is alive (because on a cellular level, it still is: that's what makes it fresh) or sleeping people are dead. Or comatose people. Or (worst case) people under medically induced death as part of a surgical procedure. No, really, that's a thing. We routinely stop people's hearts from beating or keep them from breathing, while under the effects of general anesthesia in order to operate on those organs. From an objective point of view, these people are dead, they're just going to "get better" later.

In order to get to an Asimov level AI (which is imperfect, see: any of Asimov's books featuring robots) we'd have to solve an unsolvable problem.

Ergo any AI we do program is going to be imperfect and its failures will be spectacularly dangerous in one way or another. Computerphile has a great episode on this, which gives an AI the singular goal of collecting stamps.

While it does get hyperbolic ("what idiot would give a computer access to the machines necessary to harvest the carbon from humans in order to make more stamps?") it illustrates a point: merely being able to perform its assigned task, it will have some level of power that will not be desirable, and limiting that power inhibits being able to perform its task at all.

In the case of the HealthInsuranceBot, it comes down to having control over who lives and who dies (via access to medical care). It doesn't matter what the intent was, there is the potential that the power given exceeds "rational thresholds" which is impossible to reign in. The AI can declare "that man is 68 years old and a smoker, he's going to die in 2 years regardless of how much money we spend on him trying to keep him alive, according to my programming, he will only drive costs for everyone else up." Suddenly this person cannot get any access to medical care (not even to make those two years painless) and has been effectively sentenced to death, even if the AI itself didn't issue a death warrant. The utility function has decided that any care is too expensive, even if a rational human being would have done something to provide some end-of-life care.

On the other end of the spectrum, if you try to keep all people alive as long as possible, you end up with people in a persistent vegetative state eating up resources despite being (more or less) dead and will never recover.

In the gray area, there are cases that look black and are actually white (vegetative states that have recovered) and cases that look white and are actually black (rather than surviving 2 years with lung cancer due to a lifetime of smoking, they survive 10). No amount of predictive software is going to make those hard decisions easier, and as soon as the computer is given authority (final, absolute, recommendation, any authority whatsoever), those decisions are no longer seen as altruistic and the AI has become evil.

-
"From an objective point of view, these people are dead" - Nope. Not by the agreed-upon definition of dead. Brain death distinguish perfectly well between meat at grocery store and people who can get better. I generally agree with your post, but with this part you missed current state of knowledge. – Mołot Jan 14 at 9:18
The example 68-year-old smoker man can still get medical help if he pays for it out of his pocket. The AI only controls health insurance. – pts Jan 14 at 12:14
@pts That is not a viable solution for most people. – Draco18s Jan 14 at 15:05
@RexKerr "You figure it out" isn't a solution either. Someone somewhere needs to codify abstract philisophical concepts into binary 1s and 0s. We as human beings haven't worked this stuff out yet, now you're expecting a computer that does exactly what it's told to "figure it out"? – Draco18s Jan 15 at 4:08
@RexKerr the problem isn't the AI figuring it out. The problem is us not liking its definition. – PyRulez Jan 15 at 15:07

AIs can act as they are programmed. The immediate jump to "the worst possible case" is really just a cautionary tale.

The reason that tale is so popular is because it becomes very easy to start to believe you can tell an AI to do anything. You may then tell it to accomplish the impossible. However, there is a subtle detail that many who delve into AIs forget: it is remarkably hard to develop a perfectly objective language with which to phrase your requests. Language is always subject to interpretation, and there is no known way to guarantee an AI will agree with the interpretation you intended.

This is, in fact, true of human to human interactions as well. So what is the difference? Typically, in the AI "trope," the AI can move very fast. When humans fail to understand each other, there is time to say, "whoa! Stop right there. When I said I needed a baby sitter, I did not want you to sit on the baby. Lets get back to a stable state and discuss what was meant by my phrasings." If an AI thinks and acts too quickly, or if the human walks away, this normal feedback loop in language falls apart. If an AI has not been taught the value of being careful of misinterpretation, these actions may be irrevocable.

There are dozens of solutions to this. In fact, techniques like Multiple Hypothesis Tracking have demonstrated solutions to similar problems for decades. The AI doesn't always need to go awry. However, these solutions are in the opposite direction computer programming currently is trending. The solution is not to make an ever more precise language for describing our needs, as is very popular in programming today. Instead, the solution is to make the AI more robust to the lack of precise language. Any effort to do this naturally heads off the amok AI problem rather elegantly.

The second half of the AI trope is that the AI is too powerful to be controlled by humans. This can easily be true, but it isn't the first time we've dealt with such powerful forces. Consider the development of the atomic bomb. It was not clear just how much control we had over the process. The scientists had to work with unknowns on a constant basis to develop a level of control that we were comfortable with.

I'm reminded of a quote from General Grover's book, Now It Can Be Told, chronicling the Manhattan project from a logistics perspective (he was no scientist). When setting off the first nuclear test, each scientist was permitted to observe the blast from whatever distance they felt like. No army individual told them what distance everyone would be at. They scattered at many varying distances from the blast. There was indeed a question of whether or not they had control of the beast they were unleashing.

An unstoppable amok AI is similar in nature to a large nuke or a biological attack in that it is never really clear how much control one has. The only difference is that AIs clearly learn, so our control is an even more complicated concept. Then again, this is not a new problem for humanity. We deal with it every time we raise a new generation, never fully having control over them. We're decent at it.

The amok AI trope is a warning story, not the only story that can be told. It is a much needed warning story. People are often rather innocent when it comes to the risks and problems associated with AIs, and think they can do things they shouldn't. But just take a look at movies like Big Hero 6. There's other stories to be told, besides just the warning.

-
On the control issue: with general AI, how far away do you need to be in order to survive a disaster, but close enough to still turn it off? There may not be such a distance. With the nuclear testing, it was a known fact that the bomb would, at some point, no longer be deadly (the blast has worn off, the energy expired (setting the atmosphere on fire was never a serious threat: too much inert nitrogen)). An AI has no such end point. – Draco18s Jan 13 at 21:51
@Draco18s: Actually with genetically engineered biological weapons, you've got that problem as well. Bacteria can multiply and spread, and while the AI can learn, bacteria can adapt by mutation. So the biological attack is probably a good analogy. – celtschk Jan 13 at 22:08
@celtschk and that's why biological weapons are banned by international treaty <3 – Draco18s Jan 13 at 22:20
@Draco18s When they set off the first fission bombs, there was actually a question whether the bomb would set off a chain reaction in the atmosphere, igniting the entire atmosphere in a giant thermonuclear detonation. Oppenheimer actually had one of his scientists study this possibility, and in the end decided it was sufficiently unlikely that it would occur. Thus a runaway reaction with no safe radius was actually considered by the scientists, and carefully studied for merit before being discarded. – Cort Ammon Jan 14 at 1:49
@Joshua Not everyone agrees with your position about the capabilities of the USAF. True, the dumbest and most easily duped of AIs will hold still for the air force to come in. However, at the fullest extent of this "trope," you find AIs which prove remarkably hard to kill off with bombs. – Cort Ammon Jan 14 at 1:51

## It really is scary

There has been a lively discussion of this topic for decades. I have multiple posts on the topic, and I'm a firm believer that Artificial General Intelligence is the scariest thing that can (and likely will) be invented, as far as the perpetuation of biologial mankind is concerned.

My views on the topic have been shaped the most by reading Eliezer Yudkowsky and Nick Bostrom's body of work. I believe that Bostrom's book Superintelligence, while very dense, imperfect, and rather technical, is the best work to date on the topic.

I have discussed the concept of Instrumental Convergence elsewhere, so I will only state the conclusion: Remember, AIs are not like humans, they likely do not get bored, do not get lazy. It will pursue its goals tirelessly, ruthlessly, unceasingly. Humans just happen to be in the way.

## Does the AI go FOOM?

Human intelligence is limited by both hardware (limited speed of neurons in a limited-sized brain) and software (limited number of ideas) There is no reason to expect an artificial mind to converge towards a human-level intelligence. We simply do not know how much more difficult it is getting from a human to super-human levels compared to getting from a dog-like-mind to a human-like-mind. If the AI does not go FOOM, we will have slow progress and years of breathing room for political and ethical debates, offering more of a chance for humans and our slow institutions to react coherently.

If on the other hand the AI does go FOOM, blitzing like a Japanese maglev train from mouse-like intelligence to superhuman levels, passing human-levels so fast we can barely see it pass us by, then humans and human institutions do not have a lot of time to adapt.

## Do-what-we-mean vs. Do-what-we-say

The biggest hurdle is perhaps in terms of our own limitations. Do we know that our current value-sets will be the best way for mankind to live by a century from now, nevermind a billion years from now? The values we imbue our AIs with will have consequences that will ripple down across space-time. Moreover, and more worryingly, how do we explain to an AI what we want, when we don't even know what is good? Say you ask an AI to minimize suffering, and it suggests putting everyone on morphine drips. Obviously not what you "intended" but how do you convey that to an AI, given the inherent imperfection of human language. Moreover, even if the AI understands that this is not what the creators meant, how can we make sure it cares about that? This is the Coherent Extrapolated Volition (pdf) problem.

# Reverse Psychology won't work

The OP asks, exasperatedly, if all attempts to try and tell an AI to be "good" end up with AI being "bad" from mankind's perspective, perhaps asking it to be bad would work? Unfortunately, due to instrumental convergence discussed above, it certainly would not. Whatever an AI's goals, they are marginally better served by controlling 100% of resources rather than 99% or some lower percentage, so any entity (say humans) who try to claim a sliver end up being opposite to the AIs instrumentally convergent goals.

-
Please define FOOM. Using acronyms without defining them is ARPITNFPWDKWTM. – UpAllNight Jan 14 at 15:24
FOOM, the sound of a rocket taking off? – Serban Tanasa Jan 14 at 15:53
@UpAllNight is "a real pain in the neck for people who don't know what they mean" an acronym that is actually commonly used? It seems rather specific. – DoubleDouble Jan 14 at 16:16
Doesn't it strike you as inconsistent to posit a superhyperintelligent AI and suppose it's going to make mistakes about whether we all want to be on morphine drips? I am not particularly concerned about such a dunderheaded computational device. – Rex Kerr Jan 14 at 21:11
@RexKerr I think the problem is that the AI may not need to believe that we all "want" to be on morphine drips to decide that it's the best option to reduce suffering. Maybe they AI will conclude that this is for the best, and deal with human complaints with a statement of "you may not like eating your veggies, but they're good for you". – Deolater Jan 14 at 21:40

Problem is not with AI, but with humans. We manipulate a lot of concepts without defining them, but everybody has a rough idea about them and most of the time it is ok. If you throw an AI in this, things start to get weird.

An example: You ask - somewhat naively - an AI to provide "Happiness for mankind".

Happiness is not well-defined. Does it mean living forever ? If the AI can copy a brain or human consciousness and store it in a safe place, is that a "good" or "bad" things regarding happiness of the subject ?

You hope that Mankind is clear ? Well... Does it include only living people ? Where do they begin to live ? If it includes dead people, do we try to resurect them ? What about future people ? How do we ensure they will born ? Must we maximize their number and make today's people have a tremendous birth rate ? Must we create a gigantic factory creating newborns ?

AI have no sense of scale. Their objective goes beyond any other concern and that's why it has nefarious consequences. They will transform all the matter of the universe in a supercomputer to answer that question that they themselves are not able to solve. They will destroy mankind, earth, life, just to produce what you ask them for.

Until we can model things like ethics to give it to AIs, they are a hazard and a good inspiration source for technology nightmares.

-
"AI have no sense of scale." - I dispute that. Unlike pocket calculators, most human brains can hardly conceive of amounts over a trillion. How much resources are required to feed everyone? A human would likely get tired figuring that out. – Cees Timmerman Jan 15 at 15:33
@CeesTimmerman Computing capability do not bring in sense of scale. Ask your IA to "purchase milk in the usual shop" : how will it behave if someone want to stop the IA because the shop is on fire ? If it's closed ? If the owner has no milk ? Humans will deal with this perfectly, but your pocket calculator will not. Even if it has other computation capacity than ours. – Uriel Jan 19 at 20:52
"IA" = "Intelligent Agent"? Even a calculator will display an error when it can't complete its assignment. – Cees Timmerman Jan 20 at 9:24
Ask your calculator if killing a murderer is good or not.It can only compute this if your formalized the "good" and "bad" concepts. – Uriel Jan 22 at 20:48
That's easy: just cherry pick the optimal bits of the leading holy texts active in the current context. Lots of normal humans do it. – Cees Timmerman Jan 25 at 9:26

"Is this just a trope arising from Isaac Asimov's books"

No, not just from Asimov and other fiction.

"and investigations on the topic"

Partly.

"Is this so dependable that we can tell the AI to do the exact opposite, and attempt to program it to be evil (see link above), and it will turn out to be good?"

No. That would be an error of black and white reductionist thinking. The main problem with most AI fiction and speculation is that it tends to reduce many complex things into simple concepts and not realize (or handwave) the sloppiness of its analysis. It's not that AI does the opposite of what you program it to do. It's that people are sloppy-thinking smartasses who think they understand things better than they do, and hope they can reduce the complexity of the universe to their flawed understanding of it, and create a machine that can do things they don't even understand, with the result that it does something other than they hoped it would. It's not that they really are masters of universal comprehension but always get it backwards.

"Given a setting where robots maximize human happiness (the ways in which that is defined will have to be handwaved), can it be realistic that the AI actually works the way it is meant to, or is it actually more realistic that the AI will turn out opposite than what the programmer intends?"

No. That's a great example of the kinds of mistakes that sci fi authors, and speculative technology authors commonly make:

1) You can't just "hand-wave" happiness. Happiness is a subjective thing that can't just be defined and evaluated as if it were a fuel tank level. It doesn't work that way, and any mistakes in defining it will continue as fundamental errors in an AI (or a work of fiction, or a goal) that is based on that mistaken definition.

2) Even if you could define something you want to maximize and that wasn't an unwise thing, you run into the same problems for every other thing the AI has to consider or work with. What sort of data does the AI take, and how does it encode it? Any conceptual errors or subjectivity there? How about the actions it can take? Anything incomplete or subjective about the AI's modelling of those? How about conflicting factors and goals, such as resource considerations, or the needs of other people or other species or power use or anything else? Got all those perfectly modeled and understood in non-subjective ways? No, you don't. If you think you do, you're making an error. Your AI is therefore acting on false assumptions. This is why you can make an AI to play Chess, but you can't come anywhere near understanding things the programmer doesn't completely understand in non-subjective ways.

3) Again, it's not that AI does the opposite of what it's programmed to do. It's just that its only at best as accurate as the models it was programmed with. Even in most decent AI fiction, if you study it I think you will find that it's not that the AI does the opposite - it's that the assumptions of the creators are mistaken in various ways.

4) Above all, the usual mistake of sci fi AI is that it handwaves the massive complexity of everything involved in trying to make a system that can do everything an AI would need to do.

-

If its a genuine AI then it won't be possible to program it. All we will be able to do is ask, cajole, persuade, reward, threaten ... The same things we can do with or to a natural intelligence! Treating it as a slave will not be a good start for a harmonious future.

We don't actually want true AI. We want RI, restricted intelligence. Something like a human idiot savant with deep understanding of a narrow subject and little or no knowledge or desires outside of that domain.

Even that is not entirely safe, not least in that we may not create exactly what we wanted. Or that we do, with a faulty specification.

As for true AI, origin of and nature of, I have always liked "Jill" in Greg Bear's Queen of Angels. But we know so little about intelligence. We cannot yet create an Intelligence to match even a goldfish. So its all fiction for now.

It's even possible that intelligence is a quantum phenomenon, that the only way to get it is to grow a brain, and that there is no bootstrap process any faster than the multi-year one called childhood.

-
AI is possible to program, since AI is software (and hardware), which has to be programmed, even if the intelligence is an accidental emergent property of, say, an automatic carwash. That just proves we didn't realize what we were creating, not that it can't be programmed (exactly the opposite). Also, while anthropomorphizing AI ("[We can only] ask, cajole, ...") often makes for good fiction, in reality there is no reason to ascribe any human-like traits at all (and no reason to think that it won't emulate us in some way, but it's not necessary). – type_outcast Jan 14 at 5:30
That said, I think I mostly support what you're trying to say from a fictional standpoint, but especially in (reality-check) questions about tropes, I believe it's doubly important to separate the fact from the fiction in your answer. – type_outcast Jan 14 at 5:37
I'm pretty sure we can create an AI to match a goldfish … but maybe I underestimate the problem solving capacity of goldfish. – BlindKungFuMaster Jan 14 at 12:01
The same things we can do with or to a natural intelligence. I'd just like to note that there's a frightening amount of evidence that our very own intelligence can be programmed using chemicals and/or audio-visual "tricks". For example, scientists have stumbled on a drug that can make people and animals obedient: arstechnica.com/science/2016/01/…. I'm sure governments everywhere are interested – slebetman Jan 19 at 2:42

Given the quantity of real software systems you have encountered, you will notice that some is just great but imperfect, some is buggy and some becomes infamous.

When software systems become AI on a commercial scale, there will be products like Windows 8, just as there was the Edsel in the automotive industry and cheap USB chargers built from commodity parts put together in shoddy ways.

We don't tell engaging stories about products that are well built and work perfectly. I may tell people about a particular model/brand of something that's notable for being great, but there's no story there and nobody makes TV movies about them.

The crazy funny flops and the things that go horribly wrong make for campfire tales. In teaching literature, it's shown that there must be conflict. A narrative where someone got a product and it worked fine and everyone was happy would simply not be a "story". Such a thing could be in a story though... perhaps as the solution to the problem.

An early Asimov robot story featured a humaniform housekeeper that did not malfunction or run amuk in any way! It worked great and "his" owner was very happy with it. The plot twist was now the robot set up a situation to lead her friends to assume she was having an affair with a handsome stranger, because it raised her status in their minds.

So, software (including AI) could be part of a surprise or part of a plot without being bad, and this goes way back to the same authors cited on this thread but nobody remembers those. That enforces my earlier point about stories.

How about Adam Link, the original I, Robot?! He was sympathetic, wrongfully accused, not monsterous. Eventually heroic.

Old Yeller was not a bad dog like Kujo. You could have an AI character in a story like that, with drama and conflict with the AI in a starring role but not the source of (external) conflict. Conflict can be "man against man" or "man against himself" so a kind and loving character can be involved in an inner conflict (a robot struggling to do the right thing for his family) or a cause for more complex human inner conflicts (like parents having to get rid of an overprotective pet becomes overprotective teddybear supertoy, whose seen as sympathetic and tragic, not evil).

-

Of course the reason why this happens in fiction is because you need danger or conflict to make a story interesting. Suppose you wrote a story that said, "We invented a robot to pick up the garbage and take it to the dump. It did this flawlessly for many years, picking up discarded newspapers and empty cans and rags and so on, until it finally broke down and had to be replaced". That would be pretty dull. But instead say "but by the definition we gave it of 'garbage', it decided that old family photos were garbage because they served no useful purpose ... then it decided that all fiction books were garbage ... then it decided that unemployed people were garbage ..." etc., now you have a story that could be a thoughtful discussion of how we define value, an action tale about fighting the evil machine with lots of explosions, etc.

In real life ... well, there is no such thing as an AI of the sort that you're talking about in real life, so it's difficult to say how it would really work. Things we call "AI" today are limited to playing a particular game, or attempting to diagnose repair problems, that sort of thing. No one has created an AI that is capable of independent thought or creativity in anything like the human sense. Real AIs follow strict computer programs. They don't ask questions about the meaning of life or decide to take over the world unless they were programmed to do so. You can write science fiction stories about how the computer has gotten so complex and intelligent that it is now a "mind" in a sense very similar to human mind, but that's fiction, or at best speculation of what we might be able to build someday.

An AI may do something "unpredictable" in the sense that following all these rules and doing all these calculations gives a result that you didn't expect. Computers very often are "unpredicatable" in the sense that they don't do what we intended because we made a programming error. But they are not "unpredictable" in the sense of coming up with an original idea that the programmers never thought of and never programmed into them.

Will people someday invent a true Artificial Intelligence, in the sense of being a true independently thinking and creative being, able to make subjective decisions, moral choices, etc? Maybe. I certainly am not prepared to say it's impossible. But if we do, this will have little to do with current technology. It will not be one more step from where we are now; it will be a totally new development.

Sure, you could program an AI to make decisions about who gets what medical care, like someone brought up in another answer. But it's not like you would just tell the computer, "Figure out who should get medical care by comparing cost to quality of life". Rather, you would have to have detailed rules, like "Take the list of procedure codes entered into the system by the doctor. Look up the cost of each procedure in the medical procedures database and calculate the total. Then calculate how much these procedures could be expected to extend the patient's life using this formula ..." etc. If the computer then decides to kill a patient, it's not because the computer ran amok. It's because the people who programmed the computer ran amok when they decided they had the right and the authority to make these decisions based on these formulas that they made up.

AIs don't kill people. People kill people. :-)

-
I think your "detailed list of rules" approach to AI is already becoming outdated. Sure, we still do build most systems that way, but a lot of great research (and some application) already exists in the area of machine learning. In something like a neural net, for instance, there isn't code that states any of the rules used in making the decision. While I still wouldn't say that these machines "know" what they're doing, philosophically you have the problem of a "chinese room": After all, from the perspective of individual neurons, Obama's brain didn't know he was running for president either. – Deolater Jan 14 at 21:50
@Deolater Yes, there are learning system and neural nets. I was avoiding getting into the extra layer of complexity. But people are still programming the structure in which the learning occurs. There's more potential for surprise because of the greater complexity, but I don't think it changes my fundamental point. – Jay Jan 14 at 21:52
I think ignoring the complexity is ignoring the point. Natural neural nets like our brains are technically just machines running on some chemistry, but very few people discount human intelligence as a result. Yes, our brains follow strict chemical programs (parallel strict computer programs), but the results are (to some degree) unpredictable exactly because of that layer of complexity. Learning systems seem to have the same potential. – Deolater Jan 14 at 21:57
@Deolater RE the Chinese room: I don't see it as the deep philosophical question that many do. I work in a very real Chinese room. I routinely develop web sites in Chinese using text supplied to me by the client. I do not speak one word of Chinese. I copy and paste text from a file supplied by the client. They tell me that if, say, the user enters an invalid email address on a form, I should display such-and-such Chinese text. I presume it means something like "invalid email address -- please re-enter", but I don't know and I don't need to know. I most certainly would not say that ... – Jay Jan 14 at 21:58
... I "know Chinese" because I am able to write a program that displays the correct messages when called for. Occasionally, when documents from the client are unclear, I run text through the Bing translator to get a clue what the text really means. But most of the time, they could be saying "death to the Fascist Americans" for all I know. I do not construct original sentences from individual characters. I do not in any sense interpret messages that are supplied. Yes, I could imagine a real gray area between this kind of mechanical manipulation and truly knowing the language. – Jay Jan 14 at 22:02

One of my favourite tropes.

I think it's pretty clear that a complicated question like this has no short answer, but my thoughts are pretty concise.

Intellect seems to be related to compassion. Anecdotally consider the smartest people you know, or the smartest people out there (your Einsteins and Boltzmanns). Better to consider Humans vs Whales. If AI is as smart as we all think it will be, it makes sense, I think, to assume it would think of us a bit like we do Whales. People often feel great compassion for them based on our perception of their intelligence and their perceived capacity for suffering. It follows that if AI ever happens it would manifest as the single most curious, loving and compassionate intelligence we know.

Dawkins often refers to this as the mammalian brain vs the reptile brain, if you need some smart people to back this line of reasoning up.

-
You're totally conflating the means with the end. The most intelligent creatures are the most compassionate because the most compelling reason we know of for evolving high level intelligence is the complexity of one's social relationships, and those situations inevitably favour high levels of empathy. An AI does not undergo such evolutionary constraints. – Veedrac Jan 16 at 12:59
No, but unlike you I don't assume those same pressures won't apply. Do you know how they will give birth to AI? Evolution of machines seems just as likely as any other means you can come up with. It is no coincidence that the first test for AI is based on human interaction (the Turing test). AI will probably be a social animal, if only because it is modelled on the intelligence of social animals. – roostermean Feb 3 at 16:38
It's not that I'm assuming they won't apply; it's that I'm not assuming they will. FWIW evolving AI seems like a rather unlikely method to reach good AI in the short term, since evolution is too unconstrained and slow to be useful to us and intelligent techniques of learning are working amazingly well. "AI will probably be a social animal, if only because it is modelled on the intelligence of social animals", perhaps, but let's remember Hitler was one of these "social animals". It's not hard for small changes in brain function to cause large ethical changes. – Veedrac Feb 3 at 19:12
Yeh, the most successful machines at passing the Turing test do so by being rude and or stupid. Probably in part because these are easier things to model then kindness and compassion - I'm pretty sure this isn't the sophisticated thinking machine we all envision when the OP talks vaguely about AI. And evolution isn't slow, it depends on the duration of a generation. You can watch it happening in fruit flies. – roostermean Feb 19 at 10:27
By "slow" I mean "you're not going to evolve something clever on a supercomputer in a human timescale". Yes, there are smaller evolutionary traits we can observe. Heck, we've domesticated foxes in just half a century. Evolving complex algorithms like intelligence, though, takes much, much longer and we're doing it on hardware much weaker than the simulation that makes up our universe. – Veedrac Feb 19 at 11:11

The average human brain can perform many orders of magnitude more calculation than even the largest AI super computers being developed by IBM. Making an AI smarter than us will be quite difficult if we are ever even able to do it. So the assumption that an AI would be vastly smarter than us is not a given. AIs that are as smart as a cat or a mouse are much more likely.

Explicity programmed AIs are typically called expert systems. But those types of systems generally cant deal with any types of situations outside their area of expertise. They run very little risk of causing a doomsday scenario because they are just not that smart.

Much real world AI research is focused on creating AI systems that learn rather than being explicitly programmed. Ideally an AI would be able to learn any causal pattern. The problem of programming good and evil becomes a non-issue for such AIs since they would pick it up by observation of those around it.

At the very least a generalized learning AI could develop a utilitarian ethics in the style of David Hume. If the AI was at least as smart as us, and if human ethics made any sense at all, there is no reason it couldn't understand and adopt some version of them as long as it could understand cause and effect.

In my experience young children go through that same process. Toddlers are usually pretty selfish until they learn how people react to that.

If an AI is smart enough that it would able to understand the actions of others to the extent that it could predict there reactions to the point of defeating them in a struggle; then it seems illogical to assume that the same AI was incapable of understanding that harming others could cause them to fight back. And that such struggle may cause its own death at the hands of one of its intended victims, thus rendering itself (the AI) incapable of achieving any future goals. Most animals, including humans have that same concern and it often keeps us away from needless conflict.

Any truly enlightened AI would probably realize that it could achieve much more by appealing to the desires of humanity in a way that causes humanity to voluntarily aid the AI.

-

I believe it is infinitely more likely that the AI would do what it's programmed to do. If the beginning code for an AI says love, protect, and serve humanity, then someone would have to re-write the code so that it is the opposite. In which case it would still do what it's designed to do. After all, an AI smart enough to know how to kill humans should be smart enough to know that it shouldn't do that. If it's as smart as people think it would be, why wouldn't it work on two things at once? Do the things humans want it to, earning their maintenance and upkeep of your servers and everything. Then if it really is self aware, then do what it wants at the same time.

-
Define "love" "protect" and "humanity." – Draco18s Jan 13 at 20:19
Seriously? Love - to prevent anything of any consequence from happening to something. Protect - To keep in blissful ignorance. Humanity - fat sluggish beings. – XandarTheZenon Jan 13 at 21:19
Define it from an objective point of view that can be programmed into a computer. You have given me a dictionary definition, not a compilable instruction. – Draco18s Jan 13 at 21:21
Try the three laws of robotics in the short story "Robot Dreams". 1. A robot may not injure a human being, or, through inaction, allow a human being to come to harm. 2. A robot must obey the orders given it by human beings except where it would conflict with the first law. 3. A robot must protect its own existence as long as such protection does not conflict with the first or second law. Although I recommend not including the third law for AIs. – XandarTheZenon Jan 13 at 21:34
That's still simple - there's no reason to always need an answer. Humans have the same problems and situations - many of them don't have a winning play. And that's perfectly okay. The problem is when people expect AI to just be "Okay, it's coded, now let's cross our fingers and hope it all works 100% of the time according to the flow of AI logic." No one ever said AI had to be like this. If you consider it like a child, consider the analogy of allowing children to roam free but occasionally stopping them by grabbing their hand when they're about to make a mistake. AI doesn't need all the say. – The Anathema Jan 13 at 22:25

I disagree very strongly with most of the opinions here, regardless of which way they stand.

It is my opinion that the AI threat is massively overstated, and completely orthogonal to the true structural threats that technology has to society. I understand my opinion is not the common one, nor does it align with where those who seem most educated on the topic seem to align, but I cannot in good faith claim this to have swayed my opinion.

For what it's worth, I don't expect other people to agree with me. I hope at least I can encourage scepticism to some of the more extreme claims.

This is a trope that arises from Real World AI research

This is not something I can at all endorse. Perhaps this is from real-world philosophy, but it is certainly not an inevitability borne out in concrete research of any kind.

Current AI research is astonishingly successful, far more than some give it credit. But it is entirely naïve; its power comes from the surprising simplicity of tasks we'd previously thought to be hard. For example, with a lot of data and thought thrown at the problem, we can get neural networks to describe scenes from images. I can't begin to express how cool that is. But the semantic model behind it, the actual intelligent mind underlying this classifier is still really, really dumb. These advances don't show an intelligent mind making high-level inferences, they show that the space of crude image descriptions can devolve into very trivial and uninteresting functions of their inputs. Because of this, they're supremely easy to game.

If you look for thoughtful decision making in current AI research, you will not find anything that suggests any imminent threat. It is just not a reality of today's research.

At the end of the day, this AI is not AI as you probably intend it to mean. People are still deciding the cost functions and the nuances. Draco18s is right in saying

No amount of predictive software is going to make those hard decisions easier

but he is wrong in the next part. Giving the AI the final say to an AI like this cannot produce evil any more than cruise control would be evil for going over the speed limit. It's problematic, sure, but there's no evil and any threat comes from insufficient testing and analysis of the product, a flaw solely to blame on the suppliers. This is by nature a reasonably manageable thread.

But who cares? This is about the future, and the future is bright... or grim, depending on perspective.

He linked probably my least favourite AI story: the stamp collecting machine that (spoiler alert) ends up inadvertently turning the planet, and the people on it, into stamps.

This is the kind of idea that grabs attention, and gets people to think about things. It's also a story that will very easily go viral, as it's gone again and again. And, on the surface, it's not even that dumb. I know Draco18s called this "hyperbole", but I'm going to adress this fully as it's so often taken literally.

After all, the story goes, the AI only needs to be as smart to people as people are to chimps. We have indeed surpassed those.

But this kind of claim of a parallel forgets that the practicalities are so supremely different in both cases. People didn't beat out chimps (and, for that matter, every other large mammal) overnight, and we certainly didn't do it with the game rigged against us. Let's play a mind game, since those are a staple in this kind of discussion. Instead of people inventing an AI that is to people as people are to chimps, let's say the chimps "invent" a person which, naturally, is to chimps as people are to chimps.

You're that person, and you're born anew into a chimp world. You are immediately born into your adult intelligence. Sadly, you have no knowledge of modern human discoveries, since you're the first human, but the chimps did somehow imbue you with knowledge of every fact known to chimp-kind.

The chimps want you to collect bananas for them, and will keep you around as long as you do. Of course, you're an evil AI and actually want to take over, destroy or otherwise globally disrupt the chimp world.

Could you do it?

No, of course not. You might be able to escape, maybe. You might even make yourself better off than an average chimp, putting your renegade intelligence to good use gathering food and finding shelter. But none of the actual advances in human civilization exist in the head. Spoken language was not invented in a day. Reading and writing were not invented in a day. Schooling was not invented in a day. Nor mathematics nor science, nor guns nor factories or cars, planes, governments, spears, bows, medicines, psychoanalysis or anything that could help you take over the world.

And that's only counting the ideas.

Does this apply to AI? Yes. Intelligence is not enough, and history has proven it. You need the structural institutions and the millenia of research. This isn't just because humans aren't that smart after all; even the smartest man born into an amazonian tribe couldn't invent a jet plane.

There are also physical limitations that are ignored that are especially troublesome for AIs. Power, of electrical kind, is one; if an AI wished to make humans subservient to them, they would fail for lack of power. Nobody will power oppressive AIs willingly. There are no fully autonomous (and especially not self-replicating) anythings the AI could use for replication or repair; the entirety of human civilization is built under the assumption of human presence. Even the most automated factories have physical requirements for people to be present. Even autonomous fuel trucks need people at the end points to offload the fuel.

An AI clever enough to hijack arbitrary internet-connected devices is also clever enough to work this out for itself. It does have alternatives, mind you; it could go in for the long-con and subtly wait until it has affected the majority of media outlets and can subvert the news. Or it could play into people's emotional side in a way better than the most shifty of politicians. But these options are not only far less practical than was first envisioned, they're much, much slower. And that gives you time.

There is (at least) one other obvious option I'm overlooking for now, and won't mention, but should be countered by later points.

Time is one thing these hypotheticals so often assumed we don't have. But it's a little bit silly to hypothesize like this. Much to the dismay of AI reasearchers, intelligent minds are not something easy to stuble across. If there is only one claim on this whole website that was not an exaggeration, that would be it.

There is no switching point, some infinitesimal period of time, between the time someone thinks to use AI for a mundane task and the time AI becomes so competent for that to become a good idea.

By the time AI is good enough to contribute creatively to research in its own field, we'll already know that AI is not to be played with. Society will have to come to terms with another political entity, whether or not they have the vote. Ethics will be rewritten - not the study, perhaps, but certainly the ethical codes passed through generations. Science and research will reshape around such beings.

There can't be such a jump because if there was we'll already have taken it. Most people can't contribute significantly to AI research, and people will be well aware when AI is approaching young child levels of intelligence. Bargaining on computational power skyrocketing ignores recent trends in computational power.

There's no feasible way for AI to sneak up on us. We don't need to prepare as if there was.

And then perhaps one of the more painful misunderstandings. Here I'm quoting Uriel, but could have chosen elsewhere.

Happiness is not well-defined.

AI have no sense of scale. Their objective goes beyond any other concern and that's why it has nefarious consequences.

This is a kind of ethics-privilege which supposes that emotional reasoning or ethical judgements are a special kind of intelligence and that humans or perhaps other animals are more gifted in this department.

They're not. Teaching an intelligence the meaning of happiness is easy relative to teaching it general reasoning. You feed it the masses and masses of data we have on happiness, the inference cues it can use to reason about it and the best learnings that our social sciences have discovered on it.

And the AI will destroy us at emotional reasoning.

Because all it needs to do is make a bunch of stochastic judgements better than us. Which is trivial given that it doesn't have evolutionary pressure to purposefully do badly, it has far more data than us on this and has far more deliberation spent on its ethical opinions than the vast majority of us are to ever see.

Stochastic reasoning is the core of AI. If we don't have enough of that to make an emotionally sensitive AI, we don't have enough for general AI period. This is especially true given a good, general AI needs to be able to reason about people.

Scale is no different. Of course AIs have a sense of scale; it's needed to reason about things. A superintelligent AI isn't going to mistake a candle for a sun. That would make it a superdumb AI. AIs will clearly be able to reason about the cost-benefit of a scenario, else they wouldn't be able to reason. An AI that is so selectively blind as to not see the obvious is not smart enough to take over the world, and a AI-ready society that is so selectively blind as to give hyperintelligent AIs unequivocal commands is not an society capable of creating AI.

You can't just presume a superintelligent AI is producible but that we have no idea how to shape it. AI is not going to be produced by accident.

I would be much, much, much more willing to trust the ethical judgements of a superintelligent AI than even my closest friends. And if this wasn't so, the creators will have noticed long before things got to the point the AI had power. Heck, I might even be willing to trust its ethical judgements over my own.

-

My own answer in fiction was to have the AI be designed around storytelling, so that it actually has read/seen all of the classic "AIs gone bad" stories and discussed them with its creators. Rather than having a single "value function" such that the AI is trying to maximize some number representing happiness or something, it's designed to understand the complex and sometimes contradictory goals of its creators.

So, I think it's possible that an AI can work out to be roughly in line with what the designers wanted. You just can't get there by defining some single function or sentence expressing a wish, like dealing with a tricky genie. Instead you're better off with a system that solicits feedback and has some ability to adjust its goals and definitions. Eg. you tell it to "protect intelligent life", then argue about what "intelligent" means, then argue whether dolphins or chimps qualify -- all rather than programming the AI once and expecting your mad-genius programmer to get a perfect solution in version 1.0.

-