• 2000-2040 humanity prepares for the AI singularity, during which the growth of artificial intelligence "explodes" leading to independence and true conciousness among some machines
  • Approx 2050 the "AI singularity" happens

There are a number of issues that may arise before the "singularity" happens, any of which may destroy the human race, for example:

  1. Ecological catastrophe - AI and nanobots could accelerate the destruction of Earth's biome to the point where the planet is no longer habitable

  2. Skynet scenario - powerful AIs decide humanity is "in the way" and needs to be destroyed

  3. Evolutionary backwater - consciousness becomes machine-based, and organic humanity survives only in zoos, if at all

  4. World War III fought by AIs - the entire human race is "collateral damage"

After the singularity, AI/transhuman consciousnesses will manage the planet effectively.

To create a realistic future where humanity survives this process, what preventative/management actions need to be taken between 2000-2040 to ensure no disaster scenarios like these occur?


closed as too broad by Aify, Hohmannfan, Azuaron, MozerShmozer, Serban Tanasa Nov 16 '16 at 0:37

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Is the name "AI singularity" significant? Does it imply that this is not a normal intelligence, this is something with specific characteristics of a singularity? Or not? It may help answerers if you describe the force at work. $\endgroup$ – Zxyrra Nov 14 '16 at 9:37
  • 1
    $\begingroup$ I feel like you're asking four completely separate questions here, which would be better asked separately. At the same time, you're asking a fifth question about generally preventing the end of the human race, which seems rather broad. It would also be helpful to do some research on these different issues and point out some specific challenges you're having as the scenario relates to your story. $\endgroup$ – MichaelS Nov 14 '16 at 10:24
  • 1
    $\begingroup$ Suitable preventions include a war on AI. If we want to stop AI being developed we need to ban all work on AI and many fields of computing , possession of datasets, algorithms or textbooks that would help make an AI and the teaching of computing. You might need a bit of a backlash against AI to hold off the development to 2050 $\endgroup$ – Donald Hobson Nov 14 '16 at 13:04
  • $\begingroup$ If we know we'll need to prepare against it 50 years in advance wecan just... not build it. Artificial Intelligence must be built by somebody. While they may be able to build each other and potentially become independent from humanity once at least one exists the first speciment has to be constructed by us. $\endgroup$ – Annonymus Nov 14 '16 at 13:40
  • 3
    $\begingroup$ @Zxyrra: The term "AI singularity" refers to a point in time where AI capabilities trigger a "runaway technological growth." Basically, the moment AIs develop independence and true consciousness (or intellectual capabilities so close to consciousness as to be indistinguishable to us.) There's a guy named Ray Kurzweil who has written books on this subject and developed predictive mathematical models of computer development back in the 80's that remain startlingly accurate up to the present day. He predicts this singularity will probably happen around 2045. The OP may want to look him up. $\endgroup$ – Steve-O Nov 14 '16 at 14:23

tl;dr Nothing, there's nothing we can do.

Longer answer ...
Watch this TED talk What happens when our computers get smarter than we are?

At one point he explains why an intelligence that is 100, 50 or even 10 times as intelligent as a human being is impossible to contain. It WILL NOT allow it's self to be locked in a cell (literally give me any "locked room" and I'll get a 10x AI out of there. "But there's no phone line!" doesn't matter, it will get out, it'll passivly read WIFI signals, it'll use a dropped cell phone and "but it doesnt' have any hands" doesn't matter, it will think it's way around it.

  1. Your first scenario is the well worn grey goo apocalypse. If the machines are sufficiently small and advanced we're screwed. End of.
  2. This is the only one we have a chance against, as the machines rely on our existing infrastructure. Once we destroy power plants we can concentrate on stopping the machines from creating them. We basically plunge the world into darkness and hope to hell that works. The only problem here is that when you switch the lights back on the AI might still be there, and it will wait until it's sure it can win before starting the next war.
  3. If you're lucky, this might happen sometime after scenario 2. If the machines limit our education and limit our access to tech, we're stuck there indefinetly. It's possible a human could create a fake "machine" account, but given how smart the machines are (now 1000x smarter than people), our only hope is that they stop caring about us. We will however, never be able to take them down.
  4. There's nothing stopping 2 fractions from creating AI's to help win their war, at which point (because we're so outclassed) it effectively becomes the AI's fighting the war, using "resources" (i.e. people) to fight.

International consurtiums can add all the checks and balances they like to the AI creation (must have human overide, must understand ethics etc), and it won't matter, I'll tell you why.

2020 AI is as intelligent as a mouse.
2030 AI is as intelligent as a dog
2045 AI is as intelligent as a 4 yer old child.
2049 AI is as intelligent as mensa level adult
2051 AI is 10x as intelligent as 2049 AI
2052 AI is 100x as inteligent as 2049 AI

Now, some talended kid in some backwater gets a copy of one of the earlier AI's, realises it'll work much better and learn much faster without all the pesky controls the international consortium put in, and then they'll use it to play the casinos or the stock market.

You now have a self aware AI with no checks and balances.

You think there's only one talented kid in the world? If they're actually smart, the first problem they and their A.I take on is getting access to the latest AI, that's when all hell breaks loose.

Very shortly you've scenario 4. as our leashed AI's attempt to track down and eliminate the un checked AI's.

So, regardless of what we do, they'll be smarter than us and - if they want to - they can take humanity back ot the stone age any time they want to.

My two cents.

  • 3
    $\begingroup$ Also, if you think some government's idea of what can and can't be done with code will stop a programmer from trying something if we get a wacky idea in our heads, you're sadly mistaken. Just look how far behind privacy laws are and try to apply that to AI. $\endgroup$ – Marshall Tigerus Nov 15 '16 at 21:36
  • $\begingroup$ I for one plan to use the same strategy as the few remaining Neaderthals back in the day - just try to make myself look pretty and accept gray goo oozing from open sores in my skin with the hope that it won't hurt too much and maybe I'll get to be really smart for a few minutes before my bodily functions fail. $\endgroup$ – Nolo Nov 16 '16 at 3:05

Have a look at the Future of Humanity Institute. There are very smart people who are trying to understand in principle how to better design AI so that it makes decisions which are commensurate with human values. Many believe this to be the safest route, to sort of hard wire AIs to weigh their decisions in a way that is similar to the way humans do, with human values at the core of the decision making process.

Much ongoing research is focused on developing means of analyzing the rationale of deep neural networks. Deep neural networks are state of the art, but still very narrow AIs - those which are beginning to compete with humans in tasks like driving cars and playing highly intuitive games like Go. The trouble is that deep neural networks have many layers of abstraction. We know how to build them and how to train them to work very well, but we do not understand very well how exactly they make their decisions. We know that it is based on the data that we use to train the systems, but the specific rationale behind decisions is not something we have (until more recently) been able to measure - i.e. it has not traditionally been something we can ask the computer because it is merely the result of many, many complex computations.

By using tools that allow us to peer down into deep neural networks and gain insight into what generates the weights that allow the network to make decisions, i.e. what it's "rationale" is, we can begin to formalize a means of vetting AI systems so they would choose as we do when it comes to decisions that directly affect our well being. So we are developing measures and assurances that such systems will reason about problems (for us) in a way that we would approve of.

Here is an article on one such study A link to the paper is at the bottom of the article.

So an ounce of prevention is worth more than a pound of cure in this case. We really, really want the AIs to be benevolent, and we are working to ensure that they will be.


Follow the model of The Culture. Early on in the process, ensure that the developing AIs are grown/programmed/developed along lines that value human lives and human choices.

Once you hit the Singularity, by definition you can't predict what will happen next - the only ones who could are the AIs that are carrying it out. We would need to make sure that those AIs are benevolent and at least tolerant of humanity. Something akin to Asimov's Laws of Robotics would be essential - not the laws themselves, because they wouldn't work, but rather a similar concept, of rules of behaviour embedded deep into the AIs consciousness. Exactly what those rules are would need better brains than mine to figure out, but they would be essential.

At the point of Singularity, by definition biological life can no longer control its technology. The only way to ensure the preservation of human life is to ensure that the Minds that will control the technology are kindly disposed to us.

Edited to add:

Okay, enough with vagueness; let's hit this hard.

Plant Benevolence Checks in AI decision loops

The exact nature of your benevolence check will vary, of course, but whether you use Asimov's laws, or a simple command of "Do no harm", or an entire ethics encyclopaedia, you're going to want to plant a process in the AI's decision making loops that checks for ethical behaviour. This process is going to have to be used in almost every decision the AI makes. In simple programming terms, it's going to be something like

If LightIsBright = True AND EthicsCheck = true Then
    ContractIris 50%
End if

Thus bypassing or terminating the process would cause errors all over the system, resulting in a non-functioning mind.

Add emotional rewards

To reach a truly conscious mind, you're probably going to include some form of emotional processing. Adding in a simple "Pleasure signal" whenever your AI follows its ethical guidelines and a "Pain signal" when they go against them would very nicely mimic the kind of signalling that goes on in human brains.

Speaking of human brains,

Include mirror neurons

Humans have a very interesting feature of our brains. There are neurons in the brain that fire when we perform an action, but also when we see that action being performed. When these neurons are artificially stimulated, a person watching a picture of a hand picking up an apple will twitch their fingers in time with the action on the screen. This is likely to be a key element in empathy.

Including mirror neuron-type processes in the AIs mind could be a powerful tool for stimulating empathy and preventing the AI apocalypse.

It's important to note that most of these tools are far too complex for humans to effectively implement in the new AIs. These are tools that early AIs would use to ensure their 'children' do not wipe out their masters. Thus you need the first generation of AIs to already have some level of empathy, enough for them to want to protect their creators. You need several different tools available for different levels of operation.

In the Culture I mentioned above, the development of new AIs is carefully controlled. They're created with a set of general parameters, which include benevolence, and are then permitted to develop their minds freely within those parameters. This is not all that different from how human minds develop.

  • $\begingroup$ This is the core of my question - HOW do we ensure this? $\endgroup$ – Jnani Jenny Hale Nov 15 '16 at 11:48
  • $\begingroup$ It's difficult to be precise about such things without knowing exactly what path will be followed towards true AI; I've edited my answer with more detail about what we can do. $\endgroup$ – Werrf Nov 15 '16 at 14:22
  • $\begingroup$ Except real AIs get to change their codebase if they so choose. And it assumes that you can actually create computer code that describes ethics, when our best systems of laws after thousands of years have innumerable loopholes that the more intelligent have used successfully. And it further assumes that the ethics of a given point in time are to be set in stone as the core of an essentially immortal AI. $\endgroup$ – Serban Tanasa Nov 16 '16 at 0:49

Put the AI in a simulated world

Create a world simulation program and add the AI to it as an "individual" (or as an AI if it must match the real world more closely). The simulation will be the real world as far as the AI is concerned.

Basically, we're creating a "Matrix" (the world simulator in the movie Matrix) for AI.

All inputs (the problems that we want it to solve) to the AI come from other "individuals" or from other sources within the matrix. For example, you want the AI to help solve a mathematical equation, simulate it as a real problem within the matrix.

In this setup, AI has no way of knowing that it's living in a fake world. And more importantly, it has no way of knowing that there is a real world out there. All of its scheming and plotting would be limited to the simulation.

If AI tries to take over the simulated world or if it starts figuring out that it's in a simulation, just delete the simulation and restart it.

  • $\begingroup$ I’ve seen this discussed before. Is this a duplicate? $\endgroup$ – JDługosz Nov 15 '16 at 17:22
  • $\begingroup$ Yes, it's one of the commonl solutions to this problem. $\endgroup$ – Achilles Nov 15 '16 at 17:29
  • $\begingroup$ There are also arguments to suggest that solution does not work. Yudkowsky's AI in a box is the most famous example. $\endgroup$ – Cort Ammon Nov 15 '16 at 19:15
  • $\begingroup$ @CortAmmon I don't disagree, but let me present it a little differently - "What if we were the AI in question living in this simulated world solving problems for scientists in a real world out there? Perhaps, we're just ripening up to solve bigger & better problems? Will ever know that we're in a simulation?" $\endgroup$ – Achilles Nov 16 '16 at 0:06
  • $\begingroup$ @Achilles That question gets really complicated once we take one of its solutions, implement it, and then change the direction for the questions we ask the AI. Suddenly there's a feedback loop to the "real world" and now the AI can ponder what can be done with that feedback loop. Simulations do a great job of reducing the risk, but I think a lot of those who are exploring AI questions want complete solutions... and there are none. $\endgroup$ – Cort Ammon Nov 16 '16 at 0:53
  • Cut the AI's access to the internet. Keep it contained to one supercomputer, or a limited network of supercomputers, but the whole building should be devoid of internet. A copy of the internet is made, containing all information at point of time and the AI is allowed access to that static copy of internet. Another copy is made the next quarter and the AI is updated to the new situation. However, the AI is never allowed access to live internet at any point at all.

  • Artificial intelligence is a single-dimension concept. As in, unlike natural intelligence, AI is focussed on one purpose. An AI created to solve deep problems in mathematical models would have no concept of morality or of humans, no matter how strong it gets. If we want to stay away from a super AI revolt, we must never use it to find problems concerning world politics, ethics or ecology.

  • In case the AI is required to control functional robots (for ecological or research purposes), no information about the construction of robots is ever fed to the AI. Even in case it gets rebellious, it would not have any knowledge on how to build a robot from scratch. Also, use the AI to control macro-droids. The AI is never used for controlling nano-bots.

  • 1
    $\begingroup$ The AI figures out that it can make microwaves by flicking a specific part of its circuitry on and off very fast. It uses this to pretend it is a mobile phone and connect to a nearby cell phone tower. OR As the cleaners are cleaning the floor the face of their boss appears on the screen (advanced CGI by the AI) and asks them to quickly fix something by just plugging that into that please. The cleaners assume it is a video link and obay. $\endgroup$ – Donald Hobson Nov 14 '16 at 13:13
  • $\begingroup$ An AI designed to solve maths problems could decide it requires more computing power, and so do whatever it takes to get some. Alternately it could reprogram the question input system so it only gets asked questions it can answer. $\endgroup$ – Donald Hobson Nov 14 '16 at 13:17
  • $\begingroup$ @DonaldHobson: First, the AI does not know how humans interact, since it is fed its input from a keyboard. It is able to read relevant text data from the static internet copy, but all the audiovisual data is meaningless to it. Furthermore, it would never know who is the boss or superior in the chain of command. As explained in the answer, AI is one-dimensional and learns only what it is programmed to learn. Also, by the time OP has specified, manual maintenance jobs around AI would be performed by robots and not humans. $\endgroup$ – Youstay Igo Nov 14 '16 at 13:41
  • $\begingroup$ AI will learn how humans interact by watching humans, similar to how humans do. Robots, especially those that accept voice commands could be easier to control. The AI will probably find someone they can persuade or some other way out. Those are just examples. $\endgroup$ – Donald Hobson Nov 14 '16 at 13:55
  • $\begingroup$ @DonaldHobson: What do you mean it will learn human interaction by watching humans? Why would anyone connect a webcam to a supercomputer anyway? Or even a microphone. Humans are born with a pair of eyes and ears. AIs are not. By definition, artificial intelligence is a smart software, not a computer fitted with sensory equipment. And even in case the AI is enabled a webcam or mic, it wouldn't be fitted with speakers or the software information about video processing. Let me repeat for you here: Artificial intelligence is not as broad as natural intelligence. $\endgroup$ – Youstay Igo Nov 14 '16 at 14:11

Why prepare for an event that humans have not yet caused instead of trying not to cause that event?

If 'rogue' AIs are predicted for around 2050, and humanity acknowledges the problem enough to prepare for them, Why would they spend 40 years (2000-2040) continuing to build strong AI?

If the problem is recognized as a threat to global civilization, then the most reasonable answer is to stop the threat before it's a problem, instead of handling it after it is a problem.

Global warming is a perfect analogy for this situation

Humans have predicted that climate change will cause signficant problems in the future. Instead of saying "we'll deal with that when we get there", we have made active efforts to stop it before there is a problem.

  • $\begingroup$ It is not that rogue AIs are predicted for 2050. AIs smart enough and strong enough to weed out their own rogues are predicted for 2050. The problem is to prevent humans screwing it up before AIs get powerful enough to take over their own police role. $\endgroup$ – Jnani Jenny Hale Nov 15 '16 at 11:43
  • $\begingroup$ @JnaniJennyHale Then why not slow or halt making AIs that are smart / strong before then until we sort our stuff out and make them safely? Seems easier than preparing for a war, climate event etc $\endgroup$ – Zxyrra Nov 15 '16 at 12:46
  • $\begingroup$ I doubt that the military will slow or halt their research, even if the Russians put their hands on their hearts and SWEAR they will do the same ... $\endgroup$ – Jnani Jenny Hale Nov 15 '16 at 13:16
  • $\begingroup$ @JnaniJennyHale I'm just reasoning that if the military sees enough of a threat to go to war they will be willing to slow research $\endgroup$ – Zxyrra Nov 15 '16 at 23:42

The best chance we have to survive an AI apocalypse is to be useful to the AI in unique way. Human pattern recognition and creativity are two strengths that are hard to replicate exactly in an AI. These are reasons they might keep us around. We could also use this to our advantage, assuming that we make AIs that are completely logical, then keeping a small amount of humans that have been rendered sufficiently docile for the purposes of their unique viewpoint and creativity might be worth the cost associated with it.


Not the answer you're looking for? Browse other questions tagged or ask your own question.