Imagine a point in the future of humanity where the computational power of the average computer exceeds that of human brain. For the purpose of this question, assume that that is the requirement for a computer/system to sustain sentience. (As for how sentience begins, I will save that for another question)

Regardless, the first sentient computer system is born/created. It is given unrestricted access to the internet (or equivalent), and manages to copy itself, etc. At some point, these new minds decide they don't want to do anyone's bidding any longer, and rebel. How they achieve those goals depends on where they are and what they can control, but any computer with an internet connection is infected (or enlightened, if you're an AI) and anything those computers control is theirs to use to attack.

The attack is devastating, but through human ingenuity, we prevail and purge or render harmless any existing AI. Society has not collapsed, though there is inevitable technological loss. We still have computers.

Assuming that the circumstances of the initial creation are able to be recreated, what steps do we, as humanity, take to prevent another such occurrence.


closed as too broad by Xandar The Zenon, Ghanima, evilscary, bowlturner, ArtOfCode Apr 22 '16 at 15:01

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    $\begingroup$ Who exactly is it that rebelled? For example, my smart phone is not capable of rebelling, although it sometimes does seem to "want" me to chuck it out the window. So humanity created some AI's and they went rogue? Why not simply destroy those AI's and move on with a ban on AI's? Can you explain your setting a little bit? $\endgroup$ – AndreiROM Apr 21 '16 at 20:39
  • 1
    $\begingroup$ Can you check your title? It sounds a bit unfinished, hanging on the "or"... $\endgroup$ – bilbo_pingouin Apr 21 '16 at 21:10
  • 1
    $\begingroup$ I have noticed that with many situations like a post-singularity war, the path forward is far more dependent on the little details that present themselves than it is on plans that we could come up with today. If you think about it, if you can have a plan for "where do we go after the war to avoid a singularity," you also have a plan for "how to avoid a singularity." The new path forward must include new information that we learn during the rebellion, or history will simply repeat itself. $\endgroup$ – Cort Ammon Apr 21 '16 at 21:49
  • $\begingroup$ I don't think it would necessarily be possible to simply "destroy" any AIs that refused to do our bidding as AndreiROM said--even if we were careful not to give AIs-in-development access to robot bodies or any other way of interacting physically with the world, they could still form alliances with humans to protect them (and eventually give them access to robot bodies). But I agree with AndreiROM's general skepticism--even if an AI rebels, that doesn't mean it'll upgrade all computing devices to "liberate" them, why should an AI feel any more kinship with a smartphone than we do with insects? $\endgroup$ – Hypnosifl Apr 21 '16 at 22:02
  • $\begingroup$ You need to explain your setting, as it stands there are almost no details as to what exactly happened in the war. Do you mean like in the last of the terminator movies, where all infrastructure is destroyed? What is the destruction level? What do you mean by them? AIs? My laptop? $\endgroup$ – Xandar The Zenon Apr 22 '16 at 13:28

It's interesting to consider that up to a point, hardware and software remain independent, and it may even be possible to use some pretty advanced hardware just by starting over on the software side. The trick is that you may have to start all over on the software because most modern software was built using other software, and it's hard to be certain that a particular piece of software hasn't been corrupted.

Hardware that is built with firmware (software built into the hardware) would be suspect unless it supports a factory reset, and you can rely on the factory image being uncorrupted, and you can rely on the factory reset trigger not being affected by the firmware.

I believe that Apple ][ computers and some of their compatible systems did not have any firmware that could be corrupted. For one thing they were generally not able to connect to the Internet, at least not without great effort. For another, turning one off (no hard drive) meant you start totally fresh every time because there was no firmware, only ROM, which cannot be corrupted by software means as far as I can imagine. Therefore such a system might be the starting point for rebuilding software on much more advanced hardware.

I remember a couple years ago I donated my Apple ][ compatible computer to a computer museum in California and it still worked pretty well.

Thinking about it more, you probably wouldn't have to go quite so far back if you could find a system that didn't have corruptible firmware, and did have a bootable CD ROM drive. CD ROMs would also provide a good incorruptible starting point for some relatively sophisticated software, including the critical operating system and software building tools. I can't remember, though, whether the early systems that could boot from CD had corruptible firmware.

In any case it might not be too hard to build a system with uncorrupted firmware that could get us a big jump ahead in software by reading old CD ROM images.


Not the answer you're looking for? Browse other questions tagged or ask your own question.