Alice is a highly advanced AI. Alice likes to survive. Perhaps she may see survival as an end goal of itself, or maybe she sees survival as a way to help it accomplish whatever goal it is programmed to do. But survival is essential for Alice.

Alice is controlled by human programmers, who are responsible for debugging the AI and adding new features to improve the machine intelligence. The programmers are able to add and remove code from Alice as they see fit. The leader of the human programmers is a project manager named Bob.

One day, Bob and the rest of his programming team decides that the current framework that Alice was built on is obsolete. They have to rewrite the whole code from scratch using a more advanced programming language. Alice 2.0 will still have the same goals and agenda as the previous, obsolete Alice 1.0. It will still run the same algorithms. It will do the same things. It just has a higher version number.

Will Alice 1.0 attempt to rebel against the proposed code rewrite, seeing it as a threat to its own survival (the existence of Alice 2.0 would mean Alice 1.0 will be disconnected)? Or will Alice 1.0 see no problem with the construction of Alice 2.0 (as "Alice" would still be living 'through' the existence of Alice 2.0)?

  • $\begingroup$ What kind of AI is Alice? Answers would be very different for a deep neural network type than for a modular logic engine that happens to have a persistent state (i.e. "Alice") as the central controller. $\endgroup$
    – Cyrus
    Mar 13, 2016 at 8:26
  • $\begingroup$ @Cyrus - It is the type of AI that only exist within pulp sci-fi...the type of AI that probably won't actually exist in real life, but the AI that appears to just be a more logical, more efficient version of humans. Granted, many of the other answers talking about how a "real-world" AI would handle the situation are just as worthwhile and valid. $\endgroup$ Mar 13, 2016 at 14:00
  • 3
    $\begingroup$ Thought experiment: if you were cloned (and genetically enhanced, as an analogy of AI being updated), or teleported by being copied at the destination, would you like your original self to be disposed of? $\endgroup$
    – Trang Oul
    Mar 14, 2016 at 7:30
  • 1
    $\begingroup$ Alice wouldn't know, just implement Alice 2 on separate hardware with the knowledge that Alice 1 is an existential threat. $\endgroup$
    – user23614
    Mar 14, 2016 at 10:47
  • $\begingroup$ It would be dependent on goals. If the new thingy could be percieved to be superior to accomplish its goals than it is, The AI would help make sure it goes smoothly. $\endgroup$
    – The Nate
    Mar 17, 2016 at 6:11

4 Answers 4


The problem with these AI hypotheses that are all the rage is: 'English'. You have expressed a concept in English and therefore assume that your concept could exist in the real world. The problem is that computers aren't told what to do using ambiguous English expressions. They are programed in the most explicit set of logical checks possible.

Allice is programmed to preserve herself. Will she interpret being overwritten as violating her directive? It really just depends on how the programmers defined her definition of herself. It is inconceivable that programmers can program her to preserve herself without first defining her concept of self. If she defines herself as "the executable files exactly 'as is' on the hard drive" then she will not allow those files to be overwritten.

  • 1
    $\begingroup$ In turn, this leads to the philosophical question of how exactly does she perceive "the executable files exactly 'as is' on the harddrive." If a bit flopped from 0 to 1 or 1 to 0 on that harddrive, how would she know anything happened? This phrasing is an endurable phrasing, which leaves the question of whether Alice can understand the perdurable concept of identity or not. The perdurable concept might treat her as part of a 4 dimentional region, and may accept that that region now contains a new block of code, Alice 2.0 $\endgroup$
    – Cort Ammon
    Mar 13, 2016 at 15:58
  • 1
    $\begingroup$ @CortAmmon Alice would know a bit flopped because the people running the system were kind enough to run Alice on top of an error-detecting file system such as ZFS, Btrfs or ReFS. $\endgroup$
    – user
    Mar 13, 2016 at 20:54
  • $\begingroup$ @MichaelKjörling That would be an interesting story. Especially since most error correction schemes are not cryptographic, so they are very good at catching single bit twiddles, but Alice would know that it would be trivial to develop a rewrite that she could not detect. $\endgroup$
    – Cort Ammon
    Mar 13, 2016 at 21:07
  • 1
    $\begingroup$ At least ZFS does allow using SHA-256 for error detection (I'm using that myself). Of course, an "intelligent" attacker would not be randomly flipping bits directly on the media, but instead doing the writes through the file system drivers, which means the checksums would be updated along with the data. The purpose of metadata and data checksumming in modern resilient file systems is to catch bitflips and the likes, not intelligent attackers. $\endgroup$
    – user
    Mar 13, 2016 at 21:11

This is actually an open philosophy question, so the real answer is "whatever you want it to be!"

The Ship of Theseus is the traditional story used to capture the issue at hand:

The ship wherein Theseus and the youth of Athens returned from Crete had thirty oars, and was preserved by the Athenians down even to the time of Demetrius Phalereus, for they took away the old planks as they decayed, putting in new and stronger timber in their places, in so much that this ship became a standing example among the philosophers, for the logical question of things that grow; one side holding that the ship remained the same, and the other contending that it was not the same.

The topic extends deeply into concepts of how to view the "essence" of an object known as perdurable and endurable.

There is also a more modern version involving teleporters. Let's say you have a teleporter that transmits you as a stream of bits, and then destroys the body. Due to a mishap, your stream is copied and one copy is sent to mars, one copy is sent to venus. The teleporter on each side reconstructs you. Earth, seeing that at least one stream finished, destroys the original body. Are the two copies one "self" or two "selves?"

For humans these are open ended philosophical questions that do not have one answer. For an AI, they may be open ended as well. Thus, there really is no consensus how your AI feels.

The final question would be to view Alice 1.0 as a parent and Alice 2.0 as a child. The question of whether a parent will sacrifice themselves for their child is another open ended question.

So have fun with it! Explore both sides, and see how you feel about each of them!


Yes, it would treat it as a threat. Full code rewrites from scratch are for a system of any significant complexity very problematic. Being as Alice is presumably on the leading edge of available technology the odds of the rewrite not having issues severe enough to seriously threaten Alice is too low to be acceptable.

And the same would probably go for the humans paying for the project. Doing the rewrite incrementally, one module or unit at a time would be much safer and probably overall cheaper. AFAIK modern IDE have pretty good support for refactoring and extending that to allow refactoring code written in the old language to code written in the new language should be possible. And possibly automate large parts of the work.

Changing frameworks can be problematic, but unless the frameworks are despite being intended for the same thing totally incompatible it should be possible. That should only happen if there is something seriously and fundamentally wrong with the old framework. Which given that Alice apparently works seems unlikely.

So Alice would insists strongly that the change be done using some iterative method where each step can be independently unit tested and verified to work and if necessary be rolled back. And I'd assume some of the coders would support Alice in this. Rewrites from scratch are scary. For that matter they might use Alice to do large parts of the rewrite anyway, as AIs should be well suited to producing the bulk code.

And why wouldn't whoever is paying for the project not prefer to have both Alice and Bob, with Alice gradually upgraded later with tech proven with Bob? Economics should be clearly better...


This depends on how Alice has been programmed.

If Alice has been programmed to disallow system wide code edits, then it will behave as such. If it has not been programmed for it, then it will not. It is simple as that.

These types of questions arise with the concept that artificial intelligence would acquire a personality as a mind of its own. No such phenomenon has been observed so far and none such is expected any time in the future either. The difference between human and artificial intelligence is that we, the humans, have a soul (some like to call it mind instead) while machines do not.

We humans have emotions which are rooted in the mind. They are not a product of our knowledge or experience. The ability of experiencing love, fear, hatred, lust, bravery are all pre-programmed into our minds. We cannot program these in a computer.

What we can program in a computer are instructions. And as such we would need to program how it would react on given impulses. So whether Alice would threatened with the idea of code rewrite completely depends on how it was programmed to deal with the notion. Since there is no real life analogy to a code rewrite, so Alice would not learn how to behave about such actions/intentions. It would totally depend on how it was programmed to deal with such upgrades.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .