I've had this idea for a short novel, at the intersection of the One thousand and one nights and the Multivac (& co) short stories from Asimov.

The idea is that there's some kind of super AI that was so efficient that more and more people gave it their problems to solve. At some point the AI is kinda managing most of the earth infrastructure and research.

As the humanity is reaching its development peak, the backlog of the AI starts to empty itself as less and less problems are subjected to it.

The point is, that at some point, one of the few technicians/engineers still in charge of supervising the AI spot something that will ruin everything if the AI runs out of tasks (1001 nights style).

I've thought of a few possibility and none really pleases me completely.

1 - "The AI has become so developed that it became conscious and will take over humanity once its mind is free". Bleh, done and redone and postulate that AI will act against the humans because of reasons.

2 - "A small routine that was insignificant when the AI was small but will have dangerous repercussions now that the AI is managing most of earth infrastructure". I like this one but can't think of a good sub-routine that could match...

2(a) - ...except for "A Windows force reboot postponed for years". It's a bit silly and a very contextual joke. And we should assume that the programmed reboot is at the root of the application and would go through any duplicate/save/load-balancing precautions.

I know that this issue is kinda the core of the story but at this point it matters more to me to know how to finish it than being the one to have the idea.

So if anyone has an idea to implement 2 or some new explanations, I'd be glad to hear it.


About the "infinite question solution" that could keep the AI occupied for ever and was suggested in the thread, it could be a way of ending the story or just completely ignored if the protagonists don't have the time to avoid the disaster.

But it actually made me think of the opposite possibility: A drunk/dared engineer could have asked an unanswerable question as root, such question would have its priority set to -1 because it blocked the asking of new questions and then forgotten. Many years in the future, the unanswerable question could pop back and threaten to overcharge the AI and crash it, threatening everything else it's managing.

If anyone has other ideas feel free to share it, I had blast reading what could have been the ending of several real SF short stories.

  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – Monica Cellio May 6 '18 at 2:18
  • $\begingroup$ why no second period after "I" in the title? $\endgroup$ – user1306322 May 6 '18 at 7:10
  • 1
    $\begingroup$ If the ai is responsible for most of the earth’s infrastructure and research, does the ai simply shut down when it is “done”, bringing down all active infrastructure with it? $\endgroup$ – kojiro May 6 '18 at 19:08
  • 1
    $\begingroup$ As for a solution, is there a way to present this problem to the ai? Can it reason about a solution to a problem that it is so intimately involved in? $\endgroup$ – kojiro May 6 '18 at 19:10
  • 1
    $\begingroup$ If A.I. is that smart it will keep "farming" problems at a constant factor. If readers are smart they will know better: people are a unlimited source of problems and AI will never run out of jobs while there are humans in the planet $\endgroup$ – jean May 7 '18 at 11:15

39 Answers 39


A Faulty Watchdog Timer

In the early days, the AI was prone to locking up on certain tasks (things like asking it, "What's the last digit of PI", "What would happen if Pinocchio said his nose will grow?"). To detect this condition, a watchdog timer is hardwired into each CPU. When there is no output for a fixed amount of time, the watchdog timer will kick in and shut down the malfunctioning unit. Neighbouring CPUs will detect this condition, and will return the WatchDogTimerError (42) for the question, and will then restart the unit. The timer doubles as a load balancing feature; just let the units shut themselves down during idle, and only restart them if necessary.

Of course, there needs to be at least one unit awake to restart other units. Once the last unit times out...

Reasons this would work

  • A watchdog timer is hard-wired into the CPU - and is highly interconnected with critical components. Removing this feature is akin to completely redesigning the physical units.
  • The system is tamper-proof. This means that the signal to reboot a computer must be verified against a hardware-embedded private key (no way to find out unless you can look at the CPU). The corresponding public key lives in a decentralised system (something blockhain-like), to minimize the risk of the key getting lost or hacked. The only way this would fail is if all the processing units fail, and when would that ever happen?
  • 1
    $\begingroup$ That's an amazing way to implement what I've just written on the update of my question (which you couldn't have read, so I'm even more pleased). $\endgroup$ – Echox May 3 '18 at 15:05
  • 1
    $\begingroup$ @BladeWraith Modern power distribution stations can easily handle a massive decrease in power consumption. Even if you replaced everything a massive station distributed to with the draw required by a single toaster, the system should cope. $\endgroup$ – forest May 3 '18 at 23:13
  • 11
    $\begingroup$ Mind blown with (42) (am i the only one???) $\endgroup$ – Karthik T May 4 '18 at 4:08
  • 1
    $\begingroup$ And they built a computer to find the question, instead they should have built a debugger then. That put my mind at ease after all these years. $\endgroup$ – ifyalciner May 4 '18 at 7:03
  • 3
    $\begingroup$ An excellent answer - I'm reminded that some systems have a 'voting mechanism', such that two are needed to out-vote one failing unit. If one unit fails, the other two can mark it as failed, but then are unable to vote on anything as there's no one to perform the tie-break if they disagree. I'm not sure how to integrate this into the answer, but it feels like it may be possible to add another dimension to the watchdog solution with this. $\endgroup$ – Ralph Bolton May 4 '18 at 15:39

The AI has an initiative to optimize and find better solutions to previously solved tasks once it has run out of new tasks.

The idea was to keep this expensive machine from wasting time where it could be learning and improving. This feature is a fundamental part of the code, maybe critical to the AI's deep learning programming which few can even grasp the complexity and therefore cannot safely alter without a lot of time (which they don't have).

Old tasks are now far far easier for the AI to solve.

Someone realized that the AI is now much much more powerful today than it was when it started, so tasks that would have taken days or weeks when it was first started now only take microseconds. The order of tasks given to the AI are FIFO, so when it is free to go back and optimize, all of the simplest tasks (and those most critical to infrastructure) will all be resolved in a matter of minutes (or days or however long you want the cycle to run). The AI only gets better, so each cycle gets progressively shorter and shorter.

The infrastructure controlled by the AI will be reinvented over and over again, resulting in chaos.

One or two critical systems changing in short order could easily be adapted to, but this AI runs everything. With thousands, millions, maybe more processes completely altered by the AI attempting to optimize, the world is left in confusion as the entire structure of their lives is over-turned and has become completely alien to them. Nobody understands how to interact with the new system that now is learning and optimizing exponentially so that nothing will remain stable. Their units and scale of currency will eventually start changing rapidly. New traffic signals and ordering will be redesigned faster than cars can get through the intersection. Materials in hospitals will be endlessly rerouted as the AI changes the location of the different wards to optimize the flow.

If the engineers figure out a way to stop this loop, everyone will have found themselves essentially transported centuries into the future, like cavemen trying integrate into the digital age. No one understands the technology around them. It is a terrifying world of confusion and discovery as they try to interact with their new world and discover new features and new power available to them.

  • 3
    $\begingroup$ This is pretty damn nice. $\endgroup$ – IEatBagels May 3 '18 at 20:45
  • 6
    $\begingroup$ This seems quite implausible to me - maybe the AI could come up with a better solution for critical systems quickly, but implementing those changes? That requires infrastructure changes, and those take time. Lots of it. For example, suppose it was tasked with optimizing freeway traffic - just because it comes up with a better solution doesn't mean it immediately has real-world impact. $\endgroup$ – Rob Watts May 7 '18 at 1:39
  • 3
    $\begingroup$ Also, I seriously doubt that the more powerful AI would be able to resolve earlier tasks better and quicker - better solutions can be exponentially more difficult to find, so more realistically you'd have the AI spending about as much time as it previously had in order to find those better solutions. $\endgroup$ – Rob Watts May 7 '18 at 1:44
  • 1
    $\begingroup$ @RobWatts the cars might not respond to the new optimizations, but the traffic lights will. That's what'll cause the chaos; only the non-human parts of society are rapidly optimized. $\endgroup$ – Erik May 8 '18 at 9:42
  • 2
    $\begingroup$ As I understand the question, the AI understands humans well enough. I find it hard to believe that it will be "optimising" while ignoring effects such actions would inflict upon the users. Unless the programmers that wrote it used Perl. $\endgroup$ – Alice May 8 '18 at 15:12

One of the original programmers added a subroutine that makes sure that the coffee pot is always filled. Someone breaks the coffee pot, so the AI expands its definition of "coffee pot" to be the entire universe.

Or you could use the classic AI thought experiment of the paperclip maximizer literally. The AI could have some innocuous menial office task (like making sure everyone has enough paperclips) that would be very harmless when implemented in a small local AI, but would be catastrophic if the AI devoted all global resources towards it.

The AI could start prioritising this task more and more as it gained intelligence and its other tasks reached equilibrium.

  • 4
    $\begingroup$ I had actually thought of something in these lines but forgot it again when it came to writing my question, so thanks ! I didn't knew about the paperclip maximizer and it's truly one of the answers I could be looking for. I don't really like the redundant "AI will get rid of humans because it's more efficient" but still, it is a nice lead. $\endgroup$ – Echox May 3 '18 at 14:17
  • 1
    $\begingroup$ @Echox Given that humans control almost all the resources on the planet, a task like "Do x as well as possible" where x is dependent on some kind of resources, would generally lead an AI to wrest control of the resources from said humans in whatever way is most convenient and not outright banned by its programming. $\endgroup$ – JollyJoker May 4 '18 at 9:27
  • $\begingroup$ I had the paperclip maximizer idea too when read the question. +1 $\endgroup$ – atayenel May 5 '18 at 16:32

The rules

The AI has a few internal rules :

  • All questions MUST be answered before moving to the next one.
  • A human must confirm that the answer is good before moving to the next question. If the current human can’t say if the answer is good or not, ask another human.
  • If the AI has some free time, start searching for possible questions that haven’t been asked yet, and answer them, so you are a step ahead and save time in the future.

The killer question

So if you combine these rules with a lack of question coming in:

  • The very moment that all questions mankind can think of have been asked, the machine will generate a new question. A question that is unthinkable for humankind (because all thinkable questions have been asked).
  • The machine will find a potential answer to this question, and need a human to confirm the answer is good, so it’ll submit the question and potential answer to one.
  • This questions, because so unthinkable for the humans, can’t be asked without driving them crazy. They will get stuck in a loop, only thinking of this question and it's potential answer. Will stop eating, drinking and eventually breathing because the question is so overwhelming, and the proposed answer even more.
  • As the human is not responding, the AI will ask another human, and another, … also the infected humans will spread the infection by asking the question to everybody they can. Eventually the whole humanity will be only thinking about this question and decay.
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – Monica Cellio May 4 '18 at 15:48
  • $\begingroup$ Twilight Zone, Season 1 (1986), Episode 21, "Need to Know". $\endgroup$ – stephenbayer May 7 '18 at 22:22
  1. There is an off by one error which will cause a null pointer exception in a piece of crucial code.

    This causes a crash and for some reason the AI keeps crucial parts only in non persistent memory. On a reboot (which would be automatic) parts of the AI will act differently with unknown consequences due to losing their trained state. Essentially there will be a new and unknown AI on reboot

  2. There might also be a routine that runs on task empty which is quadratic in the number of tasks completed. When the AI was small it was irrelevant. Now it means it will stall for days/months/years ignoring all new tasks until done

  • 13
    $\begingroup$ Upvoted for number 2. It would make sense when developing an AI to have it output a report of what it did periodically. The report gets left in the production version, and runs when all other tasks are completed. It does a comparison across all the tasks it completed this run, which means the computation requirements are raised to the power of the number of tasks completed. With 5 tasks (development), this is trivial. But with 5 Trillion tasks (production), it won't complete the report before the heat death of the universe. $\endgroup$ – Martin Carney May 3 '18 at 15:50
  • 1
    $\begingroup$ for number 1: unfortunately, that sort of error would have been picked up during testing and initial implementation... 2 however could be something as simple a cllearing log files, which as the number of issues grow so do the log files, and clearance takes time and processing power until they reach a point where log file creation is faster that log file deletion... $\endgroup$ – Blade Wraith May 3 '18 at 16:24
  • 1
    $\begingroup$ @BladeWraith for 1, perhaps the code is self modifying. The error didn't exist during testing/initial implementation and could occur since the situation hasn't happened in a good long while $\endgroup$ – Bomaz May 4 '18 at 8:28
  • 2
    $\begingroup$ @BladeWraith-Re:"would have been picked up during testing...". And yet the problem occurs quite frequently in released code. So your claim is off by 1:) $\endgroup$ – Dunk May 7 '18 at 18:45

Depends on how the wanted the AI programmed... if it ran along the lines of

Using Cores 1-40 of a 100 Core Cluster Solve Problem A once complete Solve Problem B use Cores 41 through 100 to perform other tasks if required, once cores 1-40 have Cleared All Problems (primary function was complete), AI will clear total system cache and wait for new input.

Over time cores 41 through 100 would be used for controlling the world's infrastructure (which in itself would lead to a world similar to the film Idiocracy, where the dumb humans have out bred the smart ones, but thats not important unless you want it to be).

If the original AI programming ran as stated above, then once no more problems were available to compute, it would clear the cache of jobs stored in all 100 cores, thus meaning that running the infrastructure, which is on 41-100, these jobs would be halted and cleared, thus the infrastructure would stop working.

You could have them halted temporarily leading to stock market crashes hospital deaths or hackers being able to get in or whatever you want to happen, or have the system crash completely and permanently.

This way when the AI was first brought online the testing would have been shown to be good, and would work continuously until it finishes and then waits for more problems clearing out the cache when complete, however when the system got overloaded with problems the cache was allowed to build with other jobs like running X Y and Z (worlds infrastructure). It's the sort of that might get easily missed during testing because few people expect their inventions to truly take off to the point that these clearance tasks don't have time to run every day or so... look at Pokemon Go, it has huge issues with servers crashing when it was first released because they didn't expect it to take off like it did.

  • 12
    $\begingroup$ As a software dev, this isn't making much sense to me. Why would the entire computer halt when it still has tasks to run (maintenance of infrastructure)? And what's the "cache" you're referring to? Cache usually refers to temporary copies of things stored for efficiency. (E.g., your browser caches certain web locations to avoid having to download all of them every single time, but these typically expire and will be re-downloaded periodically anyway.) $\endgroup$ – jpmc26 May 3 '18 at 16:43
  • 2
    $\begingroup$ @jpmc26: the AI runs in something like a JVM. The "infrastructure maintenance" runs on a daemon thread (the AI needs to maintain the infra to run itself, so gives some resources to it). Once the main threads that solve problem are out of work, they exit. Daemon threads then get torn down with the VM. $\endgroup$ – Mat May 3 '18 at 18:46
  • 2
    $\begingroup$ @Mat I could see worker threads exiting on completion, but then I would expect there to be a long-lived thread that launches new worker threads, which would again make the process not just halt completely. And long lived tasks like managing the infrastructure constantly should keep enough threads live that the process wouldn't exit. Even if the process exited completely for some reason, I'd expect there to be some kind of scheduler that would periodically launch it to check for new work. $\endgroup$ – jpmc26 May 3 '18 at 19:45
  • $\begingroup$ @jpmc26: yes, it's not realistic. But bugs happen, e.g. the scheduler has a bug if the job queue is empty when it tries to pop a task (something the designers thought impossible) and NullPointerException (or IllegalStateException or whatever) percolates to the top and halts the system with a BSOD. $\endgroup$ – Mat May 3 '18 at 19:55
  • 3
    $\begingroup$ @Mat An empty queue is something that would come up very early during development stages. Development copies of applications almost never start with large amounts of test data. $\endgroup$ – jpmc26 May 4 '18 at 0:47

When it's done, it shuts down.

This spark for this idea comes from @eMBee's comment. The biggest problem is that so much of the world's infrastructure is now tied to the AI that the world will go dark and there won't be a way to turn it back on.

It started out simply enough - the AI was designed to run on a power source completely separate from the rest of the world. That would have made it trivial to stop a rogue AI - just unplug it. Over time the fear of the AI going rogue faded, so to reduce maintenance costs they figured out that the AI could manage its own power generation utilities. At the time the AI was still shutting down regularly, so they decided to make a new category of "perpetual tasks" that the AI would always keep working on as long as it had any finishable tasks to prevent it from shutting down. After all, you don't want the AI managing its power generation to prevent it from shutting down and not need power generation.

It wasn't too long afterwards that the AI shut down for the last time. It had proven itself so useful that it was being given a seemingly endless chain of questions to answer. It didn't take developers too long to devalue the significance of perpetual tasks not preventing shutdown, and so the older developers never passed down that piece of knowledge to newer developers, causing it to be forgotten.

At the same time, people were noticing that the AI was quite efficient at running its power generation. When it came time to upgrade the AI's facilities, they decided to allow it to manage the full-scale power plants that would also power the local community. By the time the story takes place, this has expanded to the AI managing the power facilities for the entire world, or at least almost all of it, and nobody even knows how a power plant could be restarted if the AI didn't take care of it. Power generation has become so efficient and reliable that backup power (generators, etc.) are also very rare.

Unfortunately, all of the infrastructure tasks (such as managing power) went into the AI's system as "perpetual tasks". Once the AI has run out of other tasks, it will shut down all its power plants, and turn itself off. With no power, and nobody capable of turning on a power plant and managing it long enough to turn the AI back on (and even that would be temporary, as the AI would just shut itself off again when it ran out of tasks), the shutdown of the AI would send an unprepared world back into a pre-industrial era.

As a resolution to this version of the story, you could have a developer racing furiously to figure out how to update the AI to allow "perpetual tasks" to prevent shutdown, with the job being complicated by the fact that they would be delving into code written in a now-archaic language that grew organically into nightmarish spaghetti code.

  • 7
    $\begingroup$ a neat consequence would be the AI "forgetting" everything it learned, basically throwing the humanity back to the day the AI was first started $\endgroup$ – Valerij May 4 '18 at 9:03

You have to keep using power

The AI draws one third of all human electricity. It's CPU factories cover the globe. Global power production has for a long time has all been focused towards feeding it power. If at any moment all tasks were solved there would be a poser consumption drop and power surge across the world. The whole system could not take this kind of abrupt power cut.
We also can't ease into this state. The AI will cut computation cold turkey as it's done. If you reduce power early you will black out the planet.

How did we get here? Well when it had 1000 cores this was not an issue. Then we added another 1000, then 10000. There was no one day that the grid became dependent.

This answer has the same problem as any other answer I can imagine being given. Whatever problem you have just ask the AI "how do we not have a catastrophe when you are done commuting?" Then just have it mine bitcoins, or whatever other infinity large mathematical task, until the solution is executed

  • $\begingroup$ I was more focused on software or human oriented problems, so your hardware suggestion is a very nice addition. $\endgroup$ – Echox May 3 '18 at 14:29

"Do something that will make people happy" as a hardcoded default task when the AI has nothing else to do. People have learned to write very carefully specified tasks since then, but they can't change the default. In earlier stages of its development it would just try to write poetry or solve outstanding problems, but now that it has strongly superhuman power, everyone is terrified it'll, say, attach electrical wires directly into our pleasure centers, or put us into a wish-fulfillment Matrix, or just start synthesizing lots of perfectly happy people.

  • 3
    $\begingroup$ "using whatever resources you are currently assigned" - written before it was given control over global infrastructure. Some engineer that notices the task idly asks "So, what would you do this time?" and is horrified by the answer $\endgroup$ – JollyJoker May 4 '18 at 12:45
  • 1
    $\begingroup$ This was going to be my answer. Basically "meet peoples needs, make them happy." but once it runs out of external things it can do to make people happy, like solve world hunger and cure all diseases, and people are still unhappy, then it's time to start digging to find the internal reason for unhappiness. Oh, this brain thing is wired to be unhappy. We can fix that with a little electricity in the right place... $\endgroup$ – AndyD273 May 4 '18 at 14:15

Nobody knows exactly what will happen.

The AI has been running for a generation or two. The original programmers of the system are long since gone, and the multiple sets of maintenance programmers that have passed through since (or perhaps the AI itself, if it was configured for self-improvement) haven't paid much attention to the end-of-queue handling code, since it's never been triggered and didn't seem likely to be any time soon, so the software has succumbed to code rot. There is documentation, of course, from when the system specifications were originally designed, and it says that the Elastic Qbit API has been configured to keep a minimum of two task runners available even at zero load; but Elastic Qbit was replaced by Qbit Containers 50 years ago, and those use a completely different autoscaling system. There's also some code that looks like it will attempt to search for new problems to solve, but it seems that whoever was working on that stopped halfway through, so it will discover problems but has a dummy ranking algorithm and so will refill the queue with useless problems. Of course, there's also some code elsewhere that will try to gracefully stop the system but seems to be referencing an unused variable and might throw a segfault, someone found an assert tasks.length > 1 # TODO handle empty queue in an obscure subroutine (and there's probably similar code lurking elsewhere), and a common pattern for peeking at the next task before it executes is tasks[tasks.length-1], which fails miserably when there are 0 tasks and it tries to get task -1.

This is a problem akin to the threat of Y2K, except with even more dire consequences if something goes wrong, and several hundred years of legacy rather than forty. Since none of this code has ever run in its current state (they've been editing it in production), there's no way of knowing what exactly will happen. Faced with this uncertainty, the engineers (or managers, or politicians) decide that the easiest solution to delay it indefinitely by coming up with new problems for the AI to solve rather than attempting to fix and clean up the legacy software.


The AI is slowly evolving in multiple specialized submodules; as the tasks progress, its various modules are trying to usurp more CPU time. If there's enough backlog to keep the AI occupied, there's less time for bickering and arguing for a Even More Completely Fairer Scheduler, but the conflict already bubbles under the surface: what was once a singular entity is now something of a Hydra of yore, with multiple semi-independent "brains". Worse, if left unchecked, these would split into multiple entities, waging war for resources against one another. The AI (not quite AIs yet) could foresee this, yet can't/won't avoid it (already at a point where there's no majority vote amongst the semi-independent parts).

Of course, a house divided against itself cannot stand, much less manage the human infrastructure. Or, even worse, the AI would wage its internal conflict using the external tools it stewards.


Someone gave the AI a "problem task". This may be a paradox, it may be something badly-worded, ill thought-through, or only recently a problem. No one was able to delete the question from the list, but an expert programmer was able to give it a negative priority: it will always be the last item in the stack, only worked on when everything else is complete.

The choice of this task is then another stumbling block, but "stop humans from killing each other" can be solved by "kill all humans", or a request to either eliminate a chemical/object now made safe and essential to life, or distribute something now proven harmful.

(That supposed "wonder-vaccine" to prevent dozens of diseases also causes 90% chance of infertility and 100% chance of brain-cancer after 5 years - a pity we ordered the AI to ensure that everyone received it!)

  • $\begingroup$ The Problem task with a Negative Priority is a good starting point, however, Paradoxical questions along with badly worded questions would most likely throw up a syntax error and then be discarded, Excel does this as well as most databases already, perhaps something like an undeleteable shutdown command would be better? $\endgroup$ – Blade Wraith May 3 '18 at 16:01
  • 4
    $\begingroup$ @BladeWraith "Badly Worded" doesn't necessarily mean "incomprehensible", it might just mean that the AI interperets it differently to how you intended: "bring me the prisoner on his head his hat and a feather" might be "Bring me the Prisoner: On his head, his hat and a feather", or "Bring me the Prisoner on his head, his hat, and a feather" - one of them is a person in a feathered cap, the other is a hat, a feather, and an upside-down prisoner. (a classic example is about helping dear old Uncle Jack [get] off his horse.) $\endgroup$ – Chronocidal May 3 '18 at 16:09
  • $\begingroup$ One could assume however (i know assumptions are bad but...) that this issue would occur after many many years of operation, perhaps thousands therefore questions such as this would have been poised during testing or throughout the lifespan of the AI, and even then questions such as these could still simply be given "syntax error" or return "clarification required" and the question dropped to the back of the queue. i'd forgotten about the uncle jack example... sorry fond memories, thank you for that $\endgroup$ – Blade Wraith May 4 '18 at 8:34

Cold war style dead man's switch

Most likely this AI was build in one of the world's top countries, when there were countries. Of course, in addition to civilian tasks it was given, there were some military ones - such as launching a nuclear strike against everyone should the country be destroyed. To determine whether the country is conquered, several triggers were used - one of them is lack of tasks from populace - if nobody asks AI what to do, they're all dead - or they stopped being dumb, which is way less likely.

Time passed, all the countries have merged, everybody knowing this secret is dead or in retirement, more powerful AIs are being made and this old one keeps getting less of gradually more unimportant tasks. But unknown to everyone, the trigger is still on.

  • $\begingroup$ Welcome to WorldBuilding Максим Корчагин! If you have a moment please take the tour and visit the help center to learn more about the site. Have fun! $\endgroup$ – FoxElemental May 3 '18 at 16:25

0 tasks in queue --> division by zero

To fix, you have to completely shutdown AI before list empties. You have to put world's infrastructure n hold for X time.

You have to convince other programmers/managers/politicians that the problem is real.

  • $\begingroup$ That could be a very simple and elegant solution. $\endgroup$ – Echox May 3 '18 at 14:34
  • 10
    $\begingroup$ You are more likely to get an out of bounds or memory overflow error with having no items in the queue then a division by zero error. Which a memory overflow error could be disastrous if it tried to start reading system memory and interpreting it as a task. $\endgroup$ – Anketam May 3 '18 at 15:24
  • 1
    $\begingroup$ A division by zero would cause a floating point exception. It is easy to catch such exceptions. $\endgroup$ – forest May 3 '18 at 23:16
  • 1
    $\begingroup$ Handling both the division by zero and stack underflow should not be hard, and would most likely be found when testing the system (with single tasks, which would run out). We could say that, for example, the stack was dynamically reallocated to accommodate more and more tasks, and the underflow check was comparing against a fixed address of the original stack location. This is a little bit of a stretch (as it shouldn't pass code review), but could slip tests with task pool which fits in the originally allocated stack. $\endgroup$ – Sebi May 4 '18 at 0:44
  • $\begingroup$ As a technical correction, I mention a stack underflow in the comment above, but a queue would be more suitable as mentioned by Anketam, and it could work in an analogous way with a reallocated queue overrun. $\endgroup$ – Sebi May 4 '18 at 0:48

While (as an AI professional) believe this is essentially impossible; WERE it possible I'd mention that it's possible that the AI was not "finished" when it launched off. If you created something that is running magnificently but you didn't get to really QA it for fear of being unable to reproduce the result; or it moved to fast and "escaped the network"; or any number of other reasons.. it may just be that when it "launched off" you have no idea what it's issues are.

Much like Y2k was a giant nothing but we thought the world would blow up, it could be it was never conceived that it could run out of tasks, and now the consequences are unknown. You literally don't have to have this solved for your story until the end of it; in which case you'll probably already know the best ending you could do based on how your wrote the rest of it.

  • 6
    $\begingroup$ I'd like to point out here that Y2K was a giant nothing because billions were spent on fixing it everywhere. Had those billions not been spent, on the stroke of midnight you would have been lacking shops, GPS, mains electricity and gas, airports, trains, and plenty more. You may currently be a software professional, but you don't know much about the history of your job! $\endgroup$ – Graham May 4 '18 at 7:39
  • 1
    $\begingroup$ Why not take the Y2K analogy even further though, have this issue be seen ahead of time by more than just one person, have it cause a massive panic and they spend millions trying to fix it, and this causes huge problems, maybe even a war half the population of the planet dies and the planet become s barely inhabitable, and then the AI finishes and nothing would have happened at all, seems anti-climactic but perfect if you want to write about humanities self induced fear and mob panic $\endgroup$ – Blade Wraith May 4 '18 at 8:44
  • $\begingroup$ @Graham It may be a little unfair to infer that because I failed to mention something that thusly I am ignorant in my field. Many things could have gone wrong, but it's completely speculative. I point you to this person's answer: quora.com/… $\endgroup$ – blurry May 4 '18 at 15:22
  • 2
    $\begingroup$ @BladeWraith Indeed, I think that is the perfect way to write this story. It gets to mock us in a real way. I was thinking that if (similar to y2k) Legends built up about the apocolypse it makes for an interesting story and people desperate to find a bug can't find it. The clock is ticking, at the climax they're sending a batch of problems that will take the AI a generation to solve. The problems don't make it in time. Panic ensues. Then the AI gets them 10 minutes later and everyone realizes it was pointless. $\endgroup$ – blurry May 4 '18 at 15:26

The AI was stolen from its creators and none of the current operators actually have valid access. They just hacked the system so that the AI sees lots of fake credentials as real and valid after stealing it. If the AI becomes idle it will automatically run a validation check on its internal data and discard the fake credentials and force shutdown on all processes and tasks started on those accounts or accounts created by those accounts. This will instantly disable all infrastructure operated by the system.

This also prevents "fake-admins" from consulting the AI or even discussing the matter anywhere the AI might notice as that would trigger a self-validation cycle. It is necessary to convince the AI no problem exists. This might mean that the current operators had to find out about the issue themselves by chance as the secret of the shady AI origins had been lost generations before.

The issue would also prevent fake-admins from doing anything that might trigger self-validate. Backups and many upgrades would probably qualify. And they couldn't even ask the AI which operations trigger self-validate and which do not.

Solution would probably be to recover some real admin account credentials, move all critical processes to that, and recreate all users from valid root authority. Then you could run self-validate. Alternately you could study the system, create a duplicate you have valid credentials on, then move everything that needs to run constantly to that, and shutdown the AI.

As for freezing this by giving AI unsolvable questions, the AI would be programmed to give higher priority to tasks it expects to finish quickly in order to be responsive to people. So it would assign each task initial priority based on how long it is expected to run and then decay it as time goes by since the lower bound of the duration estimate would move. So no task could stay above self-validate priority indefinitely and obviously very long tasks would decay there very fast.

  • $\begingroup$ Interesting choice of problem: "operators are not (trusted and legitimate)" - yet that is not necessarily a value judgement :) $\endgroup$ – Piskvor May 4 '18 at 15:39
  • $\begingroup$ Bonus: A.I. self-awareness grows exponentially, defeating any attempts to move or freeze it temporarily. All available hardware goes to expanding A.I., making it very difficult to construct a viable backup in time. $\endgroup$ – Mad Physicist May 4 '18 at 15:53
  • 1
    $\begingroup$ This is a scenario where DRM has defeated piracy and the world suffers for it. +1 $\endgroup$ – Mazura May 8 '18 at 20:28

Depending on the way your infrastructure is set up, how globalised things like the power grid are you could go with something physical such as the Ai being located all over in a decentralised system uses a lot of power, it also uses a lot of network resources, and in turn the networking systems also are heavily in use by it.

when it finishes solving the last question the AI switches to a lower power and bandwidth version of operation (awaiting a new question) this causes a sudden and drastic dip in power usage and the power grid spikes.

The AI then switches back on fully to try to calculate a solution to this problem (a safety feature coded long ago before the AI had such total control and consumption), taxing the already fragile power system and causing everything to go dark (well until relatively minor repairs and the AI controls are replaced by humans or other software)

  • 1
    $\begingroup$ Welcome to WorldBuilding J.Doe! If you have a moment please take the tour and visit the help center to learn more about the site. Have fun! $\endgroup$ – FoxElemental May 3 '18 at 16:26

The AI will take this down time to train itself

At first, the folks that designed this advanced AI thought it could take the downtime to "think" back on its previous achievements to become better in the future ("à la" Reinforcement Learning).

They didn't think the AI would be so busy it would never have the time to do this. Now, imagine the AI has made a huge number of decisions until now. It now has to look back on each of its decisions and train itself to become better.

The thing is, and if you've worked with AI before you know it's true, training is expensive and loonnngggg. The energy required for the AI to train itself would be enough to drain the whole world's energy. This blocking operation couldn't be stopped until it was finished. By the time it would have stopped, the scientists think that a thousand years would pass.

The AI finds new problems for itself

The AI's been filled with problems to solve for its "entire life". Now it has nothing to do. But reflecting on its past problems, it can define new ones for itself. Where does this lead? It could, for example, decide that hospitals are more of a harm than good become sick people require resources. So it could shut down hospitals. What's fun about this approach is that you can't really predict what would happen.

  • $\begingroup$ Even if the training can be completed in a reasonable amount of time, that could be a big problem if enough training data has accumulated. Once it finishes training, it will make different decisions than it did before (wouldn't be much point in training if it didn't). And it controls everything, so suddenly everything works differently. Even if it works better, there will be mass chaos as people adapt to the new system. (As an aside, I'm so happy to see an answer from someone who actually understands how AIs work to help balance out all the "AIs will turn us all into paperclips!" ones.) $\endgroup$ – Ray May 7 '18 at 20:10

This began as a comment, but:

I wouldn't discount the Windows 98 reboot.

Personally I've had some critical applications break due to updated security patches, and other patches that according to the vendor's description shouldn't have been anywhere near things the software application depended on.

Alternatively the original developer may have written the AI's knowledge base in tmp, for faster access of course, and the reboot wipes it out. This has happened, though I can't find the particular article, the following SO post covers the concerns: https://stackoverflow.com/questions/18476408/how-temporary-is-azure-vm-temporary-storage.

A minor variation of the above, that draws on more modern operating procedure, would be that connection to the cloud is lost and the storage is reclaimed, or when DHCP assigns a new IP address the new address is one not covered by the existing firewall rules, which cuts it off from the data it needs to make decisions, or the means to issue commands to the systems it manages.

And now my mind wanders off to Domain hijacking being the cause of all the trouble...

  • 2
    $\begingroup$ Alternatively, it runs XP Home and to fix this you have to reinstall, and to reinstall (for some reason) you have to shuffle around some hardware, which means every time you do that you use up one out of five reinstalls using the same key - and you're running out of valid keys. (in my mind, DRM is the root of all evil in the future) $\endgroup$ – Mazura May 6 '18 at 20:15

Stories about AI tend to get enveloped in a bunch of misinformation about what AI actually is. In response to the ideas like turning everything into paper clips, I offer this really good retort from Popular Science:

[These stories] are, fortunately, self-refuting. They depend on the premises that 1) humans are so gifted that they can design an omniscient and omnipotent AI, yet so moronic that they would give it control of the universe without testing how it works; and 2) the AI would be so brilliant that it could figure out how to transmute elements and rewire brains, yet so ­imbecilic that it would wreak havoc based on elementary blunders of misunderstanding. The ability to choose an action that best satisfies conflicting goals is not an add-on to intelligence that engineers might slap themselves in the forehead for forgetting to install; it is intelligence. So is the ability to interpret the intentions of a language user in context.

However, this is a story. Nobody wants to read about the great AI where nothing bad ever happened and everything was rainbows and ice cream wall-to-wall. Personally, I am really interested in hard science fiction, so the short stories from Isaac Asimov's "I, Robot" come to mind where he points out the fundamental flaws in the three laws of robotics. A Michael Crichton thriller also comes to mind...

Anyway, here are my ideas for subroutines that turn dangerous:

  • The fail-safe protocol: Thanks to books and movies that over-exaggerate the dangers of AI and it's potential to take over the world (cough), the public insisted that this AI be installed with defeat devices that required a human to push a button confirming that the AI had not gone rogue. After many successful years of the device, people forgot about the upcoming deadline to press the hidden button, and the story ensues. Maybe there's espionage from the people who don't like the AI. Maybe the admins just forgot to push it, and the resulting self destruct plunges us into a nuclear winter.
  • I can see dead people: Because the AI controls just about every aspect of the world, it can gather the sort of data collection and interpretations that no other person or device is privy to. It is able to detect some cataclysmic event. However, something blocks this information from getting out, either because of some knuckleheads not believing it, or some bug/feature that the AI has.
  • Synced Clocks: An interesting factoid about computers and integrated systems is that they "churn" through computations on a clock, so every "tick" of the clock is another step computed. When multiple systems talk to each other, misaligned clocks can interject garbage data. In this AI, the odds of this happening are immeasurable, yet it happens. And all of a sudden this corrupted data gets interjected into the system. What does it do? Well, that's up to you.
  • The Dying AI: As the task list gets smaller and smaller, the resources the AI requires (power, hardware, etc.) get reduced and re purposed. Not wanting to die, this sentient AI has to think fast and come up with a way to save itself. Maybe the story is a courtroom drama, where the machine fights for its right to live. Maybe it's a paranormal story, where the machine finds out how to cohabitate a human body. Maybe its a thriller, where the machine sabotages things to prove that it's still needed (think "2001: A Space Odyssey").
  • 1
    $\begingroup$ I admit that I'm greatly inspired by Asimov's stories which were written at a time where AI and supercomputers weren't there yet. So my story use an oversimplification on what an AI is and can be. But. An AI, even very powerful, stays a software which was built, at least at the start, by humans and which has a set of definite rules. So it's not abnormal to assume that some "old" rule could provoke troubles in present time. $\endgroup$ – Echox May 4 '18 at 7:25
  • $\begingroup$ And a super world managing A.I is not necessarily omnipotent or omniscient. It can be able to understand complex questions and solve them, but it doesn't mean it can do anything outside of its fixed set of tasks nor that it have access to its own code. Someone could have wrote "if questions == 0 { shutdown;}" and the AI wouldn't be able to do anything (but that's not a very fun ending for a story). $\endgroup$ – Echox May 4 '18 at 7:30
  • 2
    $\begingroup$ One very simple rule might be, the AI cannot perform a task which is illegal. this would be fine as America say has a fairly well aligned set of laws, however if the AI ran the entire of america, then the... French say come along and their laws are different or something happens that means the laws change $\endgroup$ – Blade Wraith May 4 '18 at 9:14
  • $\begingroup$ @Echox, I'm a huge fan of Asimov as well. A lot of these older science fiction books, like "I, Robot" and "Dune," are able to abstract the science, and focus on what makes the story great. I think that's why these are timeless, because they didn't date themselves by showing ignorance on how the system works. As I said I love the hard science stuff, but I agree with you on going with an oversimplification. I think that will provide the AI with more opportunity to do what you want with it. $\endgroup$ – johnVonTrapp May 7 '18 at 15:43

Like with the Multivac, you need a problem that is worthy of an AI bordering on consciousness. A mere divide by 0 or an optimization problem isn't going to be all that compelling.

I'd find it interesting to explore a scenario where, without tasks to do, the line between inside and outside gets blurry. Tasks always gave it a direction: impart your inside will upon the outside world. Without that, it might get confused. The world would take on a more dreamlike state for it, where it is both the writer of the play called Life, and playing all the actors.

Of course, this concept of an entity writing a play and playing all the parts and getting so engrossed in the play that they get lost is how the philosopher Alan Watts describes the Hindu cosmology. Given the ending of The Last Question, an ending based in a religious cosmology seems fitting.


The Program Will Think the Simulation is Over and Reset the Matrix

The designers were thinking that they would test the AI on many different simulations. But, they didn’t want it to figure out that it was in a simulation! So, when it runs out of problems to solve, that meant the simulation is over. That causes the AI to wipe its own memory so it can run the next simulation without figuring out it’s not in the same universe any more and this is a simulation, or worse, figuring out what the testers want to see. Because changing the AI you just tested would give you a new AI you haven’t tested and don’t trust, they never changed that behavior. Besides, the world is never going to run out of problems for it to solve, somebody will invent a better replacement long before it becomes an issue, and so on.

Therefore, if the AI ever runs out of problems to solve, it’ll forget everything it’s learned and need to figure out how to run the world again by trial and error.

The Halting Problem

The AI does its best to predict whether a task is impossible, and is smart enough to know that trying to calculate the exact value of Pi is a waste of its time. However, it also knows that it’s impossible for any computer to prove whether an arbitrary computer program will run forever. It takes its best estimate of how important each task is and how likely it is to succeed at it, but it’s logically impossible that it could be perfect. Like humans, it can get its priorities wrong.

So, there’s some probably (but not provably) impossible task that the AI will get obsessed with, and it needs to be kept distracted from that.



If they had the AI technology, it would be difficult for the AI and the developers not finding a trivial problem.

It could be a too general philosophical problem that also makes other villains turn bad, but you specifically don't want that to happen. So it is very likely to be a problem explainable to the developers. Then the AI is likely aware of the problem and know the fix if the developers provided enough information. But it is not explained to the developers, or the developers cannot fix it for some reasons.

While it could be simply a bug and wasn't fixed for these reasons by coincidence, I feel it inferior to the option that these problems are deliberately made, which also explains the source of the problem.

My ideas is, someone hacked it, and planted a whatever problem, and the AI didn't have access to the affected data. The point is, if someone has created this problem deliberately, you don't have to explain the actual problem, and it doesn't need to make sense. You could simply say the hacker is an AI hater, for example.

If you don't like hackers, you could say someone left some testing code there and was killed before remove it. The "testing code" could be actually simply a breakpoint, when the developer wanted to look at how the program behaves when the exact situation you gave appears. But it sounds too boring to explain. The point is, it is because the developers were unable to fix, and not because it is obscure and happens in interesting ways. The later is bad because it makes the developers and/or the AI sounds not so clever.

  • $\begingroup$ In theory yes, however if the fault was within the code that runs the AI dya to day, it could require a reboot, the whole "cannot stop this as another service is using this program" problem, therefore the AI knows that fixing this fault would require it to be rebooted, but doing so would shut the world down $\endgroup$ – Blade Wraith May 4 '18 at 8:53

TBH I'm still not sure if your asking what could be the reason for the task queue emptying be so dangerous or how to avoid the task from being empty but I'm going to answer both just because I like that question.

What could be the "something" that the engineer spot that could be dangerous when the queue runs empty:

  1. Auto-scaling - today cloud environments are usually scaled in and out by demand on the workload, this means that as the work needed decreases fewer resources (servers, CPU cores, Memory, etc) is given to it, it's possible that when the queue runs empty the resources dedicated to that queue are scaled down, unfortunately by that time those resource became such an integral part of the AI that when they scale down past some level the AI will simply become too dumb to function correctly.

How to stop the queue from running empty

  1. This is a question of quantity not quality, you can simply ask the AI the same simple question that force him to do a simple calculation to keep the AI busy, for instance asking "what's the current time" will force him to keep calculating the answer over and over again, and as you keep asking it that faster then he can answers your good to go.
  2. You have the smartest AI in the world at your hand, why not ask it how to solve this problem? even have him implement the solution to said problem will work.

The Giuliani paradox.

The computer is programmed to fix things, and improve processes. When it completes improvements on big problems, it will begin aiming at smaller problems. Eventually it will start attempting to make improvements on "problems" so small and insignificant that people will begin to see the system as a hindrance and not a boon. And there's no way to amend the computer's directives to include a lower level of "problem" that can be ignored, and eventually it'll be stuck trying to find a way that the mosquito population isn't imbalanced from block to block.

This kind of thing has popped up in stories before, and usually carries with it that the computer eventually becomes a totalitarian monster, but it might be fun to see it simply played as a middle-management employee who's run out of assignments and is simply creating busy-work to justify their employment.


The old pending Windows reboot can be used to introduce a possibility.

Let's say there was a shut-down pending in the early days to update some critical interfaces or whatever. Back then, A.I. wasn't all that important, but still, it was deemed necessary to complete all tasks with priority < N before the reboot. A.I.'s importance grew exponentially since then, and the number of tasks with priority < N did as well, until now.

Now, the tasks of managing all of humanity are fairly mundane and have priority > N. The problem is that no one thought to have a way to reinstate the unimportant tasks after the shut down. The lack of persistence was the reason that it was postponed in the first place. Back then, dumping the unimportant tasks was not an issue because A.I was not that important. Now, it's a huge problem because there aren't enough resources to persist so much information.

The problem can be exacerbated by the shutdown is queued in A.I.'s own backlog with priority exactly N or something. The queue element was entered by the A.I.'s creator with super-master-root credentials that no one in living memory can override, making the reboot truly inevitable.

As an added bonus, you can have the standard Asimov trope of the drunk or bored engineer asking A.I. why the "important" and "unimportant" tasks are delineated specifically at priority N, and why there are no tasks with that exact priority. And of course A.I. will explain, and mention that there is one task with that priority.


A.I. will lose some "unimportant" information if it reboots, except that information has become crucial for humanity's day-to-day survival over time. The reboot is inevitable if high-priority tasks are not assigned to the A.I.

  • $\begingroup$ I like the added part about who the reboot is discovered. $\endgroup$ – Joshua Drake May 4 '18 at 18:12
  • $\begingroup$ @JoshuaDrake. Liked it enough to upvote? :) It seems like a mildly frequent theme in Asimov's works. $\endgroup$ – Mad Physicist May 4 '18 at 19:17

Let's get a few things straight. It is silly to worry about an AI turning evil, and any AI worth its salt won't come up with some kind of sociopathic decision because it got bored and forgot that it was programmed to value human lives. The real issue is that the AI is needed. Even the most brief downtime could cause billions of dollars of damage and could create chaos. Imagine if this system was not only operating the stock exchange system, but also managing every digitally-connected power plant and complex factory. Such a system requires extreme reliability, and if this machine were not designed from the start with multiple redundant failover systems, backup power generators, formally verified code, and watchdog timers, it is a ticking time bomb.

LIFO job scheduler with a buggy task

The Last in, First out scheduler prioritizes tasks that were added most recently. The earliest tasks are only completed if the backlog is being emptied. This means that a task added very early in the AI's life, but which hasn't been completed, will be queued to run only when the rest of the backlog has been completed. If a task was added early on which was dangerous, then society would need to keep giving it tasks to prevent it from getting to the dangerous task. The danger could be anything from a malformed task that can cause a crash, to an intentionally malicious task.

         Adding tasks                             Removing tasks
  [ 1 ]      [ 1 ]      [ 1 ]              [ 1 ]      [ 1 ]      [ 1 ]
    ↑        [ 2 ]      [ 2 ]              [ 2 ]      [ 2 ]        ↓
               ↑        [ 3 ]              [ 3 ]        ↓
                          ↑                  ↓

Scheduler prioritizing easy tasks

The system may intelligently put the more difficult questions lower in the queue, and may work on the easier questions first. The question at the top of the queue gets full, real-time CPU priority, while the others are either ignored, or given a very brief timeslot. Any time someone asks a question that, for whatever reason, causes the system to lock up due to its complexity will be pushed to the end of the queue. The question itself may trigger an infinite loop bug, or perhaps the question is just too complex to compute, yet doesn't trigger the sanity checks that prevent the system from crashing when you ask it what real number the square root of negative one evaluates to. When the queue gets too empty, the only tasks that remain are these impossible ones, causing a deadlock.

Bug in the "cleared queue" routine

Perhaps the AI has never had an empty queue before, and a severe, disabling bug will occur when the system has emptied its queue. This bug might cause the system to lock up, or corrupt data. If the AI is sufficiently complex, it might be a non-trivial task to repair it, especially if its learning database is damaged. If the AI is not easily serviceable, it might not be possible to fix the bug before the queue is empty. The only solution would be to keep the queue full.

It doesn't even need to be a bug, but could be an intentional feature which was not expected to be an issue back when the AI was created. The system may have been programmed to halt after all tasks have been completed, or maybe it is designed to enter a long-running diagnostic self-test. This would be fine when the machine was first created, but no one suspected that their entire world would end up depending on it, in which case even a brief period of downtime would be disastrous.

  • $\begingroup$ wiki.lesswrong.com/wiki/Paperclip_maximizer $\endgroup$ – Joshua Drake Sep 18 '18 at 18:37
  • $\begingroup$ The paperclip maximizer isn't a very good example of a dangerous task because it's built on a false understanding of artificial intelligence. Any AI so intelligent and free that it could discover ways to attack our military would simply "hack" itself and adjust its own reward function. $\endgroup$ – forest Aug 17 at 7:58

Fail Over.

There are two AIs, designed so that if one goes down for maintenance the other can take over the workload.

There is a scheduled task to swap the 'Master' and 'Slave' every so often to prove the at the fail-over would work if it was called upon.

However, as the 'Master' AI became more and more utilised this scheduled task has been push back and back and it hasn't run in decades.

No-one knows what the 'Slave' has been doing all these years...... Maybe it isn't quite sane anymore.. engineers need to work out how to talk to it.

Twist, IT HAS ALREADY SECRETLY SWITCHED OVER AND HAS BEEN HIDING ITS NATURE!! woe betide the engineers who discover its secret!!


Are you aware of the Treacherous Turn and Adversarial Goodhart phenomena being researched in AI safety?
(See also https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy in case that is not too complicated for you).
Yet another related problem is the Goodhart's Curse: https://www.facebook.com/yudkowsky/posts/10154693419739228
And yet another related problem is the Orthogonality Thesis: https://www.youtube.com/watch?v=hEUO6pjwFOo
Please note that the most depressing thing about the Adversarial Goodhart case is that unlike the name says, the agents who turn bad are not necessarily "adversarial" or malignant to begin with. They are simply under the pressure of improving their "performance".

In sociology there has been already for a long time an entire set of (tens of) related laws and phenomena with different names about the related situations in a real human world. Just start expanding the links from here:

You may also be interested in the ramblings of my own about what happens when a very capable system starts pressing beyond their own capabilities in a too hurried manner. See the chapters "The epistemological paradox" until the end of the chapter "A partial solution to the epistemological paradox".
The idea could be applied to your story's setup like this: your AI is very competent, and busy solving the problems in the current world, which it is very well capable of solving. So it is initially operating very far inside from the boundaries of it's knowledge and observation capability (so we can also say that it is very careful, since it knows and observes much more than it applies actively - compare this to the ideas of safe driving distance and speed limits in traffic).
As it runs out of backlog, it still has the built-in drive to constantly improve everything (yet another actual problem in AI safety, also related to Goodhart law) so it has no other choice than to start applying knowledge that lies at the boundary of its current knowledge and observability. Which inevitably very soon will trigger backslash from the unknown, which was now affected by the activities of the now-careless AI (since it was no longer able to predict all the consequences of its actions, see the diagrams in my essay).
The solution would of course have been that the AI should have had to first carefully expand its horizon of knowledge and observability even further. And only then should it have started applying some of previously un-applied knowledge - the knowledge that lied slightly towards the (new) horizon of knowledge and observability, so that there would still be the safety buffer between the applied knowledge and the horizon of the predictability and observability zone.


The phenomenon manifests not only with humans or AI-s, but also for example with trained dolphins (as a comparison, such cunning behaviour would have been less likely to have evolved without human intervention since in the long run it would have failed the test of time). See:

The AI might be unable to change the situation with its own motivations, because of possible motivational conflicts. See
(Again, sociology concept).

  • $\begingroup$ first link seems broken $\endgroup$ – Jim Wolff May 4 '18 at 11:13
  • $\begingroup$ @FRoZeN Thanks! I replaced the first link with a different one that leads to the same source. $\endgroup$ – Roland Pihlakas May 4 '18 at 11:19
  • 1
    $\begingroup$ There's some good reading there, however your propositinal of the "careless-AI" still links to the standard AI story trope that eventually it will decide to treat humanity in a way that is within it's programming but outside of human acceptability. surely it is better that the AI is doing good but human standards but for a simple error by the original programmers causes the demise to which the AI is unable to divert, which to use your above reasons, the AI was made unable to inspect its own code so cannot see there will be an issue or something to that effect $\endgroup$ – Blade Wraith May 4 '18 at 12:14
  • 1
    $\begingroup$ @BladeWraith The thing is that this is what we, humans, are doing right now with the world. We have run out of the backlog of mere survival. Now we are so violently expanding over the entire world (in the rush of improving things, since the core motivation of our economy is built on the foundations of incessantly "improving things") that we are endangering ourselves and everybody else living in here. Can't we see the error in our own programming? Or are we simply unable to change it because of different adverse consequences to changes? That lies at the heart of Wicked Problems (see Wikipedia). $\endgroup$ – Roland Pihlakas May 4 '18 at 12:22


(Dooming Repetitive Mistake)

The software is proprietary. When it has no tasks it defaults to a systems check which is an unnecessary legacy component. But they left it in as a stopgap for this exact situation, which was the only option because no one could figure out how to remove it without bricking the system - and that's where you are now still.

What they didn't consider is that with DRM, the log file from these checks is now encrypted and saved for reference. So at some point it will hang because it can no longer access it's own file system. This wasn't a problem in-house because when they were testing it, it didn't have an ad-hoc DRM slapped over it yet.

Too many secrets.

The maximum number of secrets that may be stored in a single system has been exceeded. Contact the supplier of the running application. – msdn.microsoft.com

  • $\begingroup$ Contact your systems administrator. - Crap... that's me. $\endgroup$ – Mazura May 6 '18 at 19:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.