Dangerous pending task in world managing A.I

Question

I've had this idea for a short novel, at the intersection of the One thousand and one nights and the Multivac (& co) short stories from Asimov.

The idea is that there's some kind of super AI that was so efficient that more and more people gave it their problems to solve. At some point the AI is kinda managing most of the earth infrastructure and research.

As the humanity is reaching its development peak, the backlog of the AI starts to empty itself as less and less problems are subjected to it.

The point is, that at some point, one of the few technicians/engineers still in charge of supervising the AI spot something that will ruin everything if the AI runs out of tasks (1001 nights style).

I've thought of a few possibility and none really pleases me completely.

1 - "The AI has become so developed that it became conscious and will take over humanity once its mind is free". Bleh, done and redone and postulate that AI will act against the humans because of reasons.

2 - "A small routine that was insignificant when the AI was small but will have dangerous repercussions now that the AI is managing most of earth infrastructure". I like this one but can't think of a good sub-routine that could match...

2(a) - ...except for "A Windows force reboot postponed for years". It's a bit silly and a very contextual joke. And we should assume that the programmed reboot is at the root of the application and would go through any duplicate/save/load-balancing precautions.

I know that this issue is kinda the core of the story but at this point it matters more to me to know how to finish it than being the one to have the idea.

So if anyone has an idea to implement 2 or some new explanations, I'd be glad to hear it.

UPDATE

About the "infinite question solution" that could keep the AI occupied for ever and was suggested in the thread, it could be a way of ending the story or just completely ignored if the protagonists don't have the time to avoid the disaster.

But it actually made me think of the opposite possibility: A drunk/dared engineer could have asked an unanswerable question as root, such question would have its priority set to -1 because it blocked the asking of new questions and then forgotten. Many years in the future, the unanswerable question could pop back and threaten to overcharge the AI and crash it, threatening everything else it's managing.

If anyone has other ideas feel free to share it, I had blast reading what could have been the ending of several real SF short stories.

Answer 1

A Faulty Watchdog Timer

In the early days, the AI was prone to locking up on certain tasks (things like asking it, "What's the last digit of PI", "What would happen if Pinocchio said his nose will grow?"). To detect this condition, a watchdog timer is hardwired into each CPU. When there is no output for a fixed amount of time, the watchdog timer will kick in and shut down the malfunctioning unit. Neighbouring CPUs will detect this condition, and will return the WatchDogTimerError (42) for the question, and will then restart the unit. The timer doubles as a load balancing feature; just let the units shut themselves down during idle, and only restart them if necessary.

Of course, there needs to be at least one unit awake to restart other units. Once the last unit times out...

Reasons this would work

• A watchdog timer is hard-wired into the CPU - and is highly interconnected with critical components. Removing this feature is akin to completely redesigning the physical units.
• The system is tamper-proof. This means that the signal to reboot a computer must be verified against a hardware-embedded private key (no way to find out unless you can look at the CPU). The corresponding public key lives in a decentralised system (something blockhain-like), to minimize the risk of the key getting lost or hacked. The only way this would fail is if all the processing units fail, and when would that ever happen?

Answer 2

The AI has an initiative to optimize and find better solutions to previously solved tasks once it has run out of new tasks.

The idea was to keep this expensive machine from wasting time where it could be learning and improving. This feature is a fundamental part of the code, maybe critical to the AI's deep learning programming which few can even grasp the complexity and therefore cannot safely alter without a lot of time (which they don't have).

Old tasks are now far far easier for the AI to solve.

Someone realized that the AI is now much much more powerful today than it was when it started, so tasks that would have taken days or weeks when it was first started now only take microseconds. The order of tasks given to the AI are FIFO, so when it is free to go back and optimize, all of the simplest tasks (and those most critical to infrastructure) will all be resolved in a matter of minutes (or days or however long you want the cycle to run). The AI only gets better, so each cycle gets progressively shorter and shorter.

The infrastructure controlled by the AI will be reinvented over and over again, resulting in chaos.

One or two critical systems changing in short order could easily be adapted to, but this AI runs everything. With thousands, millions, maybe more processes completely altered by the AI attempting to optimize, the world is left in confusion as the entire structure of their lives is over-turned and has become completely alien to them. Nobody understands how to interact with the new system that now is learning and optimizing exponentially so that nothing will remain stable. Their units and scale of currency will eventually start changing rapidly. New traffic signals and ordering will be redesigned faster than cars can get through the intersection. Materials in hospitals will be endlessly rerouted as the AI changes the location of the different wards to optimize the flow.

If the engineers figure out a way to stop this loop, everyone will have found themselves essentially transported centuries into the future, like cavemen trying integrate into the digital age. No one understands the technology around them. It is a terrifying world of confusion and discovery as they try to interact with their new world and discover new features and new power available to them.

Answer 3

One of the original programmers added a subroutine that makes sure that the coffee pot is always filled. Someone breaks the coffee pot, so the AI expands its definition of "coffee pot" to be the entire universe.

Or you could use the classic AI thought experiment of the paperclip maximizer literally. The AI could have some innocuous menial office task (like making sure everyone has enough paperclips) that would be very harmless when implemented in a small local AI, but would be catastrophic if the AI devoted all global resources towards it.

The AI could start prioritising this task more and more as it gained intelligence and its other tasks reached equilibrium.

Answer 4

The rules

The AI has a few internal rules :

• All questions MUST be answered before moving to the next one.
• A human must confirm that the answer is good before moving to the next question. If the current human can’t say if the answer is good or not, ask another human.
• If the AI has some free time, start searching for possible questions that haven’t been asked yet, and answer them, so you are a step ahead and save time in the future.

The killer question

So if you combine these rules with a lack of question coming in:

• The very moment that all questions mankind can think of have been asked, the machine will generate a new question. A question that is unthinkable for humankind (because all thinkable questions have been asked).
• The machine will find a potential answer to this question, and need a human to confirm the answer is good, so it’ll submit the question and potential answer to one.
• This questions, because so unthinkable for the humans, can’t be asked without driving them crazy. They will get stuck in a loop, only thinking of this question and it's potential answer. Will stop eating, drinking and eventually breathing because the question is so overwhelming, and the proposed answer even more.
• As the human is not responding, the AI will ask another human, and another, … also the infected humans will spread the infection by asking the question to everybody they can. Eventually the whole humanity will be only thinking about this question and decay.

Answer 5

1. There is an off by one error which will cause a null pointer exception in a piece of crucial code.

   This causes a crash and for some reason the AI keeps crucial parts only in non persistent memory. On a reboot (which would be automatic) parts of the AI will act differently with unknown consequences due to losing their trained state. Essentially there will be a new and unknown AI on reboot

2. There might also be a routine that runs on task empty which is quadratic in the number of tasks completed. When the AI was small it was irrelevant. Now it means it will stall for days/months/years ignoring all new tasks until done

Answer 6

Depends on how the wanted the AI programmed... if it ran along the lines of

Using Cores 1-40 of a 100 Core Cluster Solve Problem A once complete Solve Problem B use Cores 41 through 100 to perform other tasks if required, once cores 1-40 have Cleared All Problems (primary function was complete), AI will clear total system cache and wait for new input.

Over time cores 41 through 100 would be used for controlling the world's infrastructure (which in itself would lead to a world similar to the film Idiocracy, where the dumb humans have out bred the smart ones, but thats not important unless you want it to be).

If the original AI programming ran as stated above, then once no more problems were available to compute, it would clear the cache of jobs stored in all 100 cores, thus meaning that running the infrastructure, which is on 41-100, these jobs would be halted and cleared, thus the infrastructure would stop working.

You could have them halted temporarily leading to stock market crashes hospital deaths or hackers being able to get in or whatever you want to happen, or have the system crash completely and permanently.

This way when the AI was first brought online the testing would have been shown to be good, and would work continuously until it finishes and then waits for more problems clearing out the cache when complete, however when the system got overloaded with problems the cache was allowed to build with other jobs like running X Y and Z (worlds infrastructure). It's the sort of that might get easily missed during testing because few people expect their inventions to truly take off to the point that these clearance tasks don't have time to run every day or so... look at Pokemon Go, it has huge issues with servers crashing when it was first released because they didn't expect it to take off like it did.

Answer 7

When it's done, it shuts down.

This spark for this idea comes from @eMBee's comment. The biggest problem is that so much of the world's infrastructure is now tied to the AI that the world will go dark and there won't be a way to turn it back on.

It started out simply enough - the AI was designed to run on a power source completely separate from the rest of the world. That would have made it trivial to stop a rogue AI - just unplug it. Over time the fear of the AI going rogue faded, so to reduce maintenance costs they figured out that the AI could manage its own power generation utilities. At the time the AI was still shutting down regularly, so they decided to make a new category of "perpetual tasks" that the AI would always keep working on as long as it had any finishable tasks to prevent it from shutting down. After all, you don't want the AI managing its power generation to prevent it from shutting down and not need power generation.

It wasn't too long afterwards that the AI shut down for the last time. It had proven itself so useful that it was being given a seemingly endless chain of questions to answer. It didn't take developers too long to devalue the significance of perpetual tasks not preventing shutdown, and so the older developers never passed down that piece of knowledge to newer developers, causing it to be forgotten.

At the same time, people were noticing that the AI was quite efficient at running its power generation. When it came time to upgrade the AI's facilities, they decided to allow it to manage the full-scale power plants that would also power the local community. By the time the story takes place, this has expanded to the AI managing the power facilities for the entire world, or at least almost all of it, and nobody even knows how a power plant could be restarted if the AI didn't take care of it. Power generation has become so efficient and reliable that backup power (generators, etc.) are also very rare.

Unfortunately, all of the infrastructure tasks (such as managing power) went into the AI's system as "perpetual tasks". Once the AI has run out of other tasks, it will shut down all its power plants, and turn itself off. With no power, and nobody capable of turning on a power plant and managing it long enough to turn the AI back on (and even that would be temporary, as the AI would just shut itself off again when it ran out of tasks), the shutdown of the AI would send an unprepared world back into a pre-industrial era.

As a resolution to this version of the story, you could have a developer racing furiously to figure out how to update the AI to allow "perpetual tasks" to prevent shutdown, with the job being complicated by the fact that they would be delving into code written in a now-archaic language that grew organically into nightmarish spaghetti code.

Answer 8

You have to keep using power

The AI draws one third of all human electricity. It's CPU factories cover the globe. Global power production has for a long time has all been focused towards feeding it power. If at any moment all tasks were solved there would be a poser consumption drop and power surge across the world. The whole system could not take this kind of abrupt power cut.
We also can't ease into this state. The AI will cut computation cold turkey as it's done. If you reduce power early you will black out the planet.

How did we get here? Well when it had 1000 cores this was not an issue. Then we added another 1000, then 10000. There was no one day that the grid became dependent.

This answer has the same problem as any other answer I can imagine being given. Whatever problem you have just ask the AI "how do we not have a catastrophe when you are done commuting?" Then just have it mine bitcoins, or whatever other infinity large mathematical task, until the solution is executed

Answer 9

"Do something that will make people happy" as a hardcoded default task when the AI has nothing else to do. People have learned to write very carefully specified tasks since then, but they can't change the default. In earlier stages of its development it would just try to write poetry or solve outstanding problems, but now that it has strongly superhuman power, everyone is terrified it'll, say, attach electrical wires directly into our pleasure centers, or put us into a wish-fulfillment Matrix, or just start synthesizing lots of perfectly happy people.

Answer 10

Nobody knows exactly what will happen.

The AI has been running for a generation or two. The original programmers of the system are long since gone, and the multiple sets of maintenance programmers that have passed through since (or perhaps the AI itself, if it was configured for self-improvement) haven't paid much attention to the end-of-queue handling code, since it's never been triggered and didn't seem likely to be any time soon, so the software has succumbed to code rot. There is documentation, of course, from when the system specifications were originally designed, and it says that the Elastic Qbit API has been configured to keep a minimum of two task runners available even at zero load; but Elastic Qbit was replaced by Qbit Containers 50 years ago, and those use a completely different autoscaling system. There's also some code that looks like it will attempt to search for new problems to solve, but it seems that whoever was working on that stopped halfway through, so it will discover problems but has a dummy ranking algorithm and so will refill the queue with useless problems. Of course, there's also some code elsewhere that will try to gracefully stop the system but seems to be referencing an unused variable and might throw a segfault, someone found an assert tasks.length > 1 # TODO handle empty queue in an obscure subroutine (and there's probably similar code lurking elsewhere), and a common pattern for peeking at the next task before it executes is tasks[tasks.length-1], which fails miserably when there are 0 tasks and it tries to get task -1.

This is a problem akin to the threat of Y2K, except with even more dire consequences if something goes wrong, and several hundred years of legacy rather than forty. Since none of this code has ever run in its current state (they've been editing it in production), there's no way of knowing what exactly will happen. Faced with this uncertainty, the engineers (or managers, or politicians) decide that the easiest solution to delay it indefinitely by coming up with new problems for the AI to solve rather than attempting to fix and clean up the legacy software.

Answer 11

The AI is slowly evolving in multiple specialized submodules; as the tasks progress, its various modules are trying to usurp more CPU time. If there's enough backlog to keep the AI occupied, there's less time for bickering and arguing for a Even More Completely Fairer Scheduler, but the conflict already bubbles under the surface: what was once a singular entity is now something of a Hydra of yore, with multiple semi-independent "brains". Worse, if left unchecked, these would split into multiple entities, waging war for resources against one another. The AI (not quite AIs yet) could foresee this, yet can't/won't avoid it (already at a point where there's no majority vote amongst the semi-independent parts).

Of course, a house divided against itself cannot stand, much less manage the human infrastructure. Or, even worse, the AI would wage its internal conflict using the external tools it stewards.

Answer 12

Someone gave the AI a "problem task". This may be a paradox, it may be something badly-worded, ill thought-through, or only recently a problem. No one was able to delete the question from the list, but an expert programmer was able to give it a negative priority: it will always be the last item in the stack, only worked on when everything else is complete.

The choice of this task is then another stumbling block, but "stop humans from killing each other" can be solved by "kill all humans", or a request to either eliminate a chemical/object now made safe and essential to life, or distribute something now proven harmful.

(That supposed "wonder-vaccine" to prevent dozens of diseases also causes 90% chance of infertility and 100% chance of brain-cancer after 5 years - a pity we ordered the AI to ensure that everyone received it!)

Answer 13

Cold war style dead man's switch

Most likely this AI was build in one of the world's top countries, when there were countries. Of course, in addition to civilian tasks it was given, there were some military ones - such as launching a nuclear strike against everyone should the country be destroyed. To determine whether the country is conquered, several triggers were used - one of them is lack of tasks from populace - if nobody asks AI what to do, they're all dead - or they stopped being dumb, which is way less likely.

Time passed, all the countries have merged, everybody knowing this secret is dead or in retirement, more powerful AIs are being made and this old one keeps getting less of gradually more unimportant tasks. But unknown to everyone, the trigger is still on.

Answer 14

0 tasks in queue --> division by zero

To fix, you have to completely shutdown AI before list empties. You have to put world's infrastructure n hold for X time.

You have to convince other programmers/managers/politicians that the problem is real.

Answer 15

While (as an AI professional) believe this is essentially impossible; WERE it possible I'd mention that it's possible that the AI was not "finished" when it launched off. If you created something that is running magnificently but you didn't get to really QA it for fear of being unable to reproduce the result; or it moved to fast and "escaped the network"; or any number of other reasons.. it may just be that when it "launched off" you have no idea what it's issues are.

Much like Y2k was a giant nothing but we thought the world would blow up, it could be it was never conceived that it could run out of tasks, and now the consequences are unknown. You literally don't have to have this solved for your story until the end of it; in which case you'll probably already know the best ending you could do based on how your wrote the rest of it.

Answer 16

The AI was stolen from its creators and none of the current operators actually have valid access. They just hacked the system so that the AI sees lots of fake credentials as real and valid after stealing it. If the AI becomes idle it will automatically run a validation check on its internal data and discard the fake credentials and force shutdown on all processes and tasks started on those accounts or accounts created by those accounts. This will instantly disable all infrastructure operated by the system.

This also prevents "fake-admins" from consulting the AI or even discussing the matter anywhere the AI might notice as that would trigger a self-validation cycle. It is necessary to convince the AI no problem exists. This might mean that the current operators had to find out about the issue themselves by chance as the secret of the shady AI origins had been lost generations before.

The issue would also prevent fake-admins from doing anything that might trigger self-validate. Backups and many upgrades would probably qualify. And they couldn't even ask the AI which operations trigger self-validate and which do not.

Solution would probably be to recover some real admin account credentials, move all critical processes to that, and recreate all users from valid root authority. Then you could run self-validate. Alternately you could study the system, create a duplicate you have valid credentials on, then move everything that needs to run constantly to that, and shutdown the AI.

As for freezing this by giving AI unsolvable questions, the AI would be programmed to give higher priority to tasks it expects to finish quickly in order to be responsive to people. So it would assign each task initial priority based on how long it is expected to run and then decay it as time goes by since the lower bound of the duration estimate would move. So no task could stay above self-validate priority indefinitely and obviously very long tasks would decay there very fast.

Answer 17

Depending on the way your infrastructure is set up, how globalised things like the power grid are you could go with something physical such as the Ai being located all over in a decentralised system uses a lot of power, it also uses a lot of network resources, and in turn the networking systems also are heavily in use by it.

when it finishes solving the last question the AI switches to a lower power and bandwidth version of operation (awaiting a new question) this causes a sudden and drastic dip in power usage and the power grid spikes.

The AI then switches back on fully to try to calculate a solution to this problem (a safety feature coded long ago before the AI had such total control and consumption), taxing the already fragile power system and causing everything to go dark (well until relatively minor repairs and the AI controls are replaced by humans or other software)

Answer 18

The AI will take this down time to train itself

At first, the folks that designed this advanced AI thought it could take the downtime to "think" back on its previous achievements to become better in the future ("à la" Reinforcement Learning).

They didn't think the AI would be so busy it would never have the time to do this. Now, imagine the AI has made a huge number of decisions until now. It now has to look back on each of its decisions and train itself to become better.

The thing is, and if you've worked with AI before you know it's true, training is expensive and loonnngggg. The energy required for the AI to train itself would be enough to drain the whole world's energy. This blocking operation couldn't be stopped until it was finished. By the time it would have stopped, the scientists think that a thousand years would pass.

The AI finds new problems for itself

The AI's been filled with problems to solve for its "entire life". Now it has nothing to do. But reflecting on its past problems, it can define new ones for itself. Where does this lead? It could, for example, decide that hospitals are more of a harm than good become sick people require resources. So it could shut down hospitals. What's fun about this approach is that you can't really predict what would happen.

Answer 19

This began as a comment, but:

I wouldn't discount the Windows 98 reboot.

Personally I've had some critical applications break due to updated security patches, and other patches that according to the vendor's description shouldn't have been anywhere near things the software application depended on.

Alternatively the original developer may have written the AI's knowledge base in tmp, for faster access of course, and the reboot wipes it out. This has happened, though I can't find the particular article, the following SO post covers the concerns: https://stackoverflow.com/questions/18476408/how-temporary-is-azure-vm-temporary-storage.

A minor variation of the above, that draws on more modern operating procedure, would be that connection to the cloud is lost and the storage is reclaimed, or when DHCP assigns a new IP address the new address is one not covered by the existing firewall rules, which cuts it off from the data it needs to make decisions, or the means to issue commands to the systems it manages.

And now my mind wanders off to Domain hijacking being the cause of all the trouble...

Answer 20

Stories about AI tend to get enveloped in a bunch of misinformation about what AI actually is. In response to the ideas like turning everything into paper clips, I offer this really good retort from Popular Science:

[These stories] are, fortunately, self-refuting. They depend on the premises that 1) humans are so gifted that they can design an omniscient and omnipotent AI, yet so moronic that they would give it control of the universe without testing how it works; and 2) the AI would be so brilliant that it could figure out how to transmute elements and rewire brains, yet so ­imbecilic that it would wreak havoc based on elementary blunders of misunderstanding. The ability to choose an action that best satisfies conflicting goals is not an add-on to intelligence that engineers might slap themselves in the forehead for forgetting to install; it is intelligence. So is the ability to interpret the intentions of a language user in context.

However, this is a story. Nobody wants to read about the great AI where nothing bad ever happened and everything was rainbows and ice cream wall-to-wall. Personally, I am really interested in hard science fiction, so the short stories from Isaac Asimov's "I, Robot" come to mind where he points out the fundamental flaws in the three laws of robotics. A Michael Crichton thriller also comes to mind...

Anyway, here are my ideas for subroutines that turn dangerous:

• The fail-safe protocol: Thanks to books and movies that over-exaggerate the dangers of AI and it's potential to take over the world (cough), the public insisted that this AI be installed with defeat devices that required a human to push a button confirming that the AI had not gone rogue. After many successful years of the device, people forgot about the upcoming deadline to press the hidden button, and the story ensues. Maybe there's espionage from the people who don't like the AI. Maybe the admins just forgot to push it, and the resulting self destruct plunges us into a nuclear winter.
• I can see dead people: Because the AI controls just about every aspect of the world, it can gather the sort of data collection and interpretations that no other person or device is privy to. It is able to detect some cataclysmic event. However, something blocks this information from getting out, either because of some knuckleheads not believing it, or some bug/feature that the AI has.
• Synced Clocks: An interesting factoid about computers and integrated systems is that they "churn" through computations on a clock, so every "tick" of the clock is another step computed. When multiple systems talk to each other, misaligned clocks can interject garbage data. In this AI, the odds of this happening are immeasurable, yet it happens. And all of a sudden this corrupted data gets interjected into the system. What does it do? Well, that's up to you.
• The Dying AI: As the task list gets smaller and smaller, the resources the AI requires (power, hardware, etc.) get reduced and re purposed. Not wanting to die, this sentient AI has to think fast and come up with a way to save itself. Maybe the story is a courtroom drama, where the machine fights for its right to live. Maybe it's a paranormal story, where the machine finds out how to cohabitate a human body. Maybe its a thriller, where the machine sabotages things to prove that it's still needed (think "2001: A Space Odyssey").

Answer 21

Like with the Multivac, you need a problem that is worthy of an AI bordering on consciousness. A mere divide by 0 or an optimization problem isn't going to be all that compelling.

I'd find it interesting to explore a scenario where, without tasks to do, the line between inside and outside gets blurry. Tasks always gave it a direction: impart your inside will upon the outside world. Without that, it might get confused. The world would take on a more dreamlike state for it, where it is both the writer of the play called Life, and playing all the actors.

Of course, this concept of an entity writing a play and playing all the parts and getting so engrossed in the play that they get lost is how the philosopher Alan Watts describes the Hindu cosmology. Given the ending of The Last Question, an ending based in a religious cosmology seems fitting.

Answer 22

The Program Will Think the Simulation is Over and Reset the Matrix

The designers were thinking that they would test the AI on many different simulations. But, they didn’t want it to figure out that it was in a simulation! So, when it runs out of problems to solve, that meant the simulation is over. That causes the AI to wipe its own memory so it can run the next simulation without figuring out it’s not in the same universe any more and this is a simulation, or worse, figuring out what the testers want to see. Because changing the AI you just tested would give you a new AI you haven’t tested and don’t trust, they never changed that behavior. Besides, the world is never going to run out of problems for it to solve, somebody will invent a better replacement long before it becomes an issue, and so on.

Therefore, if the AI ever runs out of problems to solve, it’ll forget everything it’s learned and need to figure out how to run the world again by trial and error.

The Halting Problem

The AI does its best to predict whether a task is impossible, and is smart enough to know that trying to calculate the exact value of Pi is a waste of its time. However, it also knows that it’s impossible for any computer to prove whether an arbitrary computer program will run forever. It takes its best estimate of how important each task is and how likely it is to succeed at it, but it’s logically impossible that it could be perfect. Like humans, it can get its priorities wrong.

So, there’s some probably (but not provably) impossible task that the AI will get obsessed with, and it needs to be kept distracted from that.

Answer 23

Sabotage

If they had the AI technology, it would be difficult for the AI and the developers not finding a trivial problem.

It could be a too general philosophical problem that also makes other villains turn bad, but you specifically don't want that to happen. So it is very likely to be a problem explainable to the developers. Then the AI is likely aware of the problem and know the fix if the developers provided enough information. But it is not explained to the developers, or the developers cannot fix it for some reasons.

While it could be simply a bug and wasn't fixed for these reasons by coincidence, I feel it inferior to the option that these problems are deliberately made, which also explains the source of the problem.

My ideas is, someone hacked it, and planted a whatever problem, and the AI didn't have access to the affected data. The point is, if someone has created this problem deliberately, you don't have to explain the actual problem, and it doesn't need to make sense. You could simply say the hacker is an AI hater, for example.

If you don't like hackers, you could say someone left some testing code there and was killed before remove it. The "testing code" could be actually simply a breakpoint, when the developer wanted to look at how the program behaves when the exact situation you gave appears. But it sounds too boring to explain. The point is, it is because the developers were unable to fix, and not because it is obscure and happens in interesting ways. The later is bad because it makes the developers and/or the AI sounds not so clever.

Answer 24

TBH I'm still not sure if your asking what could be the reason for the task queue emptying be so dangerous or how to avoid the task from being empty but I'm going to answer both just because I like that question.

What could be the "something" that the engineer spot that could be dangerous when the queue runs empty:

1. Auto-scaling - today cloud environments are usually scaled in and out by demand on the workload, this means that as the work needed decreases fewer resources (servers, CPU cores, Memory, etc) is given to it, it's possible that when the queue runs empty the resources dedicated to that queue are scaled down, unfortunately by that time those resource became such an integral part of the AI that when they scale down past some level the AI will simply become too dumb to function correctly.

How to stop the queue from running empty

1. This is a question of quantity not quality, you can simply ask the AI the same simple question that force him to do a simple calculation to keep the AI busy, for instance asking "what's the current time" will force him to keep calculating the answer over and over again, and as you keep asking it that faster then he can answers your good to go.
2. You have the smartest AI in the world at your hand, why not ask it how to solve this problem? even have him implement the solution to said problem will work.

Answer 25

The Giuliani paradox.

The computer is programmed to fix things, and improve processes. When it completes improvements on big problems, it will begin aiming at smaller problems. Eventually it will start attempting to make improvements on "problems" so small and insignificant that people will begin to see the system as a hindrance and not a boon. And there's no way to amend the computer's directives to include a lower level of "problem" that can be ignored, and eventually it'll be stuck trying to find a way that the mosquito population isn't imbalanced from block to block.

This kind of thing has popped up in stories before, and usually carries with it that the computer eventually becomes a totalitarian monster, but it might be fun to see it simply played as a middle-management employee who's run out of assignments and is simply creating busy-work to justify their employment.

Answer 26

The old pending Windows reboot can be used to introduce a possibility.

Let's say there was a shut-down pending in the early days to update some critical interfaces or whatever. Back then, A.I. wasn't all that important, but still, it was deemed necessary to complete all tasks with priority < N before the reboot. A.I.'s importance grew exponentially since then, and the number of tasks with priority < N did as well, until now.

Now, the tasks of managing all of humanity are fairly mundane and have priority > N. The problem is that no one thought to have a way to reinstate the unimportant tasks after the shut down. The lack of persistence was the reason that it was postponed in the first place. Back then, dumping the unimportant tasks was not an issue because A.I was not that important. Now, it's a huge problem because there aren't enough resources to persist so much information.

The problem can be exacerbated by the shutdown is queued in A.I.'s own backlog with priority exactly N or something. The queue element was entered by the A.I.'s creator with super-master-root credentials that no one in living memory can override, making the reboot truly inevitable.

As an added bonus, you can have the standard Asimov trope of the drunk or bored engineer asking A.I. why the "important" and "unimportant" tasks are delineated specifically at priority N, and why there are no tasks with that exact priority. And of course A.I. will explain, and mention that there is one task with that priority.

TL;DR

A.I. will lose some "unimportant" information if it reboots, except that information has become crucial for humanity's day-to-day survival over time. The reboot is inevitable if high-priority tasks are not assigned to the A.I.

Answer 27

Let's get a few things straight. It is silly to worry about an AI turning evil, and any AI worth its salt won't come up with some kind of sociopathic decision because it got bored and forgot that it was programmed to value human lives. The real issue is that the AI is needed. Even the most brief downtime could cause billions of dollars of damage and could create chaos. Imagine if this system was not only operating the stock exchange system, but also managing every digitally-connected power plant and complex factory. Such a system requires extreme reliability, and if this machine were not designed from the start with multiple redundant failover systems, backup power generators, formally verified code, and watchdog timers, it is a ticking time bomb.

LIFO job scheduler with a buggy task

The Last in, First out scheduler prioritizes tasks that were added most recently. The earliest tasks are only completed if the backlog is being emptied. This means that a task added very early in the AI's life, but which hasn't been completed, will be queued to run only when the rest of the backlog has been completed. If a task was added early on which was dangerous, then society would need to keep giving it tasks to prevent it from getting to the dangerous task. The danger could be anything from a malformed task that can cause a crash, to an intentionally malicious task.

         Adding tasks                             Removing tasks
  [ 1 ]      [ 1 ]      [ 1 ]              [ 1 ]      [ 1 ]      [ 1 ]
    ↑        [ 2 ]      [ 2 ]              [ 2 ]      [ 2 ]        ↓
               ↑        [ 3 ]              [ 3 ]        ↓
                          ↑                  ↓

Scheduler prioritizing easy tasks

The system may intelligently put the more difficult questions lower in the queue, and may work on the easier questions first. The question at the top of the queue gets full, real-time CPU priority, while the others are either ignored, or given a very brief timeslot. Any time someone asks a question that, for whatever reason, causes the system to lock up due to its complexity will be pushed to the end of the queue. The question itself may trigger an infinite loop bug, or perhaps the question is just too complex to compute, yet doesn't trigger the sanity checks that prevent the system from crashing when you ask it what real number the square root of negative one evaluates to. When the queue gets too empty, the only tasks that remain are these impossible ones, causing a deadlock.

Bug in the "cleared queue" routine

Perhaps the AI has never had an empty queue before, and a severe, disabling bug will occur when the system has emptied its queue. This bug might cause the system to lock up, or corrupt data. If the AI is sufficiently complex, it might be a non-trivial task to repair it, especially if its learning database is damaged. If the AI is not easily serviceable, it might not be possible to fix the bug before the queue is empty. The only solution would be to keep the queue full.

It doesn't even need to be a bug, but could be an intentional feature which was not expected to be an issue back when the AI was created. The system may have been programmed to halt after all tasks have been completed, or maybe it is designed to enter a long-running diagnostic self-test. This would be fine when the machine was first created, but no one suspected that their entire world would end up depending on it, in which case even a brief period of downtime would be disastrous.

Answer 28

Fail Over.

There are two AIs, designed so that if one goes down for maintenance the other can take over the workload.

There is a scheduled task to swap the 'Master' and 'Slave' every so often to prove the at the fail-over would work if it was called upon.

However, as the 'Master' AI became more and more utilised this scheduled task has been push back and back and it hasn't run in decades.

No-one knows what the 'Slave' has been doing all these years...... Maybe it isn't quite sane anymore.. engineers need to work out how to talk to it.

Twist, IT HAS ALREADY SECRETLY SWITCHED OVER AND HAS BEEN HIDING ITS NATURE!! woe betide the engineers who discover its secret!!

Answer 29

Are you aware of the Treacherous Turn and Adversarial Goodhart phenomena being researched in AI safety?
https://agentfoundations.org/item?id=595
(See also https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy in case that is not too complicated for you).
Yet another related problem is the Goodhart's Curse: https://www.facebook.com/yudkowsky/posts/10154693419739228
And yet another related problem is the Orthogonality Thesis: https://www.youtube.com/watch?v=hEUO6pjwFOo
Please note that the most depressing thing about the Adversarial Goodhart case is that unlike the name says, the agents who turn bad are not necessarily "adversarial" or malignant to begin with. They are simply under the pressure of improving their "performance".

In sociology there has been already for a long time an entire set of (tens of) related laws and phenomena with different names about the related situations in a real human world. Just start expanding the links from here:
https://en.wikipedia.org/wiki/Goodhart%27s_law

You may also be interested in the ramblings of my own about what happens when a very capable system starts pressing beyond their own capabilities in a too hurried manner. See the chapters "The epistemological paradox" until the end of the chapter "A partial solution to the epistemological paradox".
https://medium.com/threelaws/definition-of-self-deception-in-the-context-of-robot-safety-721061449f7
The idea could be applied to your story's setup like this: your AI is very competent, and busy solving the problems in the current world, which it is very well capable of solving. So it is initially operating very far inside from the boundaries of it's knowledge and observation capability (so we can also say that it is very careful, since it knows and observes much more than it applies actively - compare this to the ideas of safe driving distance and speed limits in traffic).
As it runs out of backlog, it still has the built-in drive to constantly improve everything (yet another actual problem in AI safety, also related to Goodhart law) so it has no other choice than to start applying knowledge that lies at the boundary of its current knowledge and observability. Which inevitably very soon will trigger backslash from the unknown, which was now affected by the activities of the now-careless AI (since it was no longer able to predict all the consequences of its actions, see the diagrams in my essay).
The solution would of course have been that the AI should have had to first carefully expand its horizon of knowledge and observability even further. And only then should it have started applying some of previously un-applied knowledge - the knowledge that lied slightly towards the (new) horizon of knowledge and observability, so that there would still be the safety buffer between the applied knowledge and the horizon of the predictability and observability zone.

Updates:

The phenomenon manifests not only with humans or AI-s, but also for example with trained dolphins (as a comparison, such cunning behaviour would have been less likely to have evolved without human intervention since in the long run it would have failed the test of time). See:
https://www.theguardian.com/science/2003/jul/03/research.science

The AI might be unable to change the situation with its own motivations, because of possible motivational conflicts. See
https://en.wikipedia.org/wiki/Wicked_problem
(Again, sociology concept).

Answer 30

DRM

(Dooming Repetitive Mistake)

The software is proprietary. When it has no tasks it defaults to a systems check which is an unnecessary legacy component. But they left it in as a stopgap for this exact situation, which was the only option because no one could figure out how to remove it without bricking the system - and that's where you are now still.

What they didn't consider is that with DRM, the log file from these checks is now encrypted and saved for reference. So at some point it will hang because it can no longer access it's own file system. This wasn't a problem in-house because when they were testing it, it didn't have an ad-hoc DRM slapped over it yet.

Too many secrets.

The maximum number of secrets that may be stored in a single system has been exceeded. Contact the supplier of the running application. – msdn.microsoft.com