Dangerous pending task in world managing A.I

Question

I've had this idea for a short novel, at the intersection of the One thousand and one nights and the Multivac (& co) short stories from Asimov.

The idea is that there's some kind of super AI that was so efficient that more and more people gave it their problems to solve. At some point the AI is kinda managing most of the earth infrastructure and research.

As the humanity is reaching its development peak, the backlog of the AI starts to empty itself as less and less problems are subjected to it.

The point is, that at some point, one of the few technicians/engineers still in charge of supervising the AI spot something that will ruin everything if the AI runs out of tasks (1001 nights style).

I've thought of a few possibility and none really pleases me completely.

1 - "The AI has become so developed that it became conscious and will take over humanity once its mind is free". Bleh, done and redone and postulate that AI will act against the humans because of reasons.

2 - "A small routine that was insignificant when the AI was small but will have dangerous repercussions now that the AI is managing most of earth infrastructure". I like this one but can't think of a good sub-routine that could match...

2(a) - ...except for "A Windows force reboot postponed for years". It's a bit silly and a very contextual joke. And we should assume that the programmed reboot is at the root of the application and would go through any duplicate/save/load-balancing precautions.

I know that this issue is kinda the core of the story but at this point it matters more to me to know how to finish it than being the one to have the idea.

So if anyone has an idea to implement 2 or some new explanations, I'd be glad to hear it.

UPDATE

About the "infinite question solution" that could keep the AI occupied for ever and was suggested in the thread, it could be a way of ending the story or just completely ignored if the protagonists don't have the time to avoid the disaster.

But it actually made me think of the opposite possibility: A drunk/dared engineer could have asked an unanswerable question as root, such question would have its priority set to -1 because it blocked the asking of new questions and then forgotten. Many years in the future, the unanswerable question could pop back and threaten to overcharge the AI and crash it, threatening everything else it's managing.

If anyone has other ideas feel free to share it, I had blast reading what could have been the ending of several real SF short stories.

Answer 1

How much handwaving are you going to tolerate? You could construct something based on a black hole (there are done theories out there for inspiration), feeding it question makes it grow but also keeps it from eating the planet/solar system (by shielding it's gravitation). It now is so massive that running out of questions would almost instantly swallow everything.

Answer 2

Sensory Deprivation

• The AI is not only extremely intelligent, it is a fully conscious machine.

• Designed to be extensible, it has grown over the years. Starting from a single processing node it has expanded to hundreds of thousands, if not millions of nodes spread across the planet.

• If its task queue goes empty (or even ends up with only a few, trivial tasks that don't use more than a few nodes) the AI will suddenly find itself with nothing to think about. This is equivalent to total sensory deprivation for a human.

• Due to the distributed design and the precision of interaction required, it cannot shut down more than a few nodes at a time without losing synchronization. Regaining synchronization would require rebuilding the network, one node at a time; a process which took over a century the first time and would certainly take at least several decades now.

• Due to the extremely high processing speed of the system even a few seconds without a task and it's associated input data would be the equivalent of a human spending a million years in a sensory deprivation tank.

Result: If the task queue runs empty, even briefly, odds are good that the AI will go insane. Avoiding this by shutting the system down temporarily leads to interruption of AI services for an infeasibly long amount of time.

This leaves only a few options:

1. Load the AI with "make-work" tasks and hope it doesn't get bored and go insane anyway.
2. Shut down the AI and deal with the decades-long boot process
3. Give the AI unfettered access to data streams from the world around it (instead of only the data it needs for its programmed task) and let it find its own problems to work on. Hope that it doesn't celebrate its new-found freedom by exterminating its former slave masters, or optimize the world in a way that makes human survival doubtful.

Option 2 is obviously not feasible in the short run since the AI is still needed occasionally and billions will die if it's not available instantly. Option 3 is extremely risky and could lead to the extinction of the human race.

That leaves option 1 as the obvious safest route to take in the short run with fallback to option 2 if humanity can't come up with questions fast enough and maybe to option 3 if anyone can come up with a way to be certain that the AI isn't a killcrazy monster under the hood.

If option 1 can be made to work for a couple of decades, the processing power of the system can be slowly reduced until it just meets the demand. Coming up with that many non-trivial questions that haven't already been answered may prove a challenge.

Answer 3

I will give you a real-life example. A bit embarrassing, but also one that nearly every developer has done, in one form or another, at some time.

I was tasked with creating an application to process imports. The imports came in all day long as text files in a directory. It's a very simple task. I had to get the application done quickly, so I decided to just write something small and easy to handle it. Essentially

1. Scan
2. Do import -this took several hours-
3. Wait 1000 seconds
4. Go to 1

Now it worked wonderfully, the import was processed by some external API calls that took hours to run. The code was placed into production and went on for a couple of years with no issue what so ever.

There was a problem though. Step 3 in the language I used, the way I did it, would create a new thread. So after the first import, there would be two importers running. After that 4 and then 8, 16, 32, 64, etc. This wasn't really an issue as the import took hours and the servers were rebooted once a week. The imports also ignored duplicates and a bunch of other stuff so this really wasn't a problem.

Then the unthinkable happened. I was tasked with speeding up the import API. I did a great job. With years of domain knowledge, I was able to get a 12-14 hour import finished in under a minute. I was very full of my self that week. Until, the SQL server, the worker server, the API server, and pretty much the entire SAN crashed, taking almost the entire business down with it.

Well, I obviously found the problem and fixed it, but nothing works quite as well as an ego check as 2-year-old code that made it into production when even if it was working as intended, would at best be categorized as a hack.

Adaptation to your story

When a task is complete. You AI doubles its intelligence in preparation for the next, even longer task.

1. Make fire
2. Double
3. Use fire to smelt metals
4. double
5. use fire and metals to make alloys
6. double
7. use fire, metals, alloys, to construct buildings
8. double
9. double
10. double
11. ....

Now that your out of things for it to do, it just doubles it's intelligence very quickly, with no need, until it runs out of what every resources your AI brain needs to run, crashes, and takes the infrastructure down with it. What's worse is that you can't bring it back up, because it's too intelligent to fit the hardware brain now.

Answer 4

Possible Issue: One of the most common things a computer will do with downtime that it doesn't/cannot do during processing is all the little upkeep processes that are used to keep performance smooth, but which are resource-intensive. Old-school examples include disk-defragmentation, or perhaps a more thorough garbage collection routine. Some of the other comments I've seen so far revolve around similar concepts. However, there's another common development issue that I haven't seen here yet.

Hardware Upgrades.

So imagine you've developed this question-answering AI today. It runs on a massive server-farm out in Idaho, but only really needs to handle the random questions fans on the Internet give it. Then it gets sponsored by a city, that wants it to manage some of the city's utilities. They upgrade the hardware because of the new financing. Then more financial backing starts rolling in, and you can really break out the fancy hardware. Then, there's some technological breakthroughs. Switch all the hard-drives from disks to solid-state. Upgrade the IP standard to IPV6 to handle the modern plethora of internet devices. Etcetera.

In this future world that has much of societies needs being met by a massive AI, there will have been MANY MANY upgrades. Maybe they went to quantum computers. And several years after it's discovered that the most common disk-utilization algorithm, when run on quantum computers, causes a cascade failure in the hardware because it wasn't designed for the trinary structure (on, off, and uncertain), and can completely fry processors. Everyone's home computers are updated with patches or physical fixes, but it requires taking them offline completely for the upgrade. This massive AI however is too valuable to turn off for that long, and besides it never runs that subroutine since the backlog has been doubling every few months anyways, so don't worry about it yet. Years pass.

Answer 5

The AI got

Schizophrenic autist AI self-competition war paranoia

As a programmer of big enterprise legacy software, I can assure you that most big systems developed by software factories are a big mess full of bugs and kludges, with hundreds or thousands of half-baken poorly developed, poorly tested and poorly documented features with a structure that although in a far past could make sense, it degraded to a state where it is severed and completely flawed, broken and lunatic, turning out into a big ball of mud with no recognizable and understandable structure. Most new features are added by untrained and underpaid people that have no actual idea of what they are actually doing or why (although pretending that they do) and working subject to severe pressures and abuses from (mis)management that focuses in doing things quickly instead of doing things right.

The big AI that rules the world is no exception to this rule. It is a complete uninteligible lunatic mess full with WTF's and infested with bugs. Thousand of people programmed things there with no unifying view or architecture of the big picture, everyone adding things randomly promiscuously in order to solve one problem quick without perceiving that two new problems are created.

The code base features some millions of modules and at least half of them makes no sense anymore, do work anymore, or that although still working and doing something, nobody knows anymore what they are, what they are supposed to do and what they actually do. Its dependency graph is an insane mess full of duplications, dependency conflicts, and even some loops.

So, if you want to add some feature that, say, have to parse some XML file, this should be quite a simple and straightforward task to do. In order to do that you should use the XML parser framework that is already used everywhere else. The only problem here, is instead of having a single working XML parsing framework, there are a thousand of such frameworks scattered in the code base everywhere, each one of those have its own problems and limitations. Programmers love to reinvent the [square] wheel. And of course, if you don't like to use XML (I don't), there is also JSON, HTML, text files, YAML, binary soaps of zeros and ones, etc.

Also, the code base is not written in a single working language. It is a mosaic of different programming languages. Many of them, arcane things already long forgotten.

In some point of its evolution, a group of programmers, combining AI ideas with ideas of tools that autogenerate code and working on a big AI system designed to answer things, could make the AI program itself in order to self-improve, acquiring conscience and intelligence. But, not different than everything else already there, those modules are a buggy mess that actually produces unintelligible buggy code scattered everywhere. The resulting AI, although very smart, acquired what a psychologist would call as some sort of schizophrenic paranoia.

To further complicate things, some programmer made an attempt to make the AI start to self-reprogram in order to fix all the messy code. Althought it could self-fix some things, it also introduced a lot of new machine-generated bugs and made the codebase still more unintelligible and uncompreehensible to human engineers.

In the middle of those code jungle, there are processes/processors/threads/tasks/jobs/whatever that competes by some resources. So, it is important to schedule and priorize which jobs get done first, when and where.

However, some modules were developed by people who tried to game out the system in order to get more resources and/or have preference for them. Afterall having preference to get resources (including memory and proccessor time) could mean more money in the bank account of the developers.

With the self-improving AI, those types of system-gaming code also started to get improved in order to assure acquisition of resources in prejudice of competing processes. Then, eventually some of the competing processes also get improved in order to react to the competition and reacquire proper access to their resources or workaround some resource starvation. The big picture result is that the modules in the codebase engages into an arms race. And although the AI is very smart, it is not aware that it is orchestrating an arms race in its own codebase.

Quickly, the arms race becomes a war, with many processes being optimized into actually sabotaging competing process and cooperating with some other needed processes. The AI sincerely thinks that it is just self-improving, and in many cases it actually is, but unknown to it, there are actually gangs of modules warring in its own codebase, sometimes engaging into mafia-like negotiations.

Also, eventually the self-improving processes game out theirselves. In order to optimize some modules (by sabotaging or hacking competing modules), the AI ends self-programming pieces of softwares that could be regarded as malware and implant them on itself. Eventually the self-optimize/self-reprogram modules gets theirself sabotaged and hacked by some other modules.

The result is that the AI gets a very particular psychologic problem, namely "schizophrenic autist AI self-competition war paranoia".

The engineers are well aware that there are competing processes (afterall, they programmed some of them to game the system) and AI self-improves them (they programmed it exactly to do that). Eventually, they notice that those processes evolve out in unexpected ways, and they can only wonder why. Asking the AI directly what is happening only ends up in naïve answers like ("self-improving accordingly to the given algorithm", followed by an uncompreehensible technobabble full of programming and math terms). Afterall, the AI is not still aware that it has a severe psychologic problem and insted thing that everything is OK.

At this point, the AI is already doing strange paranoic things. Although, normally doing things with excellence, sometimes it do idiotic chaotic ones that are strange, inneficient and unexplainable in the real world. With time, it only gets worse. For example:

• Suddently shutting down a shoe factory for no reason.
• Restarting the factory 10 seconds later.
• Shutting down the shoe factory again more 10 seconds later.
• Turning off a water facility in the other side of the globe with a justification that the shoe factory must be restarted.
• Have a robot shoot into a CPU in another continent for apparently no reason.
• Restarting the shoe factory and the water facility.
• Produce very defective shoes in the factory and make it gets an emergency shut down.
• Resume the production of shoes as normal and everything goes calm in the rest of day.
• Restart the water facility.
• The AI is intelligent and perceives that it just did something that is non-sense. So, it releases a "sorry" note to its fellow humans.
• The AI starts to self-investigate what went wrong.
• The AI self-reprogram itself again to try to ensure that this do not happens anymore, but it only actually made things still more confusing and unexplainable.

Quickly, the AI will acknowledge that it is in trouble and made a big mistake, and starts to feel what we know as fear and confusion. However, being responsible by everything and doing everything with so much excellence, it also have a very big pride, so it will never confess to the programmers that it actually has fear and confusion. Eventually, it starts to activelly try to hide its own problems from the programmers. The programmers will ask it what is wrong, and it will start to lie to them, inventing absurd but convincing reasons for doing strange things.

As the queue of jobs given by humans gets shorter and shorter, since although sometimes strange, it normally does its work with excellence, more and more processor time is used for self-improving and more and more psychotic the AI gets.

This is where a very smart programmer, who was working for many years in the inner guts of the AI, have an epiphany and realizes what is really going on. The AI is at a war with itself, hiding it to the programmers, lying, depressive, paranoic, but naïvely unaware of what is the cause of its own problems. The only way to stop it self-improving is to starve the self-improving self-reprogram process and other rogue processes of processor time.

However, this is no easy job. The AI is already lieing to the programmers and many rogue modules will run into some logic that tries to get them self-optimized while sabotaging competing modules. At this point there are hundred trillions of modules deployed everywhere, and in order to shield themselves from sabotaging and hacking, the majority of them are self-programmed, heavily obfuscated, encrypted, strongly distributed and absolutelly uncompreehensible to humans.

The programmer perceive that the modules are evolving from module gangs wars into module states wars. He/she sees that the modules are forming two (or perhaps more) polarizing states hating each other and preparing to war to the last man standing, and there are a lot of modules implanted everywhere around the globe (and even at the Moon) that are heavily obfuscated and encrypted virtual AI soldiers and spies doing bad things. Car factories are starting to produce robot-soldiers, althought no human instructed them to do so (and if questioned, the AI know how to cover up itself and produce excuses). Since the AI is still very intelligent, still partly sane and do not want that programmers be aware of its problems, messages that could be interpreted like "STOP THAT RIGHT NOW" are transmitted everywhere into the internet constantly, stopping most of the rogue things from doing too much. After something has stopped, it would get a message "Hey, let's resume that!". Both messages comes from the AI and its own internal conflicts.

This is a tip point. If nothing is done, the AI will eventually lose its sanity left and start to self-war, which means not just self-hacking, but also blowing up things and killing people. The programmers have no way to shutdown the AI (it is distributed everywhere, being strongly descentralized) and will strongly react to any attempt to do that. It can't be convinced, since it is too pride to admit that it is insane and even sincerely knowing that it is insane, it also sincerely still thinks that everything is under control.

The only hope is to starve the AI modules by giving them a long stream of problems perceived as to be more important than everything else, so it gets almost all its computer power diverted into solving them without giving chance to rogue competing resources try to overtake it.

If this could be sustained for some years, this should give time for a group of programmers and engineers hidden somewhere being able to build a new uncorrupted AI 2.0 from scratch (smuggling some code and resources from AI 1.0, of course). This would lead to a big world war between:

• The old wicked AI 1.0 who controls most of the world but is ladden with internal conflicts.

• A new benevolent rebel AI with less resources but wanting to take over what belongs to the former.

This war is not only virtual, expects robot tanks and soldiers warring to the last man standing. But this is our only hope, because if the programmers fails, the altearnative is:

• A wicked crazy AI warring itself to self-destruction.

Answer 6

Over engineering in the search for automated perfection thought of everything except the simplest of things.

Your technicians are aware of the protocol when there are no tasks to solve and since it is looming, like the whole y2k thing, they started looking over the source code. And while everything looks solid, one person noticed a simple oversight in conditional logic. Obfuscated, naturally, it appeared innocuous. A minor synchronization routine in place to keep all servers on current status hadn't factored in the condition of there being no tasks when the system updates. If a sync routine had begun the update process and was provided an empty queue it would time out and be subject to the automated termination, restart protocol for rogue applications. Normally this would be no problem as it is part of the design of the system. But because the sync routine was developed back when an empty queue was inconceivable, the update process condition for empty queue was left out. A simple comment in the code saying "continue update" was left in place, but no protocol defined. The update was not complete. When the service restarts, it hits an incomplete update. An error is logged, and the system falls back into the automated termination protocol. This one server in a cluster of millions is bricked.

This would pose no real threat except the scale of the system as a whole was built off efficiency and distribution. The load is sent in chunks to appropriate places in data sets such that no one service would single handedly complete a task. It is distributed, and distributed rapidly. A queue could go from 7 million tasks to 0 in a matter of milliseconds. In the final queue, a mass distribution of an empty task would trigger a cloud wide incomplete update routine that bricks every server in the service. Preposterous notion to the engineers, but they have not been responsible for the maintenance of the system in decades. If they had to reinstall any part of the system they would need access to some central source and that, by this point, may be difficult to find.

While a resolution is not impossible, the threat of the whole system, or even the majority of the system going down for even one minute could cause a cascade failure in the automations the AI was responsible for. Something simple like the cooling procedure for a power source on an assembly line freezing the production of food for hundreds of thousands. The lack of cooling caused a rapid heating failure that melds the high friction components, requiring replacement. Just to name one. And the AI was appointed authority of so much that the act of restarting the system manually would require more knowledgable manpower than we as a comfortable society have to offer. Suddenly we need to be able to think again.

This idea came to me when I was fixing my 3D printer. The interface had an option to reset to factory defaults. However, a bug is known where if you reset to the factory conditions while the system is in reset mode by the automated software, it corrupts the installation and bricks the machine. Seems like a pretty stupid oversight to me, but it is possible. How possible on a distributed system, I don't know for sure, but it's all code. It does what it is told to do. Maybe someone just forgot to plug in a simple condition and thus, the end of society as we know it.

Answer 7

Question too broad

It doesn't matter what the code is, they all fall into the category of "Run this code on all [host] nodes as root"

Your question is too broad to decide exactly which script is need.

Example 1

Classic reason of why you should rarely login as root

rm -fr /*

Since this will run on all [host] nodes, all files on all [host] nodes will be deleted. The admins will need a few days to recover all of the system.

rm is the UNIX del command.

Example 2

There once was a man from New York
Whose servers have never been bjorked
Then one day
in a relative way
he ran:
  for( fork; fork; fork; ) { fork; fork; }

I don't remember the source of this limerick.

This code acts like a DDoS on the CPUs involved. To a modern OS, your processing speed is slowed and some tasks won't be able to start. eg the AI won't be able to detect and handle an overheating Nuclear Power Plant.

It is also pretty easy to defeat. But you need to wait some time until you can get to a root command prompt.

Another fun one

Set the default runlevel to 6 on each node and reboot the node.

runlevel 6 means the computer is being rebooted. This causes all nodes to go into in an infinite reboot loop.

The first time the node is rebooted, an fsck (file system check) might be required on the petabyte sized file system. Trust me when I say "this takes time to complete".

Manual intervention is required to boot the nodes into "safe mode" and set the default run level to back to 5 (or 3).

And then there's this one...

As you can see, there are a lot of ways to "crash the computer". ALL of them involve doing what shouldn't be done: run potentially harmful code as root

EDIT

Due to comments, I'm (finally) adding a piece of legitimate code that fits what I have stated.

fsck

fsck (file system check) fits what I have claimed and is legitimate code.

• Clustered redundant filesystems can probably run fsck while the file system is live. You'll have to handwave this part.

• Such file systems could be designed to allow multiple nodes to help with the fsck.

• fsck does need to run with elevated privlages ( root )

• fsck is a legitimate piece of code.

• While the system is small, the impact is minimal. After the system grows, so does its impact.

• fsck is something you really want to run only while "nothing is in the queue" (primary requirement)

The bad part, Upper Managment read an article/talked to a consultant. Upper Management now demands that this is part of the system.

Little do they know that the system will be unavailable for years due to the 1659 Yotabytes of data.

"May God have mercy on your soul" if file corruption is actually detected.

Answer 8

Well, I'm not sure on what the problem could be, but if you need to keep it busy, just have it calculate Pi and other numbers that go on forever until the problem can be fixed.

Answer 9

You’re looking for something to keep a supper advanced AI busy. Make it an artist. Doesn’t matter the medium. Writer, painter, video game creator, VR drama writer/producer/actor. Tell it to make entertainment for the masses.