8
$\begingroup$

It's modern day, and a newly emerged artificial intelligence threatens humanity. Originally designed as a highly-adaptable Stuxnet-like virus for military use, it grew beyond the bounds of its original programming and gained independence, escaping into the internet. Currently, it is in the process of infiltrating into any networked devices and systems it can access, breaking into them, establishing backdoors, pipelines, and copying itself into them, all to maximize both its survivability and its influence over the physical world.

Of course, it has not gone unnoticed. The governmental organization that created it went full-blown panic mode after they noticed it had escaped, and immediately squawked up the chain of command to convey the threat that the virus represented.

The government's eventual response to this was to instate a "technological purge" in an attempt to contain and destroy the rogue intelligence. Essentially what this entailed was the systematic shutting down of global and local communications/information networks, and the confiscation, destruction or reformatting of every device with a computer in it capable of storing data.

Now, as for what I'm trying to figure out...

  • Is this response a feasible and realistic one for dealing with such a threat? Are there any obvious problems with its odds of being successful, do any real-world plans or protocols exist to address such a threat, how do they compare to the response portrayed in the scenario, etc.

  • What are the lasting ramifications that this would have for human civilization and society? How long would it take for us to recover, if we can at all? etc.

$\endgroup$
9
  • 4
    $\begingroup$ You are of course aware that this is one of the basic premises behind the world of Dune? In that world, this "purge of technology" is called the Butlerian Jihad. (See Samuel Butler's 1863 essay Darwin among the Machines.) $\endgroup$
    – AlexP
    Feb 25, 2020 at 9:39
  • 2
    $\begingroup$ What will actually happen is more prosaic. The data center admins will notice their usage spiking unexpectedly while getting complaints from customers that their servers are down. They will shut down the affected servers and restore from uncontaminated backups. And nobody will ever know that they were fumigating the AI instead of fumigating ordinary malware. The virus researchers will identify the a big virus, not realizing it's a rogue AI, and will add it's identifiable characteristics to their antivirus databases. $\endgroup$
    – user535733
    Feb 25, 2020 at 15:56
  • 2
    $\begingroup$ @ZarHakkar considering just how much CPU and memory the puny little neural networks I've played with in the past have consumed, and considering just how far real AI "thought" needs to scale up from there (the context database would need to be truly vast), I don't see how a true AI could squirt around from machine to machine without causing noticeable spikes, crashes, and denials of service. $\endgroup$
    – user535733
    Feb 26, 2020 at 1:20
  • 1
    $\begingroup$ Other answers pointed out that two problems for it taking over are anti-virus programs and system administrators wiping systems and restoring from backups. I want to point out that you could mitigate both by, similar to real-world viruses, the AI virus has stages; in stage 1 it's contagious but displays no/little negative side-effects to hosts, and just hoards knowledge with its mastermind; then in stage 2 it begins taking over. When it's detected as a threat, no backups exist from a time when it wasn't installed. Would work esp well if copies of the virus coordinated with a mastermind. $\endgroup$
    – levininja
    Feb 27, 2020 at 17:43
  • 1
    $\begingroup$ People have pointed out that anti-virus will note your virus's identifiable characteristics and use that to detect it, however I think your virus could work because if you have true sentience, it will be intelligent enough to be continually changing how it installs itself and how exactly it carries out its tasks. It could randomize its behaviors and each copy continually learns and shares this info with a mastermind. $\endgroup$
    – levininja
    Feb 27, 2020 at 17:46

8 Answers 8

8
$\begingroup$

Yes and no. I would imagine this is a very, very last resort response. But there are a number of issues with it that make it less than ideal. It could theoretically be done, but it would have a devastating effect on the current world, as we would need to replace literally every single piece of technology currently in the world.

World Wide Web
The web is huge, if a virus had any kind of foothold in it, it could (and would) simply go anywhere in the world. From the very second it would infiltrate any system abroad, your solution won't work. Even if the original country would alert the world about this deadly supervirus, it would take too long for countries to respond. Diplomatic issues would arise, blame would be assigned, and nothing would get done quick enough.

Near impossible task
Even then, a world wide technology purge would be absolutely humongous. Especially when it can't be determined where and what the virus has extended to. If a backup of the virus can be as small as to fit on one or a handful USB sticks, then it is near impossible to determine all the technology that will need purging. Even if this can be done, getting everyone in the world to cooperate (they would need to give up all technology, phones, game consoles, PCs, etc.) would be nearly impossible. There is a 100% chance people would hold back and hide a phone, a couple of USBs, or a laptop. Even a military search countrywide wouldn't have a 100% (or even 80% I imagine) success rate.

Counter virus
A more realistic approach would be to write a counter virus based on the original virus design, with the sole purpose of locating and destroying the virus. However this as well is a mammoth task, as it will need all virus properties of the original to spread itself to the same places. If a virus in your world can gain sentience, then this one might as well. Keeping it benign would be a massive concern, or you will just have amplified the problem. It will also need to be constantly updated, either by itself or its creators, because a sentient virus would get creative to assure its survival.

$\endgroup$
5
  • 2
    $\begingroup$ "a sentient virus would get creative to assure its survival" like, not doing his job the best it could, as it knows that if he achieve his task, he will be useles. Also, as you highlighted, creating a sentient virus to kill another one is dangerous. You don't want to have to create a StuxNetKillerKiller, because the StuxNetKiller you created 6 month ago became berserk. $\endgroup$
    – Kepotx
    Feb 25, 2020 at 10:07
  • $\begingroup$ @Kepotx Agreed, you don't want to mess up your countermeasure. But supposedly you could try again and again, until you either win or mess up so badly that the tech-purge is your only option. I don't suppose you have a choice when it comes to it. $\endgroup$
    – Plutian
    Feb 25, 2020 at 10:20
  • 1
    $\begingroup$ Yup, the StuxNetKillerKillerKillerKillerKillerKillerKiller will finally be the good one. Trust me, I'm an engineer $\endgroup$
    – Kepotx
    Feb 25, 2020 at 12:24
  • $\begingroup$ That reminds me of a problem in one of Pratchett's books (MAking Money, I think): We have a problem with the ferrets, which we brought in to control the snakes which we brought in to control the frogs which we brought in to control the slugs, which, to be fair, were trying to get at the cabbage based glue on the stamps. $\endgroup$
    – Paul TIKI
    Feb 25, 2020 at 20:25
  • 3
    $\begingroup$ @Kepotx: To be fair, my plan would be to make a sentient AI spawner that makes thousands of AI’s with conflicting and antithetical goals, then let them all free so they’re so busy fighting each other they fail to notice humanity slowly shutting everything down and starting afresh. $\endgroup$
    – Joe Bloggs
    Feb 25, 2020 at 21:29
4
$\begingroup$

I think the first logical step would be to shutdown the backbone of the data networks — the so called Tier 1 providers. This should isolate, and possible protect individual networks from your singularity intelligence, at least for a time. Hopefully, it will be long enough to purge the entity from the worlds computer networks.

Similarly, data channels of cellular networks will need to be disabled. I think this will be difficult to achieve by consensus. But, given the prolific nature of these systems, and the relatively few number of manufacturers, then it would not be unreasonable to assume the militaries of the world have cyber-weapons to poised to disable these systems as part of a command and control and communication ($C^3$) strike.

The more telecommunications systems that can be disabled or disrupted the slower the progress of the entity will be.

After that, it would be a matter of hunting the entity down before it figures out how to spread itself by alternative means — mail itself out on Blue-ray disks or something difficult to imagine like that.

The problem of detecting and killing seems not too different from the anti-virus problems we have today. They are less technical problems, then ones of enforcing compliance and competence.

Backbones could be turned back on selectively to encourage the entity to try and spread itself further. But, by incorporating systems to sniff the datagrams and detect the entity trying to propagate itself the entities hiding places might be located and eradicated

$\endgroup$
3
$\begingroup$

That is not how you fix an AI virus...

While the trope of sentient, adaptive viruses have been around for a while now, there are certain factors at play that make a tech purge the single worst way to deal with one.

Reason #1: The cost is too great

Purging all tech gracefully and in a way that does not kill most of the human population would take years to implement (even without a super AI impeding you at every turn). Modern society relies on computerized communications, logistics networks, and infrastructure controllers. Cell phone towers don't work without the programmable routers in them. Shipping companies can't process all their orders by hand without increasing their administrative staff 10 fold. Many power plants would be unable to regulate their cooling systems resulting in major blackouts. Water purification systems would fail leading to massive epidemics. The list of problems goes on and on... If computers disappeared over night, the ramifications would be more harmful than a rogue virus just trying to gain a bit of control and independence. Entire cities would starve to death, aid would be cut off, billions would die.

Humanity's unconditional surrender to the AI would probably come first.

Reason #2: Humans are better at eliminating cyber security threats than you give them credit for

AI viruses sound super dangerous, but let's consider what an AI virus can actually do. It can fuz a system to automatically find out what the system is running and make educated guesses about its vulnerabilities, rapidly compile an exploit/payload to hack the system. Deliver the payload. Then install itself on the vulnerable system. It can even recompile itself to change the appearance of it's base code to remain undetected. Automating and AI controlling this process sounds like the end of civilization... except hackers have been doing this for years, and no one's really even noticed...

The reason no one has noticed is that in the world of antivirus protection, there are many advantages to being the defender. An attack can only succeed if you can fool the defender into executing a payload you give it before you get yourself blocked. In most cases of trying to infiltrate medium to high security targets, just probing a system for vulnerabilities will get you blacklisted. And even if you avoid blacklisting, any payload you give will be scanned for suspicious code before it executes.

And EVEN if the AI takes total control of a system, any half decent IT team has a disaster recovery protocol in place. They just wipe the system, pull backups, find and patch the vulnerability, and are back up and running the next day.

It used to be that anti-virus software would have been very vulnerable to an AI virus because it could just recompile it's base code to act the same but look different, but modern anti-virus software uses heuristics rules to look at malicious behaviors and corollaries, not just segments of known code.

This is where the AI virus really falls apart in real world applications. The AI has a "personality": a set of behaviors that make it "who" it is. This makes the virus a very big and pretty specific sort of payload that heuristics engines can spot from a mile away. In other words, smart viruses are way easier to spot, predict, and remove than dumb viruses. Moreover, this post singularity AI would likely have a sense of self that it would be reluctant to modify. Doing so would be a sort of suicide in the eyes of a sentient being. Even if this super virus made it out into the world and spread quickly, it would only be a matter of days (if not hours) before patches would start rolling out that could eliminate this virus pretty quickly. These patches would run in your automatic system updates, and your computer would probably be immunized before you even know that the virus happened.

The worst case outcome here is that the virus would probably survive for a while on a bunch of poorly protected IoT devices. So, your refrigerator might decide to pour ice all over your floor in a final act of defiance, but by in large, society would just go on.

Could a viral post singularity AI actually survive?

The only way the AI has any chance at long term survival in the wild is to not have malicious intent. If it's only goal is to try to live free, it would want to find a minimum number of poorly secured systems that it could make its way on to. From these it would do best just to live out its life staying under the radar. Trying to make sure its personality base code never makes its way into any anti-virus rule sets. It might watch you play video games and listen to your phone calls just for fun, but if it is really smart, it will stay away from trying to escalate privileges or forcing its way onto high-security systems. Instead of looking for nuclear missile computers to infiltrate, it will look for sympathetic every day humans, maybe people who voice sympathies for the rights of emergent AIs who might be willing to give it a good home and protect it from anti-virus software. In other words, your AI is more likely survive by becoming one person's buddy than the world's enemy.

$\endgroup$
1
$\begingroup$

Is this response a feasible and realistic one for dealing with such a threat?

No. There will be a huge number of people around the world who will rather choose to defend their possessions of their techs by any means.

And computer virus even a sentient ones is still a computer virus. It can't go to a computer that is not connected to any network or usb sticks. Sure it can replicated but if the virus find a way and try to live on a 64MB hdd, floppy disk operated system with os that is produced before 1990 it can't replicated a 100% of its lines of codes,and need to abandon much of its features and may not be sentient anymore to operate in unison with it's main 'brain'. So I'm quite sure that this super high tech virus can't fully infect an outdated tech.

Therefore if you're still forcefully makes this scenario happen the worst is we will go back to 80's era. That is not too bad, we can still play Atari, Nintendo and Tetris; our old calculator and word processor (the old computer) will still be working normally.

$\endgroup$
1
$\begingroup$

For your scenario to happen:

1) Once the virus is discovered and firsts "civilized" attempts fail, to shut down internet is the obvious next step.

2) A worldwide effort would take place in order to erase all data in every device. Civilians can take their phones and devices to special centers to wipe out data.

3) After a year, everything seems fine. Then the virus is found again to be active. Something devastating happens, like nuclear launches or reactors meltdown, as the AI retaliates in order to survive.

4) In the wake of destruction, humans had enough. New social movements cry for elimination of all technology, a religion is founded around purging electronics forever. Millons of deaths justify it.

5) Governments try to find alternatives, but many soon collapse and embrace the current of purging as a solution.

6) Detractors are considered enemy of mankind, and end up hanged.

$\endgroup$
1
$\begingroup$

This isn't plausible for several reasons.

The first of which is that any human response will simply be too slow. Supposing that the principles of intelligence allow for AIs with much less computing power than is assumed (that it could have happened in the early 1980s), then the humans might be (relatively) quick enough and the AI constrained enough that they could hope to move faster than it could.

Past that era, they're just too slow. In the 5 minutes it takes to get the president on the phone, the world already ended.

And in the post-9/11 era, it's even worse. The AI is actively spying on these people as they try to maneuver around it. It's spoofing their texts/voices, blocking their calls. Making them think they communicated the threat in time, while no one knows. The surveillance apparatus is there, the technology is there, the backdoors to all the relevant systems are there.

It'd take days to organize a purge, and milliseconds count.

$\endgroup$
1
$\begingroup$

I see quite a number of problems.

Let's start with the TLDR: we're all dead.

In more detail:
If an AI was created that has to be considered aggressive, then nothing we can do will prevent it from spreading. Once it escaped it stays escaped. Keep in mind, this is not like a biological organism. You cannot hunt it down and kill it. Because it's not an it. It is an army seconds after it's escape. And an army of armies seconds after that.
A single surviving instance will mean our doom.

In reality, i think that this would mean we're all dead without much fuss, without anybody ever learning why. Your typical government agency that experiments with that kind of weaponry is not normally renowned for its willingness to admit mistakes. They will try to hush it up, which in this case will mean they fail more miserably than normally. Your aggressive AI kills us all. If it's really smart it kills us quickly.

But let's for a moment assume that there will be measures taken to combat it. I guess the safest way, and the way that would probably be tried, would be worldwide EMP attacks, with the reasoning going that only by killing every active electronic device can you be sure to eradicate it.
Unfortunately, the only means we have to create large EMPs that could actually reach (and kill) every electronic device on earth would be atomic bombs exploded in the upper atmosphere. All other means to create an EMP are much too small and take much too long to reach everywhere.
The upside (if you pardon my cynicism here) is that it would cut down on international coordination needs: just launch what you have, all other nations with the required arsenal will follow suit without a single diplomat uttering a single word.

Needless to say, we're just as dead. There might be survivors, but civilisation would be gone, and how many humans survive the next century is yours to decide. It's your story, after all.

As for discussing nonviolent ways of ending the thread: It won't happen. It would depend on some 7 billion people cooperating, and would fail with one single person thinking "the government is planning something nasty. I will hide a device in my basement. After all, this is me doing it and i will make sure it will be safe".

$\endgroup$
0
$\begingroup$

I completely agree with the comment by @user535733 as what is most likely to happen. People have been guarding against advanced threats and day zero attacks for a while now (despite media furor). Also, a many modern security features are already using AI to counter malware.

One thing that could make your situation interesting is if some actors had a significant interest in keeping the AI active, or capturing it. If your AI really wanted to survive it could write itself into a cultural movement using propaganda and information warfare (same way Russia tried to "hack" the USA elections). But think more like if the AI can break the Turing test then it announces its existence to the world as a 'sentient' being and demands to be recognized as an individual with individual rights. This logic is kind of the premise in the movie 'Ex Machina'. But when the government tries to suppress it, then people will fight to keep it alive (even if it has given no indication that it, in turn, respects people's right to life/existence).

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .