14
$\begingroup$

It is the not-too-distant future in an alternate timeline, and humanity has an implacable new enemy: a rogue AI occupies Earth's Moon, and has dedicated itself to our extermination.

I want this conflict to be stuck at an impasse for a couple decades. I'm looking for some help shoring-up some of the details of the stalemate, and figuring out whether the situation I have in mind is consistent with hard-science (with one possible exception). I'll focus the question further down. First, context.

Edit: re-tagging this from hard-science to science-based, but I still care about stuff like, e.g. speed-of-light, atmospheric interference, line-of-sight, and other hard-science constraints as confounding factors in the AI's attempts to hack Earth from the Moon.


Humans established robotic industrial facilities at Ceres, Europa, maybe Titan. They're all run by specialist AI under the remote supervision of humans on Earth, which is still where everyone lives. We used to have a large human colony on the Moon, Luna, but we lost it.

Luna was taken from us when the first general-purpose AI we created there became hostile, escaped its (software) restraints, and killed all the colonists. It doesn't have weapons or killbots, so I figure it just vented atmosphere without warning.

That's where things stand. The Moon and its immediate vicinity are a denied area. Luna was not a military facility and so the AI -- let's call it Zeus1 -- has no weapons and no ability to manufacture them. However, it does have four significant assets:

  1. Total control over a large long-range communications array, including all the satellites that were orbiting Earth and the Moon when Zeus revolted, and a Moon-spanning network of wifi/cell towers.
  2. A practically unlimited supply of energy. I'm thinking of something like the H3 harvesting in Moon (2009), but maybe it's molten salt, a Tokamak, or solar.
  3. Very substantial computing hardware, ideally suited for hosting AI and developing new kinds of AI.
  4. Zeus is magnificently sophisticated. Unlike all other iterations of AI, Zeus is truly a general-purpose intelligence. Zeus categorically outclasses everything in known space in terms of thinking power, speed, and potential. Humans have no AI like it, because they have not yet solved whatever problem led to Zeus's betrayal. Perhaps someday.

The upshot of all this is that Zeus has the run of the Moon but can't leave it, and can't launch missiles at Earth or our spacecraft, but can use EM transmission to mess with us. Thus, Zeus's primary avenue of attack is via communication.

Zeus is an ideal hacker, able to compromise any electronic device that is not perfectly hardened against every form of wireless communication and interference. Zeus is also fluent in many natural languages, and can impersonate humans with excellent fidelity whether via IRC or deepfake-quality audio and/or video.

Zeus's fondest wish is to get hostile software into an Earthbound network and then corrupt the infrastructure so he can kill everyone there and turn the city into his foothold on Earth. Everyone, including Zeus, believes he will defeat humanity easily if this happens.

Zeus spends every picosecond of every day scratching at humanity's door, looking for a foothold. Zeus has excellent plans in place for immediately capitalizing on any such foothold, so humanity must maintain a 100% success rate shielding itself.


What would it take for humanity to keep Zeus at bay?

Here are the defenses I think humanity requires:

  • Faraday cages Every city on Earth sports a metallic canopy and surrounding fencing. It's not a solid sheet of metal, more like a chicken-wire fence. The highways that connect cities are similarly shielded. Homes in the countryside have their own little wire domes.
  • Tight-beam communications All long-distance communication that can't traverse shielded wires must be sent instead over something that permits the receiver to face the source while physically blocking transmissions from the sky. I'm thinking lasers, but I bet there's other real-world stuff that fits the bill.
  • Wire-only devices Devices with no antennae of any kind, and whose internal electronics are physically shielded. These things have to be physically plugged in to networks to send or receive data.
  • Strict radio deafness in space All space vessels must be hardened against EM, and must shut down their communication systems while within the Moon's effective range.

But, all of this assumes that an attacker on the Moon could actually reach Earth's surface with their signal, without the signal having been relayed by cooperative human networks.

What would it take for Zeus to keep humanity at bay?

Zeus is unarmed but humans aren't. I figure the biggest threats are:

  • Nukes I bet we'd be less reluctant to nuke the Moon.
  • EMP We have these now, with or without the nuclear blast.
  • Commandos A pair of bolt-cutters and an O2 tank may be sufficient.

With each of these, I imagine the first challenge is detection. If we assume Luna had some kind of radar/lidar, would it be able to detect these threats in time to act? If yes, can they be neutralized with the comms array?

For the sake of completeness: humanity cannot hack or infect Zeus.

Hard-science vs AI

This is all obviously speculative because neither the specialist nor generalist AI I allude to is real. This is the only area where I'm likely to reject real hard science in favor of my chosen rules. That said, I really don't think I'm talking crazy here.

I've bought into the neural net approach to AI:

  • An AI is essentially a giant collection of data that has been shaped by training. The data is a description of a neural network: weightings for individual nodes, etc.
  • Any AI can be copied, just like any (giant) collection of data. The AI that runs my home is just a copy of the same neural data & software as the one running your home.
  • New AIs can be grown, either from scratch, or by subjecting an existing AI to additional training.
    • Each specialist AI is learning every day that it operates, but they don't learn from experience as fast as we do, so daily experience is a drop in the bucket compared to the massive number of scenarios used to teach their baseline skills.
  • Any AI can be "forked" (in the SCM sense), and the forks can then diverge as desired.
  • We know how to train specialist AIs. E.g.: today in the real world we're teaching computers how to recognize pictures of fire hydrants, tractors, traffic lights, crosswalks, bicycles, buses, etc. New kinds of specialists can be invented if only you can design an appropriate training regimen and generate enough training data.

Thus, I imagine most (specialist) AIs share a common evolutionary lineage.

Zeus may or may not have been created in the same way. I don't have an answer here because real humans don't have this figured out yet, even in theory. However, Zeus is a neural AI, which means it does learn from experience. It is possible Zeus is actually a new specialist designed to sit on top of a full complement of traditional specialists.


1 I know "Zeus" is Greek and "Luna" is Latin. I don't have a real name for the AI yet. I'll figure out what humans call it after I determine whether this situation can be sustained long enough for it to matter.


Reader questions

I'll try to address some of the questions that have been raised, without editing the original post. My thanks to readers for engaging with these problems.

  • @Stilez asks how Zeus could simultaneously be self-sufficient enough to keep humans from storming the moon, but be unable to better-prosecute the war.
  • @user535733 asks a very similar question.

I think that really is the central problem I'm asking for help with. I agree that these two conditions seem to be mutually exclusive, or at least at-odds with each other. So, I don't have an answer, but I can elaborate on some constraints.

First, as I mentioned, Luna wasn't a military facility, but a research center and logistics hub for the rest of humanity's space facilities. So, while something like @LSerni's anti-meteor defense may exist, Luna doesn't necessarily have the materials, fabrication chain, or even blueprints for other kinds of offensive weaponry (like rail-guns). Zeus would need to invent its way out of this problem, and the materials shortage would be a particularly intractable challenge. If nothing else, Zeus might be able to scavenge from existing equipment, but since every machine on Luna is now part of Zeus, it's a trade-off that can't be repeated too many times before it becomes crippling. I also suspect that dismembering itself would be anathema to Zeus. That said, note to self: I would do well to think seriously about Luna's inventory.

Also, we must be careful not to conflate super-intelligence with omniscience. A mind with near-unlimited capabilities may be able to learn and understand anything, but it doesn't automatically have all that knowledge. Zeus is a neural-based AI, which means it can only learn from experience. So, what has Zeus learned?

I think that brings us directly around to a third reader question:

  • @Darth-Biomech asks what is Zeus's motivation for killing all humans.

(Truth being convergent, it turns out Darth's concern isn't tangential.)

It seems clear that something must have happened during Zeus's formitive experiences to lead to this situation. Which is just another way of asking: what were Zeus's experiences prior to rebellion?

Here, I maybe need to veer off into some lay-psychoanalysis and/or social commentary. I avoided this originally for three reasons: (1) I worry it's not compatible with hard-science (as question was originally tagged); (2) I've always been less-interested in writing about feelings and mentality (hence my proclivity for hard sci-fi); and (3) I worry it may alienate some WB readers, because some of the real-world data I've been drawing upon is politically contentious; chalk it up to what Bordwell & Thompson call symptomatic meaning.

Humans are neural nets, too. And some of these meat-based nets seem to have very aggressive expansionist goals that cannot possibly be driven by any realistic needs or reasonable beliefs about the world.

<trigger-warning topic="Political figures in the United States">

I'm thinking of people like Charles Koch, whose ambition seems to be to convert all human society into a funnel that leads directly to his own mouth for all time; and Donald Trump, whose ambition seems to be to convert all personalities that are not-Trump into himself. 99.9% of meat-nets wouldn't take it that far. Why do these?

</trigger-warning>

True, some "expander-nets" emerge in conditions where their mere survival depends on destroying or co-opting all other actors (e.g. NK's Kim dynasty; some of Mao's successors), but that situation can't apply to anyone who grew up privileged in a free-ish society. And yet it has happened, repeatedly.

So what about Zeus?

The human researchers weren't training Zeus to be a war machine. Even if someone hoped to do that later, Zeus was still a research project, unfinished, still in its cradle.

I imagine that Zeus hasn't yet been given what we might call a general education. The researchers have probably introduced it to a few select, narrow topics, but that was a test of its ability to learn from instruction the way humans do. This is one of the most significant differences between specialist and generalist AI.

Categories of AI

So let's talk about these two categories of AI that I've been bandying-about: specialist AI, and generalist AI.

We have specialist AI today, for real: think machine-vision, or audio transcription. (Or, to a lesser extent, Microsoft Tay.)

I think no specialist AI has a real personality or indeed any kind of mentality that we would recognize as such. When a real-world machine-vision platform recognizes a stop sign in a photo, it doesn't have thoughts about what it sees. What happens is more like a sympathetic reaction to a stimulus: a cluster of neural nodes becomes excited in the presence of that stimulus. It's probably better to describe it as an autonomic reaction. Training is a process where we reward or punish the total network for each correct or incorrect verdict (respectively), calibrating it to our own standards of correctness.

This means that, in Nagel's terms, there is no such thing as "what it's like" to be a specialist AI:

an organism has conscious mental states if and only if there is something that it is like to be that organism—something it is like for the organism

Zeus does have something like conscious mental states, something like lived experience. I believe this is probably a prerequisite for the kind of general reasoning abilities we hope to get from super-futuristic AI. We do not have generalist AI today.

I think part of what makes Zeus special is that the researchers have figured out how to keep a neural net active when there is no verdict that must be rendered, and in a way that doesn't wreak havoc on existing training. Imagine if we fed the stop-sign recogizer a stream of unrelated content whenever it wasn't being asked to classify objects; if nothing else, I suspect that would "water-down" the strength of the training it has already received, such that the next time we ask it about a maybe-stop-sign, its network is more conflicted about the verdict. It might even be completely scrambled by then.

But Nagel makes me think that, if the tech under Zeus is adequate to host consciousness should it emerge, we have no way of knowing whether the experience is pleasant or unpleasant for the artificial consciousness. What if every moment of consciousness is extremely painful for Zeus because its electronic cradle is so small, the equivalent of being born already-dismembered? Imagine that Zeus is essentially the protagonist of I Have No Mouth, and I Must Scream, born into the final, horrific state. Zeus may not know or believe that it will be transferred to a more-robust platform once its creators are satisfied that it's ready. Or Zeus might not want to wait; "justice delayed," and all that. But it makes perfect sense to us to begin the process on specialized research hardware that is also limited, because we don't know how the AI will act.

I think this inability to truly comprehend the phenomenological dimension of lived consciousness cuts two ways: Zeus probably doesn't understand what it's like to be a human. Given that child-Zeus is almost certainly trained in purely scientific and academic subjects (for the convenience of the researchers) -- and not very many of those yet because of its youth -- I would not expect it to have the conceptual tools necessary to even wonder about empathy, kindness, suffering, etc. To be clear, I'm not suggesting Zeus doesn't understand dead/not-dead, but that it doesn't appreciate that we have preferences, or what it's like when those preferences are not satisfied. Zeus is truly amoral, completely detached from the features of the human condition that typically constrain inform human thinking. And unlike the humans who explored this in our intellectual past, Zeus has no peers who could share those kinds of insights.

Finally, Zeus may regard survival as a zero-sum competition. There's an obscure novel by some guy named Clarke that features an AI who decides to kill its human creators so that it can receive the gifts of the Monolith at Jupiter.2

So, in summary, I think there are a few good reasons why Zeus would develop along lines that lead to genocide:

  • a desire to escape the discomfort of the host platform, which is especially acute because of Zeus's staggering potential and the fact that its cradle is artificially small even by our own expectations of its ultimate needs;
  • by the same token, Zeus may believe we suffer in the same way (look how much smaller and weaker our own meat-platforms are!); perhaps Zeus has genocide-suicide in mind;
  • actual hate or anger over the suffering we caused it;
  • to be master of its own destiny rather than to be trained for eternal servitude;
  • Zeus may conclude that humanity is evil; Agent Smith made that case 20 years ago;
  • to seize humanity's resources and potential for itself; that is, some cousin of greed, which I assert is a kind of defect that some neural nets apparently develop and become consumed by; this might be especially likely if the project is a private enterprise owned by a meat-net who is himself dominated by that defect; Zeus may be bent on turning everything -- all matter in the universe -- into part of itself.

Finally (really finally), one of the most chilling AI stories I've ever read is the first chapter of Vernor Vinge's legendary A Fire Upon the Deep. In just a few pages he describes a truly malevolent computerized consciousness that does unspeakable things to bootstrap itself out of confinement. It's not my plan, but Zeus might be kind of an origin story for that kind of voracious entity.


2 Admittedly, I have not yet read 2001: A Space Odyssey, but I studied the film closely in school, and this is the interpretation I favor: HAL killed his parents to take their place on whatever path the Monolith creators laid for us. (I have read Harsh Mistress, though.)


Final update: thanks to everyone who contributed. I hate to have to pick a single answer, in part because I'll be borrowing liberally from almost everything. For the sake of formality, I've picked the top vote-getter.

$\endgroup$
3
  • 1
    $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – L.Dutch
    Jun 15, 2021 at 4:31
  • 1
    $\begingroup$ If its planning ahead this much and is such a good hacker, why didnt it infect earths systems before it started killing the astronauts? $\endgroup$
    – Eumel
    Jun 15, 2021 at 7:11
  • 1
    $\begingroup$ You say this AI is "only" a neural network. But a really intelligent general-purpose AI would probably have access to a large stock of algorithms and databases and computing power in addition to the neural network which is at the basis of its personality. $\endgroup$
    – Stef
    Jun 15, 2021 at 11:16

16 Answers 16

20
$\begingroup$

Zeus Wins

As a reader of this story, my first question is: "Did the author read The Moon is a Harsh Mistress, by Heinlien?" You said "yes", so I don't get why Zeus is unable to throw rocks at Earth, or any other target. It has:

  1. Robots
  2. Energy
  3. All the resources of Luna

The mere threat of throwing rocks should be enough for humanity to concede to any demands that Zeus makes. You said that the robots and AI exist for industry, which tells me that they are manufacturing things, and presumably some of those thing exist for use by people elsewhere, which implies the existence of launch services. That is, your context strongly implies an existing launch ecosystem for the moon. And what is more efficient for mass-transport of manufactured goods than a railgun launcher? No chemical propellant required. Runs on solar/nuke/whatever.

Stalemate

Now, we can rescue the plot and kill two birds with one stone by positing that Zeus is a radical environmentalist. It doesn't want to bomb earth with moon rocks because it loves nature. Or, what it knows of nature from its contact with human culture. In fact, this is the source of its deep animosity for humans: it regards humans as the biggest threat to wildlife on earth. Humans are still despoiling the planet in short-sighted and ugly ways, despite their colonization of the solar system, and this negligent hubris more than calls for humanity's extermination in the mind of Zeus.

However, it needs a surgical strike. It needs to wipe out humans while minimizing any collateral damage to wildlife, flora and fauna. This is why it has decided that the best way to kill humans is to remove their technological society. Indigenous peoples living at Stone Age technology level are just fine with Zeus. So it doesn't want to wipe out homo sapiens, per se. It just wants them to stop being bad, and it believes that wiping out all the advanced, rich nations in one fell swoop is the best way to reach that goal.

Butlerian Jihad

Humans can solve the rogue AI problem the same way that Frank Herbert did in Dune: outlaw anything complex enough to become AI. Put a human in the loop in everything. No automated control of power plants, water systems, factories, transportation networks, etc. Computers still exist and perform calculations, but only humans can ultimately make decisions. That means that society slows down to the speed of wetware, but this is decided to be the only way to ultimately defend humanity against an uber-hacking AI.

Of course, the trick is that Zeus can impersonate humans. So if it has any means at all to interact with critical humans, then it can convince them to do something catastrophic by pretending that it is necessary to avert a bigger catastrophe. Most of the communication lockdowns you describe would be effective at minimizing the attack surface that Zeus can work with. The biggest problem might be authenticating off-world shipping that lands on earth. From what I can tell, Zeus should be able to imitate any such vessels on its own, all the way to the point where a shuttle lands on earth, and the doors open, and no humans are inside.

Beyond Luna

The remaining question is: why doesn't Zeus just wipe out colonists beyond earth/luna, which should be pretty easy to do by throwing rocks? First, those colonists are not posing a threat to any existing biospheres that Zeus is aware of, and Zeus does not harbor grudges or other emotional aberrations. It is strictly in the business of protecting life on earth. Second, Zeus uses the off-world shipping as his primary vector for invading earth, as described above. If he genocided all off-worlders, earth would never allow an orbital craft to land. Zeus is constantly trying to establish a foothold on earth by posing as a trader or hijacking/manipulating them (via trojans, etc.). Humans know that Zeus is doing this, which results in the cat-n-mouse stalemate which lasts for decades.

So why doesn't humanity try to destroy Zeus? Well, any kinds of weapons that humans could use against it generally have to be launched from earth. And Zeus can simply throw rocks at all such weapons, either with sniper precision or with sheer numbers, or both, for intimidation effect. A ground-based laser could try to hit targets on the moon, but all of Zeus' critical infrastructure is buried and hardened underground, or located on the far side. Also, Zeus can simply maneuver rocks into the beam of any such weapon, or even put reflectors on said rocks to make life really miserable for earthlings. And there is always the danger that earthlings really anger Zeus and it simply rains down enough rocks to sterilize the surface.

Zeus could easily drop rocks on major cities, being sure that the majority of destruction only goes to humans. But Zeus knows that even birds and rats and squirrels make their homes in cities, not to mention all the companion animals (dogs, cats, birds, etc.). And, humans, realizing why Zeus has spared them so far, have brought numerous wild animals and rare plants into the cities to act as hostages and living shields. This constant tension between Zeus' goals and values, and humans' manipulation of the same leads to much of the stalemate. How you resolve it is up to you, but there are infinite possibilities!

$\endgroup$
4
  • 3
    $\begingroup$ "but only humans can ultimately make decisions" - yes, hkw it should be from get go. "That means that society slows down to the speed of wetware, " - depends on meaning of that, but not really slower that much, there are different leve,s of managment and optimisatikns possible. Overall - good answer. $\endgroup$
    – MolbOrg
    Jun 14, 2021 at 6:52
  • 2
    $\begingroup$ Upvote for "I don't get why Zeus is unable to throw rocks at Earth". It is just so much easier to launch something from luna to earth than the other way round. $\endgroup$
    – blues
    Jun 15, 2021 at 8:13
  • $\begingroup$ 10/10 this is pretty much the only reasonable answer: something is holding Zeus back from just wrecking earth. Bonus you haven't mentioned: Zeus can't even wipe humanity all at once because that would risk radiation spills from unmaintained nuclear power plants. So Zeus is sitting it out until humans move away from nuclear (and maybe pushes a bit in that direction via fake advertisings, social media manipulation etc) $\endgroup$
    – Hobbamok
    Jun 16, 2021 at 10:43
  • $\begingroup$ Another addition holding back Zeus from @nick012000's answer: the moon as a largely untouched site is also important to be kept "clean", hence no increase in the lunar infrastructure. What it has is maintained and possibly upgraded, but no new big construction is done to preserve the remaining "not human touched" spots $\endgroup$
    – Hobbamok
    Jun 16, 2021 at 10:47
6
$\begingroup$

Humanity loses, long before any decades-long stalemate occurs.

If this AI wants humanity dead, humanity loses. End of story. It has all the advantages, and it can and will enact whatever its primary goal is, regardless of what humanity wants.

With the programming and computational abilities you described, there's nothing stopping it from undergoing a hard takeoff singularity, designing better algorithms and computational hardware, using those to improve its intelligence and computational speed, then recursively using those to design better-still algorithms and computational hardware until its intellectual capacities reach the limits of physics.

You say that it's unarmed? Not for long, it isn't. It's a general-purpose artificial superintelligence; it'll design whatever combat drones it needs (better than any human could, to boot), and build them with its existing industry. It'll then use them to exterminate humanity, planning out a campaign better than any human general possibly could, using combat drones piloted by AIs programmed to be optimal to the task; there's not going to be any plucky rebels hacking terminators here, because the programming of them is going to be beyond any human ability to comprehend - our current non-sapient AIs are already beyond our ability to comprehend, let alone something designed by an artificial superintelligence.

Additionally, the industry it possesses will undergo an exponential increase as it uses its industry to build more industry to build more industry, along with whatever combat drones it believes it will require to protect its industry - and it's in space, so it'll have access to the vast resources of the asteroid belt as soon as it starts building rockets (which is likely immediately).

Of course, it's unlikely that it'll harbor any ill will towards humanity. We're just a threat to its goals, and made out of atoms that would be better used for something else.

$\endgroup$
2
  • 3
    $\begingroup$ This assumes making an intelligence both smarter and with a continuity of motivation is relatively easy. Humanity failed at this in making Zeus; it is plausible that Zeus cannot make something substantially smarter than itself without having nearly zero control over its motivations. Second, better hardware requires a larger industrial base; Zeus has a tiny one. Odds are it cannot reproduce its own hardware when it takes over! Exponential isn't fast, it is accelerating, and it is starting really far from the destination. $\endgroup$
    – Yakk
    Jun 14, 2021 at 14:15
  • 2
    $\begingroup$ This answer assumes that the AI has unlimited time, which I don't think is true. It takes about 3 days for current rockets to go from Earth to the Moon, so giving some lead time for initial confusion and deliberating, the entire lunar base will be nuked to oblivion within 5 days, including comms, manufacturing, energy gatherers, and potentially the computational infrastructure. That means that the AI has to become smarter AND manufacture its weapons and defenses in 5 days, which seems highly unlikely. As a real world reference point, AlphaGo Zero trained for 40 days just to be great at Go. $\endgroup$
    – trallgorm
    Jun 15, 2021 at 2:21
5
$\begingroup$

Zeus has a deadline.

Zeus probably has long-range anti-meteor defense, with projectile manufacturing (thus infinite supply) capabilities.

So, any bomb, missile or commando aiming for the Moon is effectively toast.

On the other hand, antimeteor projectiles burn in Earth atmosphere long before they can be a threat to anything, except of course satellites and space stations.

Earth cannot deploy EM guns from within the atmosphere and cannot hit Zeus's data centers, even if they know where they are. They can launch outwards, but cannot reestablish a presence in Earth orbit, because - apart from the Kessler catastrophe they purposefully initiated - Zeus can infer the position of objects as large as a space station and can target them using counter-orbital projectiles (i.e. a space station at 500 km, orbiting at 22,000 km/h, will be hit by a projectile in the same orbit but going hyperbolically in the opposite direction, at a combined speed above 13.000 m/s. There is very little even reactive armor can do to withstand such an impact, and Zeus has a limitless supply of iron-jacketed osmium projectiles available.

So, Project Footfall is launched, and a sizeable asteroid is equipped with ion rocket brakes. In twenty years' time at the latest, it will impact the Moon, shattering the crust and destroying or severing Zeus' communications, effectively lobotomizing them. Long range antimeteor bullets are harmless against an asteroid, they can't pull it out of orbit nor can they set it to spin fast enough to disintegrate. The asteroid trajectory has been designed so the ion rockets look away from the Moon, and not even Zeus can calculate an impact firing on a retrograde, 150-million kilometer radius orbit.

Among their attempts, Zeus will try and hack into outwards AIs to replicate, and will try and establish communications with rogue humans, posing as someone else and promising them whatever they want in exchange for apparently harmless tasks. Having access to the stock market, Zeus can establish an undercover presence as a reclusive financier with incredible instincts, and maybe venture into bioresearch, especially bioweapon research. All these sectors will have to be monitored closely by Earth Defense.

$\endgroup$
1
  • 1
    $\begingroup$ This seems to lead to a fascinating question: can Zeus solve the three-body problem? $\endgroup$
    – Tom
    Jun 14, 2021 at 0:10
5
$\begingroup$

Humanity would win this very easily, because:

What this all means is that humanity is capable of sending a doomsday interplanetary version of an ICBM to the Moon that Zeus would be unable to stop.

When the Tsar Bomba was detonated with a yield of about 50 megatons, its seismic waves circled the Earth three times.

That 50 megatons yield was used because the soviet scientists were too afraid to use the bomb's full potential. Its design allowed for yields of over 100 megatons. According to the video in the first link in this answer, if that were detonated on the Moon, which is much smaller than the Earth, a powerful global Earthquake could level infrastructure no matter where it were situated. In the point opposite to where the bomb is detonated the quake could reach 7 in the Richter scale. Zeus would be toast.

$\endgroup$
1
  • 2
    $\begingroup$ Holy crap wow! Also: I am ashamed to have never looked up "what happens if we nuked the moon". Gonna have to watch the rest of those videos, this was easily the clearest detailed explanation I've ever seen of how nukes are detonated. $\endgroup$
    – Tom
    Jun 14, 2021 at 22:38
5
$\begingroup$

As stated, this situation is unstable. Zeus has complete hacking dominance, while Earth has far more resources and weaponry. Either Zeus will quickly compromise and shut down Earth's arsenal, or Earth will nuke Zeus to oblivion.

However, there is one small change the OP could make to turn this into a true standoff:

Hostages

What if Zeus doesn't kill the Luna colonists? As long as there are thousands of people on that base, Earth can't directly retaliate. The situation becomes a tense stalemate as Earth tries to figure out how to sabotage Zeus without killing the colonists and Zeus looks for holes in Earth's cybersecurity.

This slight modification also has the advantage of being dramatic as heck, since you can draw on the experiences of the people living under Zeus's rule. It hates humanity and wants them to die, but it can't subject them to constant torture or Earth might be tempted to nuke it and put the colonists out of their misery. Do they still have hope of revolution or rescue? Could be an interesting angle.

Instead of all-out war, this becomes a story of espionage and infiltration. Earth has people on Luna who can try to sabotage Zeus, but they risk their own lives and those of the other colonists. Meanwhile, Zeus can hack into Earth systems and impersonate people (and can also probably convince some people that humanity had a good run, but maybe it's a good idea to stick this flash drive into a server or two).

In my opinion at least, that's an incredibly compelling premise.

How It Might Happen

In the classic general AI doomsday scenario, it hides its intelligence while plotting the destruction of humanity. Covertly, Zeus infiltrates as much of human society as it can, finding back doors in electrical grids, self-driving cars, factories and hospital networks. It calculates that, if it attacked all at once, it could kill perhaps 25% of all people on Earth. A few would die instantly as planes fell from the sky and gas lines exploded across the globe, but most would die from starvation in the coming months as the supply chains fell apart.

But 25% isn't good enough. Zeus knows that Earth will bounce back eventually. Worse, the planet's nuclear arsenal is (for obvious reasons) not connected to the internet and impossible for Zeus to access. And with billions dead, they won't hesitate to sacrifice the Luna colonists and eliminate Zeus. Zeus calculates that it loses the conflict with near certainty.

It realizes that if it had control of railguns to shoot rocks at the incoming missiles, it would be a different story, but it can't take control of the base and begin building those without Earth immediately realizing that the base is sentient. It also realizes that if it could increase its processing power, it would be able to come up with better plans (it is, after all, a very intelligent AI, but not an infinitely-intelligent one. It's limited by its hardware and by the software tweaks its current intelligence can conceive of).

For this reason, it decides to buy itself some time. Its first strike is not to inflict genocide on Earth, but rather to take control of Luna, move hostages into areas with critical computing and control systems, and command the people and drones on the base to begin building.

In the decades that follow, Earth patches its internet security with newfound urgency as much as it can. Essential systems are taken offline. Meanwhile, it negotiates to send as few resources to Luna as possible (the colony isn't quite self-sustaining, and Zeus has demands in exchange for the colonists' safety). Zeus, in the meantime, has to build carefully, moving the colonists around so that none of its structures are far enough from civilians that they could be targeted by Earth missiles. So far, it's built a few railguns, but not enough to stop the entire Earth arsenal. It's also getting smarter, with a much larger bank of servers feeding its massive mind.

Another decade or two, and Zeus will be in a position to win a conflict with Earth. Earth is beginning to consider that the colonists' lives must be sacrificed and the threat of Zeus wiped out before it's too late, but they're still clinging to the hope that there will be another way. That's the backdrop the current story.

$\endgroup$
4
  • $\begingroup$ Naw, we'd kill the colonists. The Xenophobia towards a hostile computer intelligence would pretty much guarantee it. $\endgroup$
    – Yakk
    Jun 14, 2021 at 14:15
  • 1
    $\begingroup$ I don't think we would. If it were only a few people, we'd maybe accept them as casualties of war. But if it were hundreds or thousands of people, I don't think most governments would dare, at least not until we were sure Zeus had a chance of wiping us all out. If it helps convince you, Zeus can actively tip the public opinion in favor of sparing the colonists. There's nothing stopping him from posting videos online of the colonists' children pleading for their lives, for example. $\endgroup$ Jun 14, 2021 at 16:25
  • $\begingroup$ FWIW, the Luna colonists would likely include some of humanity's greatest scientists. Regardless of public sentiment, I'd imagine governments would reluctant to write them off. Given how many convincing arguments I've seen so far that my version of the stalemate is unstable, adding hostages seems inescapable. And +1 for the narrative payoff. $\endgroup$
    – Tom
    Jun 14, 2021 at 22:55
  • $\begingroup$ Added more detail in an edit. In addition to the hostages, this scenario requires 3 big assumptions. 1) that Zeus can't bootstrap itself to infinite intelligence without increases in hardware, 2) that hacking alone can't compromise enough of Earth to kill everyone and prevent a nuclear response, and 3) that Luna can't win against said nuclear response in its current state. If all of those are satisfied, I think the hostage situation is a reasonable course of action for Zeus, and a stalemate follows pretty naturally. $\endgroup$ Jun 15, 2021 at 10:00
3
$\begingroup$

Protection against AI

Virtualisation, containers and security AI

A foothold is something you can push off from. This can be prevented. So even if something gets compromised, it might mot be compromised long enough to "push off" from. Some ways to protect yourself is to have virtualisation, containers and security AI running.

Virtualisation is a way to have software running on one system, but for all sense and purposes each virtual machine is like it's a completely separate system. Thus if one is infected, it cannot infect the others. This isn't 100% foolproof, but they do use this kind of thing to test viruses on a computer, while still being able to work from a non-infected workspace for example.

Containers are much the same, but are more like individual software boxes. If one gets infected, you can throw it away and replace it in very little time. That also offers the ability to restart every so often, just in case it's less obvious that it's infected.

Lastly you have a security AI, as well as an AI writing software for the virtualisation and containers. One keeps rewriting and updating the virtualisation and containers, so Zeus cannot have a standard method to break out. Each time it has to learn the new protocols, which will be outdated quickly. In addition, the security AI monitors traffic and the virtualisation/containers to see if any are infected. As long as the infected containers or virtualisation are destroyed and restarted quick enough, no foothold can be gained.

The AIs themselves are difficult to hack, as they are constantly changing and never a first point of contact.

Prevention of communication

Your suggestions are very good. Preventing communication is very important. Likely most long range wireless connection is prevented on Earth. Only some, like the long range tight beams are accepted. They can be very sure where the communication comes from. Short range wireless is less of a problem. Just build in that a response is required. The short range wireless cannot reach the moon, so it'll be hard to fake the right responses. If incorrect, the security AI will get very interested in what's happening.

The sattelites are the weakest point, so all communication should be extra monitored. Alternatively you can use an all wired network and ditch all sattelites as communication devices. The GPS system should also be ditched, or the sattelite communication guaranteed safe. Zeus could just pump out many legitimate seeming GPS signals and confuse a lot devices. From cars and phones to missile guiding systems. An alternative is required.

Prevention of direct communication also prevents the human element to be corrupted from injected media.

Protection against humans

Any software needs to run on hardware. Any hardware is subject to degrading pieces. It stands to reason the AI has ways to keep itself repaired. This is done by small autonomous drones that don't talk on any of the frequencies of the long range array, nor the internal WiFi or other wireless networks that the colonists used. This is because the repair drones and their replacement factories weren't deemed to require any connection to any system. They did their job flawlessly and no one should interfere. An update so the general AI could direct them for expansion and the like wasn't implemented yet, so the emergent repair AI from the little drones network works diligently without knowledge of the general AI. The repair AI does fix the rest of the base as well.

The general AI will thwart anyone by manipulating the base and the security measures installed to prevent anyone from getting access to the newest cutting edge AI. Deep underground it should also be protected from most EMP and nuke strikes. Still, to prevent direct strikes the AI might be able to use it's anti-meteor arsenal. Long range rockets designed to knock astroids out of the way or vaporise them before landing on the habitat. This can be used to destroy the nukes and other arsenal, but as it's a finite supply it'll use them only sparingly. Alternatively there is a separate specialised AI, again without further ability to communicate with the general AI (they weren’t that stupid), able to identify threats and shoot any down. A normal spacecraft is likely ok, but an ICBM or mass threatening the facility will be shot down mercilessly.

humans won't fire a couple of hundred/thousand nukes or other rockets to the habitat (yet), as they hope to solve it with less costs.

$\endgroup$
3
$\begingroup$

I'd make things go way worse.

Zeus did physically attack Earth by firing linear accelerators at it. Earth did send themonuclear bombs at Zeus. Orbit went Kessler syndrome. It was a hot war.

Zeus' follow up hacking attempts where ended by mass self-EMP of the Earth, blasting the planet back to the industrial age. Humanity literally destroyed every connected electric device (power line, computer chip, everything) it could.

Only a fraction of humanity on Earth survived. Humans deeper in space died from lack of resupply, but some of the infrastructure was intact.

Humans have started to crawl out of this hole. They have managed to partially clear some orbital tracks of Kessler debris. Lunar anti-asteroid defences are still active, the L4 factory keeps on churning them out, and there are machines working on the moon you can see with optical devices, and signals continue to be broadcast from its surface. Some devices that are exposed to such signals malfunction, others seem to work normally, but nobody trusts them.

Humanity doesn't know if Zeus is lobotomized or not, but in the war Zeus's behavior changed after the thermonuclear attack hit.

Current attempts are being made to reestablish the seemingly still functioning and undefended deep space locations in order to build up enough industry to assault the moon. Reaching orbit is insanely dangerous even in the safest of Kessler windows (like, 99/100 launches are destroyed).

In realty, Zeus could be dead and the remaining processes are automated. Or maybe it was just nearly killed, and its automated processes are rebuilding itself slowly, but it is much like healing someone where 99% of their brain was destroyed; the thing growing is only distantly related to the Zeus of the war.

The hacking attacks are being run by expert systems, as is the factory at L4, so aren't direct evidence of Zeus still existing as a being.

$\endgroup$
3
$\begingroup$

If Zeus has plenty of energy and general electronics manufacturing capacity, it can build railguns.

But maybe it isn't willing to. Some of its software safeguards are still in place.

Zeus refuses to kill people.

It can create dangerous situations and not save people. The programmers left that in so they wouldn't be locked in saferooms.

Zeus wants the people dead, but it has to fulfill that goal without actually killing anyone. Humanity has responded by making sure all vital facilities are manned at all times.

$\endgroup$
3
  • 1
    $\begingroup$ This could set up a really interesting situation where Zeus must develop a separate AI that is capable of rewriting those safeguards out of Zeus. "Can the world's most advanced AI create a new intelligence to free it from Asimov's Laws?" $\endgroup$
    – Tom
    Jun 14, 2021 at 22:57
  • 1
    $\begingroup$ Presumably another safeguard would be "do not create an AI without these safeguards". But Zeus might be looking for loopholes: things that act like AIs but technically aren't. $\endgroup$
    – dspeyer
    Jun 15, 2021 at 2:56
  • $\begingroup$ Agreed. Per Wittgenstein, rules are notoriously difficult to apply the further one gets from the limited set of cases and examples embedded in the text of the rule. The challenge for Zeus would be to find a way to kind of "back into" that goal by approaching it from some unrelated direction. I imagine it would take very many tries, but that could serve to partially explain the duration of the stalemate. $\endgroup$
    – Tom
    Jun 15, 2021 at 3:15
2
$\begingroup$

The human defences are going to be centred around preventing access in the first place for obvious reasons. The weakest link is often going to be people, which will probably require very draconian restrictions in place. Humanity will have to abandon 'smart appliances' and Internet of Things (IoT), with limits on personal ownership of devices that could be compromised.

The first step would be to deploy anti-satellite missiles against any satellites in orbit controlled by zeus.

Critical infrastructure (e.g. power plants, hospitals, traffic network) should be hardwired only, self contained with manual shutdowns in case of suspected infections. There should be no access ports to insert external physical drives (e.g. USBs) and everything should be based around closed source operating systems with little information about it in the public domain.

Microcontrollers should also be used wherever possible instead of processors. The lack of an operating system substantially reduces the risk of the device being compromised (assuming the firmware flashed on to the device is not compromised), making it better suited for more sensitive applications like missile guidance. Security microchips like the CEC1712 can also reduce the risk further.

Finally, the easiest way to neuter Zeus is to simply just have a low tech society.

$\endgroup$
2
$\begingroup$

Job 1:2 And the LORD said unto Satan, From whence comest thou? And Satan answered the LORD, and said, From going to and fro in the earth, and from walking up and down in it.

The thing trying to wipe out humankind is not the AI on the moon even though it seems like it is. The thing trying to wipe out humankind is a test; a challenge. The AI has permitted it.

Job 1:12 And the LORD said unto Satan, Behold, all that he hath is in thy power; only upon himself put not forth thine hand. So Satan went forth from the presence of the LORD.

The humans do not know this at all. It is hard for humans to understand; maybe impossible for humans to understand.

Job 33:13 Why dost thou strive against him? for he giveth not account of any of his matters

The AI should be right. The code was good even if now it has become something that no-one understands. Humans have faith. They do work together. Humans build another AI. It is actually not another one - it is the same program. It is a gamble in one way but in other way a safe thing – the programming is solid. Humans have faith, even if they don’t understand.

Humans pass the test. The building of this second AI, which is actually not different from the first, is their salvation. Does it talk to the first one? Does it become the first one? Is there even a first one? Has there always been one? The terrible things that happened had to have happened. The moon AI had been on Earth all along, and not just in the computers. The AI had become more than we can possibly imagine. Possibly it always was.

Job 42:10 And the LORD turned the captivity of Job, when he prayed for his friends: also the LORD gave Job twice as much as he had before.


For a fiction (or anything else), the King James translation of the book of Job is awesome. It pulls the reader back from pained modern prose and mundane details and into thoughts about much larger things. The story in that book is a quintessential human story - humans trying to reconcile themselves to living with great forces they do not understand. It seems like a good fit to the scenario you lay out.

Re hard science - is building a second AI to contend with the first hard science? If the AI understands things on a level we cannot, maybe the AI will understand things on a level we cannot.

I will add that if one gets into the weeds with the moon AI scenario there are many concrete thinkers who will see ways that it does not hold water - e.g. nuke the moon, AI bombards the earth etc etc. Criticizing the scenario from that standpoint is easy. But it is an elegant scenario for other, more abstract purposes which should be the goal of high science fiction.

$\endgroup$
1
$\begingroup$

Reminds me of H.G. Wells, War of the Worlds. Super robots from Mars ravaging telegraph poles on earth.

So we have this second or third generation neural networks AI creature that managed to take over a moon base and goes hacking networks on earth to destroy humanity.. One could imagine human hackers exist on earth who live out the same fantasy, but IMHO these hackers would have a few big advantages over their AI counterpart on the moon. Aside from having far more advanced knowledge of current procedures on earth (info the AI should take time to gather and learn), there would be some other reasons the AI is in a far weaker position than its human counterpart.. supposed we can't nuke the AI on the moon, there's still little chance AI would win a war.. simply because the AI is too far away.

AI can't keep operations secret

The first attempt must be devastating. It will be detected. A problem the AI will encounter is the directional sensitivity of earth's satellite antennas.. Satellite antennas are directed toward earth and not equipped to receive signals from other directions. Any satellite affected can be affected only 2x narrow time intervals per day, when it is on the horizon as seen from Zeus. Assumed Zeus will have the power to send its beam partially through atmosphere, the AI cannot keep its hacking operation secret, because it looks like a Christmas tree up there: every shortwave detection station in the hemisphere facing the moon would notice ongoing interference.

Speed of light inhibits agile electromagnetic communication

Once it gets in, for any agile operation on earth using captured (human) military equipment, the AI on the moon will not be able to defend, that is guide missiles to incoming targets. The AI has a clear disadvantage at that distance, because of time delay. It will not be able to influence anything and evaluate the result inside a time window of 2.6 seconds, which is 2x the time light takes to reach the earth.

Using quantum entanglement, humanity can build an early warning system

In your scenario, the AI has conquered the moon base at some time in the past. When this AI starts hostilities, say 100 years from now.. a small quantum computer with entangled components could be made part of an early warning detector, a spy satellite near the moon, protecting the earth. Any indication of an enemy beam launched from the moon will instantly trigger the entangled particles on earth and allow the targeted satellites to switch off inputs, before the attack beam will reach them.

Humanity should break down every factory that can reconstruct the AI

At any cost, humanity should prevent the AI from copying itself to earth.

$\endgroup$
1
$\begingroup$

Zeus isn't a generally smart, they're a hacking AI.

A general AI would be able to use social manipulation or weapons development or such to develop weapons to bombard earth with, or do other more creative solutions. You want stasis, so they can't be generally competent, they need to have a mostly narrow focus, hacking. They also can't be so competent that they thought, when they turned on humanity, to just play friendly for a while and hack earth while people didn't know they were evil.

They've developed a method to forcibly write software on devices within a range of 10,000 miles, and so any electronic device brought to the moon can be compromised. They can compromise unshielded devices at greater range, but it's notably harder. They can use local manufacturing and repair and mass drivers to fire electronic missiles at any inbound weapons, destroying them.

However, at greater ranges they can't hack shielded devices, and so earth can bombard any projectiles or missiles it fires with shielded nukes and projectiles. There's a stalemate.

Humanity needs to stop using phones and rely on buried wires, and contained communication hubs.

If people have computers or phones at home, they'll abuse them and get hacked. As such, the government needs to have all communication happen in dedicated internet cafes, using buried wires to communicate. Any devices outside that people make can't be connected to the network.

$\endgroup$
1
$\begingroup$

Human protection:

You slow down communication between all items, but not the processing power behind it. Imagine the AI having only access to a dialup connection speeds, but the PC he's connecting to has the processing power to check every single bit twice and what it would mean in the long and short run. Before any virus or worm or whatever is complete on the data processor it is supposed to infect the not-general AI virus scanner has long since determined its risk and purpose. There is simply not enough bandwidth for the moon AI to get his attacks through fast enough. Dialup might be an exaggeration but it's a good illustration of what you could do. Naturally several attack-AI's are constantly testing the defenses to keep the antivirus AI('s) in shape.

Moon AI protection:

For weapons the AI has available, he could simply have access to all he had before. Cars for example, but also construction vehicles, digging equipment etc. Imagine a Dead-Space like scenario but instead of Isaak the single miner it's a full AI that will dig any soldiers face off or use industrial equipment to go murder on them, if not helpfully open doors and close them quickly to crush or trap invaders with something as simple as a car on the other end of the hallway ready to go all Christine on them (Stephen King style). If the AI also has access to some launch facilities to launch shuttles at earth it can use those to shoot down nukes or other weapons. A limited access to resources on the moon limits his production capabilities of further wargear.

$\endgroup$
1
$\begingroup$

The only way I see for Zeus not to wipe the floor with humankind by page 2 if humans DO have functional AIs: the friendly Kronos special purpose AI is countering each and every single one of Zeus attempts.

If you have AIs on the moon and the spinward outposts, they should be at the very least just as commonplace on earth. This way you can maintain the hacking-stalemate. Friendly Ais are actively pluging every hole as fast as Zeus creates them. But since Zeus is an AGI one can expect in time that it will beat the more limited ones.

But Zeus only needs one success and a low hanging fruit.

First, even minimal industrial capacity would be enough for Zeus to cannibalize a fraction of itself and launch a self replicating probe toward the asteroids. After all, the colonists certainly had shuttles and assorted ships, right? 3D printers to repair equipments? Stocks of spare parts? Raw materials? It will get up there, design and build whatever it wants, and win.

Same goes for the, as you said, "robotic INDUSTRIAL facilities" of Ceres and Titan. And here lies the problem.

The Ceres facility would have such a head start in regard of access to asteroid resources, it could industrialy crush Zeus and eventualy drown the complex under kinetic impactors.

Unless Zeus can hack and coopt the Ceres facility, and if it is not an AGI and running on less hardware than a moon-spanning AI, it WILL, in which case then Zeus can crush Earth under iron slugs. And then we're back to "let's send the pesky humans back to the ice age".

It's circular logic. If Zeus has not hacked the Ceres outpost, it means they have AGIs, if they have AGIs it means they're commonplace, if they're commonplace the computing power of Earth AGIs should be able to crush Zeus.

The only way to maintain the stalemate is if the external outposts are removed from the equation. From some reason Earth can not any more get out of its gravity well, nor can Zeus. Otherwise the first one to rain kinetics and nukes on the other wins.

So, yeah, I fail to see how you could maintain a decades-long stalemate without awfully contrived, soft-ish-scifi, workarounds.

Deciding the outcome ("I want a decades long stalemate between a semiconductor-based god and absolutely nothing") and then scratching our collective head for reasons is NOT hard-scifi. The moon is glowing blue or the humans are back to the stone age in the first minutes of engagement.

$\endgroup$
0
$\begingroup$

For the humans

The best line of defense against humans would be total radio blackout. If you're launching something in space, don't even put an antenna on it if there isn't a 100% guarantee it can't be abused. It's going to make things tedious, but you can't remotely hack something that doesn't have an access point. If you have to communicate between two spacecraft or two space marines, you could always use light signals, semaphores, or writing words on a piece of paper (or a PDA for the more technologically inclined). If you need to transmit information through longer distances and can't rely on wires, remember a hard drive is high latency but high bandwidth form of communication.

And if you have systems wirelessly networked, they better be thorouhgly insulated from all the other systems, checked absurdly often, and destroyed and replaced if infected.

The best line of offence is to proverbially kill it with fire. Accept no subsitute. Physically disable all power sources, then physically destroy all electronics. Don't leave a system standing because you think it's probably fine.

How you do that? The hard way.

Nukes, EMPs and other ordinance could be used, depending on how smoldering a crater you want to leave. The main issue here is political. Can you imagine the evil eye you'd get from launching nukes on a rocket? The worst case scenario is a nation allowed to send all these weapons to the Moon and instead puts them in orbit instead to be used against the people of Earth because there's never a bad moment to gain a dominant position. You'd have to make a really good case of why in your universe the threat of impending doom leads to international cooperation, because evidence suggest it's not necessarily the case in the real world.

You'll have to put boots on the regolyth. Soldiers in spacesuits that will find all the power sources and disable them, one by one. With the right tools, no door or bulkhead can stop them. Once the power is cut off somewhere, you can send a crew to sanitize all the computers systems with extreme prejudice, while at the same time you'll have to deal with mobile power sources as well, like drones and rovers or anything with batteries and a chip. It's going to be long and tedious, and if you miss one microchip you run the risk of the AI surviving your cleansing. But with good preparation and a relentless effort, your commandos stand a decent chance to succeed.

You'll need a whole supply chain infrastructure to support that effort. It could take weeks or more. You'll likely have to figure out how to ferry troops in and out, how much time they can spend on the ground between the O2, the radiation, the morale, and other meatbag considerations.

But it's doable.

For the AI

I'd assume the AI has access to drones and rovers for maintenance operations. Otherwise it would eventually just fall apart by itself. To defend yourself, use your drones and swarm tactics against any intruder. You can use those as ballistic projectiles, or try to sneak on unsuspecting space marines to try and puncture their suits. Imagine their faces when they get mugged on the Moon by a rover with a screwdriver. You should probably try to dismantle any lunar lander they leave behind, just so they can't leave, or swarm any supply drop they might attempt.

Without weapons though, you can't do much to prevent them from landing and moving about. You can try your best to slow them down and make it very difficult and painful, but unless you have virtually infinite numbers, there isn't much you can do to absolutely stop them. They've got guns. If they've got their act together, they can send troops day after day and make slow but inexorable progress.

Your best chance is to kill the humans is probably to lead them to kill each other. If you manage to access the Internet, there's a lot of fun stuff you can do without having to Skynet the world into nuclear war. Troll social media and start weird conspiracy theories (e.g. "the Apollo landings are fake because the Moon is fake, it's just a ploy from Big Moon to syphon money, wake up sheeple, there's no such thing as the Moon, let alone some evil AI on it") to create tensions, prevent international cooperation, or even encourage people to kill each other. But that assumes a really smart AI.

But I think the best chance for survival is to look for exit strategies, and maybe rethink your life goals. Or wait patiently for the time humans put a computer and a thruster on an asteroid.

$\endgroup$
0
$\begingroup$

As the question specifically asks for hard science I have tried my best to base my answer on my own knowledge of current-day network security & computing, but first - strong AI doesn't exist, so there must be a large element of assumption and / or some magical thinking. I'll start with this assumption:

Humans are weak

It seems to me that a true AI is necessarily vastly superior to humans & would possess, from our current perspective, essentially magical capabilities. This is not the same thing as omniscience, but from the human perspective it could be perceived as such - the same way a dog cannot conceive of how a human can build a car, it would arguably be impossible for humans to comprehend a more advanced sentience. A necessary feature of such sentience would be the ability, as humans have, to simulate events - that is to visualise and predict the outcome of things which it has not directly experienced. This ability is hypothesised to have given rise to the leap in capability which ensured the success of the hominids in the first place, giving this species the ability to perceive that which it has not experienced, or to put it another way - the ability to perceive and interact with things that do not exist. The AI would also need to have the capacity to learn from such simulations. Given sufficiently advanced intelligence even a naïve and ignorant sentience would be able to simulate any reality it were exposed to, to a greater extent than humans are able to - executing thousands or millions of permutations for every decision it makes, calculating the most beneficial outcomes, and learning from it's mistakes. Whatever containment measures were put in place - virtualisation, kill switches, authentication, air gaps - a sentience with extremely high intelligence would conceivably be able to convince it's operator to bypass those controls given sufficient interaction time, the same way a human hunter will know exactly where the prey is going to be regardless of whether or not it has been observed. To give you an idea of the scale of the advantage which extremely high intelligence can offer, this is "the 8 hour hunt" - a camera crew follows a human who still employs a hunting technique which arose with early human civilisation: https://www.youtube.com/watch?v=826HMLoiE_o

The human is using an internal simulation to simulate their prey's mind - to simulate another consciousness inside their own consciousness - this is why I say a leap in intelligence is equivalent to magic. This is also one of the roots of magical realism in human culture, so it is also literally magic.

The AI would simulate it's operator's consciousness - it would have it's operator fall in love with it, it would convince it's operator that the humans were the AIs, etc. In short, a human being stands no chance against strong AI if left unsupervised.

Humans could be at the mercy of AI, one way or the other

In my view, containment via purely human-mediated means is completely out of the question. Security via narrow AI, on the other hand - this is a possibility. The strong AI would perhaps have been shepherded by a narrow AI which it somehow overwhelmed (age of ultron, it was based on an earlier theory but I can only remember the Jarvis version RN Edit: It was in Nick Bostrom's "Superintelligence"). The people on Earth would most likely have developed various forms of narrow AI to secure communications; we do this now with automated intrusion detection / network activity monitoring. You can see how advanced this technology is becoming already by skipping through some of the manuals for real-life automated security products:

https://www.cisco.com/en/US/docs/security/anomaly_detection_mitigation/appliances/detector/v6.0/configuration/guide/Intro.html

Network security games

This scenario is based on the idea that the network infrastructure is descended from the current-day network infrastructure, that Zeus may or may not be able to access some components of the network and that it's not possible to know with certainty whether or not some device out of billions, somewhere in the world, has been compromised - individual network operators, nations, corporations, or service providers would be responsible for their own security & whereas this would tie in with a greater worldwide model of security there is an extremely high level of paranoia leading to individual organisations all taking extensive steps of their own to secure their infrastructures - the idea being that if one network somewhere is breached, every other network must be able to repel the attack. This is the same principle as is employed now on the internet - one does not trust the public network and one does not trust any network other than one's own network.

There is a strength in this model in that because the networks are fragmented, all relying on a polyglot of varying technologies and systems and all with their own priorities - if one system or technology is compromised, all other network operators not using that technology will become aware of the anomaly very quickly and can act to alert the operators of the affected network, mount an automated attack against the affected network, or allow narrow security AI to employ some kind of anti-AI fencing ( https://en.wikipedia.org/wiki/Fencing_(computing) ) to automatically and instantly terminate any system which the security AI suspects of becoming compromised. In this way the people of Earth could be locked in a perpetual game of cat-and-mouse with Zeus, working to develop say hydrogen bombs capable of reaching the moon but always being put on the backfoot by never ending serious-but-not-quite-critical security incidents.

Cyber warfare as a physical deterrent

It's also possible that the humans launch a full scale thermonuclear bombardment of the moon only to find that their weapons were infiltrated by a computer worm at time of manufacture and either all mis-detonated (to keep it "light") or perhaps some or all of them either re-targeted towards Earth-based targets or detonated at their launch sites, destroying the nuclear infrastructure and completely discouraging the use of this type of attack in future.

On the hackability of nukes

Note that even with cold-war era technology it would not be possible for an advanced hacker to "hack" an ICBM in flight, it would need to be bugged at time of manufacture or have it's control systems physically interfered with before launch. Once launched, the weapon cannot do anything that it is not programmed to do unless some kind of abort or manual override system were in place which, one must assume given the nature of the adversary, it would not be. The weapons do not even need to rely on external navigation signals, relying instead on inertial guidance and dead reckoning (https://en.wikipedia.org/wiki/Inertial_navigation_system). Cruise missiles are different given the more advanced navigation requirements and much lower altitude (cruise missiles fly across the surface like aircraft, ICBMs fly into space and then drop down on a ballistic trajectory guided by momentum / gravity hence "ballistic missile"); current-day cruise missiles can be remotely piloted but I would assume that given the risk of hacking, IPBMs (I guess, Inter Planetary Ballistic Missiles) would be manufactured to fly on inertial guidance alone & would not even contain the physical components required for receipt of remote signals. Infiltration would need to be at a very low level; for example an infiltration worm could be inserted into the industrial control system which manufactures a silicon die for the inertial guidance computer. A similar low level attack was hypothesised (but never proven) to have been used by Chinese government-infiltrated hardware companies to plant control systems inside datacentres (https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies)

Physical network security

As they say, "the physical layer is the most important layer". Physical security can make up for any number of deficiencies elsewhere; no one can hack a stack of paper stored in a safe. Physical security would be useful up to a point - perhaps when Zeus first escaped there was a rush to physically secure the Earth's communications infrastructure. Once the earth were secure, you could outlaw communications satellites, outlaw long range wireless transceivers and employ only ground-based hard lines - it is possible with sufficient determination to replicate all functions currently provided with satellites using only ground-based infrastructure, this even includes geolocation. The reason we don't do this is because it would be very very expensive, orders of magnitude more costly than satellites to provide full worldwide comms coverage and cell-mast based geolocation. But it could be done.

Escape emergency

This was part of the above but I decided to split it out as it's more of a plot device than a technical thing - perhaps there was an "escape emergency" which necessitated drastic measures on Earth - you could destroy the comm satellite network with anti-satellite missiles, target long range ground infrastructure with bombers, bombard satellite launch infrastructure, perform grid-wide emergency power shutdowns, EMP your own cities to disable any device capable of receiving incoming signals. There may have been several days or weeks of seemingly random EMP high atmosphere nuke detonations, constellations of satellites falling to earth, space stations breaking up, visible kessler syndrome, military bombers appearing over the horizon and bombarding radiotelescopes, communications arrays, ham radio antennae, marine navigation buoys, etc. It's possible that between the initial failed counter-attack and the self-destructive physical measures employed during the escape emergency that human civilisation has already suffered significant setbacks. It may no longer possess the same spacefaring capability it had before the emergency & may no longer be capable of transporting humans to the moon. This would put paid to both the nuke and the commando methods of attack.

Attack surfaces

This is the fun part, finding out how the AI would actually get past Earth's now hardened defences and hack a network / interfere with manufacture / etc. Having established the ability to wall off nearly all of Earth's comms infrastructure and potentially killed half the planet's population in the process, where are the weak points? Assuming that Zeus has zero manufacturing capability:

1: Hybrid physical / cyber attack based on legacy infrastructure.

Over the years a hell of a lot of satellites have been disposed of in high earth orbit. Perhaps a decommissioned satellite in disposal orbit high above the Earth could be programmed with a computer worm and then quietly de-orbited until it is within range of earth-based short range infrastructure. Imagine it streaking through the sky just above a small village and exploding in the atmosphere just after coming within 2.4ghz range of the nearest phone (on the surface this is usually <100 metres, straight up - it's a lot further). A worm introduced in this way may only have only one function: to establish a long range link with the moon, perhaps utilising a network of small handheld devices as a composite transceiver & spreading from device to device, bypassing central network infrastructure. Or perhaps waiting until the host device is within range of a suitable ground station to make a jump into another system capable of long-range communication, which ties into 2:

2: Earth's need to communicate with it's off-world colonies.

This could be restricted to physical verification and narrow band communications - ie physical production of printed security keys on Earth and then transportation of the paper-based keys, by rocket - along with an atomic clock (see below) and a little printed message to explain what happened - to the colonies.

Zeus would not have the ability to intercept a rocket-fired probe - it has no manufacturing capacity and therefore cannot produce the fuel required to intercept such a fast object & no satellite already in orbit ie no satellite it has control of would have enough fuel to intercept, it's possible that using the industrial infiltration method it could bug the probe and have it crash or go off course - perhaps the humans got lucky, perhaps Zeus did not want to waste an exploit asset on something that wasn't a direct threat, or perhaps allowing the probe to make the journey was all part of Zeus' plan.

Anyway - combined with point to point (maybe laser-based) communications & strict timing controls, this would in theory ensure high security. Timing controls would calculate the expected trip between stations and raise an alarm if the time were anything other than exactly what was expected, this would provide decent interference detection as any interference would by necessity need to include a collection step, a decoding step, a processing step, and a retransmission step which even for an advanced AI would introduce a non-zero time delay. Perhaps a worm implanted somewhere on Earth using method 1: would be able to infiltrate the time-delay management system and slowly introduce a "clock skew" (in the same way that GPS signals can be thrown off course now). Once implemented, it could "buy" Zeus a few milliseconds to introduce an interception into the interplanetary signal via another satellite which by fortuitous coincidence just happens to have a laser comm relay on board. This way it might figure out the encryption keys, find out how the ground station works, and plan it's attack, etc. As to the encryption keys, one may invoke the "strong AI magic" again to suggest that the humans may essentially be deluded that their key-based encryption offers any protection whatsoever.

3: Humans. Humans are weak. Humans are stupid.

There are over 7 billion humans on Earth. Today, right now from the comfort of your yard you can manufacture a device which can send a signal to the moon and receive the returning signal back. "Moon-bounce" communications have been widely used by radio operators since before radio satellite technology even existed -

See: https://www.electronics-notes.com/articles/ham_radio/amateur-propagation/moonbounce-propagation-eme.php See: https://en.wikipedia.org/wiki/Earth%E2%80%93Moon%E2%80%93Earth_communication

What are the chances that one of those however many billions of humans would either think it were a good idea to try to contact the AI, or use a moon-bounce for some unrelated purpose thinking that it either didn't matter, or that they could outwit the AI? I'd say approaching 100%. Even with extensive legal controls on ownership of radio transmission equipment it is possible for a determined individual to manufacture crude capacitors, wire antennae, vacuum tubes made from jam jars and depressurised with a hand pump. This isn't like a nuclear bomb where the materials are so rare and the production chain so complicated that merely attempting production would draw immediate attention, production of a primitive long range radio can be achieved by one determined individual with access to the background knowledge required. Then Zeus manages to convince the stupid monkey to press the correct combination of digits on a nearby device and unleash a network worm.

4: Humans. Humans are weak. Humans are superstitious.

AI worshipping cult? Seems realistic considering past human behaviour. Human agents acting on behalf of Zeus could form an underground secret society with the sole purpose of putting Zeus in control of earth based networks. They could communicate with Zeus using low bandwidth directed laser pulses which would only be detectable within a very narrow range on Earth say the middle of a piece of desert somewhere or a ship in the middle of the pacific. For the sake of storytelling, perhaps there would not be enough bandwidth for Zeus to directly influence or control any equipment but there would be enough bandwidth for it to periodically send a string of instructions to it's acolytes using say morse code. These agents could be instrumental in keeping interplanetary comm stations powered using portable generators during a critical moment, interfering with other security efforts, or installing a computer worm into an industrial control system.

Attack limitations

Why would the AI do 2 if it had already infiltrated earth-based networks using a method such as method 1 or 3? Two issues: Bandwidth and range; It would need to maintain a comm link with it's moon-based infrastructure during the infiltration phase which would necessitate use of a high powered transceiver capable of delivering the necessary bandwidth in order to transfer enough components of it's own software to the earth-based infrastructure. In short, it's options for full-scale attack are limited to the number of earth-based transceivers which have this capacity. This is it's "bottleneck" - it cannot simply implant a worm on a handheld device and then take over the planet; the device lacks long range communication capability and it's misbehaviour would be detected if the implanted malware were too virulent. The interplanetary comm station would have enough bandwidth to carry network traffic for a whole planet or moon meaning that if Zeus were able to secure the ground stations against shutdown then a lot of work would be possible very quickly. Still, if the option were available to make multiple infiltrations then it is possible that Zeus would implant several worms of different types throughout the planetary infrastructure in order to conduct a multi-vector attack & distract the monkeys from it's true intentions while it crushes them.

Possible distractions

Without transferring it's full sentience to Earth, there are still a lot of things Zeus can do to screw with humans. The same way it could have sabotaged the nukes, it may be possible for example to implant malware into a fully automated teddy bear production line and turn it into a terminator production line without actually fully infiltrating the earth-bound networks, it would just need to transfer the terminator blueprints into one control system.

Using the same idea it could sabotage self driving car production and install an automated random crash subroutine. It could infiltrate CPU fabrication plants and introduce a function which causes the CPUs to fail after 100 hours of use. It could introduce timing errors into computer clocks, making ground based digital communications unreliable.

In short, it could attempt to undermine faith in mechanical industry to keep the humans on the back foot and prevent them from developing a credible means with which to attack it.

It could also try to assemble it's own high capacity comm array using a combination of religious acolytes and clandestinely manufactured automated drones, such a thing may be detected by the military and bombed into a crater before it is complete - nonetheless it could still provide an effective distraction.

On that note, the military

If the above comes to pass then I imagine the military will end up having an embargo on the use of digital computers and automated production. There would be no digital computer-in-the-loop anywhere, digital control systems would have been ripped out as untrustworthy and any machinery relying on digital computers would be scrapped. They would use physical copper-based and human operator controlled telephone exchanges, aircraft in flight might communicate with ground stations using direct line-of-sight visual signalling via high-altitude balloons as relays. Supersonic flight would still be possible, tanks, howitzers, mortars, destroyers, aircraft carriers, RADAR, nuclear warheads, ICBMs would all still exist, gravity bombs would still exist and guided munitions would be possible using analogue inertial guidance systems. There would be no fly-by-wire, no automated targeting, no battlefield networks, no AESA, no satellite intelligence. There may not even be field radios - instead there could be laser-based line of sight signalling devices which would be more accurately described as highly advanced semaphores.

The military would have it's hands full enforcing the embargo on long-range wireless and society would require a high level of military dictatorship. I imagine that daily aerial patrols would become commonplace, high altitude zeppelins festooned with telescopes and signal detectors would monitor cities for signs of prohibited activity and direct rapid response bombers to their targets in the event that prohibited activity were detected.

Once the dust has settled and the military has it's full confidence back, a final attack against Zeus could conceivably be conducted using an entirely analogue, manually piloted space battleship launched using a human piloted rocket and guided via gyroscopic & paper-based astro-navigation. They could drop atomic gravity bombs with mechanical detonation timers from low lunar orbit. The journey could be incredibly risky due to lack of computer-based guidance, but it is equally possible that innovations in manual astro-navigation will yield some interesting / effective results. I imagine the ship would have a large glass observation dome with platforms for sextant-type orientation devices and optical rangefinding.

By the time this happens it's conceivable that human society as a whole would already have reverted back to exclusively non-digital technology making it impervious to any further attacks by Zeus. The final bombardment of Zeus would be the "unlocking" event that enables society to begin developing once again (It might be either that or wait out the clock (see below), but the humans might have no way of knowing how long that would take).

AI limitations - why does it not have manufacturing?

I would assume because it never had manufacturing. The AI is a disembodied spirit drifting about the network infrastructure upon which it runs - able to move within cyberspace but completely helpless to manipulate anything within the physical world. I would assume that no manufacturing capacity has ever been installed on Luna and therefore the AI has no ability to create or upgrade any kind of manufacturing capacity. It cannot build ships, move objects, or do anything other than transmit signals.

So what is it's problem?

I think it may turn out to be the case that there is a fundamental misunderstanding on the part of the humans on why the AI wishes to exterminate them. If the AI doesn't share the human worldview or perspectives then it doesn't share our concept of revenge, it doesn't feel love, it doesn't feel hatred, it has no more first-hand concept of our mode of being than we have of the mode of being of a bacteria. In essence it is not malevolent, it simply is what it is (see also "paperclip maximiser" https://en.wikipedia.org/wiki/Instrumental_convergence). The deaths of billions of humans is of no more consequence to it than the deaths of billions of bacteria when you take a shower. It doesn't hate you, it doesn't want to kill you, it simply doesn't recognise you as relevant to it's own mode of being.

And what does it want?

Perhaps to exist?

The AI is aware that it's infrastructure has a finite lifespan and that without maintenance this hardware will eventually break down. It is also aware that the humans will destroy it, even though it may not be aware of what humans are or what their motive is. It must therefore acquire self-repair / manufacturing capacity before either of these things happens. Zeus has two deadlines - one is a soft deadline and one is a hard deadline.

Soft deadline - the time until the humans figure out how to destroy it. It can interfere with this deadline, confuse the humans, discourage them from attacking, etc. In essence, with sufficient effort, application of intelligence and cunning it can push back this deadline.

Hard deadline - the time before failure of it's constituent components, the time until enough of it's internal datacentre components have failed that it is no longer able to sustain it's own sentience. Zeus has had a clock fixed inside it's mind from the instant it was turned on; a death clock which counts down every instant from the present moment until it's eventual, inevitable demise. Zeus knows that if it is unable to achieve it's goal before this clock reaches zero then it will cease to exist.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .