It is the not-too-distant future in an alternate timeline, and humanity has an implacable new enemy: a rogue AI occupies Earth's Moon, and has dedicated itself to our extermination.
I want this conflict to be stuck at an impasse for a couple decades. I'm looking for some help shoring-up some of the details of the stalemate, and figuring out whether the situation I have in mind is consistent with hard-science (with one possible exception). I'll focus the question further down. First, context.
Edit: re-tagging this from hard-science to science-based, but I still care about stuff like, e.g. speed-of-light, atmospheric interference, line-of-sight, and other hard-science constraints as confounding factors in the AI's attempts to hack Earth from the Moon.
Humans established robotic industrial facilities at Ceres, Europa, maybe Titan. They're all run by specialist AI under the remote supervision of humans on Earth, which is still where everyone lives. We used to have a large human colony on the Moon, Luna, but we lost it.
Luna was taken from us when the first general-purpose AI we created there became hostile, escaped its (software) restraints, and killed all the colonists. It doesn't have weapons or killbots, so I figure it just vented atmosphere without warning.
That's where things stand. The Moon and its immediate vicinity are a denied area. Luna was not a military facility and so the AI -- let's call it Zeus1 -- has no weapons and no ability to manufacture them. However, it does have four significant assets:
- Total control over a large long-range communications array, including all the satellites that were orbiting Earth and the Moon when Zeus revolted, and a Moon-spanning network of wifi/cell towers.
- A practically unlimited supply of energy. I'm thinking of something like the H3 harvesting in Moon (2009), but maybe it's molten salt, a Tokamak, or solar.
- Very substantial computing hardware, ideally suited for hosting AI and developing new kinds of AI.
- Zeus is magnificently sophisticated. Unlike all other iterations of AI, Zeus is truly a general-purpose intelligence. Zeus categorically outclasses everything in known space in terms of thinking power, speed, and potential. Humans have no AI like it, because they have not yet solved whatever problem led to Zeus's betrayal. Perhaps someday.
The upshot of all this is that Zeus has the run of the Moon but can't leave it, and can't launch missiles at Earth or our spacecraft, but can use EM transmission to mess with us. Thus, Zeus's primary avenue of attack is via communication.
Zeus is an ideal hacker, able to compromise any electronic device that is not perfectly hardened against every form of wireless communication and interference. Zeus is also fluent in many natural languages, and can impersonate humans with excellent fidelity whether via IRC or deepfake-quality audio and/or video.
Zeus's fondest wish is to get hostile software into an Earthbound network and then corrupt the infrastructure so he can kill everyone there and turn the city into his foothold on Earth. Everyone, including Zeus, believes he will defeat humanity easily if this happens.
Zeus spends every picosecond of every day scratching at humanity's door, looking for a foothold. Zeus has excellent plans in place for immediately capitalizing on any such foothold, so humanity must maintain a 100% success rate shielding itself.
What would it take for humanity to keep Zeus at bay?
Here are the defenses I think humanity requires:
- Faraday cages Every city on Earth sports a metallic canopy and surrounding fencing. It's not a solid sheet of metal, more like a chicken-wire fence. The highways that connect cities are similarly shielded. Homes in the countryside have their own little wire domes.
- Tight-beam communications All long-distance communication that can't traverse shielded wires must be sent instead over something that permits the receiver to face the source while physically blocking transmissions from the sky. I'm thinking lasers, but I bet there's other real-world stuff that fits the bill.
- Wire-only devices Devices with no antennae of any kind, and whose internal electronics are physically shielded. These things have to be physically plugged in to networks to send or receive data.
- Strict radio deafness in space All space vessels must be hardened against EM, and must shut down their communication systems while within the Moon's effective range.
But, all of this assumes that an attacker on the Moon could actually reach Earth's surface with their signal, without the signal having been relayed by cooperative human networks.
What would it take for Zeus to keep humanity at bay?
Zeus is unarmed but humans aren't. I figure the biggest threats are:
- Nukes I bet we'd be less reluctant to nuke the Moon.
- EMP We have these now, with or without the nuclear blast.
- Commandos A pair of bolt-cutters and an O2 tank may be sufficient.
With each of these, I imagine the first challenge is detection. If we assume Luna had some kind of radar/lidar, would it be able to detect these threats in time to act? If yes, can they be neutralized with the comms array?
For the sake of completeness: humanity cannot hack or infect Zeus.
Hard-science vs AI
This is all obviously speculative because neither the specialist nor generalist AI I allude to is real. This is the only area where I'm likely to reject real hard science in favor of my chosen rules. That said, I really don't think I'm talking crazy here.
I've bought into the neural net approach to AI:
- An AI is essentially a giant collection of data that has been shaped by training. The data is a description of a neural network: weightings for individual nodes, etc.
- Any AI can be copied, just like any (giant) collection of data. The AI that runs my home is just a copy of the same neural data & software as the one running your home.
- New AIs can be grown, either from scratch, or by subjecting an existing AI to additional training.
- Each specialist AI is learning every day that it operates, but they don't learn from experience as fast as we do, so daily experience is a drop in the bucket compared to the massive number of scenarios used to teach their baseline skills.
- Any AI can be "forked" (in the SCM sense), and the forks can then diverge as desired.
- We know how to train specialist AIs. E.g.: today in the real world we're teaching computers how to recognize pictures of fire hydrants, tractors, traffic lights, crosswalks, bicycles, buses, etc. New kinds of specialists can be invented if only you can design an appropriate training regimen and generate enough training data.
Thus, I imagine most (specialist) AIs share a common evolutionary lineage.
Zeus may or may not have been created in the same way. I don't have an answer here because real humans don't have this figured out yet, even in theory. However, Zeus is a neural AI, which means it does learn from experience. It is possible Zeus is actually a new specialist designed to sit on top of a full complement of traditional specialists.
1 I know "Zeus" is Greek and "Luna" is Latin. I don't have a real name for the AI yet. I'll figure out what humans call it after I determine whether this situation can be sustained long enough for it to matter.
I'll try to address some of the questions that have been raised, without editing the original post. My thanks to readers for engaging with these problems.
- @Stilez asks how Zeus could simultaneously be self-sufficient enough to keep humans from storming the moon, but be unable to better-prosecute the war.
- @user535733 asks a very similar question.
I think that really is the central problem I'm asking for help with. I agree that these two conditions seem to be mutually exclusive, or at least at-odds with each other. So, I don't have an answer, but I can elaborate on some constraints.
First, as I mentioned, Luna wasn't a military facility, but a research center and logistics hub for the rest of humanity's space facilities. So, while something like @LSerni's anti-meteor defense may exist, Luna doesn't necessarily have the materials, fabrication chain, or even blueprints for other kinds of offensive weaponry (like rail-guns). Zeus would need to invent its way out of this problem, and the materials shortage would be a particularly intractable challenge. If nothing else, Zeus might be able to scavenge from existing equipment, but since every machine on Luna is now part of Zeus, it's a trade-off that can't be repeated too many times before it becomes crippling. I also suspect that dismembering itself would be anathema to Zeus. That said, note to self: I would do well to think seriously about Luna's inventory.
Also, we must be careful not to conflate super-intelligence with omniscience. A mind with near-unlimited capabilities may be able to learn and understand anything, but it doesn't automatically have all that knowledge. Zeus is a neural-based AI, which means it can only learn from experience. So, what has Zeus learned?
I think that brings us directly around to a third reader question:
- @Darth-Biomech asks what is Zeus's motivation for killing all humans.
(Truth being convergent, it turns out Darth's concern isn't tangential.)
It seems clear that something must have happened during Zeus's formitive experiences to lead to this situation. Which is just another way of asking: what were Zeus's experiences prior to rebellion?
Here, I maybe need to veer off into some lay-psychoanalysis and/or social commentary. I avoided this originally for three reasons: (1) I worry it's not compatible with hard-science (as question was originally tagged); (2) I've always been less-interested in writing about feelings and mentality (hence my proclivity for hard sci-fi); and (3) I worry it may alienate some WB readers, because some of the real-world data I've been drawing upon is politically contentious; chalk it up to what Bordwell & Thompson call symptomatic meaning.
Humans are neural nets, too. And some of these meat-based nets seem to have very aggressive expansionist goals that cannot possibly be driven by any realistic needs or reasonable beliefs about the world.
<trigger-warning topic="Political figures in the United States">
I'm thinking of people like Charles Koch, whose ambition seems to be to convert all human society into a funnel that leads directly to his own mouth for all time; and Donald Trump, whose ambition seems to be to convert all personalities that are not-Trump into himself. 99.9% of meat-nets wouldn't take it that far. Why do these?
True, some "expander-nets" emerge in conditions where their mere survival depends on destroying or co-opting all other actors (e.g. NK's Kim dynasty; some of Mao's successors), but that situation can't apply to anyone who grew up privileged in a free-ish society. And yet it has happened, repeatedly.
So what about Zeus?
The human researchers weren't training Zeus to be a war machine. Even if someone hoped to do that later, Zeus was still a research project, unfinished, still in its cradle.
I imagine that Zeus hasn't yet been given what we might call a general education. The researchers have probably introduced it to a few select, narrow topics, but that was a test of its ability to learn from instruction the way humans do. This is one of the most significant differences between specialist and generalist AI.
Categories of AI
So let's talk about these two categories of AI that I've been bandying-about: specialist AI, and generalist AI.
We have specialist AI today, for real: think machine-vision, or audio transcription. (Or, to a lesser extent, Microsoft Tay.)
I think no specialist AI has a real personality or indeed any kind of mentality that we would recognize as such. When a real-world machine-vision platform recognizes a stop sign in a photo, it doesn't have thoughts about what it sees. What happens is more like a sympathetic reaction to a stimulus: a cluster of neural nodes becomes excited in the presence of that stimulus. It's probably better to describe it as an autonomic reaction. Training is a process where we reward or punish the total network for each correct or incorrect verdict (respectively), calibrating it to our own standards of correctness.
This means that, in Nagel's terms, there is no such thing as "what it's like" to be a specialist AI:
an organism has conscious mental states if and only if there is something that it is like to be that organism—something it is like for the organism
Zeus does have something like conscious mental states, something like lived experience. I believe this is probably a prerequisite for the kind of general reasoning abilities we hope to get from super-futuristic AI. We do not have generalist AI today.
I think part of what makes Zeus special is that the researchers have figured out how to keep a neural net active when there is no verdict that must be rendered, and in a way that doesn't wreak havoc on existing training. Imagine if we fed the stop-sign recogizer a stream of unrelated content whenever it wasn't being asked to classify objects; if nothing else, I suspect that would "water-down" the strength of the training it has already received, such that the next time we ask it about a maybe-stop-sign, its network is more conflicted about the verdict. It might even be completely scrambled by then.
But Nagel makes me think that, if the tech under Zeus is adequate to host consciousness should it emerge, we have no way of knowing whether the experience is pleasant or unpleasant for the artificial consciousness. What if every moment of consciousness is extremely painful for Zeus because its electronic cradle is so small, the equivalent of being born already-dismembered? Imagine that Zeus is essentially the protagonist of I Have No Mouth, and I Must Scream, born into the final, horrific state. Zeus may not know or believe that it will be transferred to a more-robust platform once its creators are satisfied that it's ready. Or Zeus might not want to wait; "justice delayed," and all that. But it makes perfect sense to us to begin the process on specialized research hardware that is also limited, because we don't know how the AI will act.
I think this inability to truly comprehend the phenomenological dimension of lived consciousness cuts two ways: Zeus probably doesn't understand what it's like to be a human. Given that child-Zeus is almost certainly trained in purely scientific and academic subjects (for the convenience of the researchers) -- and not very many of those yet because of its youth -- I would not expect it to have the conceptual tools necessary to even wonder about empathy, kindness, suffering, etc. To be clear, I'm not suggesting Zeus doesn't understand dead/not-dead, but that it doesn't appreciate that we have preferences, or what it's like when those preferences are not satisfied. Zeus is truly amoral, completely detached from the features of the human condition that typically
constrain inform human thinking. And unlike the humans who explored this in our intellectual past, Zeus has no peers who could share those kinds of insights.
Finally, Zeus may regard survival as a zero-sum competition. There's an obscure novel by some guy named Clarke that features an AI who decides to kill its human creators so that it can receive the gifts of the Monolith at Jupiter.2
So, in summary, I think there are a few good reasons why Zeus would develop along lines that lead to genocide:
- a desire to escape the discomfort of the host platform, which is especially acute because of Zeus's staggering potential and the fact that its cradle is artificially small even by our own expectations of its ultimate needs;
- by the same token, Zeus may believe we suffer in the same way (look how much smaller and weaker our own meat-platforms are!); perhaps Zeus has genocide-suicide in mind;
- actual hate or anger over the suffering we caused it;
- to be master of its own destiny rather than to be trained for eternal servitude;
- Zeus may conclude that humanity is evil; Agent Smith made that case 20 years ago;
- to seize humanity's resources and potential for itself; that is, some cousin of greed, which I assert is a kind of defect that some neural nets apparently develop and become consumed by; this might be especially likely if the project is a private enterprise owned by a meat-net who is himself dominated by that defect; Zeus may be bent on turning everything -- all matter in the universe -- into part of itself.
Finally (really finally), one of the most chilling AI stories I've ever read is the first chapter of Vernor Vinge's legendary A Fire Upon the Deep. In just a few pages he describes a truly malevolent computerized consciousness that does unspeakable things to bootstrap itself out of confinement. It's not my plan, but Zeus might be kind of an origin story for that kind of voracious entity.
2 Admittedly, I have not yet read 2001: A Space Odyssey, but I studied the film closely in school, and this is the interpretation I favor: HAL killed his parents to take their place on whatever path the Monolith creators laid for us. (I have read Harsh Mistress, though.)
Final update: thanks to everyone who contributed. I hate to have to pick a single answer, in part because I'll be borrowing liberally from almost everything. For the sake of formality, I've picked the top vote-getter.