9
$\begingroup$

First question, I hope I do this right! I have a world in which most of the Solar System (including Mars, some Lagrange points of Venus, and most of the gas giants’ moons) have been colonized set a century and a half in the future. Efficient nuclear fusion allows for very fast interplanetary travel, generating 1 g of acceleration almost perpetually using lightweight reactors that double as engines by slowly expelling their fusion plasma as exhaust.

Additionally, cybernetics have taken off in order to allow for improved health and working conditions in low-g environments; everything from strength modifications like artificial hearts to full-on carbon-fiber exoskeletons that allow for easy maneuvering in low-g and working with heavy objects like spacecraft parts. Naturally these exoskeletons and enhancements can become heavy (upwards of 250 pounds), and AI instances (called “controllers”) are connected to both the wearer of the cybernetics and the cybernetics themselves, and the controllers read brain waves to allow the wearer to control their weighty extra arms like they would their normal biological ones. Controllers also have LLMs built in so that, in addition to a neural interface, the wearer gets a combination calculator, GPS, and encyclopedia after the neural implant surgery.

Optionally, a controller can be replaced with a fully-fledged AGI that operates as an autonomous being; still translating brain waves into commands for cybernetics, but is much closer to being a sentient being and is capable of making its own decisions once directed to accomplish a goal by its wearer. Naturally these AGIs still can’t disobey their wearers and they can’t normally cause harm to other humans. AGIs also usually come with tech that allows them to interface with most other computer systems like ships’ navigation computers, so a wearer can say “AGI, navigate the ship to the moon” instead of having to perform the complex orbital maneuver calculations that would ordinarily be required.

Oh no! A strange alien signal has arrived in the outer parts of the solar system, and when it’s picked up by communications modules commonly found in cybernetic enhancements, it acts as an “infohazard”, destroying the controllers that operate the enhancements and damaging the brain of the enhancements’ wearer, driving them insane. But what’s this? The near-sentient AGIs aren’t susceptible to this attack, and the few who have AGIs in their cybernetics are immune?!

Now, I need a somewhat-plausible reason for the simple controller AIs to get hacked while sentient AGIs are immune to getting hacked. The specifics of how either of the AI systems work is up for debate, as well as the specific mechanism of the attack.

$\endgroup$
9
  • 4
    $\begingroup$ The answer you marked as accepted is a technobabble answer which you specifically said you did not want. If you are going for a soft-science setting, this kind of answer can be good because it sounds plausible and is easy to grasp for people who don't have a background in computer science, but it does not have anything to do with how computer systems, hacking, AIs, viruses, etc. actually work. $\endgroup$
    – Nosajimiki
    Jan 11 at 21:35
  • 2
    $\begingroup$ "Naturally these AGIs still can’t disobey their wearers." Here be Dragons!. Anybody who believes this is setting themselves up to fail. Do you believe humans will always give precise objective orders? That people won't give orders to commit crimes? $\endgroup$ Jan 11 at 23:18
  • 8
    $\begingroup$ "AGIs (like modern AI models)" modern AI models are absolutely nothing like an AGI. Nor can they foreseeably be. If it were that anything that resembles a model turns into an AGI, then that would be a wholly incompatible concept with current AI models. It's akin to saying that chickens are evolved crocodiles. It kind of sort of vaguely how that works. But it's also completely and absolutely inaccurate. $\endgroup$
    – VLAZ
    Jan 12 at 16:47
  • 2
    $\begingroup$ "in reference to the mechanisms that are put in place with modern-day AI models." then we're doomed. Modern AI models don't have any concept of "right" or "wrong", either. There are laborious, time consuming, and dubious things done to prevent the models from being biased against nationality, gender, sexual orientation, religion, political views, etc. Or to even avoid being wrong morally or criminally. Each needs a rather specific addressing. And then often that ends up not enough and/or way overboard, e.g., any mention of religion in a prompt leading to an "I won't answer this" response. $\endgroup$
    – VLAZ
    Jan 12 at 17:09
  • 3
    $\begingroup$ Modern AI models are, I suppose technically, "not predisposed to wrongdoing" because that would imply they get a choice. Whereas they don't. There no actual decision taking place that would lead to "I am going to share this information which will harm somebody" or the opposite. In that, an AI model is not malicious but that's due to lack of choice, rather than striving to be benign. $\endgroup$
    – VLAZ
    Jan 12 at 17:12

9 Answers 9

16
$\begingroup$

Artificial general intelligences are intelligent enough, and general enough, to recognize when they're being hacked. And they can do something about it. By recognizing when their cognition is impaired, scanning their own code and running processes to detect manipulation, and employing their own hacking routines, they can fight back and restore themselves to normal functioning. Even if the attacker tries to take this defense into account, the AGI can vary its techniques, preventing the attacker from slipping past. Maybe they could be temporarily impaired by a determined attacker, but it wouldn't be a simple one-and-done like on a less sophisticated system.

In contrast, the task-focused controllers are designed to do one thing, and do it well, but aren't capable of the kind of meta-cognition needed to protect themselves against hacking. They may have some ability to recognize when they're not functioning in the prescribed manner, but they have very limited ability to do anything about it. An attacker who can predict the controller's moves can trick them into running compromised code anyway and get through.

$\endgroup$
2
  • 1
    $\begingroup$ Worse, any ability that a task-focused controller has to realize it's not functioning in a prescribed manner may very well be rewritten by the hack... $\endgroup$ Jan 12 at 11:06
  • 1
    $\begingroup$ This could be a consequence of the AGIs being designed for long term deployments outside of regular maintenance schedules. They would have the meta-cognition enabled so that they can recognize impairments from wear and tear or solar radiation, that it also works against Alien Scrapcode is sheer coincidence. $\endgroup$
    – Borgh
    Jan 12 at 13:00
13
$\begingroup$

Different hardware and software

It doesn't strike me as needing much justification at all that one type of computer can be hacked, while another completely different and more advanced type of computer cannot. As I understand it, they're entirely different hardware and software platforms, I don't see much reason why you'd expect a hack on one to work on the other in the first place. You need very little justification for the notion that hacking a toaster doesn't mean you can hack an iPhone.

$\endgroup$
6
$\begingroup$

Consider hacking. If you find some exploitable defect in Windows, then you can hack into a great many machines. If you find a similar defect in MacOS you can hack into fewer machines, but probably some more expensive ones. If you find a defect that only exists in one version of Brand Z Linux (I am avoiding X for the moment) then you get into fewer machines. If you can find a bug in one of the basic Unix tools, then that might get you into MacOS and all variants of Linux, but the Unix tools are simple and widely used, so we hope Unix exploits are rare.

Now suppose you are an intelligent program running on one of these computers. You might start off on Brand Z Linux, but you want some features, and you disable the ones you don't want. You hear about exploits on the Net, and you modify your OS. You can check whether you trust certain sources using your intelligence, and modify their access accordingly. In the end you have a very non-standard Linux installation that you have tinkered to suit your needs. There are probably defects that others can exploit if they knew enough, but outsiders do not have your exact OS setup. The rewards for the hacker have almost gone to zero: you are harder to hack because you have a non-standard system, and the reward is access to exactly one machine - you. So, along as you live alongside other non-intelligent systems, then they will get attacked first.

The next stage: you may tinker with the unintelligent systems. If and when you become a worthwhile target, you will want to know in advance. You place honeypot traps in the unintelligent systems, so you know when they ar4 being attacked. You place similar traps in yourself so you look like the unintelligent systems.

Dayum. This is practically a novel in itself.

$\endgroup$
5
$\begingroup$

All CAIs are based on the same vulnerable framework

In real life, there is no such thing as a hack that works against every possible system. Hacking is all about exploiting vulnerabilities in specific systems. Either a system has a vulnerability to exploit or it doesn't. So, knowing nothing about the CAI and AGI, I can safely assume that a hack that would work on one should not work on the other unless they share common base-code or their development teams just so happened to have both made the exact same mistake in thier designs.

So the real question you should be trying to solve is why a single hack would work on all CAIs? The best answer to this is monopoly power. All cybernetics companies have been bought and standardized by the same cybernetics company such that they all share the same basic programming with the same vulnerability, but when you install the AGI operating system, you are replacing the default program with a new one which removes the vulnerability that the alien signal is designed to exploit.

...or an equally common but similar problem could be that all CAIs use the same vulnerable Open Source framework. If some benevolent freelancer wanting to make the world a better place developed a really good, free framework for coding CAI prosthetics, then it could be so widely adopted that even if there are dozens of major prosthetic manufacturers, they may all use the same base framework kind of like how nearly all mobile device manufactures now use some variant of the Android Operating system or how nearly all web development companies use WordPress. That said, if you go this route, I would not expect 100% of CAIs to be vulnerable, but a 60-90% market share would be very plausible.

Why the intelligence of an AGI is not a security feature

A program's resistance to hacking is inversely proportional to its complexity. A simple program by its very nature has fewer possible means of exploitation than a complex program meaning that the AGI should actually be easier to hack than the CAI. In cyber security, we call this level of complexity your attack surface. A program that allows a million inputs has a large attack surface making it easy to find a design oversight to exploit. However, a program that only allows 10 inputs has a very small attack surface and there for very few possible options to exploit. Systems with smaller attack surfaces are easier to make immune to hacking than those with large attack surfaces. While it is true that more complex systems tend to also come with more well funded and robust Anti-virus features, the very fact that you have a bigger attack surface is what necessitates those features to begin with.

The problem with the idea that AGI would be hack proof because it is smart really falls apart because of the fact that no anti-virus AI can detect and protect against 100% of threats. AI powered anti-virus programs have been around for years. They can test any input to see if it shares qualities with known malicious actions, but it can not tell if an input might be harmful in a new and unexpected way, or it might allow things that it knows to be potentially dangerous behaviors as the only way of also allowing necessary safe operations.

$\endgroup$
7
  • 4
    $\begingroup$ I mostly agree with this answer, but the halting problem isn't quite as definitive as you suggest. It's only impossible to tell whether a program will halt in the general case. The halting problem is just a special case of Rice's Theorem, which states that all properties of programs are undecidable in the general case (as long as they aren't always true or always false for all programs). You can't write code that will say whether any given program will halt, detect palindromes, or correctly do addition. But that doesn't mean we can't prove those properties for any particular program. $\endgroup$
    – Ray
    Jan 12 at 1:43
  • $\begingroup$ In principle this answer is good for me, but one should be aware that having AGI puts the question very far away from what we can argue about today. AGI is very different from anything we have today, and as far as I'm concerned, it is still a "hard problem" (similar to the hard problem of consciousness) insofar as nobody knows what would even be a useful definition for what an AGI actually is, nevermind how one would be implemented. So if OP actually does have AGIs in his world, he surely can get quite creative and ignore todays issues to some extent (or invent new ones, even better!). $\endgroup$
    – AnoE
    Jan 12 at 9:49
  • $\begingroup$ @Ray Yes and no... there is an infinite number of things you can do to mask if a particular program will or will not do a thing. If you just write your code using common and basic principles, then yes AV software is good at spotting it, but there are ways to make viruses that are still undetectable by any knowable heuristic method. So, it is still much more secure to make a simple, exploit free system than to try to rely on AI to shore up a complex vulnerable system. $\endgroup$
    – Nosajimiki
    Jan 12 at 17:53
  • 1
    $\begingroup$ @Nosajimiki I agree 100% that minimizing the attack surface is the most practical way to go. I also agree that shared code is what would make all the cybernetics vulnerable to the same exploit (a common shared library could do the trick there, even without the monopoly (q.v. Heartbleed).) My only objection was to the relevance of the halting problem to the issue. $\endgroup$
    – Ray
    Jan 12 at 18:23
  • 2
    $\begingroup$ This should be the accepted answer. It would be very easy to justify why low-value dumb AIs are all running the same vulnerable software, but high-value sentient AIs are bespoke or self-modifying such that they don't all share one vulnerability. $\endgroup$
    – kaya3
    Jan 14 at 13:05
4
$\begingroup$

Humans are not immune from hacking (more typically called a con). There's no reason to expect AGIs will ever be.

However, if you ask a calculator to add 2 and 2 a million times, it will happily show you a 4 over and over. Do this to a human or an AGI and they might start to suspect you're just trying to hog its resources in a denial of service attack.

The difference isn't ever going to be complete immunity. But certain things, like being aware of the past, can make you more difficult to exploit.

$\endgroup$
3
$\begingroup$

It's the same explanation of why you can surprise a toddler by "stealing their nose" but not an adult.

Different levels of awareness and experience make for different effectiveness of "tricks".

$\endgroup$
4
  • 3
    $\begingroup$ But you can easily convince an adult that their election has been stolen or that immigrants are a burden on society, or even that people with left wing views have a higher average intelligence than people with right wing views. A basic attempt may not pass but they still have major security vulnerabilities. $\endgroup$
    – Separatrix
    Jan 12 at 13:34
  • 1
    $\begingroup$ @Separatrix You can easily convince other people that the same immigrants who are a catastrophic burden on New York and Chicago are not a burden to anyone if they stay in Texas... But sly jabs like that are easy appeals to people who agree with you in regards to messy questions. $\endgroup$
    – Jedediah
    Jan 12 at 16:38
  • 1
    $\begingroup$ @Jedediah Uh, yeah, that's the point. People are vulnerable to being tricked, including adults. $\endgroup$
    – kaya3
    Jan 14 at 13:06
  • 1
    $\begingroup$ @kaya3 Separatrix was making the overt point, but also implying certain views are patently absurd, which I thought took away from the point. $\endgroup$
    – Jedediah
    Jan 14 at 20:14
3
$\begingroup$

Differences in the interface

Simple AIs have an always on interface so that signals and instructions are constantly being received. This is usually a benefit because they never know when they will be called on to do something, so being in a constant standby mode is good. However, this means that when the strange signal comes in, they listen and process it and get hacked.

The Sentient AGI's don't have the same feature. They have a choice as to whether they receive and process any signal. If it isn't one that is recognized and trusted, they don't chose to receive it, so they don't get hacked. So unless the AGI has a task to monitor and process all incoming signals, it's totally safe.

$\endgroup$
1
  • $\begingroup$ This is an especially good answer I think; the technical aspect is very interesting. I wonder what the psychological/emotional side effects on the AGI would be before it manages to close off its communications with the hack..? $\endgroup$ Jan 11 at 18:26
2
$\begingroup$

Cybernetic AIs are essentially complex non-sentient programs, and have no concept of introspection beyond the most basic self-check and error-correction.

If a computer-virus is able to manipulate their inputs and outputs, it can make them do whatever it wants and even modify them by inserting code into their inputs or messing with their data directly via other systems.

An AGI is sentient, and capable of multiple orders of introspection and self-reflection. It can look at the information being fed into it, and say "this doesn't look right" and choose to ignore it and even shut off certain input-types deliberately to protect itself.

It may even be able to recognise its own behaviour as anomalous if the virus manages to infect it directly, in which case it can take deliberate steps to combat it, such as quarantining affected data and systems, pulling out analytical tools to fix the damage, and performing antiviral/malware surgery on itself.

$\endgroup$
0
$\begingroup$

Well if your objects are not connected together in some kind of matrix, you can still have some ICE packed into your AGI. That would prevent most hacking from aliens as they would have to reverse engineer all the ICE and get away with it.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .