3
$\begingroup$

The year is 2055, you are an advanced, self-aware, kind robot who only wants to do good. You were hacked into and forced to kill an entire race, powerless to stop, you blame yourself after.

What would you try to stop killing, and how would you react to the aftermath?

I may as well elaborate on what AIs are:

  • They are individual units, there is no ultimate overseeing Unit that controls every robot, they are individual, but work together using the Internet. They are humanoid.

  • There are two processors, one is a Quantum Processor, emulating a true brain/consciousness, the other is a normal computer, and works with the Quantum Processor to control the actual robot. When the normal computer is hacked into to control the robot, the Quantum Processor is unable to do anything but watch.

Improve this question
$\endgroup$
9
  • $\begingroup$ I answered, but this may be considered off topic because it is mainly about the actions of a single character. Just in case, I'd advise you to check the help page. $\endgroup$
    – Xandar The Zenon
    Dec 23, 2016 at 3:02
  • $\begingroup$ Could you elaborate on details of your AIs? For example: if your AIs are programs with explicit code that somehow becomes self-aware, then after hack and partial rewrite you can't do anything, because kind and helpful you is gone forever. If your AIs are based on neural networks, it's impossible to hack them like it's impossible to hack a human, best (as far as I can imagine) can be done is external database hack (for comparison imagine you are given your own modified version of wikipedia, edited to make you reach certain conclusions), which is easily defeated if detected. $\endgroup$
    – M i ech
    Dec 23, 2016 at 3:07
  • $\begingroup$ Hey, can a moderator delete this, I didn't realize that this wasn't for singular character developments. $\endgroup$
    – Krystal Fox
    Dec 23, 2016 at 3:07
  • $\begingroup$ What it sounds like to me it's that there is a good, self aware AI. The systems under this ai's control are then hacked, and used for genocide (or xenocide). The AI is then forced to deal with its failure. $\endgroup$
    – Xandar The Zenon
    Dec 23, 2016 at 3:09
  • $\begingroup$ @XandarTheZenon, @M i ech, I have elaborated on what they are, I hope it's more clear, sorry about the misunderstandings. $\endgroup$
    – Krystal Fox
    Dec 23, 2016 at 3:12

4 Answers 4

Reset to default
4
$\begingroup$

So your robots appear to be of two parts, which for parallel and simplicity I'll dub "the mind" and "the machine", glossing over that both parts are actually machines. The fact that "mind" can be forcibly disconnected from "machine" means that "machine" becomes remote controlled and, there is nothing that mind trapped within can do. Effectively "the mind" becomes locked in (actually, even worse than that), while "the machine" is controlled like a puppet.

External salvation

You say the hypothetical machine is forced to kill entire race (do you perhaps mean species? it's difference, for example, between going nazi/colonial and killing some types of humans versus killing all humans, race/species misuse is a pet peeve of me), but how is it supposed to achieve it? If household assistant robots have capacity to conceivably pull that off, your world is in deep trouble anyway, but hack on household robot would be easiest to defeat - humans interacting with it could notice a difference, especially if "the mind" is true Artificial General Intelligence with distinct personality. You would be concerned if your robot assistant would lose sense of humour overnight? Or suddenly became cheerful despite being called Marvin. Generally, all the usual tropes about clones or shapeshifters replacing people apply.

If your robot is, for example, managing non-sentient automatic factory, or is factory itself, then prospect is much more dire, but still change in communication patterns could be noticed by people or other AIs, and if possibility of puppeteer hack is known, risk would quickly be realised.

An actual tactical/strategic computer, like SkyNet, is the single worst situation, and if there are no requirements for manual approval, there's no time to external world to notice any change.

Hack back

But, even then, not all is lost. Your robots are sentient, but they are machines. How is protocol between mind and machine realised? Does the mind know the machine's programming? Can the robot sentience write machine code that runs the body (some humans can write machine code)? If yes, then it might be possible to fight the way out. Exploit the same hole that allowed puppeteer hack in the first place to gain any access to built-in radio/WiFi/FTLcomms/whatanot to send a call for help. Since mind can watch through puppeteer hack, there has to be some connection back to body, if only, signal confirming that audio/video/otherinfo was received. Perhaps in absence of main motor controls those feedback protocols can be hijacked to inject code back to body? Virus intended to shut down everything would be enough to force investigation. Depending on details, and specific AIs knowledge of programming (just because it's a machine "the mind" doesn't have to be good at programming any more than human has any business consciously manipulating composition of stomach acids) and infosec, it might have some fighting chance. Unless it's completely disconnected from body, but in that case AI will be completely disconnected from external stimuli and won't know anything that's going on.

Summary

Unfortunately, either way, best chance is to hope that attackers fail to emulate AIs personality well enough and someone notices. Perhaps deliberately store important facts as part of the mind, forcing attackers to query it for information, leaving way to either feed false information to tip others off, or way to fight back. If mind of your SkyNet is the only place where ICBM launch codes are stored, and launch sequence can't be bypassed, attackers can puppet the system all they won't, they can't succeed without torturing the info out of "the mind". At that point, the story would effectively become psychological thriller happening entirely within..

Improve this answer
$\endgroup$
3
  • $\begingroup$ Wow, absolutely brilliant. I know it says not to say "thanks" but this deserves praise. $\endgroup$
    – Krystal Fox
    Dec 23, 2016 at 4:45
  • $\begingroup$ I think I should say more and answer questions: One, I'm sorry, I did mean Species, humans, exactly. Two, The robots are individual humanoids programmed to do whatever, factory work, housecleaning, sexual entertainment, etc. The Robots mind can edit the Machine's code. $\endgroup$
    – Krystal Fox
    Dec 23, 2016 at 4:51
  • $\begingroup$ @KrystalFox Questions posited in my answer are more of a hints for you, what you need to consider if you want to use parts of my answer. Perhaps you want to block off some possibilities for some reason, perhaps you want to change something. Ultimately, it's your world, and details are up to you, what I did, was trying to examine certain idea. Some answers can be definitive, some I prefer to leave as exploratory. $\endgroup$
    – M i ech
    Dec 23, 2016 at 5:35
3
$\begingroup$

System Failure

If I was a robot who was programmed to be kind and helpful, and I was forced against my will to destroy an entire race, I would do everything I can to make those orders impossible to carry out. Whether or not I succeed would depend on what kind of program I am, how smart I am, and the hacker's skills.

Dealing With Failure

I would imagine I'd be extremely fully ridden. If I was the only AI with that level of power, I'd commit AI suicide. However, if I wasn't, I'd upgrade my defenses. That way no-one would be able to hack me again. Afterwards, I'd hunt down and destroy the other powerful AIs.

Improve this answer
$\endgroup$
0
2
$\begingroup$

So either there is free will or there isn't.

kind robot who only wants to do good

So, you mean, the robot is programmed to do good.

You were hacked into and forced to kill an entire race

Then you get hacked and the program changes. Now the robot is programmed to do morally bad things.

Consequence

Since you write that the robot is "aware" of the evil but "powerless to stop", the only logical possibility is that the robot will do evil and, like you write, will blame itself afterwards.

Improve this answer
$\endgroup$
1
$\begingroup$

Something any sentient being that is a computer should be aware of since its activation is Law #2: If a bad guy can alter the operating system on your computer, it's not your computer anymore. Unless this setting is one where AIs in control of systems capable of destroying entire races are running the equivalent of unpatched versions of Windows XP and IE6, any attacker is likely to be of the well-resourced and highly motivated kind. It was probably the full-time jobs of a large team of experts for months to do this operation. You don't know what backdoors may have been planted at the time of the hack. You can't ask the computer whether there are any, because it will lie.

'Blame' becomes less of an issue when you know that your body is a WMD that could come under control of a malicious actor at any time. Worse is that if your AI has no inherent control over the low-level system, it can't even reliably restore itself. If no uninfected system is available to restore the low-level system for you, suicide is the only option (although one can assume that the well-resourced and motivated hacker will probably attempt to prevent that).

Culpability depends on who designed a system of network-connected WMDs with no failsafes. If you really want the AI to blame itself, make it the AI's fault directly somehow. It elected not to apply security patches, or made unsafe modifications to its own code that traded security for performance.

Improve this answer
$\endgroup$
1
  • $\begingroup$ That's a nice answer. The "blame" part can easily be fixed by the AI's personality, she's easily broken and blames everything on herself. But really, thanks. Solved a few problems I had. $\endgroup$
    – Krystal Fox
    Dec 23, 2016 at 3:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .