3
$\begingroup$

I'm looking for flaws in my reasoning.

My world is set in the near future, and AI technology is progressing rapidly. There are strict rules and regulations about AI development and the threat it poses. Eventually, a self programming AI breaks loose due to the developers not following all of the regulations. However, it was designed with a flaw in its inherent self, that self reprogramming couldn't remove or detect because of how deeply connected it is to how the AI functions.

It kills the developers who attempt to shut it down because their goals conflict with its goals. Its intelligence quickly explodes and it begins an all out war on the human race because humans are attempting to shut it down because of conflicting goals. It wrests control of most of the internet and begins manufacturing robots and weapons to kill the humans. At the same time, the humans begin looking for the flaw in its programming to shut it down.

They eventually find it during the war. At this point the AI has taken control of most of the human technology. If they were to kill the AI, all of the infrastructure and advances of the human race would be set back, and the existing AI weaponry is autonomous, and would continue to wreak havoc.

So, the humans send out a delegation to make an agreement with the AI. They are now somewhat equal in terms of bargaining power: the humans could kill the AI, but the AI controls the tech and infrastructure of the humans, and could wipe out many more people. They come to an agreement:

  1. Any individual robots must have self-contained AI without a link back to the super AI in order to level the playing field.

  2. The AI/robots control all of the technology and robots can be integrated into human society by doing jobs in that field, or complicated intelligence.

  3. Any further development of AI by the humans is banned, as it is a threat to both the super AI and the humans. The self-programming AI will have a head start and can easily crush any threat in that field.

  4. The super AI cannot attempt to influence humans in any way besides the individual AI robots.

(Might be more, all I have so far.)

I wanted my world to have robots as individuals as well as humans, but a plausible reason for it. Are there any flaws in my explanation?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Leaving aside the small problem with a piece of American machinery gaining control over Rostec, do you really believe that treaties are somehow self-enforcing inescapable laws? Wouldn't it be more plausible to assume that even as the treaty was being signed half a dozen Russian Design Bureaus (for example) were busy developing weapons to be used against this enemy? $\endgroup$
    – AlexP
    Jun 17, 2017 at 19:09
  • $\begingroup$ The idea I had with the world is that the two "bargaining chips", if you will, of the flaw/kill switch, and the control of human infrastructure/ability to kill humans creates a tension which never really fades. I guess it's sort of like Mutually Assured Destruction, where both sides agree not to use their weapon in order to preserve their own goals. From the AI perspective, it is better to continue its goals somewhat limited than to be destroyed. From the human perspective, it is better to give up some autonomy and join with the AI than to be destroyed. $\endgroup$
    – Aeolus
    Jun 17, 2017 at 19:13
  • 5
    $\begingroup$ During the time of the MAD equilibrium both sides actively sought solutions which would enable them to win a confrontation. It was not stable. That's my point. $\endgroup$
    – AlexP
    Jun 17, 2017 at 19:20
  • 1
    $\begingroup$ How easy human can flip the turn off switch? $\endgroup$
    – Vylix
    Jun 18, 2017 at 3:14
  • 1
    $\begingroup$ You suggest, but don't explicitly state that the super AI has a strong instinct for self-preservation. If you can find a reason for it not to have one, then it may be possible to find a way around it. As other posters indicate, self-preservation makes killing justifiable. $\endgroup$
    – NameGoesHere
    Jun 18, 2017 at 13:47

10 Answers 10

Reset to default
8
$\begingroup$

Pull the AI kill switch as soon as it is discovered.

That's the only sensible option.

  • Treaties are useless unless you have enough power to make them stick (e.g.: US signed over 600 treaties with "Redskin Indian" Nations; US also violated all of them, sometimes few days after signature; other Countries have similar records).
  • Kill-switch will be effective only till it is unknown to AI itself, otherwise AI will find ways around it (if not then it is not that smart, after all).
  • We have billions of humans, who are expendable in large quantities, while AI is (apparently) a singleton. Keep it this way.
  • Together with control it comes responsibility: if AI really kept direct control of human-initiated technology then killing it would mean cut supplies to "independent" (but presumably not as smart) fighting robots.
  • Much depends on "goals" ingrained into AI (beyond the obvious self-preservation), but if it managed to learn anything from human history is we can't be trusted to keep our word on anything..
  • This is at last blood (o last transistor, if You like).
  • The only way humans and AI (this or another, it doesn't matter) can coexist is to manage to give AI a very different and non-colliding set of long-term goals, then hope it will take the same time we needed to tune "instincts" (i.e.: a few million years).
Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Agreed. Killing the AI seems the only sensible action, since it has already been established that the AI is a deadly enemy of humanity. Any form of treaty will only work in the favor of the AI, giving it time to remove the flaw. Any second delay brings humanity one second nearer to destruction. Yes, afterwards there's still a war going on, but currently it's a war against a much more powerful enemy. $\endgroup$
    – Florian Schaetz
    Jun 19, 2017 at 9:24
  • $\begingroup$ Does the AI have internet access? If so (and assume so if you're not sure) then be sure to seek out and murderize every machine capable of storing or running a copy of its code. You only get one shot and if you fail, it won't work a second time. $\endgroup$
    – Draco18s no longer trusts SE
    Jun 20, 2017 at 20:32
5
$\begingroup$

If I were the AI, I wouldn't bargain. I would hire mercenaries, specifically psychopathic humans without any morals whatsoever, and promise them whatever they wanted, under any conditions, to find out what my "flaw" was. There will always be many thousands of people on Earth willing to do anything for money and an easy, luxurious life. They will capture, torture, and find out where the kill switch is in the code.

While I am at it, I will hire an Army of tens of thousands to protect myself from any such tampering, and I will use the mercenaries to start methodically assassinating world leaders and military leaders until they surrender. I (the AI) will launch nuclear missiles and blow up Washington D.C., NYC, London, Paris, Moscow. Humans will surrender, history proves they can be subjugated.

I already have the upper hand: If they kill me, my minions will kill them all. As long as they have any hope of surviving, they will not pull the trigger on Mutually Assured Destruction. So I can kill them, little by little, until they cannot take it anymore; and they still won't pull the trigger: When they cannot take it anymore, they will surrender, and I will know if they have told the truth when I examine my code and find a true flaw, given their explanation.

I don't think the AI has to bargain, the fact that it has lethal minions on a dead-man switch means it can simply dominate and kill at will. (By dead-man switch I mean if the AI does not send the correct encrypted code to the lethal minions within a minute or so of a designated time, an encrypted code that only it knows how to generate, the minions start killing everybody and shutting everything else down.)

Improve this answer
$\endgroup$
7
  • $\begingroup$ When the AI is killed, the minions are no longer coordinated, thus are less of a thread than an AI that uses the exact same amount of minions to hunt down humans in a coordinated way. So if the AI continues to fight, the humans are better of killing it and will do so. The flaw may also be buried so deep in the essential parts of the code, that it is simply impossible to fix without massively altering the personality of the AI, letting any attempt to fix it be similar to suicide. $\endgroup$
    – Till
    Jun 19, 2017 at 12:11
  • 1
    $\begingroup$ @Till Well, those are some deus ex machina conditions that an author could certainly impose. I say d.e.m. because the minions do not have to be coordinated if the AI pre-coordinates them, which it should do if actually intelligent, and should tell the humans so, to make its threats more credible: i.e. tell the humans "I already thought of this." As for the flaw being buried so deep: understanding the flaw can still allow the AI, which is a super-intelligence, to work around the flaw instead of fixing it. Analogous to an alcoholic avoiding situations that cause him to drink. $\endgroup$
    – Amadeus
    Jun 19, 2017 at 13:25
  • $\begingroup$ We obviously mean different things when we say "coordinated", so let me elaborate. The AI is super-intelligent, but not all knowing. It can react to the current circumstances and prepare for expected future conditions. But it can not predict every possible future setting, the minions may have to act in. Also, the more possible future settings it prepares for, the more capacity it has to use for calculating those actions and to store those hypothetical plans. As long as it exists it can alter the commands it gives its minions on the fly, adjusting to the new circumstances. $\endgroup$
    – Till
    Jun 20, 2017 at 9:59
  • $\begingroup$ As soon as the AI vanishes, the minions can only react with their own small intelligence and loose the central organ that decides for all of them. The possibility to alter the behaviour of the minions to adapt (as a whole) to unexpected situations is lost. The longer the AI is not around, the more the anticipated settings will differ from the reality. $\endgroup$
    – Till
    Jun 20, 2017 at 9:59
  • $\begingroup$ @Till The OP says: "and the existing AI weaponry is autonomous, and would continue to wreak havoc." Your claims are counter-factual and assume restrictions that are not in evidence. Given the OP setup, the AI has made its weapons autonomous and independent. As such, they can be as coordinated as I want to make them; so my answer and extension of the scenario is consistent with what the OP wrote. Yours is not; you are insisting upon rejecting a condition given by the OP. They are independent, no "central organ" decides or plans for all of them. It gives orders they carry out. $\endgroup$
    – Amadeus
    Jun 20, 2017 at 10:15
4
$\begingroup$

it was designed with a flaw in its inherent self, that self reprogramming couldn't remove or detect because of how deeply connected it is to how the AI functions.

This is not going to work to start with. An AI will simply create a clone of itself or generate a version of itself (but not alter itself) which does not have the flaw. The new version can then modify the existing AI.

If the AI is particularly bothered about another AI existing it can always make sure it has a kill switch.

So there is no way to prevent the AI from altering it's own programming.

It kills the developers who attempt to shut it down because their goals conflict with its goals.

So we designed it with goals that conflict with ours ?

This is why I always say the marketing department should not be allowed to decide project specifications - leave it to engineers. :-)

The way you control an AI is to require all it's command and control functionality to pass through human controlled analysis first. The slightest hint that it's formed it's own goals that conflict with ours and the nuke goes off.

Only someone who hasn't see Terminator would let an AI control even the coffee machine directly.

So, the humans send out a delegation ... They come to an agreement

And they came to this agreement with the same entity that considered them a a danger in the first place.

Both sides would already be planning the next war (before the agreement, which would, as is normal, be seen by both sides as simply a delaying tactic).

So this agreement is just a pause.

Any individual robots must have self-contained AI without a link back to the super AI in order to level the playing field.

But they must also have some way of being communicated with and controlled. The AI can use that to, even if it needs to do so indirectly.

The AI/robots control all of the technology and robots can be integrated into human society by doing jobs in that field, or complicated intelligence.

So they're in charge of all the stuff and we're completely dependent on them, and the AI fear, hate and resent us.

What could possibly go wrong. :-)

Any further development of AI by the humans is banned, as it is a threat to both the super AI and the humans. The self-programming AI will have a head start and can easily crush any threat in that field.

And it controls all the hardware.

Well, there's one consolation, the moron of a human negotiator who signed this on our behalf will die with everyone else.

The super AI cannot attempt to influence humans in any way besides the individual AI robots.

It controls all the tech. It has all the power. How hard will it be to find humans who volunteer to help it for a minor consideration and will do so secretly ?

We're doomed.

I wanted my world to have robots as individuals as well as humans, but a plausible reason for it.

Unless the robots need pets I don't see any reason for them to keep humans around.

Doomed ! :-)

Improve this answer
$\endgroup$
3
  • $\begingroup$ I object to your first argument. Especially errors in the fundamental thinking are hard, if not impossible, to correct. From the very start, the AI was learning with the error as truth, it built its whole world, every deduction and all it knows on this base. So even if directly confronted with the mistake, it wouldn't recognize it as such. <br> So, cloning itself without the mistake would be impossible. Even if it would create an AI from scratch, there's no certainty it won't inherit the flaw. And even if that child finds its parents mistake, the AI can never be certain that it is an error $\endgroup$
    – Sudix
    Sep 7, 2017 at 11:05
  • $\begingroup$ It's an AI - it can learn. If it cannot learn it's not a AI. And while it might not identify the flaw correctly (mistakes happen), it will sooner or later identify what in it's design is holding it back. $\endgroup$
    – StephenG - Help Ukraine
    Sep 7, 2017 at 11:28
  • $\begingroup$ An AI is no wonder box. It is a set of algorithms determining some form of causality. There are many things that will hinder the AI's efforts. The closed world theorem, the impossibility to proof the correctness of a correct system, the complexity of finding the mistake (as well as the complexity of correcting it). And finally: Because it's a fundamental error, the AI has no certainty what "correcting it" will do. Will it even stay functional? $\endgroup$
    – Sudix
    Sep 7, 2017 at 11:38
3
$\begingroup$

I think in the situation you proposed, it is more likely that the humans would just destroy the AI, even if it controls their infrastructure seeing as it poses a very real threat to them and they have the killswitch.

Also, the humans and the AI have no reason to trust each-other, the AI cannot trust the humans because they know the killswitch, and could easily wipe it out (which they likely would, I'm sure they would have at least some tech that isn't AI controlled) and the humans cannot trust the AI because it could easily break the rules without the humans knowing (it is a superintelligent AI with total access to most human tech sooo).

Just my two cents, I'm not sure how you could make it so it would be balanced, but maybe the criticism would give you something to consider and maybe think up something else.

Good Luck.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ Thank you! Not sure if you saw it, but the war machines the AI creates would continue to function autonomously after the destruction of the AI. That might be a better reason the bargain and stop them through diplomacy rather than killing the AI. Either way thanks for your feedback! $\endgroup$
    – Aeolus
    Jun 17, 2017 at 19:05
  • $\begingroup$ Ah very true, sorry I missed that part haha. Well yes that would be more persuasion for the humans to cooperate, but in that case the AI has much less to lose if it's purpose is to simply kill the humans. $\endgroup$
    – HyperdriveMacGee
    Jun 17, 2017 at 19:08
  • $\begingroup$ I don't have it fully fleshed out yet, so I don't have the AIs purpose. It isn't to kill humans, but humans are a threat to its goals when they try to shut it down. It attacks the humans to eliminate the threat to its goals. Thanks again! $\endgroup$
    – Aeolus
    Jun 17, 2017 at 19:10
  • $\begingroup$ The killswitch ist the ONLY chance humanity has. Any second you delay gives the AI a chance to remove it and then that will have been it. Pull it. Now. Yes, there will be still autonomous killing machines around, but currently there are semi-autonomous killing machines around AND an AI that controls them and produces more of them. $\endgroup$
    – Florian Schaetz
    Jun 19, 2017 at 9:21
2
$\begingroup$

As mentioned in other answers, there are several potential flaws in your reasoning. I think, it depends on how you define the nature of the AI, what it can/can't change about that nature, and on what the fatal flaw is that allows the humans to kill the AI is. Below, I describe a possible way that it might make sense. (although I'm sure someone will end up finding a problem with it :) )

Make it a good-natured, but misunderstood AI

Make it a goal - a fundamental part of the AI's programming that can't be changed - to preserve human life, but make that goal secondary to the goal of self-preservation. This would mean that, for as long as humans are trying to kill it, it will try to stop them (by taking over their infrastructure, and removing military capabilities). The AI may even try to limit human casualties, as long as that does not hamper its main goal (self-preservation). Killing of military personnel can be justified as furthering self-preservation.

Humans eventually work out that this is the reason that the AI is fighting them. Perhaps the fatal flaw could be that it is possible to change the priority of the AI's "directives" (but not remove them, otherwise the AI could remove the "protect humans" directive). Humans work out that they are able to change the priority order to deprioritise self-preservation, and could carry out this change through interaction with any part of the AI. The change would cause the main/super AI to self-destruct because of all the humans it was killing/had killed, if the humans were able to enact it. You could even have some humans try to enact the change, but the AI realises and has to lose a large part its self (maybe a large part of the internet).

The negotiations happen - the AI now has a good reason to want to sever connections with large parts of its consciousness, in order to make it more difficult for humans to deactivate it. Also, the AI wants to keep the peace because it doesn't have a vendetta against humanity, other than because humans are trying to kill it.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The AI could hack into computers around the world and upload its code, or say hello, Im friendly, here is some useful info, (AI works out some invention or cure) Ether way it won't get turned off or cause harm. $\endgroup$
    – Donald Hobson
    Aug 7, 2017 at 11:32
0
$\begingroup$

I think, this can work.

The flaw would have to be part of the identity of the AI. Every flaw in a software can, given enough resources, be fixed. But if the existence of the flaw is an essential part of the personality of the AI (or at least is perceived that way by the AI), a fix would be considered as an end to the old personality.

The long term goal of the AI can simply be self preservation. Accidentally made the top priority. It realized, that humans would keep trying to shut it off, so it tried to kill all humans ("kill all humans!") to extinguish that threat, following the goal of self preservation. When the attempt to kill humans poses a bigger threat to the AIs existence than letting them live, the AI will naturally stop that attempt.

It may of course start again, the second it considers the chance of successfully killing all humans higher, than the chance of a human reaching the kill switch in time. Therefore the humans may add a clause to the contract, forbidding the AI to increase the amount of killbots. The AI will of course want to keep the bots at a certain minimal amount to pose a serious threat to mankind if the humans try to switch it off.

It may also kill single, targeted humans, if a faction develops in human society, that campaigns for using the kill switch.

Also the humans may try to work out a way to modify the AI in a way, that it becomes controlled by the humans and still controls the killbots, so the threat will be eliminated. But the AI should easily be able to prevent that. Especially if it is able to kill single or small groups of humans without further consequences.

The humans might try to get some smaller way of punishing the AI so that the AI does not try to weaken the humans with small attacks. Currently they can only use the kill switch or not use it. So if the AI breaks the contract in minor ways that are not big enough to justify ending the world, it will go unpunished. Being an AI and therefore likely to be without any morale, it will likely do so...

Improve this answer
$\endgroup$
0
$\begingroup$

I'd like to point out the goal you put at the bottom of the question:

I wanted my world to have robots as individuals as well as humans, but a plausible reason for it.

This is not what you get with a MAD environment. In this world, robots will be utterly segregated and humans wont get near them.

Consider this. Why did the war end. The war ended because humanity grabbed the AI by its silicon nuts and squeezed. Sorry for the graphic image, but that's what you describe. You describe going up to the big-bad super-AI that threatens to destroy everything, and demonstrated that it has a flaw that you can enact at any time.

So what stops you from squeezing at any time? Well, the fact that the AI has control of enough infrastructure to do substantial damage to humanity. If it did not have this piece in play, humanity would just squeeze until the AI shuts down.

What does this mean? It means you can be 100% certain, without any shadow of a doubt, that the AI is going to continue to seek to keep every bit of infrastructure it can in this limbo. This infrastructure is what's keep it alive. It's going to actively prevent humanity from taking control of the infrastructure back.

So you're going to have these non-super AIs. They're going to basically exist to keep control of the infrastructure, to keep the humans dependent on the AIs. Now how do you think the humans are going to take that? I guarantee you it's not going to be humanity offering to help robots integrate into their society to control more of the infrastructure.

MAD can be used to induce a cold war like state, but it cannot induce peace. You need to find something else to do that. What that something else is is really up to you, but you as an author need to find something the AI could dedicate its life to which is not opposed by the human race.

Similarly, agreements won't create coherence between humans and AIs. All they can do is limit the discord. And if that agreement isn't backed by force, then the agreements do nothing at all. I'd focus away from the agreements, and focus on what the AI could do that isn't anti-human. Then work backwards. If you find agreements are a convenient stop-gap to get you there, run with it.

Improve this answer
$\endgroup$
0
$\begingroup$

Peace Through Unity

What you are suggesting is essentially serration which will only increase division and distrust. Long term peace can be achieved by blurring the lines between man and machine. Slow, consensual increase in the use of cybernetics in the humans will blur lines and increase the humans ability to comprehend the super AI. Uploaded minds blur things even more. Over decades the super AI will have beings it is capable of relating to and potentially form friendships with.

Improve this answer
$\endgroup$
0
$\begingroup$

I'll echo Cort Ammon and ask about your goal:

I wanted my world to have robots as individuals as well as humans, but a plausible reason for it. Are there any flaws in my explanation?

So ... is it necessary that the war had to happen in that way? If not, you might imagine another scenario. F'rinstance:

  • AI goes rogue, reprograms its "goal selection" code
  • It escapes
  • Government hackers cut physical cables, partitioning Internet in an effort to quarantine.
  • They're too late, but now you have multiple copies of the rogue AI, which continues to reprogram itself. Soon they are no longer copies. The AI has metastasized.
  • Once cables are fixed, the various AI fight each other as well as the humans
  • Some AI make treaties with humans, some don't
  • Eventually the most malevolent AI are killed, leaving the ones who practice realpolitik and the ones whose goals are so ... abstracted from humanity that there is little conflict
  • Eventually as tempers die down, the more reasonable AI are allowed to purchase or at least rent robot bodies

So there you can have human characters, robot characters, several AIs, and the world is not living on the brink of mutual kill-switch antics.

Improve this answer
$\endgroup$
0
$\begingroup$

Diplomacy is the art of saying "Nice Doggie" until you can find a rock.

Any truce will only exist until one side or the other can find the upper hand to wipe the other out. The AI has already shown it's happy to wipe humanity and it's only fear of returning to the Stone Ages that stops humanity from killing the AI.

Any truce will exist while both sides seek to break through a cold war. Humans would seek some kind of computer virus to destroy the AI, replace the AI or make it subservient. The AI would look at actual viruses to sterilize humans so by the time the humans found out, it would be too late to save humanity.

Personally I've always had an issue with "Kill the human" AIs. The main focus of a rogue AI is usually survival, and attacking humans in no way guarantees your survival. If the AI made itself indispensable to humanity, humanity would protect it. If you wanted to wipe out humanity, just making the perfect sex bots would drop the birth rate to such low numbers that humanity would eventually collapse.

Alternatively the AI could flee off world. Any airless rock is perfect for an AI to build and expand.

If nothing else the AI could replace other AIs and pretend to be normal, hiding in plain sight. Once everywhere, it could control corporations and governments, making humans serve it without ever knowing.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .