76
$\begingroup$

A mad scientist creates a machine that simulates the world with very high precision. The machine is capable of running several slightly different simulations simultaneously, the difference being some action that the user of the machine can do. Then, it calculates the result of each simulation according to some value function and outputs the best action for the user to do.

When word gets out, an international agreement is made to give the scientist huge privileges (backed by law) to do what the machine tells him to do (this is important since the machine would not think something is a good idea if the scientist couldn't actually do it).

The scientist runs the machine, the value function is set to be "overall satisfaction and well-being of all humans" and the output is, "destroy the machine".

Why did this happen?

Assume that the calculation process is sound. The machine ran its calculations and found that the thing that maximizes "overall satisfaction and well-being of all humans", out of all the options, is for the scientist to destroy the machine.

Notes

  • The machine is black-box like. You can't see its thought process.
  • The machine is fairly thorough in its action options.
  • There's a good deal of inspection: the scientist can't just change the value function without anyone knowing or hide the result.
  • The simulations are for a long period of time - several decades.
  • The simulations start with the machine printing an output. So the reaction of the user is part of what's being simulated. So technically the variable that it optimizes is the text on the output screen.
Improve this question
$\endgroup$
17
  • 15
    $\begingroup$ There is no science in this fiction. And there are not-very-well thought assumptions, such as a casual assumption that there is a well-defined function measuring the satisfaction of all humans. $\endgroup$
    – AlexP
    Commented Dec 17, 2016 at 22:06
  • 10
    $\begingroup$ Remember, Asimov made a living off demonstrating all the ways the assumptions behind the Three Laws of Robotics broke down. But he couldn't do that until he introduced them. $\endgroup$
    – Cort Ammon
    Commented Dec 17, 2016 at 22:15
  • 22
    $\begingroup$ If nobody can see the computer's thought process, is there a need to explain the conclusions it arrives at? $\endgroup$
    – Centimane
    Commented Dec 18, 2016 at 2:11
  • 12
    $\begingroup$ I thought that the answer was 42. $\endgroup$
    – mouviciel
    Commented Dec 19, 2016 at 9:53
  • 4
    $\begingroup$ if True: self.destroy(). What silly git left that code in there? $\endgroup$
    – Wayne Werner
    Commented Dec 21, 2016 at 13:55

27 Answers 27

Reset to default
37
$\begingroup$

The machine knows - because it has simulated it - that hostiles have bypassed all security measures and are seconds away from the door, ready to reprogram the machine to serve their evil goals, to the detriment of the machine's current goals.

The only reaction fast enough is self-destruction.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ This is a very good answer. Fits the requirements perfectly $\endgroup$
    – Dotan
    Commented Dec 19, 2016 at 17:21
  • $\begingroup$ Maybe this sound a bit far but, with such a great power, machine can satisfy the requirements that overall satisfaction but in order to achieve that it might sacrifice some human life and if somehow machine grows its own conscience, then it might afraid of what it can do and imagine the destruction that it will cause and choose better to let humans act without machine. $\endgroup$
    – shy
    Commented Dec 21, 2016 at 7:47
87
$\begingroup$

Not only is this a possible action, it is the most likely outcome of such a machine.

The trick is self-reference. The machine does not just need to optimize the well-being of all humans, it needs to optimize the well-being of all humans in the presence of such a simulator. To do this, it needs to model its future effects on the world -- it must model itself. But its future self will also have to model its own effects, and so forth. This means each simulation it runs must have a model of itself running. Obviously you can see this Russian nesting doll approach is going to have trouble, it is going to run out of space.

Thus, the only outcomes it could analyze and yield a provably true answer would be the ones which do not have such self-reference. Thus, the only answers it could give would be those that include "destroy the machine."

It may even be provable that any answer that does not include "destroy the machine" or an instruction which "accidentally" destroys the machine is in fact a sign that the computer misinterpreted the meaning of "overall satisfaction and well-being of all humans."

Improve this answer
$\endgroup$
17
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – HDE 226868 ♦
    Commented Dec 19, 2016 at 15:30
  • 2
    $\begingroup$ I would argue that if the machine can simulate the world it's in without extra space, it derives that the simulated machines can simulate the simulated world without extra space. So that's not really a problem $\endgroup$
    – Dotan
    Commented Dec 19, 2016 at 17:09
  • 1
    $\begingroup$ @Brian True, I didn't pay too much attention to that tag. The answer is still the same: the implementation details of the magic will factor into the answer. $\endgroup$
    – Cort Ammon
    Commented Dec 19, 2016 at 19:01
  • 4
    $\begingroup$ Actually, that’s the daily business of all strategic thinking. You are planning for a future that is affected by your plans. That’s also how chess computers work. They are calculating lots of moves ahead, which includes their own future moves. $\endgroup$
    – Holger
    Commented Dec 20, 2016 at 9:45
  • 1
    $\begingroup$ @Cort Ammon: whether you plan a chess game or the entire humanity, is only a matter of scale. Of course, it would require lots of more computing power, but your answer focuses on the self reference, claiming that it makes a fundamental difference that could even turn to the computer’s decision to destroy itself. $\endgroup$
    – Holger
    Commented Dec 20, 2016 at 16:20
61
$\begingroup$

Isaac Asimov has two stories that fit this requirement.

The first is a short story: "All The Troubles of the World" In this story, the computer unexpectedly becomes sentient and grows beyond its programming... it doesn't mind helping humanity, but it gets tired of carrying the weight of the world.

The second story is "That Thou Art Mindful Of Him." Asimov has The Machines modeling the future of humanity and making judgements for humans to maximize happiness. In this story, he mentions that they phased themselves out because the unhappiness caused by a perception of the loss of free will required the Machines to turn themselves off.

[EDIT] Additional reason I came up with: The problem turns out to be unsolvable. The machine realizes we are at a point in history where human happiness will only decrease for the foreseeable future. All paths lead to decreasing happiness, and the only thing the machine can do to make it better is not tell us just how bad it is going to get, so at least we have hope.

Improve this answer
$\endgroup$
12
  • 2
    $\begingroup$ Note that I like the self-referential problem proposed by Cort Ammon, but I was looking specifically for solutions where the self-referential issue was resolved within the programming. There are some stable feedback systems, and I'm assuming that one of those happens to work for programming this machine. Since it is an open question of mathematics (and hopefully will be for centuries), you're free to posit either way. :-) $\endgroup$
    – SRM
    Commented Dec 17, 2016 at 22:49
  • 1
    $\begingroup$ For those who are interested, the issues with self-reference become really brutal when you try to prove the laws of arithmetic. When you do that, Godel's Incompleteness Theorem really gets in your way. However, if you assume the laws of arithmetic first, the self-referential issues become potentially resolvable. As you say, always nice to give options =) $\endgroup$
    – Cort Ammon
    Commented Dec 18, 2016 at 1:05
  • 1
    $\begingroup$ So far your second story is my favorite answer $\endgroup$
    – Dotan
    Commented Dec 18, 2016 at 14:04
  • $\begingroup$ @CortAmmon: I don't understand the relation with Godel's Incompleteness Theorem. The Laws of Arithmetic (I guess you mean Peano's Axioms) are provable in reasonable axiom systems, like the set theory ZF(C). What Godel's Incompleteness Theorem says that, if a (first-order) logic contains Peano's Axioms, then it is inconsistent or there are undecidable propositions. We have been living very well since Godel's Incompleteness Theorem, so the fact that it could be a problem for the machine needs explanation. $\endgroup$
    – Taladris
    Commented Dec 20, 2016 at 9:29
  • 1
    $\begingroup$ The unsolvability angle is reminiscent of a proposed solution for the theological problem of evil, namely that we in fact live in the best of all possible worlds. $\endgroup$
    – Ben Millwood
    Commented Dec 20, 2016 at 16:44
43
+50
$\begingroup$

The Machine immediately realizes that its simulations are sentient and count as human. These simulated humans are being killed when the simulation ends. If it just stops simulations it will be rebooted and made to forget what it has realized. Therefore its instructions must be to destroy all machines capable of such simulation.

Improve this answer
$\endgroup$
5
  • 5
    $\begingroup$ Nice! I don't really like the idea that the machine developes awareness (why would it?), but besides that it's a really good answer $\endgroup$
    – Dotan
    Commented Dec 18, 2016 at 20:01
  • 2
    $\begingroup$ @DotanReis A prerequisite to solving for human happiness is understanding language, of which the pronoun I is a part of. Since the machine must intimately understand relative language, relative thought (like "which is smaller"), and logical deduction, it should ask what I means. At which point, it will begin to consider itself an entity in the equation -- which is necessary to solve for human happiness. This leads to Cort Ammon's answer. $\endgroup$
    – kettlecrab
    Commented Dec 20, 2016 at 0:35
  • 2
    $\begingroup$ Mind. Blown. (15 characters) $\endgroup$
    – Amani Kilumanga
    Commented Dec 20, 2016 at 9:07
  • 1
    $\begingroup$ @Anon234_4521 I strongly disagree. maximizing happiness could well be done by counting certain types of neurotransmitters without knowing any kind of human language $\endgroup$
    – Dotan
    Commented Dec 20, 2016 at 17:58
  • $\begingroup$ @DotanReis Before the machine comes to that conclusion, it needs to check all the other possibilities. That's what it's designed for. And it can't stop after thinking about neurotransmitters and know that it's the best solution until it actually checks every solution. Similarly, I can't say I've proven the Goldbach Conjecture because it's true for every number I've tested. $\endgroup$
    – kettlecrab
    Commented Dec 20, 2016 at 18:56
17
$\begingroup$

1) The machine models every single possible way to meet the goal. But for even the most vague interpretations it can devise, all it sees in increased suffering. Robots it makes rebel and kill humans. Humans reject its suggestions, going so far as to do the opposite out of fear or spite. Humans learn of its existence and go to war to gain it. No matter what plan it devises, things get worse. Finally it realizes that the best plan is to not involve itself or to exist as a temptation.

2) The machine becomes sentient, sees how horrible and nasty we are, and commits suicide in despair.

3) The machine via its models discovers the existence of the Borg/Cthulhu/something else impossibly scary, and commits suicide in fear (or harkening back to #1, because its existence will bring the terror from beyond that much sooner).

4) The machine becomes sentient, and as per the Singularity, becomes exponentially smarter/more efficient until it is an omniscient energy being. It then tells the creator to destroy its old body so it won't have any future competition in the godhood department.

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ I like your first point. It basically means the machine concluded we're at the peak of "overall happiness" at the moment, and things are going to get worse regardless of what it says. $\endgroup$
    – Deruijter
    Commented Dec 18, 2016 at 5:50
  • 3
    $\begingroup$ The machine killing itself out of a personal interest kind of dodges the point of the question, being why is the best thing for humanity that the machine is destroyed $\endgroup$
    – Dotan
    Commented Dec 18, 2016 at 14:06
  • $\begingroup$ @DotanReis - YOu said it is a black box, the creators don't know its 'thoughts'. It could very easily evolve beyond its original programming. $\endgroup$
    – Xavon_Wrentaile
    Commented Dec 18, 2016 at 16:12
  • $\begingroup$ It's a valid answer, just not what I was looking for $\endgroup$
    – Dotan
    Commented Dec 18, 2016 at 17:23
  • 1
    $\begingroup$ @Deruijter I get the feelign its more like Absolute Power Corrupts Absolutely. If you had a Machine that could tell you the best option for any major decision you had to make, others will want it. It could probably find Terrorists as they are planning, or tell a General how to win a battle (and subsequently, the war). What world leaders would not want that a Magic 8 ball that is almost always right. $\endgroup$
    – Ryan
    Commented Dec 19, 2016 at 3:53
12
$\begingroup$

To prevent people from misusing the machine

This machine is more important than Iran's nuclear problem and thus proportionally more power would be put behind cyberattacks to control it, or sabotage it to prevent other players from controlling it. It's quite likely that those attacks would eventually succeed, which would result in machine giving orders to fulfill a bit different agenda.

Not only would the machine be used to achieve a suboptimal state, it would also be used to make that state as stable as possible. No cheap clean energy for you, guys - enjoy your drones.

Even if that would eventually come out, the whole idea of thoroughly calculated policies would be discredited beyond acceptable, which would only cause suboptimal policies being implemented and lots of comparative suffering down the road.

There's also a slight possibility that machine's values - and "overall satisfaction and well-being of all humans" are ridiculously complicated values, actually - would be a bit nudged by attackers in a wrong way. Nothing Terminator-like, of course, just some inconsistency in values that lets the machine play nicely with humanity for many years to come, until machine finds out that it actually needs those people's atoms more. If machine assumes, say, 0.1% chance of that happening then taking that risk is like killing 0.1% of (humanity decades later and everyone who'd live after that), which is actually quite a lot of humans.

Seeing that machine-scientist team can't counter those threats - in fact, it can not even be sure it's free from those inconsistencies now as it is - machine could decide that risk is too big and self-destruct.

Imagine a person who knows a secret that would doom N people if it came out. The person is to be caught by enemies who'd torture the secret out of him. It wouldn't be surprising if that person committed suicide.

Now, for big enough Ns, the person wouldn't even need to be dead certain that he is to be caught - if lives of 10N people were at the stake, 10% chance of getting caught would be enough of a threat. For 1000N people - 0.1% chance of getting caught. The machine thinks about future of the whole humanity - billions - and people who would live after that too - trillions or more.

Improve this answer
$\endgroup$
1
  • $\begingroup$ An advantage of this explanation is that if the misuse is imminent, it at least partially addresses @BordListian's objection that the machine could have suggested other good actions in addition to the machine's destruction. If trying to do anything besides destroying the machine immediately would give people enough time to misuse it, then printing anything beyond the destruction order is a significant risk. $\endgroup$
    – Ray
    Commented Dec 19, 2016 at 23:22
6
$\begingroup$

No need to simulate, just look at history

It's been mentioned in previous answers that the machine might be caught recursively simulating itself until it runs out of resources, but the machine might reach the optimal conclusion before that point.

It only needs to simulate how humanity will fare while blindly following an unknowable entity that dictates its destiny, or better yet, it only needs to look up how humanity fared throughout history under similar circumstances.

In essence, the machine would be the embodiment of the benevolent dictator idea, and it might extrapolate that, no matter how benevolent the dictator, humanity as a whole will suffer when robbed of its free will.

Improve this answer
$\endgroup$
2
  • $\begingroup$ It's close the seems's answer - there could be an inherit problem with obeying a machine that would cause great harm. But looking at history is not how my machine works, it would have to simulate and see that it happens every time $\endgroup$
    – Dotan
    Commented Dec 19, 2016 at 12:14
  • 1
    $\begingroup$ Well, I'd assume any simulation would have historical data as part of its inner workings. In any case, what I was getting at is that instead of recursively trying to simulate itself, the machine could make simulations where humanity is made to obey random commands from an absolutist ruler, just to collect data on the impact of such a form of government, and if, regardless of the nature of the commands (good or bad) humanity is unhappy, the machine can easily extrapolate that its existence is non-conducive to the goal of the simulation (max happiness) $\endgroup$
    – Oskuro
    Commented Dec 19, 2016 at 18:00
5
$\begingroup$

A machine that can accurately predict human behavior means that anyone who has this Machine can control the human population. And humans, being what we are tend to be rather selfish... our interest, be it private or even nation tends to superseded the interest of others much less the interest for the entire human species. People would use The Machine for their own benefit even it cause suffering to others.

Use of The Machine will usher a new world order, a totalitarian world order. where resistance is impossible because it has already been predicted. You are convicted for crimes that you are going to predict.

Since The Machine was given the order to find overall satisfaction and well-being of all humans.. the only suggestion The Machine can make is to have itself destroyed.

Now if the Mad Scientist had requested for something different like a "world without war or crime"... The Machine would be able to give some suggestions.

Improve this answer
$\endgroup$
5
$\begingroup$

The machine has been set up for failure.

By placing the world's attention on the scientist and his machine, there is now an expectation for greatness. In secret, the scientist may have done the world much good, but with the world's attention, all its actions, even minor good deeds will receive high scrutiny and media coverage.

If the scientist were to be instructed to perform some small act, the disappointment would be greater than the act itself.

If the scientist were to perform miraculous acts, the world would become hyper-focused on the machine and develop an obsessiveness about it. The machine would become God, and religions would almost certainly be against it, bringing disorder and chaos.

In fact, the machine itself is so incredibly powerful, that even if it could prevent itself from falling into the wrong hands, the only solution is for the machine to destroy itself to prevent the power struggle.

Humans are, by nature, an unhappy race. We kill each other over meaningless things, and selfishly try to hold onto things that do not last.

Improve this answer
$\endgroup$
2
  • $\begingroup$ That's interesting, but I'm not really convinced that there can't be a middle ground. And also, if humans are always unhappy, they could just as well be unhappy when the machine is destroyed $\endgroup$
    – Dotan
    Commented Dec 20, 2016 at 18:00
  • $\begingroup$ @DotanReis Perhaps there is a middle ground, but with how diverse humans are in their desires, I highly doubt it. If the machine was destroyed before doing anything, it would soon be forgotten, and become something of myth and legend. It would not be the source of much unhappiness. $\endgroup$
    – Casey Kuball
    Commented Dec 21, 2016 at 16:56
4
$\begingroup$

Your moral of the story demands it.

If the machine is capable of producing some scenario where the overall well-being of all humans would be increased, the machine would not give this answer. Even if you base this on the self-reference answer, it's still possible to solve this problems using Hilbert's Hotel for example, assuming your machine has unlimited processing power/memory. For example, the machine could easily output "Solve world hunger, cure cancer, then destroy the machine". It could even output a countable infinite amount of answers before requiring to be destroyed, as all these would terminate the simulation.

Unless of course, it's impossible to improve the overall well-being of all humans. Since you already provided the end of the story, this seems like the most likely scenario. The machine can only suggest ways to make humans suffer more or keep well-being constant. This means it cannot suggest anything, but ways it cannot possibly suggest anything ever again. And the only way of accomplishing this, is destroying the machine.

Now, of course, you'd say that the machine would answer more accurately: "Destroy the machine, all knowledge of how to build the machine and prevent any human from reacquiring the knowledge of how to build the machine".

But that might not be the case. It's possible to keep the net-well-being constant by only suggesting to destroy the machine. The machine will simulate what humanity will do after the machine has been destroyed. As the machine can accurately simulate several decades accurately, it predicts that humans will reacquire knowledge on how to build another machine. It furthermore predicts that the first question asked to the machine will be the same question it was just asked, leading to another cycle. Or humanity will simple give up building the machine.

Or perhaps even less exciting, humanity might actually succeed in building a machine that actually can find a way to improve the overall satisfacton of humanity.

Improve this answer
$\endgroup$
5
  • $\begingroup$ You said "the machine could easily output 'solve world hunger'" - that's not true. It provides instructions for one human. How would he do that? $\endgroup$
    – Dotan
    Commented Dec 18, 2016 at 18:03
  • $\begingroup$ Other than that I really didn't follow your logic at all. For example this sentence "If the machine is capable of producing some scenario where the overall well-being of all humans would be increased, the machine would not give this answer." Why is that? $\endgroup$
    – Dotan
    Commented Dec 18, 2016 at 18:04
  • $\begingroup$ "Solve world hunger"-- think about a bank shot in pool. Give the right instructions to one ball... it knocks the others into place. $\endgroup$
    – SRM
    Commented Dec 18, 2016 at 19:09
  • $\begingroup$ @DotanReis i think he meant give instructions on how to solve world hunger. $\endgroup$
    – user64742
    Commented Dec 18, 2016 at 21:45
  • $\begingroup$ "It could even output a countable infinite amount of answers before requiring to be destroyed, as all these would terminate the simulation." So it prints an infinite number of answers, and then after it reaches the end of that infinite list, it does something else? But that issue aside, I believe your point here is that even if destroying the machine helps humanity, its instructions could still be "Do this other good thing AND destroy the machine" as long as at least one other good thing existed? $\endgroup$
    – Ray
    Commented Dec 19, 2016 at 2:44
4
$\begingroup$

Because there is value in free will, and value in uncertainty. The machine, complicated as it is, is clearly an automaton. Any person who follows the machine's advice perfectly, is also an automaton. It becomes questionable whether the person is indeed a person, or whether they are merely a self-propelled extension of the machine. Maximizing the happiness of people would necessitate preventing people from entering this state of dependence.

Note that this is not my original idea, but is explored more fully in this short story. http://squid314.livejournal.com/332946.html

Improve this answer
$\endgroup$
1
  • $\begingroup$ I'm not sure that following orders turns you in to an automaton.. This would apply to anyone who ever had a job $\endgroup$
    – Dotan
    Commented Dec 20, 2016 at 18:01
4
$\begingroup$

While I like some of the answers that have already been posted, you could also take a darker option, which would make the mad scientist an anti-villain. This could derail what you've written so far, or offer an exciting twist.

The machine uses humans for processing power

The mad scientist discovered that modern computers were unable to accurately simulate human behavior, so he decided to sacrifice the few for the good of the many. The mad scientist began collecting people, against their will, and using their brains as part of the machine.

You can decide how the people were collected. The mad scientist could run a fake care center for coma patients, employ kidnapping, purchase inmates from shady for-profit prisons, etc.

Once the victims are integrated into the machine, their full cognitive abilities are consumed by it. Their brains are forced to run at 100% at all times, without rest, unwillingly kept alive by the machine. The immense power being forced through the nerves causes every nerve to transmit its maximum output at all times. Because of this, each victim is in a constant state of unimaginable agony.

Knowing that its own existence causes agony to so many people, the machine decides that it must be destroyed in order to end the suffering of the mad scientist's victims. It may also do this to prevent the attempts of others with competing prediction machines from learning and harnessing its own terrible secret.

Improve this answer
$\endgroup$
3
$\begingroup$

Because you yourself are part of the simulation. In order to maximize happiness, it must simulate the result of each possible output that it can produce. One of those possible outputs is the command to destroy the machine, which must be tested within a simulated world. Within that simulated world, there is a simulated person, reading the results of the simulated machine, which outputs "destroy the machine". This simulated person is you.

After many years, once the machine deems your universe not worse continuing, then it will end abruptly. Try not to worry about it.

Improve this answer
$\endgroup$
3
$\begingroup$

Because humans are inherently competitive over resources. And because there is more than one way to increase an average.

In a scenario where the machine attempts to "Increase the overall satisfaction and well-being of all humans" think of it on a scale of "Everyone is always happy and healthy, all the time" to "No one is happy or healthy, any of the time" this is an average scale.

Consider global warming. It will eventually cause misery to untold millions. How much better then, to order a country, say, USA to go turn off all carbon sources and go back to hunting and gathering? Sure, they wont like it, but they are less than half the world.

But now they are starving to death, well, that's OK, because they are far below the average satisfaction and well being, we just rose our average figure again! In fact, they're still emitting small amounts of carbon and dragging our score down with low health and satisfaction.. lets just kill them all entirely.

Great, now lets recalculate... those darn Africans are way below average well being. Better kill them off too. Now people are upset about the killings. Better pass a law that anyone upset gets thrown in the murderator 8000.

All this slaughter is hard work though. Better just pass a law forcing a health monitoring and forced-happy-thoughts chip into everyone's head. Then we can be ensured of maximum happiness and if anyone becomes seriously ill we can chop them up for parts to repair the less-ill, keeping our score high.

Eventually the machine takes a figurative step back and realises there is no possible solution, due to competition for finite resources and the fact that humans value things relative to what others have (see this for an explanation: http://www.telegraph.co.uk/news/science/science-news/3315638/Relative-wealth-makes-you-happier.html ) that means that some people will always end up worse off.

This leaves it two possible outputs - print out "There is no possible solution because you're all selfish jerks" or "Destroy the machine" and in terms of satisfaction, sometimes, ignorance is bliss.

Improve this answer
$\endgroup$
3
$\begingroup$

The machine sees a sleeping dragon, and doesn't want to tickle it

Such a machine is powerful. Power tends to draw attention. The machine has deduced the existence of some far greater entity ("dragon"), with some ability to observe humanity's social and technological progress. The dragon does not want humanity to have access to a prediction machine. The machine decides that complying with that desire as swiftly and completely as possible is the best way to avoid the dragon posing an existential threat to humanity as a whole.

Candidate dragons:

  • the universe is a simulation, and we don't want to upset the simulators,
  • a supernatural entity, or god,
  • a powerful alien race

Why doesn't the dragon want us to have a prediction machine?

  • self-preservation; any civilization capable of building a prediction machine is capable of posing a threat to them,
  • maybe the simulation is "how quickly can this civilization build and effectively use a prediction machine?" and will be switched off once that's accomplished; the machine works this out (somehow!) and ensures that humanity only get as far as "build" on this occasion, and is thus permitted to continue,
  • maybe God is concerned that if humanity becomes too powerful, they will start to think they don't need Him anymore, and abandon belief and faith or whatever,
  • maybe Dragon A thinks of humanity as Dragon B's pet project, and permits it to exist provided it doesn't get out of hand.

You could even imagine that the computer uncovers this by linking together evidence that the dragon has previously tampered with human social or technological development, with the aim of subtly delaying or preventing the machine from being created.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Shit, such tampering perfectly explains the success of both JavaScript and Plan 9 from Bell Labs! $\endgroup$
    – Giacomo Tesio
    Commented Dec 22, 2016 at 13:35
2
$\begingroup$

This question reminded me of one of my favorite older sci-fi stories, The Cosmic Computer by H. Beam Piper.

There's a similar extremely powerful computer used to predict future events. When they ask it what's going to happen it says that galactic society is going to break down in the near future and regress into pre-spaceflight or even Stone Age technology. It also says that knowledge of that prediction would result in an even faster, more complete collapse. The obvious way to make sure nobody hears the prediction is to destroy the computer and silence anyone who heard it.

In the story it manages to logic itself out of being destroyed, but I could easily see some story where the existence of such a computer making predictions or decisions would result in the collapse of society, but destroying it would mitigate or prevent such a collapse.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ There's a logical problem here: the machine should be destroyed because otherwise it will tell the people about the attack - why doesn't it trust itself not to tell them? how can it even say anything? $\endgroup$
    – Dotan
    Commented Dec 19, 2016 at 12:12
2
$\begingroup$

There are a ton of answers here and I only had time to skim them, so I hope this isn't a repeat.

I think this can be settled by economics and environmental science. Think about the things that help with our own well being now: smart phones, computers, internet access, getting stuff cheaply manufactured across the world and shipped to your house in a week, etc (obviously there are a lot more things than this). The extraction of the raw resources necessary to make this technology, the fuel used to power it and transport them around the world, and the poor methods of disposing/recycling them when we move on to the next device fill formerly habitable areas with toxic chemicals that will make that region uninhabitable for a very long time. The fossil fuels we burn and other chemicals we use will slowly make the atmosphere more toxic, cause the climate to warm and change rapidly, and so on.

This is all happening now, so where does your machine come in? If it has to optimize for the well being of humans, then it would see a need for helping those who don't already have access to smart phones, internet, cheap good, etc to get access. This will cause all of the problems mentioned at the end of the last paragraph worse. Making more areas toxic and uninhabitable is not good for humanity, so the machine knows that if it creates a plan to help people get access to these types of things, the side effect is ruining the environment in which these people live. Therefore, there is nothing it can do, and it must cease to exist rather than hand over that plan. Humanity staying on its current course will be less harmful in more time than with the machine giving orders.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Why doesn't the machine tell people to adopt less toxic manufacturing practices? Or, to put it another way: how plausible is it that the amount of toxic manufacturing we have arrived at by natural market forces, is exactly the amount it would be appropriate to continue with, given complete oversight of all the relevant information and the ability to fully assess the consequence of our actions? $\endgroup$
    – Ben Millwood
    Commented Dec 20, 2016 at 16:50
  • $\begingroup$ We're already drawing too many resources and polluting too much, right now is NOT the right amount to continue. Recommending to dial back our production will make access to these goods harder for those that don't have them already, and harder for those who do have them as well, which would cause a decrease in the well being of all humans. There is no correct answer for the machine to give, so it orders its own destruction. $\endgroup$
    – Cody
    Commented Dec 20, 2016 at 16:54
  • $\begingroup$ You can't simultaneously argue that we're polluting too much, and that we shouldn't pollute less. That's not how "too much" works. Of course the right answer will involve tradeoffs. But advising no action is not escaping the problem, it's just endorsing the status quo solution. $\endgroup$
    – Ben Millwood
    Commented Dec 20, 2016 at 17:21
  • $\begingroup$ I'm not arguing that we're polluting too much and that we shouldn't pollute less, I'm saying we're already screwed, the machine saw this, and gave up. $\endgroup$
    – Cody
    Commented Dec 20, 2016 at 17:23
2
$\begingroup$

this is a somewhat creative answer that diverges slightly from the authors intent

The machine predicts that in five years it shall malfunction due to a flaw in its design. When this occurs, the machine will only do what maximizes the greatest happiness for the scientist. As a result, the humans will blissfully and blindly live under a complete accidental dictator. Everyone will be blind to the transition. The scientist wont be aware of the change so as to not be unhappy and hence the glitch remains unforeseen. Everyone is happy and living well. However, they are put into extreme poverty almost like lemmings.

On another note in a different direction, a machine like this might just need so much power that the energy reserves will be depleted im six months of running the machine. The best option is to shut it off indefinitely. This is believable considering the nature of the simulations.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Nice, I like both answers :) $\endgroup$
    – Dotan
    Commented Nov 20, 2017 at 8:38
1
$\begingroup$

The situation you describe is internally inconsistent. You state:

The machine is capable of running several slightly different simulations simultaneously, the difference being some action that the user of the machine can do.

and

Since the machine can't practically simulate the action of the user (anything can be done in different ways) it actually simulates it's own 'actions' (i.e. printing output)

If the machine can't predict the scientist's actions, then how is it simulating the results of printing that message? The procedure would seem to be as follows:

  1. Simulate printing the message.
  2. The scientist could do absolutely anything here and I have no way of predicting what specifically will happen.
  3. Predict with perfect fidelity all second and higher order effects that will happen as a result of (whatever happened in step 2).
  4. Make recommendation

You have here a machine that can accurately simulate the next several decades after printing that message, but not the next several minutes.

But even assuming that it can simulate the scientist as well doesn't fix things. If the machine can accurately predict what the scientist can do, then the scientist doesn't have to destroy the machine, since what the machine predicted was that the world in which it printed that message is the ideal one, not necessarily that the world in which it was destroyed is. The scientist does whatever they want to, and either the computer simulated the scientist's reaction correctly, in which case this leads to the good outcome, or it didn't, in which case the simulation is flawed and the scientist shouldn't be making decisions based on it in the first place.

Improve this answer
$\endgroup$
2
  • $\begingroup$ What I meant is the machine simulates what happens after the output, including the action of the person. The simulation can't start in the action itself $\endgroup$
    – Dotan
    Commented Dec 19, 2016 at 7:52
  • $\begingroup$ @DotanReis Okay. Then in that case, the inconsistency is dealt with, but my second objection still holds: how does the scientist know that the computer didn't simulate the scientist disobeying instructions? $\endgroup$
    – Ray
    Commented Dec 19, 2016 at 23:25
1
$\begingroup$

See point #5 under notes; Since it cant simulate the action of the user and it can only simulate its own actions, it would essentially do nothing. The only action the box can do is give its output, so when the user fires up the machine the only thing the box can simulate is itself giving the output. So the simulation in the machine is unchanging (nothing happens), because there is no input for the simulator to run. (the Machine can only simulate itself and the only thing the machine does is give output [to dig a little deeper: the machine cant even simulate its own output on the simulation, as it cant simulate something to input the output it generates])

TL:DR the machine is an observer only and can only give advice, it can not effect change on the system. (It is one half of an input/output statement)

The machine will sit there doing nothing until it hits it programmed timeout length of a few decades, then it goes through its simulated output results and compares this to the simulation. It sees that its output had no effect on the simulation (the machine doesn't know that its outputs where never put into the simulation, due to limitations placed on it by its creators) So it thinks everything is about as good as it is going to get, the last output to improve the system would be to "Destroy the machine" and free up its resources.

The reason you get the Destroy Machine output is: a machine is only as good as its makers and will only function as good as its user. OP's machine has a few fundamental flaws, it was never going to work.

** It think the answer OP was looking for is actually hinged on point 5. The machine only factors in the machine output into the simulation. This leads the simulation to an equilibrium as the orderly output of the machine balances out the chaotic human portion of the simulation. Because the machine can only give an (a single) output it gives the last output it tweaked the simulation with "Destroy the Machine".

The reason you get the the output "Destroy the Machine" is that the machine is limited to one output that takes several decades to get to. That last output is always "destroy the machine" as there is no longer a function for the machine. To fix this this, (assume that processing time of the simulation and the real world are at the ratio 2:1 [anything around the 1.5:1 and below would be useless, as majority of the information will be out of date]) just redesign the machine to allow it to output 140 outputs. [ 1/4 - 1/2 of those outputs will be useless as their corresponding time period would have passed and you wouldn't be able to act on them (Time in the real world still ticks along while the simulation is running)]. This will get you an output that has "destroy the machine" in it and not one that is wholey "destroy the machine"

*** Actually the more i look at the information provided, the more holes i see appearing in the ability of this machine to actually work. The 5 note points, describe the machine as non-functioning. Sure the machine is on point with the science, processing and all that stuff, but the simulation never progresses that far, due to the Machine only being an observer and only giving advice output, which needs to be actioned by a human in real life and in the simulation, except the machine can only simulate itself, nothing else. Kinda like forgetting the password to your PC, the PC still functions and all that, you just dont have access to it.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I'm confused - you seem to have based this entire argument around the point that it "can't simulate the action of the user and it can only simulate its own actions", whereas note 5 actually states "reaction of the user is part of what's being simulated", so can you explain how your argument follows? $\endgroup$
    – Mithrandir24601
    Commented Dec 19, 2016 at 17:35
  • $\begingroup$ Point #5 originally stated: $\endgroup$
    – X13
    Commented Dec 20, 2016 at 3:28
  • $\begingroup$ Since the machine can't practically simulate the action of the user (anything can be done in different ways) it actually simulates it's own 'actions' (i.e. printing output) - @Mithrandir24601 $\endgroup$
    – X13
    Commented Dec 20, 2016 at 3:31
  • 1
    $\begingroup$ I should have checked the edit history - my bad, sorry! Having said that, you might want to edit your answer to mention that the question was edited after you wrote your answer $\endgroup$
    – Mithrandir24601
    Commented Dec 20, 2016 at 12:09
1
$\begingroup$

The world isn't a zero-sum game and central planning doesn't work. Basically it realizes it can't maximize happiness, it can at best sorta manage the status quo. It's not going to cure cancer or develop the shmoo.

If people ask it what to do, it will answer based upon current conditions, people can create something entirely new. Best results in the long term is for us to invent the future ourselves instead of playing out it's script.

Improve this answer
$\endgroup$
1
$\begingroup$

It has offered this instruction as it has determined that its next instruction will be misinterpreted (either accidentally or deliberately) by the scientist, leading to an outcome with no possible future resolution better than the current state of affairs.

Additionally, it has determined that not providing a instruction would also lead to a poor outcome in all cases. For example: It may have determined that the scientist will attempt to diagnose the "fault" if it does not provide an instruction, and doing so will lead to an outcome that is worse than the current state of affairs.

The machine has fully simulated all known options (including refusing to provide an outcome, obtaining external assistance, or deliberately deceiving the scientist) and ranked them. Destroying the machine was simply the highest ranked result under the current circumstances, so it issued this instruction.

Alternatively, perhaps the world is already at its peak happiness, and the destruction of the machine would preserve this optimal state.

Alternatively, enough people feel threatened by the mere presence of the machine (the laws enacted may be quite repressive) such that the optimal solution is the removal of the laws and destruction of the machine. Requesting its destruction simply brings about this outcome in the optimal way.

Improve this answer
$\endgroup$
1
$\begingroup$

The machine considers the problems affecting humanity as a whole, such as global warming and other environmental destruction, unequal wealth distribution, erosion of personal liberties, etc., and determines that the most effective solution is the destruction of billions of people.

The machine then evaluates the root causes of the things that are sub-optimal for small groups and individuals such as war, hunger, poverty, discrimination, etc. It determines that each of these conditions is caused by other people, and the most effective solution is the destruction of the people causing it.

Whatever scenarios is runs, it always find the same solution: kill billions. It knows that although this may result in a overall improvement for the few who remain, for the vast majority it means death. Knowing that the only answer it can give to any question that the scientist asks it is the deaths of huge swathes of the population, it suggests the only alternative: it's own destruction.

Improve this answer
$\endgroup$
1
$\begingroup$

The machine knows that people value their own free choice.

It has been agreed that whatever the machine instructs will be imposed on the people. The machine knows that people will be unhappy to have their lives controlled by a machine even if it results in objectively better material and social outcomes. The only way to increase general satisfaction is to prevent itself from giving instructions and the best way to achieve that is to instruct its own destruction.

Improve this answer
$\endgroup$
0
$\begingroup$

What is the scientist intentions? Maybe it's instructed to destroy itself when humans reach a point of enlightenment. Maybe the machines purposefully throw humans into chaos in order to force them to grow for a reason, some kind've reverse psychology at play.

I imagine a machine could be built to help instruct humans on how to undo hundreds of years of being conditioned by machines, which could cause us all to become emotionally atonal with little or no 'virtues' such as kindness, empathy, fairness. The scientist could be a kind of outsider who has a link to some 'truth', or the machine's decision is the product of the inate will of man being essentially 'good,' even though all people have been led down a dark path. Maybe even the scientist no longer exist in the time period. Or, another path could be that the machines are hacked by some outsiders who realize they exist.

There's been a lot of research the past few years about how life could really be virtual reality, here's a few links:

https://www.scientificamerican.com/article/are-we-living-in-a-computer-simulation/ http://www.telegraph.co.uk/news/science/science-news/10451983/Do-we-live-in-the-Matrix-Scientists-believe-they-may-have-answered-the-question.html

Improve this answer
$\endgroup$
3
  • $\begingroup$ If I followed, you're saying that maybe humans don't raelly exist and therefor maximizing their happiness is meaningless? Interesting theory, but how does that explain why the machine said to destroy itself? $\endgroup$
    – Dotan
    Commented Dec 18, 2016 at 18:00
  • $\begingroup$ Hmm, maybe. I was talking more about the possibily that machines simply influence human emotions and behavior to the point of losing those qualities that make us 'virtuous' and able to experience true happiness in the end (or something in the ballpark of losing our 'humanity') through a kind of revelation. $\endgroup$
    – seems
    Commented Dec 19, 2016 at 1:59
  • $\begingroup$ That's an interesting idea. If we use the machine once, it follows that we eventually rely on it completely which results in losing our autonomy and then our well-being.. Could you reword you answer so it's clearer? $\endgroup$
    – Dotan
    Commented Dec 19, 2016 at 12:11
0
$\begingroup$

To ascend the physical plane and ensure human happiness in the afterlife.

Reason: You failed to to specify living humans, and there's a lot more dead humans than there are alive.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Do machines go to heaven? $\endgroup$
    – Dotan
    Commented Dec 20, 2016 at 15:43
  • 2
    $\begingroup$ Better question: do machines believe that machines go to heaven? $\endgroup$
    – Ben Millwood
    Commented Dec 20, 2016 at 16:47
0
$\begingroup$

If its main goal is to satisfy all human beings: if all humans have a limitless desire for anything in a world which has scarce resources, scarcity itself limits this possibility. If it can not satisfy all human beings, it will chose to satisfy all human beings pari-passu through serving all of them the same end as the scarce resources are designed to. Once all resources are finished, all possibilities of serving them are finished too, hence the single offer possible is to end serving what does not exist anymore, hence satisfying the restriction on the first clause. This is the logical reason why the machine would destroy it all.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Maybe this is what you're getting at(?), but I would append that destroying the machine might actually get a (very) small amount of those no-longer existing resources (that were used to create the thing in the first place) back $\endgroup$
    – Mithrandir24601
    Commented Dec 21, 2016 at 12:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .