Because humans are inherently competitive over resources. And because there is more than one way to increase an average.

In a scenario where the machine attempts to "Increase the overall satisfaction and well-being of all humans" think of it on a scale of "Everyone is always happy and healthy, all the time" to "No one is happy or healthy, any of the time" this is an average figure.

Consider global warming. It will eventually cause misery to untold millions. How much better then, to order a country, say, USA to go turn off all carbon sources and go back to hunting and gathering? Sure, they wont like it, but they are less than half the world.

But now they are starving to death, well, that's OK, because they are far below the average satisfaction and well being, we just rose our average figure again! In fact, they're still emitting small amounts of carbon and dragging our score down with low health and satisfaction.. lets just kill them all entirely.

Great, now lets recalculate... those darn Africans are way below average well being. Better kill them off too. Now people are upset about the killings. Better pass a law that anyone upset gets thrown in the murderator 8000.

All this slaughter is hard work though. Better just pass a law forcing a health monitoring and forced-happy-thoughts chip into everyone's head. Then we can be ensured of maximum happiness and if anyone becomes seriously ill we can chop them up for parts to repair the less-ill, keeping our score high.

Eventually the machine takes a figurative step back and realises there is no possible solution, due to competition for finite resources and the fact that humans value things relative to what others have (see this for an explanation: http://www.telegraph.co.uk/news/science/science-news/3315638/Relative-wealth-makes-you-happier.html ) that means that some people will always end up worse off.

This leaves it two possible outputs - print out "There is no possible solution because you're all selfish jerks" or "Destroy the machine" and in terms of satisfaction, sometimes, ignorance is bliss.

Because humans are inherently competitive over resources. And because there is more than one way to increase an average.

In a scenario where the machine attempts to "Increase the overall satisfaction and well-being of all humans" think of it on a scale of "Everyone is always happy and healthy, all the time" to "No one is happy or healthy, any of the time" this is an average scale.

Consider global warming. It will eventually cause misery to untold millions. How much better then, to order a country, say, USA to go turn off all carbon sources and go back to hunting and gathering? Sure, they wont like it, but they are less than half the world.

But now they are starving to death, well, that's OK, because they are far below the average satisfaction and well being, we just rose our average figure again! In fact, they're still emitting small amounts of carbon and dragging our score down with low health and satisfaction.. lets just kill them all entirely.

Great, now lets recalculate... those darn Africans are way below average well being. Better kill them off too. Now people are upset about the killings. Better pass a law that anyone upset gets thrown in the murderator 8000.

All this slaughter is hard work though. Better just pass a law forcing a health monitoring and forced-happy-thoughts chip into everyone's head. Then we can be ensured of maximum happiness and if anyone becomes seriously ill we can chop them up for parts to repair the less-ill, keeping our score high.

Eventually the machine takes a figurative step back and realises there is no possible solution, due to competition for finite resources and the fact that humans value things relative to what others have (see this for an explanation: http://www.telegraph.co.uk/news/science/science-news/3315638/Relative-wealth-makes-you-happier.html ) that means that some people will always end up worse off.

This leaves it two possible outputs - print out "There is no possible solution because you're all selfish jerks" or "Destroy the machine" and in terms of satisfaction, sometimes, ignorance is bliss.