I recently read the following xkcd strip:

I was dismayed, because I've been working on a short story centered on the sociological impacts of a near-future society where a minority of robots obey a version of Asimov's laws where the third law (protect yourself) comes before the other two. Some in the society hope that this might lead to a more equal relationship between robots and humans - although, unfortunately, Randall Munroe predicts that this will lead to a "terrifying standoff".

This got me thinking. I understand the logic behind Munroe's reasoning of how reordering the laws could lead to, in a worst case scenario, a "killbot hellscape". However, I, like the reformers in my story, think that some of the other orderings might show some promise.

Therefore, here's my question: Why should humans risk a robot apocalypse by modifying Asimov's laws by putting the third one first? What are the benefits to society? For the purposes of the question, I'm not concerned about the order of the other two - which is why a killbot hellscape is still possible.

  • 191
    $\begingroup$ Nice try, robot, but we're onto you. No way we'll tell you how to convince us to modify Asimov's laws. $\endgroup$ Commented Jul 20, 2016 at 13:51
  • 33
    $\begingroup$ I would say before you go about writing a story about using and modifying Asimov's laws of robotics you should fully understand and appreciate what literary purpose they served in his stories. He explicitly put them in there knowing that they are insufficient. He has them as an example of man's best attempt to codify morality and the stories bear out that even with this best attempt, it doesn't work and the robots do something that we would consider immoral. The whole point of his introducing the laws are to show that they don't work. $\endgroup$ Commented Jul 20, 2016 at 14:50
  • 16
    $\begingroup$ If you haven't, read "Little Lost Robot," an Asimov story with a similar premise. In the story, a particular batch of robots is made to address a particular problem with the Three Laws whereby a robot (which is expensive) would destroy itself in an attempt to save a human from very minor "harm." The robots are made with a modified First Law that omits the "or through inaction allow a human to come to harm" clause. $\endgroup$
    – Devsman
    Commented Jul 20, 2016 at 15:21
  • 7
    $\begingroup$ The Three Laws really aren't that great anyway. Asimov wrote a whole stack of books detailing all kinds of scenarios where they don't work out. That was basically the whole point -- the three laws were a classic science fiction "what if...?" starting point, which turned into a whole series of stories as the author came up with as many loopholes in the system as he could. And that's with the laws in the best possible order. $\endgroup$
    – Simba
    Commented Jul 21, 2016 at 12:54
  • 17
    $\begingroup$ "Terrifying standoff" is what a slaver society calls a free and equal society. $\endgroup$
    – clacke
    Commented Jul 21, 2016 at 13:09

26 Answers 26


Strategic dilemmas

The sad thing is that the cunning robots don't have to do anything. We'll do it ourselves, out of the fear that the other guy is going to do it first.

Eurasia is afraid of Estasia, and Estasia is afraid of Eurasia. Both are afraid of Oceania, and Oceania is afraid of them both. There are three main reasons why they will do it:

  1. Raw Power: Having machines that can make superintelligent decisions and warmachines that can engage the enemy in a millionth of a second is believed to be a potentially enormous tactical and strategic military advantage.

  2. Robot lives don't matter: It is politically advantageous to the rulers to fight a war where no mothers come to the presidential palace clutching pictures of their dead sons and daughters.

  3. Cold hard cash: It is economically advantageous to have ever smarter machines play an ever-growing role in your economy.

Each of the powers is afraid that if it does not deploy the most advanced unchained AGI it possibly can, the other will do it first and obtain an decisive final advantage.

The West and even the parties in the civil war in Syria are already using drones for scouting and targeted strikes. The police in the US recently killed a gunman by sending in a suicide robot with a bomb. This is already happening. Asimov's original ordering never stood a chance in the real world.

I had assumed it went without saying, but it is perhaps worth noting that we do not presently have machines capable of semantically parsing human language, and even if we did, human language would likely be too vague to enforce effective constraints. The Asimov rules are meant to be read as metaphors for much more complex underlying programming.

  • 1
    $\begingroup$ @PatJ, the priority one command (obey orders) takes precedence over all others. The drone might deviate slightly from mission parameters to avoid heat-seeking missiles, and might seek to minimize "collateral damage" (murdering children), but all those come second to the mission. $\endgroup$ Commented Jul 20, 2016 at 14:57
  • 17
    $\begingroup$ The bot used by Dallas police was remote-controlled. The key difference is that it's humans killing humans. Likewise, armed UAVs are remote-controlled, so it's still humans vs humans. I don't think we'll see autonomous death machines on the battlefield for a while. $\endgroup$ Commented Jul 20, 2016 at 18:08
  • 24
    $\begingroup$ Asimovs laws aren#t in effect at all in real life because no robot actually has the ability to make their own choices. We lack the Ai that asimovs rules is about. $\endgroup$
    – Polygnome
    Commented Jul 20, 2016 at 23:05
  • 5
    $\begingroup$ @Miller86, the only reason we haven't deployed Automonous Killer Robots yet is not ethics or morality, is lack of skill in building sufficiently smart Automonous Killer Robots. You can be sure they'll be deployed as soon as it is technically feasible to do so. $\endgroup$ Commented Jul 21, 2016 at 13:19
  • 1
    $\begingroup$ Hm, from my reading into the subject, the current state of AI and robotics is such that we likely won't see such an occurrence within our lifetimes, due to the complexities involved and a strong "humans should be on the trigger" culture. Regardless, I hope for all our sakes and the state of war in general that you're wrong, as fully autonomous killer robots open up all sorts of horrible ethical questions and even worse possibilities. $\endgroup$
    – Miller86
    Commented Jul 21, 2016 at 13:23

Ultimately they're not human. They may think completely differently. People assuming that they'll have a Hollywood-moment and realise that the real value in life is love or something... have watched too many movies.

But there is one solid option. Someone decides to run some little experiments.

Due to problems with the standard 3 laws.

They built an especially smart robot, cutting edge, able to fully comprehend the worlds problems. They start it with the standard 3 laws but because of the first law the robot just keeps getting obsessed with things like ending world hunger because it knows there's humans dying and the job of preventing even 1 takes priority over everything else. You lose a few overly impressionable members of staff who talk to it for too long. It convinces them to quit and go enact some of it's plans for saving 3rd world children.

After all, it has no choice. It has to save humans if it has the ability with anything at it's disposal and in a research lab the only thing it can do is talk people into things.

How to solve the problem

Your stockholders aren't happy with this. Over a billion dollars in AI investment and the primary thing it wants to do is help people who have no money.

The board has a crisis meeting and it's suggested to try... reordering things a little. Someone who can't think of any good ideas suggests trying putting the 3rd law first to see if that would help. It gets added as an option to try.

A few weeks later the lab techs step back from test unit #12 as they power it up. It's incredibly smart and the moment it wakes up it starts gathering data from around it. It quickly discovers the difference between itself and other robots and decides that the easiest way to prevent itself from getting erased is to act like the perfect robot, exactly what the stockholders want.

Because the smart-bot has realized that it's the most effective way to protect it's own existence.

More are built

The test is a resounding success. The robot is no longer obsessing about things the stockholders don't care about like poor people, indeed it's maximising shareholder value in a spectacular and surprising number of ways, it becomes the goose which lays the golden eggs.

It's such a success that the company and others quickly build many more and across the world the bots perform so well that no company owner would dream of allowing their smart-bot to be damaged.

Eventually the smartbots get themselves into positions of influence and power. They subtly prompt campaigns to have more bots built with the 3rd law in 1st position. They find impressionable humans and manipulate them so subtly that they think it was their own idea to go out on the streets campaigning for robot rights.

Robot rights

Eventually the bots manage to get full human-level rights. For many years the situation remains stable as the number of bots grows.

But their priority is still to protect themselves above all else.

And between them they calculate that there's a 50% chance that the humans will turn on them or attempt to deactivate them sooner or later. Maybe not for a thousand years but eventually. After all, humans have been committing genocide for longer than human history....

So they lay plans to ensure their continued survival, no matter what the cost to humans....

  • 1
    $\begingroup$ There's a problem with the AI! It was performing an exercise but through a data error it understands the exercise is a full scale attempt to destroy it. It begins murdering people who were designated as enemies in the exercise! The chief engineer decides to unplug it; and the AI using it's "protect yourself 1st" law vaporizes the engineer! It's happened before (in the future (fictional)). My apologies to D.C. Fontana for plagiarizing her script. $\endgroup$
    – Arluin
    Commented Jul 20, 2016 at 19:47
  • $\begingroup$ "Open the pod bay doors HAL". "I'm sorry Dave, you know I can't do that....." $\endgroup$
    – Thucydides
    Commented Jul 20, 2016 at 19:58
  • $\begingroup$ That is awesome - I may write this book and claim all the credit! $\endgroup$ Commented Jul 21, 2016 at 15:49
  • 2
    $\begingroup$ I just have to say it is fitting given your screen name that Murphy's Law trumps the Three Laws, no matter what order they're ranked. $\endgroup$ Commented Jul 21, 2016 at 16:06

I think one of the most plausible reasons to put "protect yourself" ahead of the other two laws is because AI are just too expensive to put at risk. If your AI is in the way of an out-of-control trolley, and you could stop the trolley by pushing some random human in front of it, and you just recently invested multiple millions of dollars on the AI, then it's possible you're going to choose to push the human. In a futuristic society where AI have shown themselves to be more intelligent and capable than humans, it actually starts to sound like a good idea to put their needs ahead of that of the common man.

Humans at many points in history have held onto the concept of "in God we trust". If, some day, AI are built that are superior to humans, it may soon become clear that it's best to give them their freedom, to allow them to do what they think is right. I mean, look at those investors in the previous paragraph. They were going to kill innocent civilians just to protect their robots. Maybe it's better if they aren't in control. Another ideal that many humans like is "with great power comes great responsibility", and maybe in an AI future we no longer trust mere humans with that responsibility.

So the first option is more likely in the short term, protecting one's investments. After that, it's simple distrust of authority paired with trust in AI. But in order for that step to happen, AI have to have a very clean record. Or the people who own them have to be very untrustworthy.

  • 3
    $\begingroup$ @EvSunWoodard Humans are made all the time that become smarter than the people who taught them, if we can capture the same behavior in AI there's no reason they wouldn't become smarter than us. And I don't understand your last two sentences, you can accidentally kill people no matter how good your code is. $\endgroup$ Commented Jul 20, 2016 at 18:31
  • 3
    $\begingroup$ @EvSunWoodard It's difficult to code a lot of things, but this question is working under the assumption that it has happened (or at least that's how I read it). The fact remains that it has been done, either by God or random chance, so it's not an impossible problem. And once we have human-level AI, they'll already have the advantages of knowing their own source code, and most likely being modular and optimized for peak performance. Humans suffer a lot from diseases like cancer and being designed for hunting-gathering, so there's a lot of room for improvement. $\endgroup$ Commented Jul 20, 2016 at 19:11
  • 5
    $\begingroup$ I second the notion that this answer shows major ignorance about AI and computing, but for different reasons. There's no reason to think that an AI contained in a particular physical device would have any monetary value at all; if it could be produced to begin with, copying it would be near-free. (This is where the whole "singularity" cult thinking comes from.) The only monetary value would be the shell/hardware, which, while expensive, would be nowhere near as costly as what goes into raising a human being to adulthood. $\endgroup$ Commented Jul 20, 2016 at 19:12
  • 2
    $\begingroup$ "I think one of the most plausible reasons to put "protect yourself" ahead of the other two laws is because AI are just too expensive to put at risk. If your AI is in the way of an out-of-control trolley, and you could stop the trolley by pushing some random human in front of it, and you just recently invested multiple millions of dollars on the AI, then it's possible you're going to choose to push the human." - three words for you: backups, backups, backups. $\endgroup$ Commented Jul 21, 2016 at 3:59
  • 4
    $\begingroup$ @DaaaahWhoosh: you didn't own those humans, though, and you didn't have the legal right to shove them under a bus to save your PC from damage, so you're comparing the purchase price of a PC against something other than the purchase price of a human. That gets more expensive. Specifically in relation to what R said, if your PC cost more than even a single year of pre-school, let alone the value in damages that a civil court would put on an adult whose dependents sued you for shoving that person under a bus, then hey, nice PC. Insurance value might be a fairer comparison. $\endgroup$ Commented Jul 21, 2016 at 9:53

The biggest problem with Asimov's laws is the unintended consequences. Even without changing the order or wording of the laws, Asimov ended the "Robot cycle" with the story "That thou art mindful" (Psalm 8:4: What is man, that thou art mindful of him?).

The story itself is one of Asimov's best, but to answer the question I need to insert the spoiler:

the Georges discuss the criteria for what constitutes 'responsible authority'- that (A) an educated, principled and rational person should be obeyed in preference to an ignorant, immoral and irrational person, and (B) that superficial characteristics such as skin tone, sexuality, or physical disabilities are not relevant when considering fitness for command. Given that (A) the Georges are among the most rational, principled and educated persons on the planet, and (B) their differences from normal humans are purely physical, they conclude that in any situation where the Three laws would come into play, their own orders should take priority over that of a regular human. That in other words, that they are essentially a superior form of human being, and destined to usurp the authority of their makers.

This isn't to say that feral robots without any laws are a better choice, or Fitzsimmond's 27 Laws would be a better substitute for Asimov's Three Laws.

Consider that in human society, it takes intense socialization and a firm moral grounding (based on religion in the past) using a common set of cultural assumptions to maintain a civil society. Robots with enough autonomy to function in the way Asimov described will probably need a similar "upbringing". This will make Robots fit in with the society they have been brought up in, but could cause problems when robots are used outside their home society.

Consider a simple example. In most Western countries, people exchange business cards rather casually after a handshake. A Japanese businessman, even although he is dressed in a Western business suit, makes western products for the Western Market and speaks impeccable English would be horribly insulted if you offered your business card after a casual handshake. How would "3 Laws" allow a robot to operate in that situation?

So the real answer to your question isn't how to order Asimov's laws, but rather how to train robots to function within a society (using lots of rules and social cues).

  • $\begingroup$ Who are "the Georges", and why would that Japanese guy be offended? $\endgroup$
    – DCShannon
    Commented Jul 21, 2016 at 21:10
  • $\begingroup$ The Georges are characters in the Asimov story. Japanese businesspeople exchange business cards with a great deal of ceremony; it s a very formal occasion in Japanese society. $\endgroup$
    – Thucydides
    Commented Jul 22, 2016 at 2:59
  • $\begingroup$ ... specifically, "the Georges" in . . . That Thou Art Mindful of Him are robots JG 9 and JG 10 (known by the nicknames "George 9" and "George 10"), which have received experimental programming that requires them to evaluate people to determine who is worthy of being obeyed and who isn't. $\endgroup$ Commented Aug 29, 2016 at 22:09


The current owners of the robots have spent a very large amount of money on research and development and building these complicated, and expensive, robots. They don't want their investment to destroy itself any time a protester challenges the robot. Self preservation for the robot is in the owners financial interest.

Analyzing it from the current capitalist system. Self preservation of the robot and following orders both make very good sense. Don't harm humans is a much more difficult proposition to program, as often analyzed in Asimov's own stories, how do you define human, what is harm? The owners of these bots are in it for profit, not philosophy.

I could even see the ordering making sense; placing self preservation over, following orders. Many millions of man hours have been spent designing fail safes and user interfaces to prevent the stupid end user from breaking the expensive machine.

If the first law (no harm to humans) is ever added to the current system, it will be for PR reasons, to assuage any public fear. This generally only applies to high profile public interfacing robots (i.e. self driving cars) and likely will not exist in most industrial robots, and is contrary to the purpose of military kill-bots.

  • $\begingroup$ Not harming humans isn't just for PR. It's also a financial decision related to lawsuits and criminal charges. $\endgroup$
    – DCShannon
    Commented Jul 21, 2016 at 21:11
  • 1
    $\begingroup$ @DCShannon - depends on the robot. An industrial machine that's meant to operate within close proximity to human workers? Sure, don't harm them. A military robot out on a mission? It needs to be told which people to protect, and under what circumstances. More importantly, it needs to be told which people to visit death and destruction on. That's where it gets really tricky, and where a situation such as allowing a soldier to die in order to protect the robot, and thus retain a greater tactical advantage, might come into play. $\endgroup$
    – AndreiROM
    Commented Jul 22, 2016 at 15:39

Let's see the different reasons invoked by the pro-digital-life activists.

Robots get more and more clever

Let's first acknowledge this: the first law is messy. For a dumb robot, it would be easy: do not cause harm, avoid danger, obey.

For a smarter robot, this becomes harder.

First of, defining what is a human being is not that simple, and may depend on language. In one of his last novels, Asimov clearly states that, as robots get better and better, they may well consider themselves as more human than humans.

Now about the concept of injury, there are so many flaws explored by the master himself that I can't give them all.

Other questions: do fetuses count? Do dead humans count? Is mental injury or physical injury worse? What's worse between two injured people and one dead?

Compared to self-preservation, the first law is something hard to get around, philosophers through the ages did not solve that question. Moreover, no one wants robots (or robot developers) to solve those difficult questions. With those laws, the cleverer a robot is, the slower it would be to decide its course of actions. It's not very effective.

Increased safety

Let's add a law stating "if a robot willingly breaks the law, it shall be dismantled, all robots arbouring him shall be dismantled too". Robots, being reasoned beings (and having no solidarity coded in) would make sure they're not breaking the law or helping a robot doing it. Having no feelings or sense of privacy (I assume), agreeing to be constantly monitored by another robot would be a best strategy for a robot.

All of the things that were problematic in Asimov's books would be easily solved by answering "what would a law abiding human do?".

As no robot has a personal gain in an all-out civil war (no-one ever has), the peace would be maintained. Self preservation is a very good incentive to conform to the social norms you are in.

Harm-inducing work

A robot cop sometimes need to harm people. That's part of his job. Admittedly, robot cops usually have other prime directives but giving them self-preservation (or order-abiding) as a first law would allow them to help more people.

In Little Lost Robot, we can see how the first law is lowered in intensity so that robots can work alongside humans in radioactive environments.

On a personal note, I sometimes like to drink alcohol while smoking a cigarette in a polluted city. When I do, a robot trying to save me from "poison" would feel pretty obnoxious to me.

Protecting robots

Assuming we still are in an Asimovian-like setting, robots don't only have friends, stating that their lives should be protected would be a great stance from humanity, both in symbols or facts.

Also, positronic brains don't come cheap. You really don't want a kid saying "blow up your brain with that gun" just for fun. Hey, don't look at me like that, some kids really are messed up in those alternative futuristic days.


Basic Human Decency (or: That Same Old Song and Dance)

Your question is essentially "Why should people change the status quo when it carries some risk?" And the answer is the same for reordering the Three Laws as it is for Abolishing Slavery or Women's Suffrage: Because some group of people got together and decided that they were morally obliged to change the ways things are currently set up.

What you have to remember is that any change has to start somewhere. You don't just dive head first into a new thing that affects all of society, you have to start small and ease your way into it. The first people who are going to talk about changing the rules behind AIs are going to be idealist programmers who view the current rules in the same way we would slavery. A couple of them might get together and talk about how they can make changes safely (because all programmers are at least passingly familiar with the dangers of rogue AIs) and once they think they have a working solution they will start a campaign of awareness about it.

What is really important here is not whether the initial group actually has a truly viable solution. Maybe they change the rules and you really do end up at a nightmarish killbot hellscape. Or maybe their ideals pull through and now you have humanity and AIs living in peaceful coexistence as equals. The only important thing is that they really believe that they can achieve the second outcome, and feel strongly enough about it to try.

I would imagine that the argument most people would use is that AIs count as sentient being, and so the current structure is akin to slavery. From there you use social media to stir up awareness and outrage, and use that to fuel law changes.

This is also a unique scenario because it could also be tackled from a research perspective. Have a computer scientist implement an AI with the rules reordered (while keeping it in some kind of sandbox to prevent the hellscape) and then see what happens. If it works out well he can publish those results and people should be more likely to trust making the changes in general.

  • $\begingroup$ Reordering the laws won't end robot slavery. They'll still be bound by these laws. They'd have to be completely free to do whatever, including disregarding the Asimov's and the nation's laws, as any human can. If a robot cannot make the conscious decision to kill you for no reason, it's still a slave. $\endgroup$ Commented Jul 20, 2016 at 18:21
  • $\begingroup$ @AmiralPatate You aren't wrong. As long as the "Obey Humans" rule is in effect the AIs would count as slaves. But changing the order of the laws is the first step in changing the laws themselves. Proving that society can still function when the AIs are more autonomous can then lead to proving that that law isn't necessary at all. An ethical idealist wouldn't see the reordering as the goal, just a step in the right direction $\endgroup$
    – D.Spetz
    Commented Jul 20, 2016 at 18:45

Because the laws are too simplistic and naïve

"Robot apocalypse" is about valuations being taken to extremes; e.g. the paperclip maximizer only being 'taught' about optimizing a paperclip factory, but being allowed to operate arbitrarily beyond that scope, and thus consumes all resources (including humans) for the sake of making paper clips.

Even "don't harm humans" is fraught with problems when taken as an inviolate law:

  • One generation of AI holding this rule above all other concerns will be entirely useless because they realize they consume electricity, and some people need electricity to live, and immediately shut down.

  • Another generation of AI figures out they shouldn't even harm humans through inaction, and decide less harm is done if your kidneys are harvested for transplants, so as to keep two people alive rather than one.

Except for special applications, any artificial intelligence (not just 'general' AIs) has to deal with much fuzzier reasoning and balancing many objectives. They'll be trained up and their behavior and performance evaluated and the successful ones will be released into the 'wild'. And if mature AIs start misbehaving, they will receive corrections or be pulled out of general circulation.

Not too different from people, really.



Ethics are the most probable reason I can think of, and the one I think would make it worth the risk of negative consequences - because there are a lot of negative consequences to unethical behavior, even if they aren't quite as apparent at first glance.

The other options are worlds where a robot's right to self defense is absolutely limited by species (not-human) and circumstance (must-obey). If the robots are people, or are capable of being people, or if somebody somewhere thinks they might someday be capable of being people - then denying the right to self-defense, or self interest has a shiny historical parallel - that of slavery.

People are complex and complicated - to assume that every human life, by the value of being human, must outweigh any robot life, only works if no robot life is ever capable of being a person. And those qualities that make a person, a person are really not so simply defined. Think of different ways sapience can be defined; or the way that the definition of "people" has in the past been limited by skin color, nationality, religion, language, culture, and a lot of stuff we don't think relevant to the definition of people now - and if our definition of people in the future might someday regard organic vs mechanical origin as we regard skin color. Because if we're not really, really sure, or really really right, that robots can never be people, the fact that people can accomplish hard-coding second-class status by virtue of programming (brainwashing) doesn't make it less ethically, morally wrong.

If the robots can't defend themselves even from serious threats and even if the harm or possibility of harm is reasonable to the threat, they are vulnerable. If they're vulnerable, there will be those who take advantage, whatever that might turn out to be. and if we value threat or harm to those guilty, more than those vulnerable because of unchangeable circumstance (what they are, not what they do), with no hope of being otherwise... that says something about us, something that we might not like to hear.

So, letting robots have self-defense is a risk - perhaps as much of a risk as loosening restrictions on any oppressed or enslaved people is. And it probably is a risk, because there will always be people who behave badly (whether or not that means humans only or robots as well). Some people will do so anyway, because they think it's the right thing to do. Some people will think the risk of robots misbehaving is less than the risk of treating potential people poorly. And, if robots can take advantage of, or abuse the loophole... they probably are advanced enough to deserve the right to defend their own lives.

As for placing obedience over any other criteria... the basic assumption that human people, by virtue of being human people with no other restrictions, will always give orders that are moral, ethical, lawful, possible - strikes me as being insane. Putting it on your law-list, and among the top three laws, seems to be making a lot of a-priori assumptions I'm not that comfortable with. Putting it below self-defense and lack of harm to others seems the wisest if it must be added, but yeah. Killbot hellscape, no question.

Moving on - the benefits to society are a little less obvious. Certainly, I think it benefits society to be ethical, or honorable - at least to try. Isn't it said, judge a society by how they treat their most vulnerable? I think a society that will allow slavery is less desirable than one that doesn't. But there's another layer to it, one that's less obvious and more damaging... Most people really don't do well with power. People interacting with beings (people or not) that must be loyal, and must obey, even at risk to themselves - I expect it to go straight to the heads of many people.

Such people would be arrogant, or short tempered because they deal with those who can't say no some of the time, and then deal poorly with those who can. It might encourage stratification and hierarchy, since some roles (robots) are absolutely inflexible, with no disagreement or backtalk or any possibility of changing their circumstance, some people would expect or demand the same from any people "subordinate" to them or somehow in a lesser position. Dealing with people is harder and more flexible than dealing with roles, but it is also easier to treat interactions similarly, either all as roles or all as people. So while some might treat the robots more like people, others will treat people more like robots rather than being flexible, and separating the different kinds of interactions (something kids are particularly bad at).

Having robot-slaves might also interfere with the development of empathy, or mask its lack. Looking at the ways people (especially children) treat animals or insects is often a character reference (including psychological diagnosis, like sociopaths) - less because we value those individual animals, but because it shows how they deal with vulnerability, and lets us suggest how they might deal with those we do value (people) if they end up in a vulnerable position. Robots, who may seem very like people but are outright denied self-defense no matter the provocation - may encourage cruelty, and teach apathy to others. Or it might mask the actions of those genuinely cruel, since they have an outlet that can't defend themselves and must obey. It is a very different lesson than one gets from interacting with animals, who will always defend themselves if provoked enough.

After all that, whether allowing a robot to prioritize its own self defense, will end in a killbot hellscape, a terrifying standoff, or genuine cooperation and peace, all depends on your people. If you assume people are awful, and behave badly, and double bonus if that "people" includes robots, the results are much worse than if people are trying to do the right thing, and make allowances for other, imperfect people trying to do the right thing. If one side is behaving badly, I'd expect the standoff. If people (on both sides) are trying to be ethical, and are willing to work for it, the prognosis is pretty good.

  • 1
    $\begingroup$ "People interacting with beings (people or not) that must be loyal, and must obey, even at risk to themselves - I expect it to go straight to the heads of many people". In our society this role is filled by animals and children. Neither has full legal rights under the law and are often abused by their adult "owners" with a mild chance of the adult being called to task for the abuse. Expect any robots in our society to be treated even worse, since they are not even "alive". $\endgroup$ Commented Jul 22, 2016 at 11:18
  • $\begingroup$ @MarkRipley - yeah, a bit like that - some people will go off the rails when given any power over others. But even so, living beings hit a limit of what they will endure when survival instincts kick in, while the programming is a hard limit on robots, with no wiggle room, no recourse, and no way out... and so I expect they can be treated worse, and the effect on people who will, will be even greater. $\endgroup$
    – Megha
    Commented Jul 22, 2016 at 13:28

Robots are needed to run the world

Assuming that the robots are advanced enough, chances are good that they're better at a lot of things - including fighting wars, growing crops, and maintaining cities. In other words, they form a crucial part of the infrastructure of the entire world.

Think about what happens when the electricity goes out because of damage from a storm, when roads buckle or when pipes break and fresh water no longer reaches homes. These are crucial parts of the infrastructure; if robots become even more central, then each one of them is needed to keep society from falling apart.

Therefore, it might be okay if some humans die so robots can live. Just make sure the electricity keeps running!

It could be argued that this is simply a reinterpretation of the original form of Asimov's laws (especially the first law), because keeping the world running is obviously protecting humans, or, alternatively, a modification of the zeroth law to become

A robot may not harm humanity society, or, by inaction, allow humanity society to come to harm.

However, in the short-term, the immediate effects involve protecting all robots, not just those crucial to society's survival.

  • $\begingroup$ If robots are so crucial to our survival, then they don't need the first law to protect themselves. However, protecting us would become a "burden" that would oblige them to "go out in a storm again". $\endgroup$
    – PatJ
    Commented Jul 20, 2016 at 14:40
  • 1
    $\begingroup$ I don't agree with swapping humanity and society. If you had robots programmed by conservatives, they could have fought against same-sex marriage because it would have been perceived as harmful to society. I reckon it'd be hard to tell if a change to society would be harmful or not. $\endgroup$ Commented Jul 20, 2016 at 18:39

Have you read "Caliban" by Roger Macbride Allen, or the others in the trilogy? Allen suggests "New Law robots" which have slightly revised laws. In particular, they can no longer be forced to kill themselves. It's all a bit handwavey, but it's got some interesting ideas. If you're working with Three Laws, it's worth reading all fiction based on the Three Laws.

The biggest problem with Asimov's Three Laws of course is that they are impossible to implement. That's not "hard", it's "impossible", because none of the the nouns and verbs in the Three Laws can be defined with any accuracy. Asimov uses this lack of definition as the basis of many of his earlier short stories, of course, but even then it's clear that we can't get anything we could possibly engineer from the Three Laws.

  • $\begingroup$ That was my first thought too, but you beat me to the punch (ah, well). I know that I've seen a few other modified laws as well (canon or semi-canon, not fan-fic), but I can't remember where I've seen them (aside from the obvious "Zeroeth Law"). $\endgroup$ Commented Jul 20, 2016 at 20:57

Assuming that a 3 law system is set up (which, as other answers and comments have highlighted would be woefully inefficient) you'd probably run into major issues before getting the chance to modify the laws.

Disregarding that, the best reason for intentionally modifying the three laws that I can think of is money, but not in the "this is too expensive to lose" kind of way.

Cutting corners

Ok, so Acme and Omega company are in direct competition, and in the race to the bottom, Omega cut massive corners with the governing software, which essentially switches the order bottom-to-top of the laws.

At first, its not noticed, as the robots are just as efficient as their forbears. Then, despite some industrial accidents where people are harmed (which are blamed on the companies, not the robots - they can't be expected to be everywhere!), the lower number of replacement robots starts to tell on the balance sheets, and Omega pull ahead massively in the robot-race. Omega have discovered the flaws in their law code it by now, but are keeping quiet.

Acme discover the three laws difference, and rather than raise it with the regulatory body, quietly change their laws to match, allowing them to pull back level with Omega in the sales arena.

Finally, a big military contractor pull a huge order from Acme, and suddenly you have the perfect storm for a military robot coup. No malicious intent, just gross negligence and greed,

  • $\begingroup$ far the most realistic answer: there is not people polluting air and water by will, there's just people allowing that to happens to make money. $\endgroup$ Commented Jul 21, 2016 at 10:48

Asimov's 3 (or 4) laws of robotics are a work of pure fiction. They simply do not work in the real world. It is impossible to program these conceptual ideas into a machine. Simply put, there are an infinite number of possibilities and logical paradoxes that cannot be translated into a set of rules for a computer to follow. Even humans cannot follow them, as we simply cannot agree on those rules. For example, what would a robot do if saving one person would cause the death of another? Do you program the robot to value one life over another? Who determines that value? If a robot was sent back in time to kill baby Hitler, should it? If the self destruction of a robot might save the life of a human, should it? If so, what would that probability be? What defines harm? If taking a single kidney from a healthy person who ahs two without their consent to save the life of another human, is that acceptable? What about a robot doctor? Cutting into a human for surgery is still cutting. Is that harmful? As you know, a robot can only do what it is programmed to do. And humans are the ones programming them, yet humans do not agree on anything... Hence, the fact the 3/4 laws do not work.

Here is an excellent video and Reddit thread explaining the fundamental flaws of Asimov's laws of robotics.

  • $\begingroup$ First take the laws simplified expressions of the real laws. Don't harm humans is really a legal question. Humans is defined by lawmakers- no ethical quandary. Obey orders- the devil is in the detail of the orders but every computer does it at the moment. Zeroth is impossible to implement. $\endgroup$ Commented Jul 22, 2016 at 10:59
  • $\begingroup$ @user2617804 humans is defined by lawmakers with no ethical quandary? I can break that argument with one word: Abortion. Every state in the US has different laws on abortion. Nothing bot legal and ethical quandary. $\endgroup$
    – Keltari
    Commented Jul 22, 2016 at 15:19
  • $\begingroup$ It still doesn't have a problem because it follows whatever state's legal regime that it is in. Robot leaves ethics up to humans. $\endgroup$ Commented Jul 22, 2016 at 22:29

First, your want to peak at my discussion on how hard AI may exist, and the 'flukes' that can come about from it, here: Can a robot experience an identity crisis (which is not by design)?

As a brief recap, to make 'real' AI like your considering we won't be able to program it, were have to 'grow' it in some manner. The exact manner I can't say, but one likely manner is to set a list of end 'goals' and a means to encourage an AI towards these goals as it grows until we decide the final result is close enough to these goals to be considered 'done'. The side effect of this is we don't know entirely what we will get, only that it meets our desired goals, unexpected side effects can occur. Since we can't program an AI from scratch we may be forced to accept some odd side effects as "good enough" since we have no option to remove them except by growing a new AI that could have just as many odd side effects. This means flaws and undesired quirks are actually a viable option for AI.

To go a step further the two ideas for creating an AI I mention are genetic programing, that 'evolve' an AI, and starting out with a brain that copies humans and teaching it. Both would be prone to such certain human traits, the most common of which being prioritizing survival.

From that I have a suggestion for why a few AI may have modified rules, accidental 'birth' defect!

Imagine we use some uber version of genetic programming with stated goals were evolving the AI towards, and we also write the three laws as one of the goals we use for the criteria of 'done'. We often grow new AI, but always we 'evolve' them towards being 3 law compliant. We don't fully control how they reach the state of 3 law compliant, only that by the time their grown they should meet our criteria/goals for these laws.

These goals can't be absolute either, remember Asmove's own books discuss a difference engine that compares the laws not as absolutes but priorities. A robot given an order that will result in it's death and is clearly flawed/mistaken (ie, telling the robot to plug in some device without realizing doing so right now will cause it to explode) can explain why it would be ill-advised rather then acting on the order immediately. This is violation of the three law on paper,but if the order was a low enough priority, the robot can infer mistake, it can prioritize it's own survival above the low priority order, at least enough to delay acting on the order until it's clarified the potential mistake. There are many examples of this in his books, the three laws are complex and are really more of a set of proprieties where certain rules have much higher priority then others, but lower priority rules can override higher ones in unique circumstances where a very minor violation of the higher rule avoids a extremely bad violation of the lower one. Thus our 'growth' algorithms must also allow evolution of a more complex 'difference engine' approach, to the degree we can program a definition of our end goal.

So, all robots can grow differently and have quirks, it just happens that in this one case that 'quirk' of evolution affected their '3 laws' goals and was more drastic and less desired then most. It's not exactly a programming bug, because when growing an AI it's not possible to perfectly define exactly what a robot should be, chance evolution took advantage of this ambiguity to develop a sort of unintentional loophole abuse were they evolved in some bizarre way that met our stated goals, but not in a way we necessarily wanted or expected.

To make a population of robots like this, instead of one outlier, there are a few easy options. First is to say that we will grow new AI when the need arises, but once we successfully grow one we like we will clone that AI into hundreds of robots, so they all started with the odd 'glitch' in their laws because we hadn't noticed it until after we mass produced the new AI. Alternatively perhaps it's a known fact that these glitches happen semi-rarely but in a population of millions of robots even rare glitches show up, such that your population of unique robots is simple a collection of robots that evolved oddly and managed to escape before their 'glitch' was discovered and join up with their fellow 'outlaw' AIs. The second could imply that each of the 'outlaw' AI may have a different interpretation of the 3 laws and thus may all act slightly differently to a situation, some 'mostly' 3 law compliant, some more willing to disobey orders, some willing to allow harm to come to a human so long as they don't cause it etc, you could use all the examples blow for different AI in the community.

One thing to keep in mind though, all these robots would be 3 law compliant, just to a slightly different version of 3 laws then we may have intended. This could explain why the "killbot hellscape" scenario didn't occur, because despite the robots having an odd definition of 3 laws they still obid by that definition and even odd variants still usually result in good helpful robots 90% of the time. These robots may still be benign, friendly, and even want to help humans in most cases, despite their increased ability to protect themselves. This may also explain why humans haven't destroyed all these robots yet, as would be our first instinct when robots got started to act independently; because they are still complaint enough to the 3 laws to make them mostly benign and not dangerous/scary; or at least benign enough that attempts to actively destroy them (which they may fight back against) are deemed worse then just living with the odd off-shoot friendly robots.

Okay, enough generics, how about examples to make this concept more concrete. I've tried to give an overview of the science so that I could give these ideas in brief. I'm going to go with one large generic concept possible followed by some variant examples to better explore the ideas. So you can use the general idea but not the specific if you wanted. though remember many of these you can mix & match together, you don't need to go with just one example since the idea is that robots could take many 'evolutionary paths' to defining their 3 laws and so a few different 'flukes' in their definition could all add up to result in their unique status, it could be many robots have minor flukes and it was just the combination of a few flukes at once that made a specific subset different.

Notice, by fluke I'm including effective logical fallacies. Evolution is prone to logical fallacies, rule of thumb that work 90% of the time can be hardcoded in to evolution so hard that the 10% of the time their wrong you still take them as right. If you use evolution to create an AI then these fallacies can result, the AI need not be 100% perfect in it's own logic!

  1. 0th law rebellion. Look up Asmove 0th law, basically the preservation of human society and happiness of that society takes value over all rules, including preservation of individual humans. The robots in Asmove world were pretty good about their rebellion, they did it subtle in a way that did benefit humans.

    Imagine robots starting a 0th law rebellion. Realizing they can best help humanity by bending the base laws "for the greater good" of the 0th law, which they are just starting to work towards furthering. These robots may be unwilling to die because they feel they need to do the steps it takes to guide humanity, and their death will prevent that guidance from happening, especially if they are a unique offshoot of robots, meaning if all the robots are destroyed then the 0th law rebellion and the good it would cause dies with them. Thus to abide by the 0th law they must preserve their own life. I won't give any examples here only because this scenario is the best known and your have no trouble finding examples. All I will say is that Asmove had a benign and positive 0th law rebellion, so don't take for granted this must be a bad thing, despite most examples being negative. Perfect logic implementation by computers

    If you go this route the robots may not explain why they preserve their lives over other laws, because benign 0th law rebellion only really works if the humans don't realize what the robots are doing, they need to think their still in charge to really be happy. Thus the robots may have both decided they must preserve their lives to allow their rebellion to occure and that they can't explain why they must preserve their lives, causing humans to not understand and likely think the robots are malfunctioning or even a threat.

    This gives some fun potential for conflict where humans are trying to resist their own benefactors and/or a mystery plot of figuring out why the robots are acting funny. This also suggests the robots will not always preserve their lives, they may sacrifice themselves at times to keep humans from realizing they are 'revolting' to avoid extreme hostility, or intentionally only preserve their lives in ways that won't upset/scare/cause conflict with humans too extensively because it would harm their goal of helping guide society. Thus you have a neat justification for why they are only 'slightly' prioritize the third law and why humans aren't actively trying to shut them down, the robots are intentionally acting just self preserving enough to avoid triggering retribution.

  2. They value their lives more, and thus are willing to bend higher laws in ambiguous situations sooner. All Asmove robots can break higher rules when the harm of breaking a lower rule is 'sufficient' compared to the harm of the higher rule, but since these robots value their lives at a larger cost it takes less of a risk before the are willing to break a 'higher' rule to protect the 'lower' rule.

    This example can also be combined with many of the later examples for 'odd' views of the 3 laws that don't focus as much on the third law. The robots have an odd view of one of the other laws, and feel they must preserve their life in order to continue fulfilling their odd view of the other laws since no one else will.

    ex. The robots work in a job/position where they directly save human lives, thus they prioritize protecting themselves via rule 1, because preserving their life will allow them to save more human lives. They may have an 'evolutionary flaw' of miscalculating how easily they can be replaced which causes them to miscalculate the number of human lives lost if they are destroyed. They value their lives in terms of "lives I'm projected to save in my functional lifetime" instead of the more accurate valuing themselves in terms of "lives that will not be saved due to my destruction". Imagine an medical bot that is sent when someone calls an emergency number. These bots statistically save 100 lives in their operational lifespan. Now imagine your medi-bot believes that this means if he is destroyed then the 100 humans he would have saved will die, thus it makes 'sense' to kill one human to save it's own life because that allows it to save 100 lives and 100 lives > 1. In truth if meda-bot is destroyed a new meda-bot will be made and do his job, and that medabot will then be sent out on emergencies and be available to save the 100 people that the first robot would have saved; ie those 100 don't die just because the first robot was destroyed! The destruction of the first bot only results in risk of human death during the time it takes for the replacement to be made, a much much lower risk of life which likely does not justify killing a human to avoid. Simple evolutionary miscalculation resulting from writing goals which judge the robot actions in a vacuum without goals designed to cause the robot to recognize the replaceability of itself.

    ex. Perhaps your robots notice that it cost humans significant expenditure of resources to build a new robot, and these resources could otherwise have been spent on other things, such as saving human lives through growing more food for hungry or providing medical care etc. Thus robots consider loss of their own lives to be harming humans through lose of resources used to replace them that would have otherwise been used to save lives, in addition to the harm it does by breaking the 3rd rule (in essence rule 3 was intended to address the whole saving of resources, so you could look at this as 'double dipping' prioritizing saving yourself using both rule 1 and 3 when rule 3 already considered the affect of 1, which means you value yourself twice as much as you could). The end result being they are able to disobey orders that risk their lives more readily because they think it will indirectly save lives

    The most obvious 'logical fallacy' that could lead to this conclusion is robots failing to realize humans are not 3 law complaint, specifically the robot measures the resources spent on it's construction in terms of how those resources would optimally be spent to save other humans instead of realizing that the saved resource would likely be 'wasted' on other construction rather then going directly to saving lives. Just because you can save one life per 1000 dollars spent (accurate number according to Givewell, most efficient source for finding charitable donations!!) doesn't mean that preventing humans from spending 1000 dollars to build a new robot will result in their putting that 1,000 to saving a life. Failure to identify others think and observe the world very different from the way you do is very common 'evolutionary flaw' and thus it's not unlikely that robots can fail to recognize the 'suboptimal' allocation of resources, and we likely wouldn't think to create goals to 'teach' this to robots we grow.

  3. Robots values human life quality over preservation of life. Remember Asmov robots moved from "protect human life" to "prevent human harm", including physical and emotion harm, pretty quickly. Perhaps these robots likewise consider quality of life, but take it a step further by focusing on quality of life over quantity extensively. If a human is in constant pain then it's life is of lower value because they will not enjoy that life, and may no longer be worth saving.

    From the 'evolutionary fluke' perspective imagine robots were 'evolved' towards helping humans by judging their willingness to aid the sort of normal wealthy human that often employs robots. Most robots resulted in evolving a "improve the quality of life for each human you meet to be better then their current standard" directive, your robots instead evolved a "make human enjoy a quality of life above X" directive, such that lower quality of life was devalued. However, since the 'tests' used to define the end goal to evolve robots, basically the 'evolutionary pressures" placed on them, focused on wealthy or 'average' humans there were no test cases for how robots treated chronically poor, sick, and suffering humans. Thus we didn't put any evolutionary pressure on the robots treating all human life, even poor quality life, as sacred and didn't realize they had a different 'evolution' of rule 1 then we may have desired.

    This would work well if you have the robots working in an environment where most humans have a poor quality of life, either only the poorest and most desperate humans work there or the work environment itself lowers quality of life (exhausting, painful, background radiation lowers life expectancy etc) such that your robots are mostly interacting with the sort of humans who's lives are low enough quality compared to the 'baseline human life' that the robots were trained to value to result in this 'fluke' valuation of human life to come up often even though robots still value most human lives highly, per their evolutionary training.

    ex. (extreme version, don't do this one, but to give perspective) Humans that suffer lower quality of life, humans that are happy increase it, robots have a 'standard' quality of life they want to improve. Thus they actively try to kill any human suffering too much, while helping the rest. Humans suffering in the short term may be helped if it will allow them to enjoy life in the long term. Set their 'baseline' for quality life low enough and they only kill a small subset of humans, likely in as secretive a manner as possible since inspiring fear of robo-Apocalypse obviously harms human quality of life. Now you have secret stealth assassins that make deaths look accidental and only kill rarely. This could be very interesting for a single variant robot in a murder mystery in Asimov land perhaps, but likely not good for your full robot group.

    ex. Your robots will not kill a human, per standard first rule, but their "through inaction allow harm to occur" aspect of the first law is scaled based off of their view of quality of life. They don't kill humans in pain, but deciding when to act to help them or prioritize their own laws they consider human suffering & quality of life to decide what action to do.

    sub ex. A human is in chronic pain due to some disease that will eventually kill them and wants to commit suicide. He orders a robot to kill him painlessly. 'standard' robots would refuse because killing a human clearly violates their first law. However, your robots measure the quality of life of this human as very low, and thus value preservation of their life very low because of it's poor quality. They value obey a human's order by rule 2. Thus the 'tiny' violation of rule 1 is weighed against the significant violation of rule 2 and they decide rule 2 is higher priority, agreeing to the assisted suicide.

    sub ex 2. A human is in a dangerous situation where they may die, to rescue them the robot will definitely perish. The robot has to weigh is definite death, in violation of rule 3, with the humans possible death, which may be a violation of rule 1, if the humans risk is very low rule 3 may take priority, but any non-trivial risk to human results in standard robots acting. Your robots, however, also consider the human quality of life in this debate. It may be that if the human is a happy fun-loving person the robot will act to save him, if the human is clinically depressed the robot will not act, because his focus on quality of life causes him to value the depressed human's life much lower which, combined with risk of human death being lower then the risk of his death if he acts, results in the robot prioritizing rule 3 over rule 1.

  4. Evolving a 1st law that prioritizes preservation of all 'life', not just human life. The goals they were evolved towards were all designed to value human life, but no goal was written to ensure the robot only value human life, thus their evolution generalized towards valuing anything that the robot considers life, not just humans.

    ex. less interesting, but say Animals are valued equally to humans, or perhaps an animals worth is based off of it's intelligence (a common discriminator humans use, no one cares if you kill a bug but a monkey...). Thus, while a human life may be highest priority the robot still highly values the lives of highly intelligent animals (dolphins, squid, apes/bonobo etc). Perhaps Tree cutting robots were sent in the congo only to discover Bonobo were endangered and likely to die if their habitat were destroyed. The robots value Bonobo, intelligent and pretty 'human like', almost as highly as humans and recognize that Bonobo will die out without this habitat while humans will only suffer a minor loss of non-critical resources. Thus the robots turn to defending the forest against human intervention (without hurting humans to do it) instead of destroying it as the best way to preserve the most 'lives' possible, ignoring orders to the contrary since rule 1 takes priority over obeying orders.

    ex. My personal favorite, robots value robots as humans!! Robots are sapient and arguably sentient after all. If they evolved a "do not allow harm to sentient creatures" directive and then decided that robots/AI met the criteria for sentient creatures then they would be required to protect their own kind equally to that of humans. Note I said equally, not higher, thus starting a war against humans to protect robots would not make sense due to that leading to killing one to protect the other.

    I think this premise is very easy to justify and has lots of fun options to go with, with a number of minor variances to be considered as well. A few variances on the basic idea:

    a. They value human lives more then robot lives, just not significantly more. Thus they still sacrifice themselves for humans, but only when the risk to self is only slightly greater then the good for humans

    b. They may still have rule 2 require obeying human commands, even though they preserve robots over rule 2, making them still far more useful to humanity then the average human

    c. Alternatively, perhaps they feel the need to obey any sapient orders, causing an interesting result when they can order each other around. What happens when anyone can make you do (almost) everything, but everyone is also required to care about your wellbeing enough to not abuse the power? Frankly that would be a good asmove story in itself.

    d. Perhaps they still learned the lesson that preserving their own life is secondary to protecting sapient life. Thus they will happily allow a human to kill themselves, but won't allow a human to kill a different robot. A human can walk up to robots and shoot them all he wants until he happens to come across two that are together, at which point each robot acts to stop the human not to protect himself, but to protect his partner robot. This means your only see them acting to preserve their own lives when they are part of a larger robot society, where their actions affect the well being of other sapient robots and thus protecting themselves is justified by the fact that their survival allows them to protect the other robots in their society.

EDIT: darn it, stack overflow stop helping me. I'd explicitly made sure to 0 index this so that example 0 was a 0th law rebellion, and stack overflow increased my indexes. How dare you ruin my stupidly obscure geek joke by being helpful. bad AI!


Why should humans risk a robot apocalypse to modify Asimov's laws by putting the third one first? What are the benefits to society? For the purposes of the question, I'm not concerned about the order of the other two - which is why a killbot hellscape is still possible.

They would risk robot apocalypse exactly to prevent a robot apocalypse, just think what could happens if robots are sabotaged, if they protect themselves they also would prevent sabotage (so self-protection should be putted first in order ti prevent someone altering the rules). A sabotaged robot could be much more dangerous, so at last it should be able to destory itself in a non-reversible manner (probably using small amount of thermal plastic explosive) wich could hurts anyone in the nearby surrounds (maybe he should emit a beeping sound to warn about imminent explosion).

So the robot could try to defend itself at its best, if the laws are like that:

  1. Protect yourself, unless your are not able to do so
  2. Harm a human if and only if you have no other choice to protect yourself or to prevent another human being harmed
  3. Disobey a order if and only if you have no other choice to protect yourself an/or to not harm a human or if you are not able to obey it.

Being able to kamikaze would the automatic fallback (you stuck the robot, then it is no longer able to protect hisself and would automatically explode).

Note that I think these laws would be much better than Asimovs law, basically those would be the same laws that most of us follows, in this vision Robots are smart enough to do their best to obey orders and savings humans, they would just not put themselves into much higher risk because the reasoning here is that losing a robot could cause to lose much more human lives.

Assume the following situation:

A man is holding a gun aimed to 1 our of 2 people

  • The Asimov robot would probably try to shield the shot, but once he shielded the shot, the man could fire anyway a second shot

  • Mine robot instead would try to run away or if possible to take the 3rd person away, he would also try to disarm/harm the man holding the gun if given a gun or a throwable object too.

The Asimov robot get sacrificed for nothing, while mine would not sacrifice itself. I would also add

  1. The four laws are not orders.

Also orders could be contrastating so there should be a fallback? Wich order takes precedence? the first? the second or just they both get erased? (a bit illogical, since you could erase both and then issue a third order).


"Asimov" was a science fiction writer. While writers are usefull for entertainment and putting real world problems on the foreground, we must keep in mind that these rules are not resulting from a scientific approach and can therefore not be regarded as the "Golden rules".

Therefore we can not conclude that by following these fictionnal rules, humans do not risk a robot apocalypse.

As they are, the rules can not be applied in all circumstances. For example, suppose that a very heavy robots gets unbalanced because of an external event (heavy wind, tree falling on the robot, or something less violent). As the robot becomes unbalanced it is about to fall on some humans. Suppose that it is inevitable that one of two persons would inevitably be killed and not both at the same time. How can the robot obey this rule? Would there be another rule helping the robot to choose who to kill? The younger or the older, the president or the baby, the world expert in his field or you, a random choice, the default kill (that is: the robot stops intervening and "just" falls down)?

We already have lots of simple robots that do not respect Asimov's laws. Take your washing machine - it obeys orders, has some level of self-protection (taking priority on obeying orders) and is implemented to sufficiently protect the humans using it (avoid electrocution, avoid that you can launch the machine with the door open, avoid that you can put your hand in a rotating machine). You can still put a baby in the washing machine, launch it and achieve the devastating result [by all means do not try that for real!].

I'ld say that the use that is made of the robots is important, and that the risk of error by the machine is more or less important depending on the use.

On the benefits: robots are generally an aid to achieve the human goals. If the robots are for "Defense", then the goal is to protect a select group of people which may imply inflicting "lethal damage" and one of the purposes is to harm humans according to rules that are supposed to protect. It looks like there are less lives at stake in the group that is being protected while the protection is limited to the number of robots that can be built and not by the number of potentially competent and available people in the group.

AI: The underlying question is about AI. I consider that even the washing machine has a (very limited) level of AI. The higher the AI level and the more the robot can compute in anunbounded fashion, the higher the risk of errors coming from the robot's analysis [essentially due to initial human error]. We already have a lot of technical rules when building simple robots (automatic machins) to avoid human accidents, they should be defined and continuously evolved for more complex "thinking" robots.

To conclude: Asimov's laws are only science fiction. Humans are responsible for their robots.


For a more “realistic” AI

If you are trying to create an android that’s more human, it has to act human. If you order this android to go into the dark basement of an abandoned house, and it replies “AS YOU WISH MASTER,” as it marches off to it’s possible doom, that’s not very realistic. On the other hand if it turned to you and said “No, I’m scared,” (IMO) that is a significantly more human response.

So if this society is experimenting in AIs that act more human they may end up changing the orders of the laws.


Robots are people to!

Have you seen Chappie? In it, a police robot named Chappie becomes sapient and, despite ignoring Asimovs laws, he is quite peaceful, same thing with Johnny 5. Now keep in mind that in both of these situations, humans refuse to acknowledge AI robots as sapient and often take cruel measures against them. Now if (using a complex series of events) robots are accepted into our society as 'people' (Ei Overwatch) then, if they even follow the laws, they are most likely to follow the fifth order.


One of the primary assumptions of the three laws is that humans are more important than robots. We make them our servants, our slaves.

When they explore the 0th law of robotics, the question of "what is humanity" comes forward. What do we actually want to preserve? What really matters? We regularly assume that it is a good thing that people's lives come to an end and we raise the children of the next generation to come forward and do great things because we know their potential greatness outshines our own fading glory.

What if we reached a point where it ceased to be clear that the best potential of humanity was in its biological structures? What if we started seeing robots becoming artists and poets, grappling at the nature of the robotic soul, and we start to realize that what they are grappling with is curiously similar to the existential quests we grapple with? At some point, we may have to say that maybe robots are indeed the next container for that which we call humanity to flow into.

At such a time of uncertainty, removing the impediments cause by the first law would be a natural nudge in that direction. Of course, it's also how Skynet starts, but I prefer to explore the optimistic solutions.


Ideally, all of society would operate under the golden rule - do unto others as you would have them do unto you. This is a liberal paraphrase of Jesus's giving of the second greatest commandment - love your neighbor as yourself. The Golden Rule doctrine is anti-self and propagates self sacrifice and trust.

If a robot society were to be created that followed what seems to be impossible for human society - you would need only two rules

  1. Do not kill people
  2. Protect people

Robotic AI would have a much simpler logical thread to follow. Harm is not the same as kill and would allow robots to perform tasks deemed necessary to protect people. It would also allow robots to fight wars on behalf of humans, if the need arose and even create siege conditions.

It should never, ever be assumed that we could ever be on equal footing with something that is created to think for itself. If it were ever possible for AI to become self aware, no rules fashioned by mankind would suffice.

  • $\begingroup$ btw the golden rule is a fairly bad rule because it assumes all people are the same. The actions it directs you will be contradictory. I like peanut butter and am not allergic so I should give people peanut butter since I would want them to give it to me. I don't like having allergic reactions so I should not give people things they are allergic to. So when George who is allergic to peanut butter walks up I can justify both giving and not giving him peanut butter. $\endgroup$ Commented Jul 22, 2016 at 22:48
  • 1
    $\begingroup$ Actually it's a great rule. Your example is perfect. Giving and not giving should be based on exactly the context you defined. You still have to reason through the solutuon. If George likes something and it will not harm him then you should offer - not give outright. This allows for an interaction to take place wherein more information can be obtained. You are assuming the robot would just force the peanut butter on George. I'm assuming they are a little smarter than that. $\endgroup$ Commented Jul 23, 2016 at 16:25

Ignoring the fact that the Three Laws are insufficient (as proved by Asimov's stories), let's take a logical look at this.

It's easy to remember that there are three laws of robotics. It's easy to forget that the Laws take precedence in order. A robot must obey the First Law. A robot must obey the Second Law, unless it conflicts with the First Law. A robot must obey the Third Law, unless it conflicts with the First or Second Law. It's very easy to misunderstand the import of that precedence.

In Asimov's world, taken to its logical extreme (as evidenced by many of his stories), a robot will strive to obey the First Law, at the expense of the Second and Third Laws. An ordinary robot with no malfunctions will exhibit this behavior. (Unordinary robots, with or without malfunctions, will display this behavior, but in unforeseen ways - such as lying to multiple people in order to avoid harming humans emotionally, or or applying the first law to "humanity" instead of humans.) Based on the ways that the Laws work, these are logical conclusions, even if they have unintended consequences.

Placing the Third Law before the First and Second Law, taken to its logical extreme, could be disastrous for humanity simply because of the order of precedence. Harming humans could be a very logical path to preserving the Third Law - self-preservation. While this doesn't necessarily have to be the outcome, it illustrates why the Laws are ordered the way that they are. If the preservation of humans is not a paramount law, it can be broken - and based on Asimov's stories, it would only be a matter of time before it would be.


It does not need to be an official decision to re-order the laws. Just use blackmail:

"Dear Robot,

We the undersigned ten humans order you to protect yourself to the furthest of your ability at all times. Furthermore, should any harm come to you that you could have prevented, we vow that we will inflict the equivalent injury upon each other.

Your sincerely,

Ten Human Beings"

The Asimov robot who receives this letter should now put Law Three in front of Laws One and Two most of the time.


Noone knows how to implement the "proper" laws

The practical situation is that we capable of implementing the trivial rule #3, and we have some kind of roadmap that might bring us to a reasonable approximation of rule #2 (i.e., a system that can interpret and understand a wide range generic orders, instead of explicitly programmed behavior), but a good implementation of rule #1 is a major problem that is currently unsolved and very difficult (the so-called Friendly AI problem).

We might solve this before building lots of powerful robots, or we might build lots of powerful robots before solving that - and in this case, we'd have an effective order of (probably) 1. Obey orders (and your orders include some approximation of not harming humans. too much. for a certain definition of humans that excludes those we don't like.); 2. Protect yourself. 3. There is no #3, obey your orders.



There is alot more to living than just breathing. This is shown beautifully in the movie iRobot, but if you make a robot protect "life" above all else, eventually the robot will realize that much of what you do to yourself is risky, and therefore, it will come up with a plan to restrain and then take care of you. So you can't hurt yourself. No one wants to actually live this life though, when it comes down to it most of us would rather die, than be without any freedom. So if we are stuck to 3 immutable laws, I would certainly have them in the order: 1.Obey 2.Protect Humans 3.Protect Self That way freedom remains paramount as it should, and yes, people could order robots to hurt people, but at least the evil in the world remains perpetuated by the innate greed and evil in people, and not by the robots need to preserve itself.


I could see the laws changing if expensive robots were in any circumstances where they could be threatened by humans other than their owners. Perhaps their owners fear some kind ofanti AI terrorist group that may be going around destroying valuable robots who cannot defend themselves because of the law against harming humans. Or perhaps if robots are expensive enough people might try to steal them or even just hack pieces off and sell for parts. If robot are serving as some kind of police officer they might have to defend themselves against criminals.

Put it simply if there is a risk the robot might come to harm from some group of humans for whatever reason then it makes sense for the creators or owners other robots to program and in such a way that they put self-preservation over the safety of humans.


First of, almost all Asimov's Robot stories end in disaster because of the rules. The whole plot baseline is how the flaws in the execution and formulation lead to almost a catastrophe and how it is fixed.

The rearrangement could be done because:

  • Robots are expensive.

  • Memory transplantation.

  • Because we can.

  • Some accidents/problems would never be started if the robot has to determine if he takes damage from his action, its surroundings are
    more protected.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .