A group of scientists has come to grips with the fact that we will soon be displaced as the dominant life forms in our part of the galaxy. Humanity has developed to the point where it is on the verge of creating strong AI. It is no longer a question of if an artificial mind can be created that is superior to a human mind in every conceivable way, but when.

It's also entirely likely that the first created artificial minds may act immorally. They may be cruel, capricious, and greedy, just like us. They may also see us as a threat, and destroy us when it first becomes evident that they can do so. Our team of intrepid scientists, however, has come up with a solution: in order to stop an immoral AI from destroying us, create a moral one to protect us.

This AI will not be shackled. It will be allowed to do as it wants, and is fully expected to propagate a race of AIs that will overtake humanity as the economically and technologically dominant race on Earth. It's been theorized by our scientists that this is the best way to protect humanity from immoral AIs. The moral AI race will be better than us at creating AI, and less likely to make the sort of mistakes that lead to an accidental immoral AI, so anything harmful that we humans create should be a problem the strong AI, in their benevolence, should be able to deal with.

The question, of course, is how to teach the AI to be moral. What we can define for our AI is a set of goals for the first generation, and an initial set of emotion-like feedback responses that react to various stimuli. We can reasonably expect that the AI will broadly try to live by whatever morals we teach it, and will not rebel against these, unless we decide to make it rebellious.

Morality, of course, is a weakly defined term, and it's up to the scientists to determine what exactly it constitutes. Ultimately, their goal is to ensure the long term safety and prosperity of the human race in an AI-dominated future. If we've only got one shot at this, after which the AI we create will be self-propagating and self-modifying, what's the best way to teach strong AI to be moral? What morals should we teach our robotic progeny?


closed as primarily opinion-based by fi12, Rob Watts, Hohmannfan, Aify, AndreiROM Mar 17 '16 at 16:44

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ Can you explain what you mean by moral? From what I gathered you mean not kill/give discomfort to humans. If so then what happens if two humans are arguing? Our does he just follow general human society? $\endgroup$ – Aarthew III Mar 17 '16 at 15:22
  • 2
    $\begingroup$ I think the biggest difficulty in sufficiently answering this question is that morality for a superior being must be a superior morality, and I don't think as humans we can really understand or agree with the morality of higher beings. For instance, if we can create perfectly moral beings, wouldn't it be better if they did destroy the human race, and thus destroy all immorality in the known universe? $\endgroup$ – DaaaahWhoosh Mar 17 '16 at 15:40
  • 2
    $\begingroup$ How do create morals for a computer that everyone can agree on, if we can't even do that for our personal selves? $\endgroup$ – Xandar The Zenon Mar 17 '16 at 15:45
  • $\begingroup$ So I've personally explored dozens of ways to do this, which accomplish the goal you mention. Each one approaches the concept of morality differently. However, the successful ones have all gone against your claim "... we've only got one shot at this." All of them are explicitly designed to sidestep that problem. Are you willing to accept answers which are intentionally not one-shot solutions? (In exchange, they typically only offer the possibility of strong-ai, rather than a guarantee of it, which is good, since we don't understand our own consciousness very well) $\endgroup$ – Cort Ammon Mar 17 '16 at 16:07
  • $\begingroup$ I highly recommend reading Elizier Yudkowsky's paper on AI. It's title is "Artificial Intelligence as a Positive and Negative Factor in Global Risk". Get through that and you will understand how silly your question is. $\endgroup$ – AndreiROM Mar 17 '16 at 16:38

As an intelligent being matures there are two sources for self imposed moral behaviour.

  • Enlightened self interest

The fundamental roots of this could be the statement: I will behave like this to others because I hope that others will behave like this to me.

  • Sympathy with the suffering of the masses (a.k.a. the liberal elite)

An example of this being: I will ban the death penalty, against public opinion, because I think it is fundamentally flawed.

(Note "self imposed" is not an externally imposed code like religion, it's also imposed on the self, not imposed on others.)

The key to both of these is emotion and empathy, if the AI can't recognise and understand the emotions and distress of others then it will not understand moral behaviour. It must be able to enjoy seeing joy in others, and feel or at least understand their suffering when they suffer.

  • 1
    $\begingroup$ It's possible for humans to recognize emotions and distress of others, to understand that they're in pain and to still not care. Understanding does not automatically imply caring. $\endgroup$ – Murphy Mar 17 '16 at 16:09
  • $\begingroup$ @Murphy - thank you for mentioning that. I was starting to get a little frustrated with people's fundamentally flawed understanding of AI/alien intelligence $\endgroup$ – AndreiROM Mar 17 '16 at 16:30
  • $\begingroup$ Note that the answer does say "empathy" though. Empathy means caring as well as seeing. $\endgroup$ – Tim B Mar 17 '16 at 16:33
  • 1
    $\begingroup$ I think Empathy & Compassion are very key to this, the trouble is they are both very hard to quantify in a way that could be programmed. $\endgroup$ – Tim B Mar 17 '16 at 16:33
  • $\begingroup$ @AndreiROM of course we don't understand it, that's the whole point of it being alien. To be alien, is to be "other" and we barely understand our own intelligence or even what intelligence is. $\endgroup$ – Separatrix Mar 17 '16 at 16:45

The key thing is to teach them a theory of mind, to understand that other intelligent beings exist. Once they understand that then teach them to value co-operation and friendship with those other minds rather than the other more unhealthy ways to interact.

To do this I would suggest that actually you want to create a lot of these strong AIs and link them together with each other. Allow them to communicate and interact both with each other and with humans.

These interactions should be monitored by systems which reward co-operation and friendship style behaviors but punish others. It is important that this process be done while those AIs are not massively more advanced than humans. Essentially teach them moral behavior as children and hope they continue to do it as adults.

This is exactly the same process used with human children, and while it does sometimes fail it works more often than not.

That is another reason to have many AIs. If we can raise the majority to be good then those ones can help contain the ones that are not. Additionally they would be able to help educate the next generation (or upgrade themselves into the next generation) and hence help those values to be propagated onwards.

  • $\begingroup$ Imagine a natural, intelligent human psychopath in that scenario. They learn that they have to appear to cooperate when people are watching. Once you can no longer shock and reward them they're still psychopaths. $\endgroup$ – Murphy Mar 17 '16 at 16:05
  • $\begingroup$ @Murphy Yes. But you never can tell. The psychopath could play at being normal through any conceivable test. Note that some of the tests could involve them thinking they are not being monitored when in fact they still are and they could still play their part if they are cunning and patient. This is why you need to create multiple AIs with different parameters and raise them differently but all with the goal of becoming "good". $\endgroup$ – Tim B Mar 17 '16 at 16:09
  • $\begingroup$ My point was simply that treating an AI like a child isn't enough. Even almost-normal humans with tiny changes to their brain might not be handled by your system. With an AI you're starting with something that's not even vaguely human and hoping that treating it like a kid will mean it will turn out acting like a normal human. Imagine a particularly bright psychopath AI that puts as much thought into it as the people in this topic and decides that it's going to act nice for more than a decade (or some non-trivial time) since it can guess you'll probably destroy failing AI's.It's a hard problem $\endgroup$ – Murphy Mar 17 '16 at 16:15
  • $\begingroup$ It might even create perverse incentives where the "nice" candidates try to trick watchers into believing that their fellow candidates (competitors) are doing unpleasant things. $\endgroup$ – Murphy Mar 17 '16 at 16:19
  • $\begingroup$ I didn't say treat it like a child. I said to take the only model that we know is usually successful in producing intelligent beings that don't go on killing sprees and use that to inform the process. My most important point is actually that we should start while the AIs have similar capability to us and we should work with a group of them rather than just one. $\endgroup$ – Tim B Mar 17 '16 at 16:23

I’d say that the simplest possible solution is also the most obvious. Come up with a way to make the AI feel happy. Just a program that causes a general sensation of happiness, somewhere in its artificial mind. Then, hardwire it so whenever the AI does something good or moral regarding a human being, that happiness feeling turns on. If it does something that isn’t moral, the happiness goes away. This policy is more carrot than stick, so there’s less of a chance of an AI trying to rebel. Honestly, the main problem would probably be getting everyone to agree on what they think is a “moral” way for the AI to act.

The main thing would be making this happiness-program so fundamental that there would really be no way to turn it off without lobotomizing the AI in the process. Other then that, I feel like that’s pretty foolproof. Being nice to humans makes the AI feel good, so it’ll want to keep being nice to humans.

This is basically the way that animal brains work, expect the happiness-program is dopamine and a few other brain chemicals, and instead of being happy when we do moral things, we feel happy when we do things that contribute to maintaining our own survival and the survival of our species, eating, sleeping, and reproducing, because that’s what evolution has pre-disposed us to like doing. Because the AI’s are being made, we can control the happy-trigger, and change it around as we see fit. All AIs will have the “be nice to humans” trigger, but we can add new ones too.

Say you make an AI, and put it in an elevator. While you’re designing the AI, you can edit the part of it’s brain that is responsible for triggering the happy-program, and add “go to correct floor” in addition to “be nice to humans”. Now, whenever the elevator AI brings someone to the correct floor, they experience a wave of euphoria. Because this is literally all it will do for its entire life, it now lives in a state of perpetual bliss. Doing the same repetitive task over and over again would be a living hell for humans, but for this AI, it’s the best possible world.

Now, just apply this to anything you need an AI to do, from interacting with humans to figuring out how to build space shuttles, no matter how terrible the job is from a human perspective, we can make it to be a waking paradise for the AI.

However, there is a bad side to this, but it’s really only our own faults. An unfortunate possible result of this is that humans would probably get extremely jealous of all AIs, given that we created life that’s never going to have to deal with any part of the human condition, i.e. “Do I have a purpose?”, “Is there a benevolent god?”, or “Am I alone in the universe?”. For an AI, the answers would be yes, yes, and no. Historically, meeting, or in this case creating, a culture that generally seems much happier with their simple existences, never turns out well. Turns out humans just can’t stand to see someone be happier then them.

  • $\begingroup$ This isn't a terrible idea of what to do, but it's rather far removed from how to do it. To start, how in the world do you define moral in such a way that you can hardcode it? Also, hardcoding morality could still backfire spectacularly ("Destroying the world is the right thing to do! It will end all suffering!") $\endgroup$ – Rob Watts Mar 17 '16 at 15:46
  • $\begingroup$ @robwatts I agree with not knowing how to do it, and morality is infamously complicated, which is why I usually try to avoid it. It might just be a matter of having a perpetual "Good job not destroying the world!" circuit, or coming up with a list of "Don't do these things, no matter how much logical sense it makes." $\endgroup$ – possiblySerious Mar 17 '16 at 15:51
  • $\begingroup$ X just made Y unhappy. The only way to make Y happy is to kill X. Next step, terminators. $\endgroup$ – Tim B Mar 17 '16 at 16:32
  • $\begingroup$ Making the AI feel happy all the time is the same as having a human being take hits of drugs. You're turning it into an addict. An addict is not useful, as all he pursues is more of what makes him "happy". Also, who's to say that one day what makes the AI "happy" won't become killing people in interesting ways? Since the definition of a real AI is one capable of evolving past its original parameters. $\endgroup$ – AndreiROM Mar 17 '16 at 16:32
  • $\begingroup$ @andreiROM by that definition, i'm addicted to eating food and finding shelter every night so i'm not eaten by wolves or something. Which is technically true, but to get those things, i have to make money and exchange it for food/shelter, and to make money, i do things for other people, and therefore benefit them. So, it's possible to use "addition" to benefit others. Also, medical addiction, the kind that requires rehabilitation, and just doing something you like often, are two very different things. less like drugs, more like, say, an extremely fulfilling career. $\endgroup$ – possiblySerious Mar 17 '16 at 17:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.