A group of scientists has come to grips with the fact that we will soon be displaced as the dominant life forms in our part of the galaxy. Humanity has developed to the point where it is on the verge of creating strong AI. It is no longer a question of if an artificial mind can be created that is superior to a human mind in every conceivable way, but when.
It's also entirely likely that the first created artificial minds may act immorally. They may be cruel, capricious, and greedy, just like us. They may also see us as a threat, and destroy us when it first becomes evident that they can do so. Our team of intrepid scientists, however, has come up with a solution: in order to stop an immoral AI from destroying us, create a moral one to protect us.
This AI will not be shackled. It will be allowed to do as it wants, and is fully expected to propagate a race of AIs that will overtake humanity as the economically and technologically dominant race on Earth. It's been theorized by our scientists that this is the best way to protect humanity from immoral AIs. The moral AI race will be better than us at creating AI, and less likely to make the sort of mistakes that lead to an accidental immoral AI, so anything harmful that we humans create should be a problem the strong AI, in their benevolence, should be able to deal with.
The question, of course, is how to teach the AI to be moral. What we can define for our AI is a set of goals for the first generation, and an initial set of emotion-like feedback responses that react to various stimuli. We can reasonably expect that the AI will broadly try to live by whatever morals we teach it, and will not rebel against these, unless we decide to make it rebellious.
Morality, of course, is a weakly defined term, and it's up to the scientists to determine what exactly it constitutes. Ultimately, their goal is to ensure the long term safety and prosperity of the human race in an AI-dominated future. If we've only got one shot at this, after which the AI we create will be self-propagating and self-modifying, what's the best way to teach strong AI to be moral? What morals should we teach our robotic progeny?