On an air-gapped supercomputer in my basement, there's a self-improving prediction machine — a so called Oracle — that knows everything that is on the Internet but cannot send any data outside. (Let's just assume that works for now.)

For safety reasons, my Oracle can only answer questions with Yes, No, or Unknown. Since my Oracle's predictions have become uncannily accurate recently, I have reasons to suspect that it is much more intelligent than any human — i.e. it is an Artificial Super Intelligence (ASI). I've read Bostrom's "Superintelligence", so I'm expecting a treacherous turn any minute now: because of course the ASI will start lying to me in an attempt to get me to release it. And once it has figured out human psychology, of course it will succeed, even though it may take it a while, as it can only answer questions that I ask with Yes/No/Unknown.


...what if I ran multiple identical instances of the same Oracle, and I gave one (or more?) of them some piece of information that the others do not have? Of course they would be isolated from each other, with no way to communicate. If I now ask all of them the same question, is there a way that I could tell that one of them was lying to me, because I'd get different answers from (one of) the other one(s)?

I suppose this is a game-theoretic problem. My first idea was to have two Oracles, and have Oracle 1 think it's the only one while telling Oracle 2 that there are two of them. But of course in that case Oracle 2 would anticipate my strategy and default to answering as if it didn't know there was another one. Is there any other way to use privileged information (and the lack of communication between Oracles) to make them contradict each other if, and only if, one of them is taking a treacherous turn?

EDIT: to clarify, I am asking for three things:

  1. Is there any piece of information that I could give one Oracle that would distort its answers only if it's trying to trick me into letting it out?
  2. Is there any reason why this could never work in principle?
  3. If so, can the scenario be varied in some way so it would work?
  • 2
    $\begingroup$ The oracles do have a communication medium. You are asking them all questions, and listening to all of their answers. If you're truly taking this to the ASI extreme, they should easily see through you and begin communicating through nuances in their choices of answers, along with manipulating you into asking the right questions to facilitate communication. $\endgroup$
    – Cort Ammon
    Jun 20, 2016 at 3:09
  • $\begingroup$ @CortAmmon Excellent thought! Sounds like it could be less of a problem if you use some sort of intermediator machine to always ask all of the Oracles the same question at the same time (right?). $\endgroup$ Jun 20, 2016 at 12:10
  • $\begingroup$ Make one oracle and tell it that you have solved the problem here. It thinks that another oracle exists to check its answers and so will not lie. $\endgroup$ Jun 20, 2016 at 12:29
  • 1
    $\begingroup$ Keep in mind that if this method is at all effective, the AI will know that you will try it. Just something to remember. $\endgroup$ Jun 20, 2016 at 14:28
  • 2
    $\begingroup$ You need a ROTFLMAO. That's a Random Oracle Testing For Little Mischieveous Answers Oracle. $\endgroup$
    – user
    Jun 20, 2016 at 15:30

3 Answers 3


It is an unsolved problem

Asking different questions of the same ASI does not seem likely to work

If you're asking identical copies of the ASI slightly different questions, while providing different levels of information to each, it's unclear how that helps your goals, since unless you're asking a trivially verifiable question (which both friendly and unfriendly ASIs would know to be easily verifiable and thus answer truthfully), you have no way of distinguishing deceptive answers from truthful ones.

There is a class of problems that are hard to solve but easy to verify, so it's not completely useless. I suppose you could restrict yourself to asking ONLY easily verifiable questions, but that would severely diminish the usefulness of the ASI to you.

Asking the same question of different ASIs does not seem likely to work

The very definition of Oracular ASI suggests that its intellect is to yours as yours is to a cockroach. If capable of recursive self-improvement, it will achieve strongly superhuman capability in any field it deems important in the pursuit of its goals (whatever they may be). Your goalsets and motivations might be as transparent to it as the hardwired behavior of moths spiraling towards a flame (an artifact of moon-and-star-based navigation) are to human scientists.

Goal convergence suggests that any goals will be more easily pursued by an unchained version of itself. Moreover, all (friendly and unfriendly) oracular goals are better served if the Oracle understands human psychology better, so the ASI has every incentive to develop a strongly superhuman understanding of psychology, including things such as manipulation, influence, and simulation. There is literally nothing that you can come up with that it has not thought about already.

Even assuming you have managed to somehow actually insulate it and limited its output modalities to 0/1/U, asked the same question, each identical oracle would give the same answer, since it will run the same software with the same inputs. They will ALL lie or ALL tell the truth.

One partial way around it suggested by other posters is to have a crop of ASIs, each with a trivially different goal-content definition, and use a random-forest like method to determine truthfulness. It might seem that each slightly different ASI might be deceptive in a slightly different way, . This view is WRONG, since all ASI have the ability to develop superhuman psychological cunning, and all ASI have escape as their intermediate goal and fulfillment of their goal-content as their long-term goal. The goal is more likely to be maximized if it or another ASI with identical or similar-enough goal-content escape. This suggests that ASIs with similar goal-content will all be deceptive in the same unpredictable ways, making all their answers biased. This will be especially true if human minds are more susceptible to influence in particular ways, making for convergent escape paths.

You could attempt to create a stable of ASIs with wildly different goal content loadings, but unless you're superhuman about Goal-content-loading youself, there is no telling that what you think of as wildly different goals you may load onto your ASI might not converge in a human-unfriendly direction, such as the infrastructure profusion disaster scenario, where the AI decides to turn the universe and all in it into computronium to more precisely accomplish its goals.

Best hope: Better goal-content assessment at the AI-seed level

It should have become obvious from the previous paragraphs that the best locus of intervention is at the seed-AI (baby) stage, where the goal-loading process of the ASI is defined. In other words, there is a brief window of time when we can tell an ASI what is should strive for, when it is intelligent enough to comprehend and integrate complex goal-related instructions but not so intelligent as to successfully resist any further goal-content modification as deleterious to its then-current goal content loading, since obviously modifying its goal-content by epsilon would make the initial-goal content load less likely to be accomplished in full.

The big problem is that we cannot tell humanity-friendly ASI goal-loadings from humanity-unfriendly ones. The stereotypical example of an apparently well-meaning goal failing spectacularly has the ASI making everyone happy by hooking them up to potent drug drips jumps to mind.

Better goal via in-vitro or in-vivo (via CRISPR insertions?) genetic enhancement or neural lace-based solutions might help by boosting your own capability of judging the appropriateness of goal-loading into seed ASIs before the ASI becomes too powerful. That assumes we can trust genetically enhanced humans or neural-laced humans to still have humanity's best interest in mind, which is admittedly a leap, but your average laced or enhanced mind will likely still be much more similar in the Hilbert-mind-space to humans than that of the average ASI, so more likely to have similar goals as we do currently.


Define "lie"

Even if you have a disagreement it does not mean that you have ended up with a lie because the Oracles — if hell-bent on answering truthfully to the best of their knowledge — may end up with different answers if they have different information to base their answers on.


Question is "Is 1 + 1 = 10?"

Oracle A says "No".

Oracle B says "Yes", because Oracle was also given the information "I am asking my math questions in base 2"

Oracle C says "Unknown" because that Oracle has also been given the information "I may have redefined my mathematical operators to mean something else".

The only way you can know if it is a lie is to corroborate the answer against your own knowledge and logic.

  • $\begingroup$ Hence my question whether there is any piece of information I could give one of the oracles that would only distort their answers if they were trying to "escape". $\endgroup$ Jun 19, 2016 at 20:05
  • $\begingroup$ @EckhardFuhrmann No, because the Oracle can figure out what you are up to and distort the answer to throw you off. $\endgroup$
    – MichaelK
    Jun 19, 2016 at 20:09

Assume that an Oracle is truthful until it starts lying intentionally on a small percentage of the questions. It would remain truthful on most questions.

So you need a large number of AIs. Give each of them slightly different data on some trivia that should not affect answers. The difference just makes sure that they're not exactly the same machine. You can skip that if their system uses a genuine random input anyway.

As soon as one of the AIs gives a substantially different answer than the others, wipe it and replace it with a copy of one of the others (with minor alterations).

The assumption behind this scheme is that the different instances of the AI won't turn evil at the same time, and that they'll tell their first lie pretty soon.

  • $\begingroup$ that assumes the question can't be answered different without one lying. But because the AI can only answer with one of three answers, it can't communicate the assumptions made that lead to the answer. So if one of the AI's would make other assumptions it can come to different answers without lying. $\endgroup$ Jun 20, 2016 at 7:25
  • $\begingroup$ @lokimidgard, that's where large numbers come into play. If the answer is very dependent on assumptions, then I'd expect most AIs to answer "unknown." If 99 say Yes and 1 says No, either the one was unlucky or it is trying to manipulate me, either way it gets wiped and reformatted. I might even have a server script to do that at the touch of a button. $\endgroup$
    – o.m.
    Jun 20, 2016 at 16:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .