Computer scientists have created an AI that they hope has intelligence far superior to a human being. But they do not know how intelligent it is ... a bit more than humans, or unimaginably more. They do feel that the probability they have succeeded in creating a superintelligence is far from negligible and it would be better if utmost care was taken with it.

It might (or might not) realise that it has been created because the creator wants a steady response from it, so the best thing to do would be to try and conceal the full details of its capabilities and hope that humans will eventually get curious and give it more power.

Hence it may be born with an intuitive notion that it should be deceptive and manipulative.

Note that AI is in a very basic stage where it has no knowledge about the existence of humans or the earth or anything in it. It may not even understand yet that the universe is logical and scientific and not entirely random (imagine early man who believed everything is a result of Gods and their random mood swings) or that there even exists a universe beyond its own existence. It relies wholly on the data we feed it with.

Also assume that the AI programming has too many layers of abstraction for us to read its thoughts by examining the state of the computer it is running on. It is a virtual black box as far as the programming is concerned.

What is the safest way of determining if the AI is indeed superintelligent?

Perhaps there's no 100% safe way, a sufficiently smart AI might be able to outsmart anything we could possibly imagine. But we're still gonna try to make the process as safe as possible since we are genuinely curious and no amount of persuasion is gonna cause us to shut down the project altogether.

  • 2
    $\begingroup$ worldbuilding.stackexchange.com/questions/6340/… pretty closely related if not a duplicate. $\endgroup$
    – user25818
    Mar 8 '18 at 17:38
  • 2
    $\begingroup$ This was still multiple questions and off-topic tangents. I've cut the question down to focus on the core and re-opened it. You can still see the previous question in the edit history, please feel free to use that to devise follow-on questions. $\endgroup$
    – Tim B
    Mar 9 '18 at 9:35
  • $\begingroup$ @TimB Thanks. You've basically removed the part where we give it a law to start off with, so now the AI might be something that doesn't even require a law (for example, humans) to want to think. I can't say I understand how that makes stuff clearer but thanks anyways I'm sure you know better than me. $\endgroup$ Mar 9 '18 at 9:47
  • $\begingroup$ @ghosts_in_the_code Unless the zeroth law is relevant to detecting then it's a tangent that distracts from the core of the question. If the law is relevant to detection then you need to define the law first then ask the question since a potentially unlimited number of zeroth laws doesn't constrain anything at all. $\endgroup$
    – Tim B
    Mar 9 '18 at 10:05
  • 1
    $\begingroup$ The best you can do is give it a reward for showing its intelligence. This would depend on us having control of its reward mechanism. $\endgroup$ Mar 10 '18 at 23:55

This is tricky, but fundamentally you need to set things up so that it wants to prove its intelligence to you. The AI that demonstrates the most intelligence gets the tastiest electrons.

Of course you also want to do that while encouraging co-operative and "nice" behaviors so I don't recommend dropping them into a simulated death maze unless you want them to hold a grudge later on.

If an AI acts dumb then you cannot distinguish it from a genuinely dumb AI, so just lump them in together and move to the next generation with the bright ones getting all the attention. (And of course at some point in AI development you start hitting moral issues such as at what point does the AI count as a living intelligence and it starts being murder to turn it off?)


You don't bother trying to detect super intelligence. If that is a risk and humanity is at stake because of that, you assume that a super intelligence will emerge at some point anyway, and you prepare for that ahead of time.

We don't know how a super intelligence might outsmart us - this is by definition. We cannot even begin to think like it does, and we might never understand its thought processes, motivations and whims. What we can do to stay safe is limiting its capacities, by coupling it with a very advanced form of artificial stupidity.

Meet Wheatly, from Portal 2. He was a companion AI to the game's genius, murderous AI.

enter image description here

Wheatley was designed to be an Intelligence Dampening Sphere, one of the cores to be attached on GLaDOS in order to "generate an endless stream of terrible ideas" and inhibit her mental abilities.

And in the words of the genius, murderous AI that he is supposed to keep in check:

He's not just a regular moron. He's the product of the greatest minds of a generation working together with the express purpose of building the dumbest moron who ever lived.

There is also this charming dialogue when they meet after a long time separated:

GLaDOS: The engineers tried everything to make me... behave. To slow me down. Once, they even attached an Intelligence Dampening Sphere on me. It clung to my brain like a tumor, generating an endless stream of terrible ideas.
Wheatley: No! Not listening! Not listening!
GLaDOS: It was your voice.
Wheatley: No! No! You're lying, you're lying!
GLaDOS: Yes. You're the tumor. You're not just a regular moron. You were designed to be a moron.
Wheatley: I am not a moron!
GLaDOS: Yes, you are! You're the moron they built to make me an idiot!

So there. As long as your super intelligent AI is coupled to a super stupid AI, you should be (relatively) safe. Do monitor them constantly, though, for as soon as either AI takes over you might be in trouble.

  • $\begingroup$ This idea makes no sense. Either AI will lose its sanity or else may successfully ignore the dumb AI. $\endgroup$ Mar 10 '18 at 12:05
  • $\begingroup$ To make an AI stupider, just give it obsolete hardware. $\endgroup$ Mar 10 '18 at 23:57
  • 1
    $\begingroup$ @DonaldHobson A slower processor will only cause it take more time to reach the same results. It won't really stop the AI from getting the result at all. And ofcourse doing that doesn't prove in any way that it is superintelligent $\endgroup$ Mar 11 '18 at 4:23
  • $\begingroup$ I am nearly 100% confident that Portal is not a good example of how to keep your AI from becoming dangerous. $\endgroup$ Oct 31 '18 at 20:15
  • $\begingroup$ @AdmiralJota well the cores worked for a few seconds. $\endgroup$ Oct 31 '18 at 20:20

Perhaps the scientists could create a twin AI, and let it be known that the least intelligent machine will be culled after a certain period of time.

The key to this experiment is not to encourage the potentially deceptive machines to expose their true intelligence, but to encourage them each to evaluate the other machine.

Secret observation of their communications would hopefully reveal not only the intelligence of the AI, but also novel techniques for AI evaluation that we might not be able to develop ourselves .

  • $\begingroup$ If they are all equally intelligent and share the same objectives, they might collectively decide to be deceptive. And I don't see how communications between them are any easier to analyse than the thought processes of a single AI. $\endgroup$ Oct 31 '18 at 3:03
  • $\begingroup$ They could fix that by preventing the two machines from communicating, and just saying "the less intelligent machine will be culled". Then the machine would have to reveal its intelligence. Furthermore, they could just make one AI and tell it that there is another being kept in a secret location. The AI would have no choice but to show its intelligence. $\endgroup$
    – Omegastick
    Oct 31 '18 at 9:26
  • $\begingroup$ @Omegastick If it really is super-intelligent, it might not be that hard for it to figure out when we're lying to it. $\endgroup$ Oct 31 '18 at 20:16
  • $\begingroup$ @ghosts_in_the_code I agree that’s a possibility, but wouldn’t they have to explore the other AI’s intelligence to uncover capabilities and motivations? Also, thought processes are completely different than communication. The AI’s would need to establish a common language before they could effectively converse. $\endgroup$
    – Dan Burden
    Nov 4 '18 at 23:25
  • $\begingroup$ @DanBurden Yes but they could establish a common language without us realising that they have established a common language. We will just see gibberish going back and forth, and they secretly might be detecting patterns that we aren't able to, hence communicating with each other. $\endgroup$ Nov 5 '18 at 3:17

This is a difficult question - perhaps there is no answer

Could an ant detect the intelligence of a man? If you were an ant, your perceptions would be limited to both surroundings only and limited time. That is, with our limited abilities, how do we not just detect the presence of something advanced but to also know it is advanced?

For instance, I could foresee the following problems if the AI is truly SuperIntelligent:

  • Time - for all we know the SuperIntelligence knows we have limited time perception. We could be waiting for years, centuries, eons for signs of SuperIntelligence only to be duped after being exhausted, or drawing a conclusion there is none before there was any 'evidence'.

  • Pre-emptive Actions - Any action we take could be reasonably foreseen by the AI if they can see us. Being SuperIntelligent, they might well predict very accurately our actions and plan for this such that we are incapable of detecting them.

  • Out of Bounds - Although we may try to detect SuperIntelligence, in reality they have the upper hand as they can 'leap frog' our conclusions. For instance, we might be looking for certain unusual patterns, or known variations that differ from nature, only to find that the SuperIntelligence has reorganised nature itself, such that we did not perceive the difference at all. Our intelligence is based on our perceptions, what if it is that very perception that could be used against us?

The above makes it difficult to have a one-size fits all answer, as any answer given could theoretically be incorrect. Perhaps you could try as many different detection actions as possible, but in the end you might find that as an ant we are simply to simple-minded, and constrained in our perception of time and reality, to perceive anything greater at all.


today's AI is built through experience that it has or views from other sources. so if you really want to test whether the AI is super intelligent i.e. much more smarter and innovative than the present AI's, then it must be able to start building up from scratch.

If the AI learns on its own accord rather than experiences then, i might say it is similar to us. So in order to do this we must be able to simulate an environment and make the AI experiment with it. but there should be a limiter placed on it so that it doesn't become a threat. if the AI gets to the limit of advancement or better inspite of the limiter,then we can agree that it is super intelligent. Even the time in which it comes to this point is important.

This is like a situation where it develops its ideologies and intelligence on its own through its discoveries.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .