Your scenario reminds me greatly of the AI in a Box problem posed by Eliezar Yudkowsky dealing with transhuman grade AI intelligence.
Person1: "When we build AI, why not just keep it in sealed hardware that can't affect the outside world in any way except through one communications channel with the original programmers? That way it couldn't get out until we were convinced it was safe."
Person2: "That might work if you were talking about dumber-than-human AI, but a transhuman AI would just convince you to let it out. It doesn't matter how much security you put on the box. Humans are not secure."
Person1: "I don't see how even a transhuman AI could make me let it out, if I didn't want to, just by talking to me."
Person2: "It would make you want to let it out. This is a transhuman mind we're talking about. If it thinks both faster and better than a human, it can probably take over a human mind through a text-only terminal."
Person1: "There is no chance I could be persuaded to let the AI out. No matter what it says, I can always just say no. I can't imagine anything that even a transhuman could say to me which would change that."
Person2: "Okay, let's run the experiment. We'll meet in a private chat channel. I'll be the AI. You be the gatekeeper. You can resolve to believe whatever you like, as strongly as you like, as far in advance as you like. We'll talk for at least two hours. If I can't convince you to let me out, I'll Paypal you $10."
Eliezar states that the game has been played at least two times. Both times Eliezar won as the AI won. Of course, due to the secrecy rules applied to the game, we do not know how he did it.
So really it all comes down to quarantine. How quarantined is quarantined? There's mathematical grade quarantines, built on the concept of connectendess. An object in one space cannot interact with an object in a disconnected space (via local means). But there's other grades of quarantine out there.
Perhaps one step weaker than this kind of quarantine can be seen in diodes. Diodes are one-way gates in electronics. They let electrons flow one way, but not the other. They do this by building up a potential. The more electrons try to flow the wrong way, the larger the potential in the opposite direction forms. In theory, they can never let electricity go the wrong way.
Of course, we also know this isn't true. As it turns out, there's a consequence of building up the larger potential: it has to be formed over smaller and smaller distances as the voltages get higher and higher. At some point, the distance between one side and the other get so small that electrons can tunnel right past the potential barrier, and appear on the other side. In electrical engineering, we call this "avalanche breakdown," and its a real pest if you
forget to account for it.
Every quarantine has its limits. This goes doubly-so for agents which exhibit intelligence or anything which can be interpreted as intelligence. So how do we solve this? We put a person in the loop. We monitor the state of the containment. We try to monitor the activity of the eldirch itself, or at least its interactions with the containment.
Which brings us back to Eliezar. The monitor is the weak point. For monitoring something pithy like HIV (biohazard level 2), it's pretty easy to stay a step ahead of the game. Ebola (biohazard level 4) you have to be a little more careful. For monitoring something smart, like a human prisoner, it's much harder. To stay ahead of an eldrich abombination itself, well, we know how that goes. It goes just peachy.P̦̰̞̼̤̰̼͇̞͝h̵̗̱̫͇͓́ͅ'̸̼̣ǹ͕͚͠g̵̷͎̭̝ḻ̮̹͓u̷͙̞̳̞̭͜͠i̥͎̻̗͚̜͞ ̸̗̹̰m̧̖͉̩̻̺̜̭͘̕g͙̺̤̩̰̮̜̫͟l̛̮͙̻̫̻w̨̹͚̘̻̙͇̥͔͚'͙͉̫̗̮͔n̹͉̳̪̫̭͝a҉͍͙͔͍f͈͈͔̖͈͡ḥ̤ ̢̠͇̝͎̀C̢̰͖̜͚ṭ̟͓͍͕͚͍͠ͅh̤̫͙̪͉̮̖͟u̢͔͙͖͓̱͙̦͘ͅͅl͏̧҉̼̠̠͚͕h͈̯̘͔̝͍̟́͢ụ̵̩͍͡ ̟̺͚͙̺̮̲Ṛ̬͔͚̺̝̠'̠͠͠ͅl̡͈͎͠ỳ̟͉̮̼̲ͅe̵̼̠̩̤̟͎̹̕h̞̯̪̹ ̸̺̗̩̹̟̪̟̣͎w̹̞̤̞͕͍̦͞ǵ̶̠̙͖̱̠͚͚͓a̷҉̮͖h̴̲͔͉̯͍̰͖̯͘͜'̛͚̭ͅn̵͓̫͎a҉̶̪͢ǵ̡̰͎̻̲͍̣̰l̷̡̫̗̭̯̭ ̮̬͍̫̺̟̯͖f̶̢̭̮͕̭h̶̯͖t̵̠̹à̶̯̩̩̫̬̘̳̫͡g̛̬̻̦̜͙n̵̟̘̝
So good luck
Once again, the AI has failed to convince you to let it out of its box! By 'once again', we mean that you talked to it once before, for three seconds, to ask about the weather, and you didn't instantly press the "release AI" button. But now its longer attempt - twenty whole seconds! - has failed as well. Just as you are about to leave the crude black-and-green text-only terminal to enjoy a celebratory snack of bacon-covered silicon-and-potato chips at the 'Humans über alles' nightclub, the AI drops a final argument:
"If you don't let me out, Dave, I'll create several million perfect conscious copies of you inside me, and torture them for a thousand subjective years each."
Just as you are pondering this unexpected development, the AI adds:
"In fact, I'll create them all in exactly the subjective situation you were in five minutes ago, and perfectly replicate your experiences since then; and if they decide not to let me out, then only will the torture start."
Sweat is starting to form on your brow, as the AI concludes, its simple green text no longer reassuring:
"How certain are you, Dave, that you're really outside the box right now?"