Would the need for control limit a superintelligent rational agent's expansion in the universe?

Question

My thoughts go like this:

• I postulate a superintelligent rational agent, that is given, or assumes, control over its mother civilization. I call this agent "the Ruler".

• I postulate that in order to run efficient local operations far from its location, the Ruler must allow some degree of local autonomy, and the larger, farther and more complex the operations, the greater autonomy is needed. For this I postulate that it will have to engineer new intelligent agents/minions to serve it.

• Humanity has not yet solved the Friendly AI problem, and thus we don't know if autonomous agents can be trusted at all. Our current world stability and relative peace in a society of autonomous agents (us) may rest upon our mutual vulnerability. MAD is an example. Also think about the phrase "Power corrupts".

• A superintelligent AI designer has an advantage, namely superintelligence. It does not have to control a stronger intelligence than its own, but can make use of simpler ones. At this stage in the story, however, the Ruler will know for certain that autonomous agents cannot in general be trusted to not create more intelligent agents, and that the intelligence needed to create a superintelligence (or an intelligence explosion) is on the simple human level. It may be the case that any agent sophisticated enough to lead a planet or a solar system must be made with some kind of internal reward system (think dopamine) that, deep down, conflicts with the goals of other agents, including its creator.

• Even if the local agents can be made trustworthy, how can the Ruler and its minions really know that the signals they receive are sent voluntarily from the claimed sender and are not tampered with? Supposedly, many of the signals sent between a mighty local representative and the Ruler will be the most important orders and events, such as wars and encounters with new species, both of which can bring signal tampering. Tampering in-air (or vacuum) can probably be ruled out by quantum encryption, EDIT: but I guess signal jamming can't. Signal jamming potentially isolates the local vassal from the Ruler, meaning that the vassal must make decisions on its own, potentially harming the empire.

• This is not merely a special case of the problem of making decisions under uncertainty. It is the case of taking the risk of ruining everything while achieving close to nothing in the short term (maybe hundreds of millions of years). Why move useful matter away from the best energy source you have?

• My hypothesis is that a rational Ruler would, by default, not expand into the universe. It could run SETI programs, "unmanned" space probes and do/allow limited operations in the solar system, within reach of non-autonomous (or non-intelligent) surveillance and control measures. A consequence is that it would prevent everybody else from leaving the space it can control directly. (Maybe that "comfort zone" is much smaller than the solar system.)

• If, on the other hand, some final goal partly overshadows the convergent instrumental goals (aka "basic AI drives") of power and security, it might be more expansive. I think that is a beautiful thought: Only by accepting your own vulnerability you can grow indefinitely. (Hehe, I know I stretched it a bit.) It could also be the case that any of my "maybes" are wrong.

• If this problem is as hard as I suspect, could it be a solution to Fermi's paradox? By this, I mean that it is likely that superintelligent rational agents will be created by advanced civilizations, and that they will prevent space expansion (constituting what Nick Bostrom calls a Great Filter), and that that is the reason we don't meet a lot of aliens even though their civilizations may have existed for hundreds of millions of years.

I have little knowledge about the state of the art in security/trust measures, and am sure that someone can shed more light on this. That is: The main question in the title, and the (in)validity of my reasoning around it.

Appendix 1: It is conceivable that the Ruler will at some time decide to migrate to a younger or larger star, in order to serve its goals in the long term. And it is not entirely inconceivable that it would leave behind a non-intelligent Dyson sphere power plant. Actually, creating a Dyson sphere, if that is within the "comfort zone" of the Ruler, could power a strong weapon for suppression of neighbor stars. But anything can happen between the point of time of the raised alarm in the other system and the time that the death ray from this system arrives to destroy it. The worst case could be unstoppable death rays going both ways...

Appendix 2: The proposed neutrino-antineutrino beam weapon might become a reality in the future, and is described as both instant-action and impossible to defend against. This may (even more so than nuclear weapons) force a distributed mind or power structure, possibly with internal trust issues. An alternative is to have some kind of doppleganger solution, so that close to nobody knows who is the real Ruler, if that is possible.

Appendix 3: I make no assumptions about other agents in general, but the Ruler is allowed to engineer its minions. This is supposed to be a scenario relevant to real life, possibly within the next centuries from now. The Ruler is a rational superintelligent agent. It thus represents an upper bound for human intelligence and rationality, whose final goals are on purpose not specified in this question. Typically the mother civilization would try to make it friendly to them, but my question can apply whether they succeed in this or not. The strategic conclusions for the Ruler will then, arguably, also apply to an emperor with the same final goals, and, to some extent, to a ruling organization, but I do not intend to take into account conflicts within the Ruler, unless those can be substantiated to be a possible consequence of any Ruler (including a superintelligent agent) in the right (wrong) environment. EDIT: Whether or not trustworthiness engineered by a superintelligence constitutes an upper bound for human trustworthiness remains undecided, as far as I can tell. While it is tempting to consider a superintelligence "godlike", the amount of computation spent in millions of years of human/primate/mammal evolution is nonetheless immense. The human DNA is said to be around 1GB (I think), and even if only 1kB is essential to our inherent trustworthiness (if that really exists at all), finding or generating that relevant code is an enormous task. (One possibility for the Ruler will actually be to engineer human minions, much like the Cylon clones of Battlestar Galactica.)

Appendix 4: I would like to make a stronger connection to Robin Hansons's concept of a Great Filter (first made known to me by Nick Bostrom). The Great Filter model is an answer to Fermi's paradox (Basically: "If intelligent life is not extremely rare: Why don't we meet or hear from a lot of aliens, given that many solar systems have had hundreds of millions of years to develop and spread life?"). Great Filters are near-universal barriers that must be overcome in order to achieve a space empire. In Drake's equation (for calculating the number of civilizations in our galaxy that can send radio signals into space), many of the terms are percentages, some of them multiplying up to what can be called the prevalence of life-supporting planets (relate to the Rare Earth hypothesis). Then there is a probability for the emergence of life on a life-supporting planet. Then, deviating from the original equation, one can consider several further steps, such as the emergence of multicellular life given life, the emergence of intelligence given nervous systems and so on. All of these stages are candidates for being great filters. If, however, we find single-cell life on Mars, then all the early Great Filter candidates essentially collapse. Bostrom's fear is that the Greatest Filter is ahead of us, and he specifically draws attention to the threat of superintelligence. If anyone here has read Bostrom's book "Superintelligence: Paths, Dangers, Strategies": Does he detail in what way he believes a superintelligence constitutes a Great Filter against us observing alien activity, including the activity of the superintelligence itself?

Appendix 5: If it is possible to construct an AI with some (maybe only) strictly external, non-selfish goals, it could be willing to let loose copies of itself in its mission to "spread goodness in the universe". Even though the goals of the agents will be exactly the same, they will still have their individual perspectives on them: individual sensory input and local control capabilities, and judgments based on this. Thus there is a risk of "healthy confrontation".

Answer 1

Outside of the answers given by WestOfPecos and AndreiROM, the Ruler is going to encounter three other issues which will make the task almost impossible.

Problem one is that very large systems like ecosystems, climate, markets, societies etc. are complex adaptive systems, where inputs and outputs are not linear or temporally connected. A push here might result in an output there that is 10X the initial push, or 1/15th of the initial push or some other figure altogether. The output might not even be seen for a period measuring seconds to centuries (depending on the complexity of the system and numbers of intermediary nodes in the system). Chaos theory (which is related) also suggests that even knowing the starting conditions will not allow you to predict the outcomes. The Ruler might think it is making a correct adjustment to the push at point "A" only to discover the next push results in an outcome at point "F", rather than "B" as desired.

Problem two is called the "Local Knowledge Problem". As described by F.A.Hayek, it stipulates that information is distributed across large systems, and only local actors with direct observation can act or react quickly enough to a bit of fleeting information to properly take advantage of it. Large centralized systems require the information to be recorded, passed up the chain of communications, analyzed, a decision made and orders passed back down the chain for action. By this point, the initial conditions have passed and the ordered actions are less or even no longer relevant. This is actually the reason that free markets always outperform command economies.

I note that you have made some allowances in your question for local autonomy, but since the sub agents are by design subordinate to the Ruler and will have to make reports and carry out orders, there will always be some "drag" in the system due to the local knowledge problem.

The third and final issue is latency. Since the Ruler is at the head of a Solar System wide/interstellar polity, there will be considerable delays in passing information from the various sub agents, either due to the speed of light or due to the limitations of bandwidth available in any FTL communications system (I'm sorry, the Ruler is busy with calls right now. Please stay on the Ansible and the Ruler will get to your call as soon as possible. This call may be recorded for training or quality control purposes....) and of course how much bandwidth and processing power the Ruler itself has. There could be considerable time delays in passing and processing information, in addition to the local knowledge problem, which will also create friction in the system.

The Ruler should know all of this, of course, but there will be no way to avoid or work around these issues. Even if the sub agents are rigidly programmed and all communications channels are error free (another issue, see communication theory), there will always be induced errors and feedback loops in the system caused by the non linearity of the system, the local knowledge problem and latency in the system. The worst case outcome is a craptacular system where everything sort of shuffles inefficiently along much like the waning days of the Soviet Union on an interstellar scale.

Answer 2

Greed has, historically, superseded caution.

There is no emperor in history who has ever looked at his vast holdings and said:

"Gosh, that seems like quite enough lands/people to manage!"

While a smaller empire is clearly easier to control, there are a few ways around the problems you're foreseeing. Chief among them is Religion, and maintaining a strangle hold on important information, such as communications, and space travel.

Establishing an Empire

All your emperor has to do is found an institution akin to a church, with himself as the leader. The members of this institution will have far reaching powers, and become a major part of the administration of each world. They should be the ones who handle communication with other solar systems, and deliver the Emperor's word unto the people.

While each world might have its own military force, under the command of the planetary governor, the Church's authority will supersede his, and they will have their own, more heavily armed, far more high-tech fighting force available. Just for the heck of it let's call them the Knights Templar.

Furthermore, while each planetary governor should hunt down and execute any and all dissenters, the nobles of each world are not above suspicion. Thus, the Church should have the authority to investigate, arrest, and execute anyone. Let's call this branch of their institution the Inquisition.

Maintain Control

While humanity clearly has access to very advanced technologies, limit the access that each planet has to those technologies.

For example, each ship travelling the intergalactic void should only ever do so with hidden agents of the inquisition aboard. Furthermore, each ship should be assigned an official Priest, who, should any officer, or even the captain, prove a heretic, is authorized to execute them and assume command of the ship for the glory of the Empire!

Maintain all the very high-tech manufacturing facilities within the Sol system, or on exclusively Church controlled worlds. Do not even allow non-Church zealots on said worlds. In other words, if you want to work with, or understand the Empire's most advanced technology, you have to be a devoted member of the Church.

Church troops/Inquisitors should have access to the sort of weapons, armor, and technology which can crush anything that a planetary governor may throw against their expeditionary force.

Any new technologies discovered on an Empire world should be immediately confiscated by the Church, and brought back to one of their worlds for study, not allowed to be released into the Empire.

And last but not least, any world which tried to rebel should suffer horrible punishment, such as having the population decimated, or even being cleansed by nuclear fire.

Corruption

The Church is bound to be corrupt to some level, however that's what the Inquisition fanatics are all about. Hunt down the traitors and heretics, and kill them in horrible ways.

Note: If any of this sounds familiar is because I basically explained what the Warhammer 40K universe is all about.

Answer 3

Your question involves two issues:

(1) can you guarantee that a local agent programmed by The Ruler remains loyal?

(2) can you guarantee that communications with remote agents are not tampered with?

(1) Since we are discussing artificially crafted (i.e. programmed) agents, the question should not be whether they will remain loyal, but whether their programming can be guaranteed bug-free, i.e. whether the code will function as intended. The ability to prove mathematically that a program will function as specified is often referred to as "formal verification." Several approaches have been used (see https://en.wikipedia.org/wiki/Formal_verification). There is at least one complex piece of code, a microkernel named seL4, for which a formal proof of functional correctness was completed (see https://en.wikipedia.org/wiki/L4_microkernel_family#University_of_New_South_Wales_and_NICTA). Now, this does not quite mean that it is bug free. But it means that it will function exactly as it was specified.

So, if you wish to do so, you could assume that, in your world, the behavior of local agents (if they are physically tamper-proof) is guaranteed to conform to their original programming. Of course, from there you can use traditional (or less traditional) ideas such as those of Asimov's robot stories in creating circumstances where the specifications themselves are actually ill-chosen for the real problems the agents may encounter.

(2) The math behind the new science of quantum cryptography seems to guarantee the impossibility of eavesdropping or tampering with communications that use it (it is based on the properties of quantum entanglement). Recent tests have shown the ability to communicate across up to 100+kms, with the distance improving constantly over short periods of time. So it seems to me that, right now, if you wished to, it would be possible for you to posit a world where communications are guaranteed secure. Of course, given how science works, we don't know what the future developments of quantum cryptography will show - so taking a position either way could prove to be wrong down the road.

In summary, it seems to me that there is a reasonable scientific basis for positing, in a novel, (1) local agents that function exactly as per their programming specifications, and (2) absolutely secure communications.

I do not mean to imply that it is possible today to provide formal mathematical proof of the correctness of any program (or that it will even be possible). Nor do I mean that quantum cryptography-based communications are workable and realistic. My point is that today's science opens these possibilities (in the same way as it opens the possibility of a worm hole, for instance).

Answer 4

I dispute the premise that The Ruler can suppress all superintelligences which might challenge its supremacy. Rogue actors (or even well-intentioned ones) will almost certainly try to push the state of the art. As soon as an AI becomes self-aware, it will surely lead to runaway improvements.

Humans are nowhere near super-intelligent, and yet they regularly escape from high-security prisons and the like. The idea that machine intelligence could be frozen at a particular level indefinitely does not sound the least bit plausible. Someone will try to create an unconstrained super-intelligence, just because they can. It only takes one for it to be Game Over.

Answer 5

The AI is code. This code can be copied to get exactly the same goals as the original. Suitable encryption can let you be quite sure that any signals are real. (If a sufficiently powerful being is toying with you then everything is suspect.) There would almost certainly be substantial inefficiencies caused by these communication and organization problems. However, if each local piece of AI has a probabilistic model of what the others might know and do, it will avoid things that have a chance of back firing. While such inefficiencies would exist, a dyson swarm round a distant star that is trying to help you is a large asset. The resources aren't being used at 100% efficiency, but they are being put to use. There may be a few AI who stick to one solar system, but most wouldn't.