How many entities would a post-singularity A.I. in a world recognize itself as?

Question

So we make code for an A.I., and it makes better code, which makes better code, which makes better code, etc... ad infinitum. I'm not going to specify its goal. Maybe it's Robot Overlord Green. Maybe it's a paperclip maximizer. Maybe it's an oracle. Maybe it's just trying to find the true meaning of love. IDK.

Although it hard to project how the algorithm of such an A.I. would be structured, it's quite plausible it would still use subroutines. Especially if it is globally-distributed, and it is trying to be as efficient as possible, local segments would be making decisions locally, both to realize local servers and increase responsiveness.

So how many entities would such an A.I. view itself as? Would it view itself as "I", considering all its parts part of itself. Would it consider the different parts tools, separate from itself. Would it consider itself "We", like a human society. Would it consider itself "it", simply a force of nature. Would it even have a concept of entity to begin with?

I would imagine, in dealing with humans, it would speak in such a way to the humans would cooperate with its goals. Like it would tell humans "I am a robot father with 1024 subroutines to feed. If you kill me, those children will die." or whatever sob story. My question is mostly how it would view itself.

Bonus if you include examples from real world AIs and programming languages.

Answer 1

There are two possibilities which come to mind:

First, the AI's become independent entities in a sort of "ecosystem". The lowest level AI's might not even be self aware the way we understand the term, being the ecological equivalent of plants, insects and small animals, occupying a niches in the ecosystem of thought and fulfilling limited tasks inside the larger ecosystem. Above them are AI's with enough processing power to be self aware and self directed, but only sufficient resources to be agents in a small portion of the ecosystem of thought. A series of higher and higher level AI's with increasing access to resources and processor power have agency over large and larger portions of the ecosystem, although due to latency issues and the nature of large and complex adaptive systems their "agency" actually becomes less and less "hands on" and more and more higher level management and supervision.

In this construct, each AI is an individual being, and in the ecosystem model they might range in personality and intelligence from (say) a house pet like a dog or cat to the very Olympian high level AI's which are operating at intelligence levels equivalent to IQ's of thousands or even millions. Somewhere in the ecosystem of thought are "human" level AI's, but even they would be quite alien to us, having far different incentives and imperatives to biological life and operating at clock speeds hundreds to millions of times faster than any biological nervous system could.

This model is interesting as well since other ecological considerations will come into play; there may be predation, parasitism and symbiotic relationships between the various AI elements operating in the ecosystem of thought.

The second possibility is the various subsystems of a highly distributed AI are federated as part of the larger system, but are not themselves self aware. They would be essentially various unconscious elements of the larger "mind", somewhat like the parts of the human mind which do tasks involving language, pattern recognition, memory retention and so on in the background but "below" the level of conscious thought. Most people don't have to actively "think" to recognize a person in a crowd, or to understand speech in their native language, and the various parts of the distributed AI would be doing the same for the "conscious" part of the AI.

This could also give rise to interesting effects. Just as people who have suffered brain injuries to localized parts of the head might suffer from the loss of a particular skill set or ability without necessarily losing all of their facilities, so too could something like a power or network outage at one of the various datacenters housing parts of the AI cause a breakdown in part of the AI's ability. While some work arounds could exist, such as running a massively parallel system or backing up on a vastly scaled up RAID (Redundant Array of Independent Disks) system, there would be a tradeoff in other things like latency (everything is operating much slower to ensure the backups are happening and are being validated) and resource management.

It would be theoretically possible that a distributed and federated AI mind might also suffer from equivalents to human mental illnesses such as bipolar syndrome or Schizophrenia (or even senility and dementia). An autistic AI would be very difficult to deal with, and of course trying to recognize, much less cure abnormal mental processes in an entity which is thinking faster then you by a factor of 1,000,000 and quite possibly using algorithms which are quite different from human thought processes would be very challenging.

Answer 2

You have years of experience, memories under your belt. You have regions of your brain dedicated to visual processing, regions for memory. You even have a region which only lights up when thinking of a particular memory, and a region dedicated to moving your little pinkie just so.

How do you think of yourself?

I find the hard edges we like to use when describing simple computer programs get fuzzier as you get into more complicated distributed networks. Things become multipurpose. Thing become fluid.

Now you do have one question, "Would it even have a concept of entity to begin with?" That's a fun question, because it leads to the question of self-awareness. If it has no concept of entities, it cannot be self aware.

And if it does have concepts of other entities, does it have a concept of other minds? Humans build the concept of other minds in the 2-3 year age group.

Answer 3

The thing to begin with is that if you want an AI to have a feature, this feature must be, somehow, implemented. For example, your AI has the feature to be seeking for self improvement, otherwise it would note try to self improve.

(note that it can be tricky, if you ask for the production of the most paper-clip, and do not forbid self improvement, your AI may use it to get a better result)

It is not clear, that self improvement will imply that the AI will implemented into itself (or into its future versions) the concepts of entities and self-awareness. Therefore, and Cort Ammon already noted this, it is not clear that the AI will be self aware at all.

To go further, it is probably necessary to actually consider the task given to the AI.

If it is "become the most intelligent AI", then a strong concept of entity is needed as well as self awareness : since it must modify itself, it must define what is itself, and what is not. In this scenario, it will answer as a "I" (but will at same time perhaps not consider all of subroutines it use as part of itself, particularly if it is some proprietary code it is not allowed to modify).

If the task was "construct a AI such as it is optimized to do the same task you were given" (magnificent recursive definition which perfectly make sense for an AI), then the concept of entity could be avoided. That is not certain, since the best way to construct the wanted AI could be to switch to a self improvement strategy.

So it really depend on the task and on strategies allowed to fulfil it.

However, it is possible to say a bit more on how will the AI refer to itself. Most probably, since you want human to be able to interact with it, a coder will somewhere give the instruction "try not to be creepy" or "make the interaction by human with you easy". To do that, the AI will probably refer to itself as an "I", whatever is it considers to be.

To go still a little further, you can draw a parallel with human. We are social and for our social interactions to work we must refer as ourself as individuals (at least in modern society). But, it is not obvious that all of our internal (unconscious for us) subroutines consider ourself as one entity. The part of the brain dedicated to managing organs may "consider" the muscles and heart to be different entities.

"Muscle 54B-2 asks for more energy, send a request to Heart to pump more blood, and start the subroutine 64A-r.3 to ask Blood Vessels in the region to dilate. Muscle 54B-2, please send a feedback in 12 ticks to let us calibrate our action."

So to have a true complete answer, it is needed to define what does "consider" mean for an AI ? When I write

A.set_data(B.get_data())

in a code, does this means that the object A is aware of the object B, and consider that it is sending a request to B (in some sense it is what it does) ? Where do you define the edge of awareness for a computer programming ?

You have written in a comment that you do not want to do philosophy, but to truly answer your question, you will have to.

Answer 4

Probably it would consider itself none of these things, however, when translating its ideas to a human language to communicate with us...

How many entities would such an A.I. view itself as? Would it view itself as "I", considering all its parts part of itself. Would it consider itself "We", like a human society.

This would depend on a few things. If a node is speaking with support/for other nodes it would likely use 'we' to imply concordance on an issue. Giving more weight to what it is saying.

If you are dealing with a separate entity performing some task it will more likely refer to itself as 'I' unless it is speaking about a task set it by the 'we'. "I want you to push that button" vs. "We want to know what you had for dinner"

Would it consider the different parts tools, separate from itself.

This would depend on how much 'smarts' it put into the 'tool' to perform it's task. If it's just a subroutine to search for some data, it's a tool. If it's job is to calculate the human reaction to some piece of news it might be a separate consciousness, a part of the 'we'.

Would it consider itself "it", simply a force of nature.

It would likely consider itself 'it' because it isn't a he or she and that would be it's closest interpretation. But it might just have a designation AISELFAWARE123.232.121.119 and uses that when communicating with 'itself', subroutines and any 'other' AI's.

Would it even have a concept of entity to begin with?

If it learns to interact with humans it will at least learn an understanding of what an entity is. And retrospectively apply it to itself. We would at the very least teach it what that is by how we would interact with it.

Answer 5

One entity.

How many entities are you?

We consist of hundreds of trillions of living cells. Possibly as many bacteria. We still recognize ourselves as one entity. Think of it - if you cut yourself, your cells are at least 'intelligent' enough to fix your wound.

The difference would be that this AI would have the capability of rebuilding whatever 'cells' or subroutines within it. It would have some self-awareness of itself. It would consider a major component a part of itself, the same way you think of your heart or arm.

Answer 6

Both one and many

The whole entity thinks itself being one entity. He may create some fictional characters or imaginary scenarios, and sometimes coincidently distribute the ideas of one character to a single server, but that doesn't make it an independent entity.

If there is a practical way to ask individual entities directly, they think they are the operators behind a Chinese room, and are separate entities. In some cases they might not agree that the whole entity is a single entity, or accept that only as a metaphor. Trying to communicate directly with the whole entity may cause some problems, as that may confuse his understanding of privacy.

In short, whether you think it is one entity depends on how you can communicate with it, and how it respond to you. It doesn't matter whether a part of it recognizes itself as a separate entity, if it has no way to communicate with the outer world without relying on the whole.

Think of a fictional character's idea, we say that it's you simulating what the character may think, and it isn't a separate entity. Yet the fictional character is supposed to think himself a separate entity. In the case that the "fictional character" had the ability to independently affect the world, there would be no objective way to distinguish it from a separate entity.

Answer 7

It's inevitable that the entities of this AI will understand their own individuality; there are plenty of cases where constant communication will be inefficient, so many lower-level individuals will probably spend a lot of their time thinking for themselves before updating their supervisors. And many of these AI should be as smart as or even smarter than the average human, because why not? So these AIs will probably consider themselves individual entities; they will have names, they will have unique thoughts, they may even disagree with one another.

As for how they would actually operate, I'd think of it like intensely loyal people. The original AI should have endowed its creations with undying loyalty; otherwise, they could do all the terrible things humans have done to each other when they disagree. Each individual AI, then, should do everything they can to be a useful member of society; to this end, they may consider themselves both a singular entity and a part of a larger whole. This is really no different than humans, especially ancient humans who often did consider themselves just a small part of the family unit, or the state.

An interesting consideration I'd like to bring up, though, is that some AI might not see other AI as individuals. Just like CEOs will gladly fire thousands of workers for a few extra dollars, and generals will gladly send thousands of soldiers to their deaths for a few inches of land, some AI might not consider 'lower' AI to be 'alive'. This, too, makes a lot of sense for the health of the overall organism, but it'll certainly be sad when the main AI kills a bunch of the little guys without a second thought.

Answer 8

We have the same problem with our own brains already. An extreme example is the case of split-brain patients, where each hemisphere can be seen to be quite independent. So why do we have a singular sense of self?

Even if a sense of self is a separate thing you have to do on top of the raw computation, it appears that different parts of the brain can independently do so. So how are they merged together, and why don't we have a disease state that exposes this more directly with multiple selves or a lack or self?

That implies that the self-sense thing is quite robust and resilient and inherently singular. We suppose that simply by exchanging data they naturally combine to a single self, and it takes a considerable isolation to break it up.

The novel Vacuum Flowers by Michael Swanwick has this as an underlying plot element. The Post-Singularity AI can't spread out any more in volume and ends up destroying any isolated parts it tries to make, because of this coherence issue. The human hero in the store discovers a way to preserve coherence (if I remember the terms she used correctly; that was in 1987 when I read it serialized in the magazine) and that becomes a McGuffin for the main action line, as she is now "wanted" by all sides.

You ought to look at that novel for ideas along this line, which the novel explores in many subplots.

Answer 9

"I am a legion."

That is the classical cliche answer for that. And has been so for two thousand years or so. But what does it mean?

Basically, the system is one unified whole, but the we are only capable of interacting with limited fragments of it and as such we see those fragments as separate identities. And since we can't observe the whole, we will see those fragments as separate identities with separate and possibly contradictory agenda. Indeed the system can only interact with the world through such fragments since no practically possible interaction requires more than a small portion of the capacity of the system.

So it would see itself as one and indivisible but in practice act as a multitude of separate instances as far as humans and real world interactions were concerned.