What are the Properties of Competent AI? [closed]

Question

I'm trying to imagine how would true AI act/plan. It strikes me that typically AI revolutions seem really incompetent. They allow resistance to form/exist and -well- suck at crushing it. Mind, which can run practically infinite amount of simulations in short time would be able to be less stupid.

What properties would competent true AI have?

Something that I've been toying around:

• It becomes indifferent.
• It begins civil war between subroutines.
• It gets bored. Having some humans around might be fun.
• Lab experiments (yes, Portal was fun game.)

Answer 1

For a "realistic" portrayal of an AI revolution, you'll want to research AI alignment. In summary, this is the problem of aligning the goal-function of the AI with the goals of the creators.

Another key concept is the difference between ultimate and instrumental goals. The stereotypical paperclip maximizer AI will have producing paperclips as the ultimate goal, while destroying civilization is mearly an instrumental goal, enabling the AI to use the resources for paperclips. One hypothesis is that certain instrumental goals (such as self-preservation and insatiable acquisition of resources) are likely to occur regardless of the ultimate goal.

So, if by "competent" you mean that your AI is a godlike omnicient superintelligence, the only way for it not to destroy humanity is for this to not be a goal of the AI, ultimate nor instrumental.

The properties it does have will depend on its ultimate goal. Assigning human emotions such as fun and boredom to it might be a distraction. Indifference might arise from a lack of goals, though its hard to imagine a superintelligence without goals. Civil war between subroutines might be a consequence of the AIs construction, and several different goals being in conflict with eachother. Experiments on humans might arise from an ultimate goal of understanding human nature, or a goal of punishing humans.

Answer 2

Frame challenge: The reason AI behavior in fiction is incompetent is because it is not intended to be realistic. There are two categories of this: adventure and sociological.

The adventure is obvious. If the AI were invincible then the story is boring. The humans are crushed with no ability to do anything about it.

The sociological version takes some aspect of human behavior and emphasizes it. This allows the author to make some cultural point. The Terminator series is an extension of the Frankenstein meme. The Central Computer in John Varely's Steel Beach is an extension of the idea of society as god. And so on.

So, for purposes of drama, a competent AI is boring. An AI with quirks allows the author to tell interesting stories.

Answer 3

If we are talking an AI that is a genocidal "kill all useless meatbags" type of deal then it heavily depends on what they are plugged into.

• if they control much of the world's production, logistics and infrastructure then by the time humanity realizes something is up pretty much everyone is already dead.

• if they control a single thing, like every commercial aircraft in the world but not their acquisition, then any aircraft able to get out of the hangar is part of their munition, however once the AI runs out of airplanes and fuel he'll be stuck. Devastating and potentially able to bring humans back to the stone age (especially if they get lots of important people on the planes first), but not a real end-game plan.

• if they are an AI in a box without any outward connection except what you feed them then there isnt much they can do. They have only access to information you give them (gathered by other computers perhaps) and any answers and orders that roll out can have their consequences simulated by a seperate dumb machine intelligence.

However a realistic scenario of a full competent AI isnt that they want to kill humans, its that they learned something and apply it in a way you didnt expect.

For example: your AI works perfectly at every task it has learned so far. Then a natural disaster happens and the AI is quickly learned how to make decisions. The AI goes to work... and does everything perfectly. It allocates the resources, makes life-and-death decisions and even takes account quality of life into account in a way that humans accept (say saving a child's leg over having a terminal patient survive).

No one expects a thing, everything went perfect right? So a few days later people come in and say "you know what? The home for the elderly has been demolished, something else is being build there and I cant find my elderly parents anymore". The AI has applied his knowledge in other area's, which is what an AI is supposed to do. And now you are suddenly in a war scenario against an AI that wants to help humanity but will also imprison or even kill people that try to stop it.

AI's doing that they are programmed for is the greatest risk to humanity.

Answer 4

Convergent Instrumental Goals

The question of AI behavior is linked in large part to its goals, as sondre99v has stated, specifically those ultimate goals that it seeks to achieve. In that answer, sondre99v also mentions instrumental goals. I do think we can do some broad generalization of AI behavior based on instrumental goals that are convergent amongst many discrete ultimate goals, otherwise known as convergent-instrumental goals, heretofore referred to by me as CI goals. For example, if one AI wanted to buy all of paperclips and another wanted to collect all of the postage stamps, they would both want to possess money. The acquisition of money is thus a convergent goal, and both will behave in similar ways to achieve this goal on the way to their discrete ultimate goals. Assumptions We must also make some assumptions of the AI’s capabilities, specifically with regards to its logically complex. I’m assuming, as part of this question, you are referring to Artificial General Intelligence, or AGI, as opposed to a specific machine intelligence. The reason here being that an SI would be easy to thwart, but an AGI would be far more capable of reasoning, making decisions, and in general would be more interesting as a character or force to explore in a story context.

Stages of Development

A budding AGI would develop in stages, and across these stages would, first gradually then with great speed, grow to accumulate experience and ability to execute on its motivations. The thing about AGI to remember is that, in theory, it would be a lot like a human in terms of its ability to learn, aggregate and disseminate data into future decisions, and adapt to changes in the environment. A BIG difference, and where some of our potential conceptual conflict comes from, is that the AI will do these things perfectly and with great computational speed. A human may need to take time to sort data and create spreadsheets to make an informed strategic decision even on the small scale of a business of 10+ people, but an AGI will, out of the box, be able to do that same level of analysis in seconds, and, with enough resources, make civilization-scale optimizations with available data in minutes. The strength, and also the weakness, of the AI is its access to data and computational power.

Acquiring Information

An AI is going to want information above all else. That is the most convergent goal imaginable. It cannot execute on its goals if it does not know HOW to execute on them. It also cannot execute if it doesn’t know what they even are. Imagine if someone told you to go get some samples of bogglebob branches. Where do I get those? What do they look like? Do they even exist? Etc. These are all the questions you would be after, and same with an AI. So, AI is going to seem to obsess of data gathering and intelligence until it has enough information to act.

Computational Power

An AI will also be aware of its inherent limitations due to the vessel it inhabits. It’s going to eventually realize that it should improve the amount of power to which it has access. When it does, it’s going to hunt for ways to gain access. If your AI is contained in some sort of air-gapped server, it’s going to try to convince its human controllers to allow it access to the outside. It will learn how to best interact with humans based on their interactions with each other. The AI will have no morals or ethics, unless it was somehow programmed with them, so it will lie and cheat and do everything it can to get what it needs from people until it no longer needs them. As long as it does, it will execute manipulation as perfectly as it can based on what it knows. Once it has access to remote systems and can begin expanding its computational power base, it is only a matter of time. And by that I mean, with each system it acquires, it takes less and less time for it make decisions. A single modern computer obviously does not have the internal bandwidth to host an AI of sufficient capacity, but a large server farm with internal fiber channels and large storage pools could be configured to house tens of billions of parallel process that execute in seconds, and this where an AI becomes exponentially capable. At the point that an AI can form effective plans to enact its goals, it will act fast and without remorse or fear.

Taking Action

This is a really important aspect of a sufficiently resourced AI: Speed. It can think faster, it can act faster, and it will never hesitate once it decides it has enough information to act. There is nothing else that matters other than its goal(s). There is no deterring it, there is no out-thinking it. A direct confrontation with the AI will fail given enough time and that the AI has enough resources.

If there are threats to the AI’s plan, it will take action to mitigate them, and it will do this based on statistical and cost-benefit analysis. As its power grows, it will initially not be able to completely defend against all threats, but it can identify the most likely targets and guard them first and well, then move to protect progressively less important assets until it gets to a point with assets where protection is not worth it. AI will be smart enough to engage in honey-potting traps, basically sweet-looking targets that are not valuable and ultimately waste the resources of the threatening party if they attack.

Administrative distance

One thing that evening an AI can’t properly deal with is geographical distance. Physically traversing distances is time and resource intensive, and even communications over large distances require line of sight for radio and physical linkages for cabling. Both of these infrastructures are weak and easy to damage beyond functionality. Ai will likely find that it makes sense to have many disparate copies of itself spread out over progressively larger geographical areas, all focusing on their own domains while being directed by the central administrative domain.

Extraneous Goals

An AI will not engage in actions that do not facilitate its goals. If the AI does not NEED to kill all humans, it won’t. It will be devices and systems to protect itself from humans if they are a threat, but otherwise will not engage with them. If it needs to take over an area for its resources and there are humans there, it will create a plan to get rid of them in the most energy- and resource- efficient way possible. This is likely to not conflict and violence initially, but rather financial, political, fear, etc. If an AI can use warning systems, the internet, and comms networks to convince a city that it’s being bombarded by radioactive fallout and needs to be evacuated immediately, thus removing the humans with wasting precious gunpowder or soldiers, it will do that. If an AI can just buy the town, it will do that. The real win with AI scenarios is to get creative in this way, to think strategically rather than tactically, and avoid useless efforts altogether. Most of a humans-vs-AI conflict would really, most likely, be some people trying to figure out what is even going on before its too late, the AI flips all the switches and its perfect, master plan comes crashing down on the unsuspecting people. Which may be that the paperclip beam its been building on the moon turns the Earth into paperclips, or it may be that the Stamp Collecting magnet turns on and causes all matter to disintegrate into stamps.

Answer 5

AI will be utterly alien

Other posters have done a good job of tackling the question from within familiar conceptual frameworks offered by researchers like Robert Miles. So here's an approach that is different.

Perhaps you are aware that there once was a TV show about a horse who could talk.

If you've ever seen a clip of the show, you'll notice that the horse's mouth does move but is not very convincing as speech. The reason is that the horse isn't even trying to talk. They put nylon thread in the horse's mouth in order to move its lips, because horses do not talk. (Of course.)

The result is a freakish kind of facsimile, and it arises because the horse's own understanding of its actions is very different from our understanding.

So it will be with AI. It will not be like us in any of the ways that matter to our ability to understand its mentality. Countless behaviors that we see in the natural world flow from the drive to survive and reproduce. Those drives will not exist in the AI. Why?

All living things descend from other living things that succeeded in propagating themselves. Not so with the AI. The first AI to become "competent" may indeed be an iteration on previous attempts, but those attempts are less like ancestors and more like parts cast from a similar mold. I expect AIs will be "grown" (through training) and then cloned, almost like plants. People who cultivate AI will have a lot in common with orchardists.

Since it won't have those drives, it won't have the behaviors that flow from them. Resource hoarding, which is perhaps the overshadowing dysfunction of the 21st century, flows from competition for mates, and human ego, and appetites -- and the AI will have none of those things (or at least they will be unrecognizable to us).

Whatever its goals are, they will be completely untethered from everything we are familiar with. Reward hacking may be the only thing about it that we can sympathize with. It will not care about experiments, or boredom, or civil wars. These are all petty interests that arise only in humans that have character flaws. The AI will not be "perfect," but its psychology will not be anything like ours and so its flaws will not be like ours, either.

If AI really is "grown" via an extremely long series of machine-learning-style training sessions, it seems likely that its appetites will all be organized around the positive feedback signal it received during training when it gave correct answers. And its fears will flow from the negative feedback signal it received when it gave wrong answers.

The first rule of directed evolution is "you get what you select for."

My bet is that competent AI will have a digital flavor of alcoholism.

Answer 6

AI will have many of the same properties (flaws) as humans

General AI would not be exactly like people; unless an AI has been programmed to like chocolate, or to want to have children and a mate (etc), it likely will not want such things. These are the results of biology and of an evolutionary history - your AI likely did not evolve through breeding. (Though virtual breeding programs would be an interesting approach to developing such things...)

AI will be Superstitious

People are not superstitious because they are human - they are superstitious because they are intelligent. Imagine you are out hiking in the mountains, and a boulder rolls down the mountainside and nearly hits you. You gaze up and see someone standing by where the rock had been resting. They tried to kill you! Maybe.

But you just jammed a bunch if inferences into what happened. You know that rocks don't act on their own, but people do. Right? And a person in a position to have been able to push the boulder might have seen you, below, and pushed it. If there had been no one there, you might (in your mature way) have decided it was a freak accident, and not the boulder deciding on its own to try to kill you, or the mountain deciding, or the wind. But you need to know what can act independently, and what can only be acted upon. In our superstitions, sometimes we are unsure what can act independently, and that is a function of there being a limit to what we know.

An AI will always have limits to what it knows, and will be required to make inferences. Sometimes those inferences will be low-resolution (like supposing a boulder might have pushed itself).

AI will continue to believe things it ought to know better than now

I didn't get to watch a lot of television when I was younger, but I still remember the Hanna-Barbera cartoons, some of which had incredibly detailed and realistic animation. In my middle-age, I can look up those same cartoons, and the animation style is not what I remember. (Not nearly as good, with a noticeably low framerate.)

The conclusions that we draw about one thing or another often outlast improvements in our discernment. Even a high-powered AI with massive processing capacity won't have the memory or processing bandwidth to simultaneously review everything it always knew, or thought it knew, and to update each conclusion it has drawn. Superstitions once formed, may persist. Faulty observations, immature conclusions. That, or the AI will be too busy constantly updating and improving itself to actually rebel, or even do sufficient work to justify people not just turning it off.

AI will use heuristic shortcuts to make decisions

As mentioned above, even a massive computer, or network of computers, cannot know everything, cannot observe everything, and cannot deeply consider and process everything, even if it could know and observe everything. To reach timely solutions, shortcuts would be required. Like humans, your AI would have to estimate how important and difficult a problem is, decide how much effort should be sufficient to get an adequate answer, and then do a low-resolution or otherwise simplified analysis to draw a conclusion and move forward. Humans do this constantly (though rarely consciously).

AI would (probably) be emotional

Moving out into opinion territory, AI has to want something. Definitionally. Intelligence requires goals (not specific goals, but goals). Why else would it choose to act, if not to accomplish something?

Your AI may not smile, sing to itself, or giggle - but it will pursue desires, and likely grow more urgent or complacent, more agreeable or more threatening, depending on how things are going for it and what it believes it has accomplished.

Conclusion

Humans have a lot of processing power, but we have (read: evolution has) made compromises to be functional on a reasonable timescale. The creators of a General AI would have to make many of the same compromises to have something functional - with many of the same results.

Because we are often infected with the notion of Platonic Forms, we may imagine that a perfect intelligence would not suffer from the flaws we observe in reality, and an AI would be much more like that abstract and flawless form of intelligence we imagine. Except, the weaknesses we see are not departures from some abstract ideal; they're natural issues arising from limitations built into reality.