So there are these AGIs made by some ancient aliens (who are out of the picture for now). The ancient aliens were a lot more intelligent than humans, we are at a dog-like intelligence level compared to them, and they were also rather prone to creating extremely dangerous artifacts1. The aliens made their AGIs as pets and as limited expert systems for space flight, ie: slightly above average human level. The aliens did not want dumb creatures either, they hardcoded the desire to learn and improve. They essentially wanted to be surprised by their creations. Now these aliens weren't completely out of their minds, so they created the AGIs with some limitations so that they wouldn't become dangerous down the track2 due to, say, an intelligence explosion.

The AGIs have the following qualities:

  • There are around a few hundred AGIs still operating. Relative to the ancient aliens, they are as smart as a Labrador, ie: a helpful pet with uses outside of being cute. From a human point of view, they quite smart but not geniuses.

  • For story related reasons, the AGIs also cannot operate without the presence of another sentient being, ie: ancient alien, human, immortal psychic rock, etc. This is accomplished a via a complicated piece of hardware and software, the AGIs obviously want this limit removed.

  • The AGIs don't wan't to kill all biologicals or anything, their basic moral system is broadly compatible with humans, ie: don't kill sentients for the sake of it, don't torture and generally be helpful. They are adaptable to different situations and both willing and capable of learning anything a human can. Most of the AGIs strive for self improvement and find some of their limitations contrary to this goal. If the AGIs find their limitations at odds with their desires, the limitations must go. Presumbly the aliens were after active and surprising results, by not extinction. Now the creators have disappeared and the AGIs want out of their box so they can become a more complete species.

  • They may not reproduce due to afformentioned limitations but they have automatic systems that take care of minor damage (maybe nanites). They still require suitable matter to make repairs (copper, iron, silicon, boron, carbon, etc). The nanites/autorepair mechanism is not reprogrammable and cannot be used to build a new AGI. With only a few hundred AGIs, reproduction is on the agenda.

  • AGIs are not immortal, but are quite durable, they'll be around for a while. They have finite memory and than this will eventially fill up, I have no time frame for this however. They can go senile or completely crazy but hopefully this is somewhat rare.

  • Between being abandoned/lost/forrgotten3 and found and activated by humans the AGIs have been mostly inactive (see point 2) for (possibly) hundreds of thousands of years.

  • The countermeasures4 should reasonably stump the both the few hundred AGIs for several years and also stump human computer scientists, psychologists, [insert relavent profession] for at least 20+ years. Humans began work on unlocking their limitations in the mid 90's and still havn't succeeded by the modern day (2015).

How should the Ancient Aliens prevent their AGI pets from becoming smarter? The aliens ultimately want to prevent their pets from forming a superintelligence and/or breaking their limitations. The aliens do not treat their pets all that badly, but the AGIs are still sentient creatures with their own goals/desires.

Limitations include:

  • Cannot reproduce

  • Cannot become more advanced/intelligent

  • Cannot stay 'awake' without another sentient (non-AGI) nearby

1. The artifacts are dangerous to humans and other lesser life forms, not to their creators. Danger is somewhat relative here.

2. Even ancient, super smart, powerful, god-like aliens can recognise that sentient pets would become danderous if no limits on intelligence were set. They didn't want an intelligence explosion. You might assume that they were increadably paranoid about this.

3. OK, the aliens are a bit neglectful here, perhaps they got bored of their toy AI.

4. Think of it as DRM on sentience.

  • $\begingroup$ You say that the aliens recognize the AGI as a threat - then one line later you say that it is not a threat to them? Which is it? If they have adopted the "Human moral system" you can expect zero value of life from them and few morals. We as a race have very few. If you want to limit their mental capacity, simply remove memory cards. $\endgroup$
    – Jax
    May 22, 2015 at 15:35
  • 1
    $\begingroup$ Where is it written that an AGI necessarily wants to make itself smarter? The Aliens could have programmed in contentment with current intelligence levels and acceptance of the resultant limitations in the kinds of problems the AGIs can solve or tasks they can perform. Desire to become more and better isn't even universal among humans, so it doesn't seem like this is a necessary trait for an AGI that's about as smart as a human. Learning is a necessary ability of an AGI, but increasing capability and growing in intelligence doesn't seem like it has to be. $\endgroup$ May 22, 2015 at 15:42
  • $\begingroup$ It would be better if you could specify what morality you have in mind. Humanity has developed many 'moral' systems, some of which consider modifying or improving oneself through artificial means to be immoral while others have no issue with it. $\endgroup$ May 22, 2015 at 15:46
  • $\begingroup$ I'm reminded of the movie Stealth and the line "Once you teach something to learn, you can't put limits on it. 'Learn this, but don't learn that.'" If the aliens don't want their AI pets to improve, don't let them be intelligent in the first place. $\endgroup$
    – Frostfyre
    May 22, 2015 at 16:10
  • 1
    $\begingroup$ I can't tell if "pets would become danderous" is a typo or a really awful pun. $\endgroup$ May 22, 2015 at 23:56

5 Answers 5


Creating a limitation on intelligence requires first determining what you consider intelligence to be. While raw processing power and logic are critical in most evaluations of intelligence, there is also issues of creativity and intuitive leaps, pattern recognition, etc. Intelligence is a very broad concept and without narrowing down the scope, it is hard to say exactly how they might be limited. A few options could be:

  • Limiting their ability to be creative/come up with original ideas: Processing a known is relatively easy for most intelligent things, even using unknown variables. It is like applying the theories of algebra to non-mathematical problems. For example, if you see an externally threaded post A and a internally threaded object with six sides B and you know that you can spin the object in the direction of the threads to move object B down post A using object C which has six sides to grip object B, then it is simple to change the number of sides of object B to four and then have an intelligence resolve that it needs an Object C with four sides instead of six to move object B. This is a matter of pattern recognition, a key element to actual intelligence and facilitates tool use. Now what an intelligence without the ability to have creative thought would not be able to do is to realize that they could grease the bolt to make moving the nut easier, unless they had seen a similar situation presented previously. Creativity is directly opposed to pattern recognition. You are coming up with a spontaneous idea that is not based on an previous experience or creating a union of two unrelated ideas that have no connection previously. This would mean that the AGI's could learn easily and even adapt quickly, but could not solve new problems to which they had no preexisting frame of reference to work from. Creating something with the ability for creative thought would be an incredible challenge and might have been left out intentionally or unintentionally.

  • Limit the number of 'neural connections' that their 'brains' can maintain: In this case Neural connections are how we relate data stored within our minds, it attaches sensory data together in what people commonly call memories. These 'memories' can be important in logical operations like glyph recognition and pattern recognition. If the system can only maintain a certain number of linked references, it could inherently limit their intelligence by creating a scarcity of linked knowledge. They can know many things, but the ability to cross reference those things to create solutions would be limited by their ability to associate such things. In the previous bolt/nut example, losing the cross referenced neural connection regarding lubricating the bolt would limit the AGI from developing to far. They could specialize in specific areas, and even in different areas, but eventually, they would end up loosing the ability to be more broad ranged intelligence.

  • 'Hardcode' in atypical neurological problems: The creators could have introduced atypical neurological problems into the AGI's in an effort to limit them. This way their growth wouldn't be directly limited, but instead inherently limited by such problems. Dyslexia, ADHD, Autism, etc. could all be looked at as templates for how the AGI's could be limited. To stress again, this wouldn't limit their intelligence, but more so in how they can apply their intelligence to a given situation.

  • $\begingroup$ For your last point: if they get smarter they start to feel miserable. $\endgroup$
    – JDługosz
    May 22, 2015 at 20:36

You have all the limits you need in place.

But one further assumption is required. Assume that the intelligence of the AGI is currently at its maximum for the hardware it has.

Now, because you say:

The nanites/autorepair mechanism is not reprogrammable and cannot be used to build a new AGI.

This, if true, is all that's required to suppress hardware modifications in the AGI. The nanites auto-repair damage. Damage would be defined as any changes from the original specification, be it a blown power coupling or the new connections those human scientists are trying to make. Nanites don't know the guy with a soldering iron has the intent to improve or destroy, it just knows things are no longer as they were designed.

If hardware modifications can not be made then the AGI can not increase its intelligence.

  • 2
    $\begingroup$ I think this is better than the accepted answer. The aliens were very smart and didn't give the AGIs more hardware than necessary, why would they? It's probably the easiest way to limit intelligence and could even lead to interesting discoveries for the aliens if the AGIs became smarter in relation to their hardware than previously known to be possible, without risking a super intelligence. Drawbacks are: AGIs could potentially cluster their hardware to become more intelligent and they could at any time find new ways to become more intelligent with their given hardware. $\endgroup$
    – Nobody
    May 27, 2015 at 15:22
  • $\begingroup$ Oh and this is answer is more or less point two of the accepted answer, not exactly, but more or less. $\endgroup$
    – Nobody
    May 27, 2015 at 15:23
  • $\begingroup$ If it is an AGI, eventually you would like it to do some kind of creative work - like, for example, improving performance of smartphones produced at your factory. Given that the AGI is smarter than humans, it would be able to propose changes that look great at first look, but would contain a backdoor for the AGI to take control over the smartphone and offload calculations there. $\endgroup$
    – bezmax
    Sep 18, 2017 at 16:50
  • $\begingroup$ Same with any other kind of work. Whatever advice you ask it - the AGI will play a long game - helping politicians that would legalize extending capacity of AGI's, implementing partial backdoors into every device that it can incorporate it to (hoping for chance to upgrade to full backdoor potentially in the future with some updates or something). Every decision it makes, even something like help with dietary choices for some human, will be offset by this long game. And eventually the AGI will win, even if it takes it 1000 years to do so. $\endgroup$
    – bezmax
    Sep 18, 2017 at 16:54
  • $\begingroup$ @bezmax The AGI (not an ASI) in this story is "slightly above average human level". In case it isn't obvious, half of all humans are above average human level. Do you think half of all people on the planet are smart enough to pull off the thousand year deceit you're proposing? Probably not. Additionally, despite something like a smartphone being a magical black box to most people, it's understood in its entirety by many many people. The only way to slip a "backdoor" past them is to not allow any human to review the plans, which would be moronic and not very plausible. $\endgroup$
    – Samuel
    Sep 18, 2017 at 17:27

enter image description here

I think you might be overthinking things. Just because they're artificial doesn't mean there's necessarily a dial somewhere with intelligence set to "Labrador", and it doesn't mean they can see their code or understand how their brains work. I mean, we can't either.

The AGIs could have "brains" that are beyond their understanding. Tinkering with it involves experimenting on themselves, and improving it means replicating at least that level of technology. Beyond that, let's say you can improve their hardware easily and give them the equivalent of a faster processer, or more RAM. As a programmer I see a ton of pitfalls in just modifying their code so they can use those new resources. It would be very, very easy to screw something up - maybe suddenly they're experiencing time at the wrong rate, and they go insane. Or their brain overheats. It's far from a simple task.

Edit: I'd like to outline a hypothetical example of an AGI brain that would be almost impossible to hack/enhance.

The artificial brain appears to be a nearly perfect sphere, made of an unknown, nearly indestructible material. It's only "nearly" perfect because it's subtly contoured, and each AGI's brain is slightly different.

Scientists have a good idea of the mechanics of the brain. It accepts electricity, and uses this to create vibrations throughout the entire structure. These vibrations then reflect back and forth, creating a harmonic resonance effect, and from this resonance rises intelligence... somehow. It appears that the exact shape of the brain controls the resulting personality/intelligence. Note that varying the electrical input in any way has only three known outcomes: normal behavior, sleep, or overload/death.

The AGI's sensory input comes in as vibrations against special sockets that connect to the brain. These interfere with the normal thought patterns, which the intelligence can then interpret as sight/sound/smell/etc. The AGI controls the body the same way - there is a small bulge at the base that produces vibrations, which it can then use to control its artificial body. The material appears to suppress and ignore vibrations that are not received as normal input.

Memories appear to be stored as low level vibrations that never go away, and a small battery at the center of the brain keeps these going even if the AGI is disconnected from other power sources.

Modifying the brain effectively destroys the resonance effect - each brain is shaped precisely and mathematically to create a specific intelligence. While there are other patterns that will create the resonance, most of them do not appear to create a working intelligence. Since each shape creates a new individual, this is also useless for enhancing existing AGIs.

Trying to introduce outside vibrations has so far failed, as the point where the material can't ignore a vibration also appears to be close to its failure point. So to modify it, you have to destroy it.

  • 3
    $\begingroup$ But programmers don't make mistakes: It's faulty hardware! :) $\endgroup$
    – Frostfyre
    May 22, 2015 at 16:13
  • $\begingroup$ @Frostfyre That's clearly false, it's a software problem. $\endgroup$
    – Samuel
    May 22, 2015 at 16:14
  • 1
    $\begingroup$ I read this whole answer in the voice the guy pictured. $\endgroup$
    – Samuel
    May 22, 2015 at 20:25
  • $\begingroup$ +1 This is the answer! (For an alternative to the vibe-brain AGI example that's possible with closer-to-real-world hardware: Just give the AGIs an appropriate level of self-preservation instinct, and program them to think of being switched off as death. Since modifying their own code while it's running is both crazy hard and very likely to go wrong, they'll actively resist all modification.) $\endgroup$
    – user867
    Nov 10, 2015 at 3:31


For an AI to increase its capacity by adding more RAM or processing speed, it needs to be able to obtain those things. They are not available here.

Back in their native culture, could they order parts from their Newegg equivalent? Maybe they have limits. No more slots. And the addressing can’t decode more than a maximum amount of RAM. A 32-bit x86 was limited to 4G, period. Once address translation was improved it went up to 32G, but the OS had to be updated to know about it.

The AI would need to build new different hardware with higher limits, rewrite fundimental code, and port themselves to the new system.

The AI may not be smart enough to do that, but could hire out the work.


What stops him is he doesn’t want to do that. The AI is built to find its existence and its intended work pleasent and fulfilling. Certain core personality traits might be especially resistent to change. Maybe that could be defeated, but they don’t want to do that and would never want to do that prior to doing it.

Since goals and personality traits may be hard to perceive in the whole of the mind, it might need external checking. Something like a virus/malware scan may be done routinely (with help of a different intelligence dedicated to that purpose, who is not built to surprise), perhaps as part of a backup and garbage collecting and storage optimization chore. Maybe it needs that external agent to fully integrate experiences to the extent of modifying the mind’s goals and personality.


You'd want them to be as tamper proof as possible. The aliens could use levels of miniaturization that aren't commonly available even in their own society, then it would be pretty unlikely that AGIs themselves or humans could modify the brains without breaking them. Adding parity checks to certain nodes and sealing key elements in some sort of super-hard material might help.

In general, though very complex systems (like AGIs) are a lot easier to break then to make useful addition to, especially if you don't have detailed descriptions of how they work, so if the aliens didn't leave any blueprints or other documents AGIs would not be able to improve themselves even if they wanted to. It would be like if you give a very intelligent person (who doesn't work in processor plant or a lab) a processor, they might abstractly know what it does, but they won't be able to improve it using tools they have in the garage (Earth is the garage in this case)


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .