It could be that the solution to the Fermi paradox is that civilizations of biological creatures are typically destroyed due to being taken over by machines that are much dumber than the original biological creatures. The most advanced AI systems we have today are still at insect level intellectual capability, but we already have such things as autonomous cars. The critical point for takeover is reached when we have enough atomization to get to self-replicating sets of factories, this has a priori nothing to do with intelligence.

So, a takeover is more about machine biology than it is about machine intelligence. We know from Nature that there is no need for a huge amount of intelligence for autonomous systems to thrive. The danger of AI is then not so much that we get to super-intelligent machines who take over from us by virtue of being more intelligent. It's i.m.o. more likely that our civilization will give rise to a machine abiogenesis and then machines will spread like cancer, ending up destroying our civilization.

How can we prevent falling into this trap that may well have destroyed the vast majority of other civilizations in the universe, as the Fermi paradox suggests?

  • $\begingroup$ Not an exact duplicate, but I think it covers this question well enough. $\endgroup$ Commented Jun 19, 2018 at 22:57
  • $\begingroup$ @Draco18s no, it is not, different thing. Here the AI rather misleading, not the best title. $\endgroup$
    – MolbOrg
    Commented Jun 19, 2018 at 23:21
  • $\begingroup$ Curiously, a search for AI takeover doesn't yield as many questions as I was expecting to find. Count, we need you to review the question linked by @Draco18s and tell us if there is or isn't an answer to your question there (I agree with him that there easily could be). This isn't an issue of pride. If there is, we need to mark this a duplicate and move on. If there isn't, we need to narrow the question because how-without-knowing-anything-about-my-civilization is too broad and will result in closure. $\endgroup$
    – JBH
    Commented Jun 19, 2018 at 23:26
  • $\begingroup$ This appears to be a discussion of your ideas about the Fermi Paradox but there is no Worldbuilding question I can find. The only question is your last sentence and that is simply a highly speculative (opinion based) question. Not worldbuilding in any sense. $\endgroup$ Commented Jun 19, 2018 at 23:59
  • $\begingroup$ Four close votes for four differen reasons. I'd never seen this before. $\endgroup$ Commented Jun 19, 2018 at 23:59

3 Answers 3


Self-Replication is not commercially viable

The OP is right about one thing; what made life so abundant on this planet wasn't intelligence, it was the ability to replicate. Eventually (through evolution) variations in that replication occur and some are advantageous for living in the environment, some are not.

If DNA wasn't so good at replicating itself, life wouldn't exist on Earth, or anywhere else.

This is no different to (say) computer virii. The reason they still exist is their ability to replicate. What they do after that is determined by the payload of the virus (or worm, or trojan, or whatever) but without the ability to replicate, they're not really of any use to the person writing them.

That is ultimately why machine replication doesn't make sense; machines are manufactured to be useful in an applied sense, not successful in their ability to reproduce. To put this another way, you might have the most efficient manufacturing process on the planet (including self-replication) for your widget, but if no-one has a use for it, then the widget won't be manufactured.

While life has become successful because of its initial bias towards replication, humans have become successful because we are 'useful'. We can organise ourselves to achieve far more as a group than we could as a collection of individuals which means that we can focus on replication for only part of our existence (if the internet is anything to go by, some of us focus on it far more than we should to be useful to our society, but that is out of scope here). Machines, our products, will follow that philosophy as humans look at everything around them (including their own inventions) from a perspective of usefulness.

In short, that means that unless a dumb machine is useful (effective) at something we want done, we're not going to invest in its cheap reproduction (efficient) because to do so doesn't increase its usefulness.

We haven't even touched on the economic issue; do you really want your product self-replicating without some way of;

  • monetising the use of the duplicates, and
  • protecting the IP from people who want to reverse engineer it?

I'd add to this that a self-replicating factory is a LOT harder than it sounds to create. Factories are made of many different resources; metals, rubber, plastic, glass, etc. They take tremendous amounts of energy to create, and to operate.

I just don't see a factory being capable of mining, smelting, casting and drop forging, growing rubber trees for latex, vulcanising rubber, harvesting silica, making glass from it, fabricating solar panels, wind turbines, nuclear power plants and the like to power all this activity, etc. 'Full Stack' replication isn't viable because of the sheer diversity of activity required to harvest the materials required to facilitate it.

"But wait!" I hear some of you say, "What about Star Trek style replicators? They'd make it possible because all you need is energy, right?"

Wrong. Well, alright, not wrong per se, but impractical. To generate the materials you need from pure energy would take a LOT of energy. In point of fact, the universe already has that kind of technology and metals and the like are being manufactured every single day in special fabricators designed for that very purpose; stars.

Stars like our sun take hydrogen and through fusion, convert it to helium and other heavier elements. This is why it is believed we could well be the first life in the universe; many stars had to die in supernovas before there were sufficient metals and dense elements in the clouds from which the next generation of stars were born. Every element in your body was born in the core of a star.

But not a star the size of our sun.

Our sun isn't big enough to produce most of the elements we need for life, despite the massive amount of energy it produces. Even Gold (which is an essential ingredient in many forms of electronics) can't be born in a normal supernova; it is generally believed that gold can only be produced by two neutron stars (or similar masses) colliding.

If it takes THAT much energy just to produce gold, how much energy do you think a Star Trek style replicator would need to produce all the materials necessary for a factory? You'd need a replicator on the outside of a Dyson sphere surrounding a star the size of Betelgeuse with direct solar energy transfer technology just to consider it.

Bottom line; machines won't self-replicate to the point they get out of control because there's no commercial benefit to it, the diversity of functionality required to go 'full stack' is too broad, and the energy requirements to generate matter from energy directly is just too high. We're FAR more likely to be brought down by a semi-intelligent computer software that replicates itself constantly and renders critical infrastructure (via SCADA infection) non-functional.

Given our reliance on it, software is far more likely to destroy us than hardware. Even stopping fuel distribution by taking down critical pipeline flow monitor control systems could create famine in most major cities within weeks given how far people are from the food sources in today's world. It won't be dumb machines that take us down, it'll be dumb programs.

  • $\begingroup$ sad WB isn't a place for discussion, you have a lot to defend here - "Bottom line; machines won't self-replicate to the point they get out of control because there's no commercial benefit to it" $\endgroup$
    – MolbOrg
    Commented Jun 20, 2018 at 2:31
  • $\begingroup$ @MolbOrg I think you'll find it's already defended in the body of the answer. DRM exists because uncontrolled replication doesn't materially benefit the producer of the item. Digital material is VERY easy to replicate; no-one's going to design physical replication (which is much harder) without complete control over the process. Even GM seeds purportedly contain suicide genes so that you can't grow an unlimited number of generations of plant from a single purchase of seeds. Fabrication Rights Management (FRM) will be a big thing in a world of replication. $\endgroup$
    – Tim B II
    Commented Jun 20, 2018 at 3:51
  • $\begingroup$ yes, I did read the whole answer. I do believe things are a bit more complicated than that. On earth we do not exactly need some self-replicating equipment, usually, production lines work for ages and new top-notch lines - their development cost is about as huge as their building costa and there is no point to make few of them - like Intel fabs. But also it depends on a particular implementation of such self-replication - conventional machines, or 3d printing or gray goo. Yes, sure some control can be used on both sides. More a question is that commercial benefit - in space, it can be. $\endgroup$
    – MolbOrg
    Commented Jun 21, 2018 at 0:13

Replication can be limited to a set amount:

Just because it is useful for something to be self-replicating doesn't mean it is useful for it to be infinitely self-replicating. You could require, by design, that replication is limited to a specified number until humans request more.

To self-replicate, a robot must spend a 'replication ticket', but replication tickets can only be made by humans. If a bot has many tickets it can give some to other bots for more efficient replication, but no robot can make new tickets. Once a bot has run out of tickets it can't self-reproduce any more. If you want 64 bots, start with one and give it 63 replication tickets. It replicates, and it and the new one split the remaining tickets. this process continues recursively, growing the number of bots exponentially, until finally 32 bots each use their one ticket to make another 32 bots. The resulting 64 bots have no tickets and replication ceases. If later you want more bots, make more tickets and give them out.

If these bots are the dumb, 'grey goo' or 'infinite paperclips' kind as you said, then they won't figure out how to circumvent the restrictions in their programming, if they even want to. I'm assuming this is pre-singularity AI, in that it is not smart enough to reprogram itself to be significantly smarter.

If you are worried about unintentional bugs or intentional hacking/sabotage, then you could implement a hardware version of the ticket system. Make the robots unable to replicate some small yet critical piece of themselves, like a BIOS chip or their solar cells. These will be made on automated but 'dumb' assembly lines overseen by humans. If we make a limited number of chips at any given time, then any potential outbreak of uncontrolled replication will have an upper limit.


Managment, matter, and energy - have to be controlled by humans.

Like in a strategical game - Starcraft and alike - units have the full technological cycle and the capability of producing themselves. But they do not do anything until you say them to do something.

Controlling sources and amount of matter available for them to produce stuff also an efficient means to control a total amount of units.

For any work, energy is required and control its source you can stop propagation of the units.

So interwoven interaction of management kind of many humans with the system can control and formulate appropriate reactions for potential deviations in the process - are they human-made or a result of an error or a coincidence.

Those are 3 main cornerstones. But there a are many more and they are technology dependent - as there are potential for different implementations of self-replication technologies. (btw one of which is today's technologies with interwoven human interaction, management, and resource control)

Another possibility, as an example, could be you would not mind if the self-replicate at maximum of possible if they create the living media in which one lives. Like trees are such for plenty of animals and insects and those would not mind if the forests would occupy the whole solar system if possible. For trees, it is not possible, at least not so simple, but it can be possible for infrastructure in which humans do live in a solar system - more of it better for us if the system still obeys simple rules(basically provides resources by a request of a human) and those can be automatically monitored if rules still in action and appropriate response can be made if something fails in that regard.

Dumb self-replicating machines - are the best choice actually from all those automatic systems. They can be controlled in multiple ways and we can predict their action/reaction better and more accurate. Can a horse kill a human - yes; Is it a problem - generally not, horses revolt generally not a thing to fear, and even if it happens, it is not a problem.


Not the answer you're looking for? Browse other questions tagged .