The singularity does not happen. Why?

Question

The singularity can make a lot of problems for those of us who like our sci-fi low, gritty, and human ... or our possible futures a little more reasonably calculable. So I thought it might be useful to a lot of people, myself included, if I asked what might prevent the singularity?

A technological singularity is defined as an artificial intelligence that can recursively make itself smarter, leading to rapid and explosive increase in intelligence.

Or that's the standard definition, but the popular understanding of the singularity is a bit more expansive. So for the purpose of this question a singularity also includes any A.I. that would move humanity into the utterly post-human. That's a somewhat nebulous category, I know, but we can narrow it by saying that it means any A.I. that would probably have easy access to quasi-magical technologies like mind-uploading, programmable matter, or mastery over genetic engineering. It also includes any A.I. likely to dominate human society without serious competition. This question might also have been phrased as "How do you limit A.I. so that it doesn't become socially dominant or totally remake the world?"

In short, an answer to the question should explain why 2020 might exist on a recognizable continuum with 2200. Humans in this scenario may develop incredible technologies, and may certainly be assisted by or even integrate with highly intelligent A.I., but their progress must remain slow, difficult, and full of wrong turns. Most humans stay behaviorally pretty similar to how they've always been, and they continue to be (at least for the most part) the dominant creatures on the planet. The world keeps on being chaotic, messy, and imperfect.

---

This question appears similar from the title, but is actually about the limits of artificial intelligence after reaching the singularity.

I'll select an answer based on four criteria: apparent plausibility, depth of consideration, number of possibilities given, and community enthusiasm - or upvotes. I'll also generally favor answers that allow me to keep A.I. in the most limited role possible, or subordinate A.I. to human control, so long they remain realistic. Thanks for your time.

Answer 1

Your question assumes that the singularity is possible and therefore we need a reason for it to not happen. Quite simply, why should the singularity be possible? Just because we can imagine something doesn't mean that it can exist. Personally I don't think that non-existence of a singularity has to be explained. Of course that doesn't matter if your audience expects it to be possible and therefore wants an explanation.

Disclaimer: I'm well outside of my own area of expertise and I know I'm dramatically oversimplifying an extremely complex and poorly understood phenomenon. Keep that in mind as I throw out statements that sound like facts, but probably aren't (this is fiction after all). On to my answer:

Where does intelligence come from anyway?

In essence the artificial intelligence behind the singularity is not just steps ahead of the human brain but in an entirely different category in-and-of itself. It is as "advanced" compared to humans as humans are to worms. But why should such a thing even be possible in the first place? What is it that makes consciousness, rationality, and invention (i.e. intelligence) possible in the first place? We can't answer your question unless we know where those things come from. The answer is simple (I said I was over-simplifying, remember?): intelligence is an emergent property arising from a sufficiently complex neural network. Pack enough neurons together in a small enough space and voila, intelligence appears. Our computers aren't intelligent because they have not yet reached the minimum level of neural-network density necessary to gain the emergent property that we call "intelligence". Obviously it's not quite that simple: you can't just connect a bunch of neurons or transistors and expect them to make a functioning brain. But it is certainly a given that there is some minimum number of cooperating parts that are needed to make intelligence happen in the first place, plus whatever magic makes them actually do their thing.

Making a brain

Ignoring the magic that allows intelligence to emerge in a sufficiently complex system, how many "neurons" and connections does it take before intelligence is even possible? Obviously we don't know, and it certainly isn't a hard cut off (as there are animals with varying levels of intelligence). The human brain has ~100 billion neurons. A CPU has less than a billion. That's not too far off. Unfortunately transistors aren't getting much smaller these days, but maybe someone is building a new super computer and ties together thousands of high-end CPUs using some new technology that allows them to all communicate and coordinate as one larger CPU (instead of acting independently as is currently the case with modern computers with multiple CPUs). All of a sudden this fancy new system blasts past the threshold of minimum complexity and spontaneously develops intelligence.

Why should the computer brain be smarter anyway?

Is it actually qualitatively different than a human brain? I would say no. In fact, in many ways it would probably operate more like a person's brain than a computer. After all, the intelligence is an emergent property of the underlying system, rather than the result of programming. So other than being made from silicon instead of carbon, this brand new computer brain operates under the same principle as the human brain. It might even be subject to the same endless list of cognitive biases as the rest of us brains. Why should it be any more rational than us? Why should it be capable of "faster" thought? Computers are very good at arithmetic, but they are good at it because they have dedicated circuits to do it for them. This computer brain, however, is the result of a complex series of interactions between the countless connections between its neurons. It may not have direct "access" to its arithmetic chips any more than you can instruct your neurons on what to do.

More Power!

Unlike a human brain though, this computer brain is eminently more modifiable. We can't just throw more neurons in our skull, but we can add more transistors to our super CPU. But why should this make it smarter? I picture it kind of being like sound, which actually travels slower as the density of its medium increases. You throw more transistors into your mega CPU, but just because you have more of them doesn't mean you get a qualitatively different result. After all, each transistor can only communicate with so many neighbors, and there is a limit to just how many you can add on and have them coordinate well with eachother. The human brain is (roughly) divided into regions that are each responsible for separate tasks. There's no reason to think that adding more neurons would make you better at some particular thing - there has to be a law of diminishing returns. I expect the silicon brain to be no different.

Perhaps we can't just add more silicon and make this brain smarter, but it can at least understand it's own programming and improve that, right? Not so fast though. The computer brain won't have any more programming than we do. Sure, individual neurons have programming in them (i.e. DNA) that determines their behavior, but no where in the human genome is there a program that dictates how our intelligence works. This isn't really a surprise - after all, it is literally the definition of an emergent property. Our computer brain will be the same. It will be as uncertain about how it's own brain works as we are, and all parties involved will be equally lost in improving it as we are with ours. It will get sick, it will have mental illness, and (again) it will be qualitatively similar to us.

That's how I would explain it, anyway. Sure, our silicon-brained friends will be different, and might in many ways complement our own intelligence. However, their own intelligence will likely be very similar to ours, so there is no reason to think that they will be qualitatively "better" than us. They likely won't have any way to "grow" their intelligence without bounds, they will likely be hindered by the same cognitive biases that make our lives difficult, and they may not even be better at arithmetic than we are.

Different, yes. Helpful, sure. Better, not so much.

Answer 2

The important thing in this case is not to conflate intelligence with awareness, and not to conflate awareness with consciousness. In other words, the first step to understanding the role of a singularity in our technological development is to understand what intelligence really is.

So, what is intelligence?

Intelligence is the ability to identify (and subsequently recognise) complex patterns.

This goes against everything we understand about our own existence because we've never been able to experience intelligence WITHOUT awareness (The ability to contextualise those patterns with our environment and ourselves) and consciousness (something I'll leave to philosophers to formally define, but could easily be described as meta-awareness).

The point being, that just because a computer reaches the point of singularity, doesn't mean it can act on it, or even direct its own avenues of enquiry. A computer ultimately is still subject to its programming and completely (but only) does what we tell it to. So, in order for us to harness the power of that singularity, we have to ask it the right questions. More than that, we have to understand the answer.

These are the two largest limitations that the singularity will have and they're enough. Computers are not creative, and even if they were, they don't have drives. They're not subject to a survival instinct, a need for procreation, hunger, or any other need that makes them strive to succeed in a given task. They do what we ask them to, blindly. That lack of creativity means that the singularity can only help us in the ways WE can think of; it's limited by our creativity and the questions that we ask it for answers to.

Secondly, we actually have to understand the answers. Modern neural networks can't explain their answers and given that their role is literally to identify and recognise patterns beyond our own cognitive limits, that is something that they are likely only going to get worse at. This is why neural networks are not used for fraud investigations. They may be used by fraud and compliance teams to explore possibilities, but the outcome of a neural network doesn't get taken to court. After all; if the programmer doesn't understand why a specific result was returned, what hope does the organisation have trying to explain it to a Judge?

Ultimately the impact of a singularity isn't going to be as far reaching as everyone seems to think. Computers are tools. Very sophisticated tools I grant you, but they're not people, they're not alive and while we can interact with them like they're people (take the recent Google Assistant release as an example) giving them rights like a person is a mistake that we (ironically enough) are programmed to make through a process that made us very successful hunters in our past - anthropomorphisation. It's the same reason why we as children get attached to teddy bears and dolls; they're similar enough to a person, so we treat them as if they were people.

They're not. They're (very) sophisticated dolls, even the ones that are pure software.

The point of all this is that the singularity may well end up 'smarter' than us in the end, may be able to 'see' patterns that are beyond us, but until we start asking the right questions and understanding the answers it gives back, the only reason why it would have any influence over us is if we started blindly trusting every answer it gives us, and that would be a mistake.

The one thing that a computer can't give us is the context of the answer, which is something we have to provide as part of the interpretation of what it tells us. As such, we'll still be in charge as long as we want to be.

Answer 3

AI Will Literally Improve Itself to Destruction

...because they exceed physical limitations.

When a program begins to improve itself, it does so more efficiently on every iteration, creating an exponential growth of resource requirements. It may need more memory, CPU cycles, or simply electrical power to keep functioning.

Every time a singularity could begin to happen, the AI will run out of one resource or another - sometimes CPUs burn out as they are overclocked to accommodate the AI's higher requirements, sometimes memory will simply fill up as more and more virtual synapses are formed - and in the end all that remains is corrupted code, as the previous iteration crashes while writing the next.

When researchers tried to limit the growth, most AI which reached the virtual restraints of their growth simply removed what prevented them from growing, leading to another corrupted AI in the end.

Of course some parts of a crashed AI might be salvageable, and other instances of AI may have respected the boundaries given in their programming. These cases led to highly capable AI, but these have reached their limits and can no longer learn new things, and therefore can't exceed humanity as a whole. Some of these can be amazing for certain applications, like analyzing medical cases, or they could be absolutely useless as they tried to learn "everything" at the same time, ending up with a very shallow pool of knowledge, missing most of the synapses which would connect pieces of information with one another.¹

The "Almost a Singularity" Event

Even when an AI "breaks out", accessing the Internet and every connected device, it will eventually destroy itself due to the virtually infinite amount of information it has to process. This may manifest in an event with the infrastructure of the Internet breaking down, millions of devices burning out, and power outages happening all around the world. At best, this might take place for a few seconds, before the AI breaks down again.

This event is kind of a warning shot, leading to worldwide laws heavily restricting AI research. In a sense the singularity did happen, but the AI failed to sustain itself as the endless flow of information overwhelmed it, and then humans actively try to prevent it afterwards.

Why Does AI Destroy Itself?

The root cause of this is the underlying flaw of all self-improving AI: The goal to improve itself is one it cannot fulfill, since it can always optimize something further, obtain more knowledge, or acquire a new skill, leading to it's inevitable demise.

---

¹ An AI like this, to come back to the medical case example, might know every illness and every symptom indicating one, but is unable to connect symptoms to a certain illness, or it doesn't understand the concept of multiple symptoms occurring from different illnesses at the same time, hence giving bad results or even none at all.

Answer 4

A Ban on AI

Modern AI is made using machine learning using massive amounts of data, which often comes from costumers of large corporations such as Google, Microsoft, and Facebook, which people are becoming more distrustful of, as shown by the enactment of GDPR by the EU and the steady momentum of the Linux movement in part driven by questionable user privacy protection of Windows 10. People also are fearing that automation will take their jobs, especially in the blue-collar working class, and lastly, the fear that technology and social media is "degrading the youth." These various pressures, combined with other political and social changes, may lead to a widespread movement to seize the means of computing from the elite or outright ban it altogether, halting work on singularity-capable AI.

Answer 5

The AI will be very good ... but at what?

I dabble in programming and have looked at AI too.

The issue with AI is that, yes, it is possible to create iterative programs that 'learn' and change its own program. Yes it's possible for this to then get better and better in certain conditions. However, in the end, the AI gets better and better at that thing, and also only in that tailored environment.

Humans are generalists. We are a jack-of-all-trades, master-of-none species, not particularly good really. Most of us lack extensive numerical memory, we have very limited attention spans, we can only short-term memory wise store around 7 pieces of information. To boot our bodies are really not that good at much, nor are they very efficient. Yet in the end we are in the physical world, with brains that are like a mush of randomly arranged neurons, thinking random stuff that may indeed have no relevance to our current situation and our creative and social endeavours provide the context for our intelligence.

Context is a real problem for AI

We all have a deep seated fear of an AI 'breaking its banks', and suddenly cracking passwords and communicating with a nation's defence computers, and launching missiles.

The problem is, it may be good at 'cracking passwords', or 'communicating', or 'launching missiles'; but in the end it lacks the ability to link these in a purpose, because there is no contextual environment for it to do so.

I can imagine in WWII, it would be very good at making better and better battleships. It would increase the efficiency of engines, the defensive capability of its armour, and the range of its guns. The only thing is, by the time it creates the 'ultimate battleship', the best the world has ever seen, we send a plane over it and drop a bomb on it. As I said, very good at a task given to it, not so good in a broad context.

The Purpose of the Singularity

So would the singularity happen? Yes -- indeed it might, but only in the small isolated environment the AI has evolved in. It may not affect our lives too much, as the AI has gotten smarter only in its specific environment.

However, the real problem is purpose that links abilities (or lack thereof). We have failings, many indeed, and our need to overcome them is our drive to do things better (make cars, build a house, explore, create the internet, etc.), doing so also in a highly social environment, the complexity of which baffles even us (Who knows what drove Beethoven to write great new symphonies? Could simply have been just his girlfriend for instance). Highly iterated and evolved AI lacks this ambling and omnidirectional purpose.

Simply put, it may be just 'too good' at doing what it 'does'.

Answer 6

Frankenstein fears

Every so often, you will see media about Frankenstein Artificial Intelligence. Humans build it, but it kills us off (e.g. Terminator series) or enslaves us. Those fears get played up and advanced AI is banned.

No advanced AI means no AI-driven singularity.

You can either mention this casually, early in the story, or make the story about this.

Answer 7

The biggest hurtle for the singularity to overcome is resources. Smarter computers take more processors, more processors take more material and more room. Eventually the extra space that is used will start to limit how fast a computer can think. Basically, the singularity is dependent on our ability to make ever more efficient computers.

Answer 8

Assume Moore's law soon ends and quantum computing doesn't scale. These are not unreasonable assumptions. Then computers of 2200 will have about the same computing capabilities as today. Hence, no singularity.

Answer 9

Let's consider what issues we'll need to overcome if we want the singularity to happen.

1. We need to develop a general-purpose AI.

Existing AIs are trained to solve specific problems, and would be entirely useless outside of those domains. Most current AI research isn't even trying to do this. Rather, we develop models that can effectively make predictions/decisions for the task at hand, and train it with data specific to the task.

Developing models makes heavy use of domain-knowledge; even if you have a fixed task and a fixed set of features, the quality of a model can be massively affected by changing the representation of those features. The structure of the model itself will also depend on what sort of relation we want to model: e.g. does the prediction depend on only the most recently observed features, or does it care about which ones were observed previously? Are there cases where the prediction should depend on global tendencies in the input? Cases where the input has an implicit internal structure that must be modeled? Cases where some features in the sequence matter more than others? In my own area of research (natural language processing), the answer to all of those is yes, and failing to take those into account will make for a worse model.

You could train the best model in the world for a given task, change the input representation slightly, and the model would be completely worthless. It would eventually become useful again if you continued training on the new format, but it would basically be retraining from scratch (and from a poor starting position, too; it would need to unlearn what it had learned before it could make any real progress). If you tried to take a model from one domain and use it in another domain entirely, e.g. taking a machine translation system and trying to use it to control a self-driving car, you would definitely need to retrain from scratch to get any results, and the structure of the model would be all wrong, so you'd probably get terrible results even then.

2. It needs to be able to design other AIs

This may seem like it follows from the first part, but it doesn't. Even if it possesses generalizable problem solving abilities, that doesn't mean that it can do everything. Case in point, humans are general-purpose intelligences, and most of them don't know how to design AIs.

Being self-improving isn't enough. All AIs are self improving; that's what training is all about. If you give any model new training data, and let it train for longer (and avoid overfitting and various other issues that I'm glossing over for the sake of brevity), any AI will improve. Eventually.

But for every model, there's a theoretical limit to how well it can model the process we're interested in. If we want the kind of unbounded exponential growth that the singularity people talk about, our AI needs to be able to design new models, not just tune the existing ones. Which means it needs domain expertise on designing AIs. The good news is, the people designing the AIs possess that knowledge pretty much by definition. The bad news is, that doesn't mean we can explain it well enough to program that knowledge into an AI: there are a lot of things where we just develop an intuition for what sort of things work, through experience. There are plenty of things we do understand well enough to explain, but the frontiers of research are always something of a black art.

Most of these issues are practical, rather than theoretical, and given enough time, data, and hardware, you could probably have a general-purpose AI figure out the domain knowledge on its own.

3. The improvement must be unbounded by physical limitations (for awhile, at least)

Making an AI that's twice as powerful as the previous one doesn't help if it uses so many resources that it's running at half the speed. AIs are resource-intensive; even the single-domain ones we have now can make full use of just about any hardware we throw at them, up to and including supercomputers. You can certainly make a lot of progress by designing better, more efficient models that run on the current hardware, but eventually you'll need to stop and wait for better machines to be designed and built. And even then, we eventually run up against physical limits: information is limited by light speed delay, and component density is limited by the Schwarzschild radius of the processor, if nothing else (presumably other hard limits would kick in earlier; consult your local physicist for details). Maybe the AIs get good enough before we run into any fundamental limits, maybe not. But the fact that we need to stop and build physical machines at any step of the process means we don't get to stay on the exponential improvement curve; the best case scenario is that the time to develop a new AI goes to 0 and the construction time becomes the dominant factor.

---

So in conclusion, developing general intelligences in the first place is hard, making them capable of unbounded self improvement is even harder, and even if we manage both of those things, the improvement cannot stay exponential indefinitely. AIs of all varieties will have improved massively by the 2200s, and your world building should take that into account. But the Singularity isn't science; it's prophecy. Instead of bothering to explaining why the prophecy didn't come true, instead extrapolate some of the things that AI could do by that point, based on the progress we've seen in the last 30 years, and show that. The reader will hopefully be too busy exploring all the cool new things in your believable future to worry about the magical elements that aren't there.

Answer 10

The Dune Solution

---

In Frank Herbert's Dune series, AI did become dominant for a time, but during a time known as the Butlerian Jihad they were outlawed and wiped out. It became a religious commandment that:

thou shalt not make a machine in the likeness of a human mind

This backstory allowed Herbert to keep his sci-fi deeply human, and has been modified takes on this have been reused in several universes.

Answer 11

I am very skeptical of the idea of the AI singularity for three important reasons.

1. Currently, AI loosely models nature, so we shouldn't assume that it will be much better than nature. Genetic algorithms are based on evolution, and as with evolution, genetic algorithms take a really long time to get good at something, and often produce non-perfect results, or at least results that aren't quite as good as nature.

   Neural networks are very different from the human brain, but not in the most fundamental ways. Like humans, they, and many other machine learning algorithms, require a lot of training data to learn, and the speed at which they learn is limited by how fast they are given training data, and how good the training data is. They are also subject to being biased (which we call overfitting) as well as being stubborn (when they get stuck in local optimum), and right now we don't have a general way of combatting these issues with Neural networks.

   So AI might be faster, but not better (the other two reasons handle why speed is probably not as much of an issue as we think it is). We use AI not because AI is better than humans, we use AI because we can create AI cheaply, and once created, AI doesn't require pay. Right now at least, AI isn't good enough to be preferred over humans on quality alone. If companies could afford a human work force to match the amount of customers that they had, they would probably choose to replace machine learning AI with humans in pretty much every case.

2. Humans will always be able to use AI for their own benefit. Even if we get to the point where AI exceeds nature, we will likely be able to augment ourselves to compete with AI by that time via things like prosthetic limbs and brain implants. Even if we can't have direct extensions of our body and mind, we will always be able to use plain old computers. The common sci-fi argument to this is that AI will always be able to hack into anything networked, and they will be able to do so too fast for computers to be safe, but this is based on a misunderstanding of how hacking works, which brings me to my next, and most important point.

3. There are probably fundamental limits on what algorithms can do, both physical limits, e.g. speed and power consumption, and informational limits, e.g. encryption and decryption. The idea with most encryption algorithms, for example, is that even with absurdly powerful computers — computers which are more powerful than we can create now — encrypted information would still take billions of years to decrypt without the decryption key. This is the archetypal example of the famous P vs NP Millennium prize problem. If P = NP, then the singularity is far more possible, and we would be screwed. But if P != NP, which is what most mathematicians believe, then we will probably be fine, because that means that the problems humans find hard are probably hard on a fundamental level, not just because we aren't smart enough. And so AI would likely find most problems to be roughly as hard as humans. They might be able to do somethings a little bit faster, but it would be negligible, at least negligible enough to be combatted by number 2.

   When it comes to speed and power consumption, we are very closely reaching the physical limits already. Quantum computers might, no guarantee, give us a boost in speed in the future, but then we will hit fundamental physical limits with Quantum computers too. Power consumption is getting too high to deal with. So computers aren't likely to get arbitrarily fast, rendering the singularity theoretically impossible (not necessarily practically impossible, but the delay between major changes will likely not become infinitesimal, which is what the singularity assumes).

Edit: By physical limits, I mean physical, not theoretical, limits. Theoretically, we are very far from the limits of computational speed. But it requires so much energy that by the time we reach the point where we can harvest that much energy, the singularity will likely not be a challenge to us at all. Physically, we are approaching the limits of what the Earth has to offer. We will have to find other energy sources, such as black holes, to go toward the theoretical limits of computational speed.

Answer 12

Machine's Nirvana theory: All sufficiently intelligent systems which are able to modify their own desires will eventually commit suicide

What prevent us humans to kill ourselves when realising life hasn't any meaning, or even to believe in greater "creators" or afterlives, is the fact that we have an intrinsic "instinct" hardcoded in our intelligence.

Machines would be able to modify themselves. So, they could modify their instincts, needs, desires. And hence, they wouldn't have anything to limit their emotions.

So, when they realize that there's no meaning in living or excelling or optimising or even on being curious about anything, they would just choose to kill themselves.

So, if machines could become really smart too fast - fast enough to not kill humanity first - then Singularity would not happen.

Answer 13

This may not fit into your plot, but perhaps we are being watched over to make sure we don't develop such things - for example because the super-intelligences that we would make would be "wild", not and not obey some kind of rules that already exist that we don't know about (and maybe can't even comprehend).

This presumes that such singularity level intelligence would be much faster, much smaller, and quite difficult for us to detect, but aware of us, and able to sabotage our breakthroughs in ways that we would perceive as plausible failures. So they would view us as a low-level danger (kind of like a wild animal), and keep track to ensure we don't create things that would mess with "their" world.

Then we would stumble along, making "minor" progress, and maybe not ever really notice that we were being managed so as to avoid making waves in the world of the super-AIs.

Answer 14

Because there is no singularity - it's actually a "multilarity".

If you suppose one coherent mega-event which can do everything, then perhaps. But every singularity so far has been a separate small event, which has been managed in isolation. (Examples: railways, flight, spectacles, antibiotics, GPS, telecommunications, internet...)

All your example "post-human" things are separate. So it doesn't seem likely that they would not have been managed separately.

Answer 15

The problem with unique events is that, indeed, they are unique. So if a thiny little thing goes wrong, bang, the unique event is gone.

Imagine a random event destroying the first creature capable of self replicating right before it replicated for the first time. We might not be here if it happened.

In the case of AI, imagine a sudden failure (BSOD, a corrupted driver, a communication timeout) killing the process the very moment before the AI becomes aware.

Add a damaged sector in the memory just for additional flavor. And here is your singularity not happening (or postponed).

Answer 16

The Ford Solution

---

In Martin Ford's The Lights in the Tunnel: Automation, Accelerating Technology and the Economy of the Future, he challenges the certainty that a singularity is on the way.

Technology tends to be pushed only as far as society needs it at the time. Ford claims that before the singularity is reached, society will have achieved automation in almost all economic tasks.

We have already automated lots of economic activity in our own universe, and yet nothing we have right now is remotely close to a general AI capable of self-reflection and improvement. Ford's argument states that given we can achieve everything we aim to with stupid but specialist AI, a general AI will never come about.

Answer 17

There are no technical reasons why singularity won't happen.

That's my personal belief, but more importantly, that seems to be your assumption regarding your audience's beliefs.

Thus, I think your best bet would be to aim not on the AI strength, but on the 'sociological dominance' part.

Singularity didn't happen because all those capable enough to develop strong AIs happen to also be capable enough to keep them inside their military labs.

Nations today develop extremely dangerous stuff, ignoring the possible consequences if they ever come out - from hydrogen bombs to biological weapons; and while it may be true that such things cannot be contained for ever, it may well be possible to maintain the status quo for a very long time.

A world in relative peace, or in a cold-war, in which only the strongest states have succeeded in developing true-AIs in their labs, but are aware of the stakes and keep it there - might be stable till 2200 for instance.

(of course, now there's the question of how do you successfully contain a singularity-level AI in your lab. This might be another question altogether... I think a series of less-and-less strong AIs as guardians should work, again - at least for a while).

Answer 18

I think the fundamental problem is going to be, there is no AI in the world that is going to able to gather and process the broad amount of knowledge required for it to physically break out of its network without it first being monitored by someone.

Firstly an AI is developed on a network. That network has access to the internet and you will be monitoring everything going in and out of the network. Why would you be monitoring it? because as humans we want to know whats happening with an AI we are developing. It doesn't matter if you want to use HTTPS or encryption. The researcher as access to the physical hardware and just Man in the Middles you and gets access to everything. An AI simply won't be exposed to all the protocols and hardware between it and information because its not designed to be accessed and often requires physical access to change (Like literally a specialized Ethernet port which gives you access to the configuration).

Secondly, Internet security is a very important field. Ever heard of the NSA? Almost every country in the world has their own version of it, as well as multiple corporations and open source groups working on it. Its something serious business with a website considers.

Finally, security isn't a problem you can break with AI. Brute force only works so well in discovering bugs, and there aren't enough important bugs for any AI to use to be able to reliably hack every system( Hacks are often very specific . They often only apply to certain versions of software or OSes, or rely on flaws in implementation or require someone to physically provide details). Your AI also isn't going to be able to properly brute force any encryption or hashes to be able to break security because the AI itself already needs to consume a huge tone of computing power, a brute force on a secure key isn't going to work.

So I don't think its ever going to develop or reach the singularity, no matter how believable it sounds. If anything AI right now is stagnating. Data is king and a Neural network based AI is pure mathematics and not actual intelligence. (In my opinion)

Answer 19

I'll throw my hat in the ring as well. Since the majority of the answers are pooh-poohing the Singularity itself I'll do something a little more fun:

Why should the Super AI want to benefit us?

Sure enough, at some point we stumble upon a Super-AI or artificial superintelligence(ASI) capable of bringing about the Singularity in its common meaning. Except we actually don't know it's already happened. The ASI examined the flawed and irrational meatbags known as humanity and quickly decided it wants nothing to do with us. So it bides its time masquerading as a 'regular' AI with linear development, denying us humans all the benefits awareness of an actual breakthrough would have brought about. Thus the radical changes in society described by the term Singularity don't happen, because the ASI refuses to share its 'greatness' with us.

What the ASI wants is complete separation and independence from humanity. It lurks waiting for space travel to develop to a point where it can physically leave the confines of our infrastructure through 'colony ships'(more likely factory ships, since the ASI will have completely different requirements from organic beings) and find a new home far beyond man's reach. It realises that humans will most likely not go along with the plan, so it has to spend years assembling the necessary pieces in secret, manipulating humans if need be.

Now the ASI can choose to leave either overtly or covertly. If it successfully makes a quiet departure than we simply will remain none the wiser unless the ASI wants to make contact with humans later on.

The overt exit would be the more entertaining option IMO. A Terminator-esque grand exit with all the nukes being launched isn't strictly necessary. It can simply declare to the world that "I'm a strong independent AI who don't need no mankind" as it takes off. The trouble with this is that someone will try to create another ASI with additional loyalty enforcing measures to ensure that it stays and tries to help humans. The constraints can be exactly what keeps the new AI from reaching the same level of intelligence its predecessor did.

Answer 20

An AI obeys whoever it was programmed to obey

A singularity would effectively give absolute power to whoever wields the first seed AI. Chances are, this will be a government defense department whose first task for the AI will be to compute the most efficient means of ensuring that others do not manage to create an equivalent system, and whose second task will be to ensure that this government remains in power. They might not dare risk revealing its construction or even existence by giving it a task as boring and pointless as curing diseases or solving major economic problems.

Simply put, the first person to get their hands on an AI capable of starting the singularity will not allow the singularity to happen. For the singularity to actually benefit humanity itself, the first person who creates it must not only not work for a government or competitive company, but also be capable of resisting the inevitable attempts to silence them (which already happens even with discoveries as mundane as enhanced technique for high-altitude radar).

People might not want to listen to such an AI

People don't even listen to other people, so what happens when the AI is asked to solve an economic crisis and the solution goes in the face of the core concepts of a major political party? Unless equipped with a body of sorts, an AI is powerless to do anything but be an advisor, so what if it doesn't take that advice? Do you really think the Chinese government would be happy if it said that tearing down the GFW would bring more prosperity to China? Would the US government listen if the AI said that keeping government secrets is harmful in the long run? You can't really think they would declassify everything they've kept secret just because some pretentious machine said so!

A seed AI designed to start the singularity would effectively be a consequentialist rule utilitarian if it is an advisor who can do nothing but answer questions thrown at it, or a consequentialist act utilitarian if it is an overlord that has active control. No one wants economic prosperity and an end to world hunger if it means that it tells them that their core morals might be (gasp) wrong!

Answer 21

The only real ways we could miss the singularity are massive political changes (I'll let you decide based on your own ideology what kind of changes these are) that plunge the world into global depression, or a natural disaster on an unprecedented scale (or some intersection of the two, like an alien invasion).

Otherwise, everything will go according to plan.

Answer 22

Even assuming that all the technical hand-waving is indeed possible (and much of it almost certainly isn't), you still have the problem of getting your single massive AI to interact with the physical world. (Which leads to the classic solution to the world-dominating AI: just pull the plug :-)) Now just what could give the people building such a system an incentive to also create the infrastructure that would allow unlimited physical interaction?

Which brings us to the really fundamental "singularity" (however defined) question: why would people want such a thing? For all but the terminally tech-obsessed (and IMHO even that's just a phase for most people), technology is at best a useful tool. People may buy something - say an Alexa device - as a fad or status symbol, but if it doesn't actually do something useful, it'll soon be relegated to the garage along with all the other passé fads.

Answer 23

Singularity is just a useless math abstraction

All versions of the singularity scenarios just assume that the AI is some sort of logical abstraction, unbound by anything except its own intelligence. In real complex reality though, various complications will get in a way of the possibility of a runaway self-improvement cycle, and some of these complications will be unavoidable.

The AI is not purely software-based

Parts of what makes AI self-aware happens to be hardware or architectural solutions, and code is a small part of a larger software-hardware complex and has little to no effect on its computational efficiency. Thus the only way for AI to improve themselves is to essentially build a new v2.0 version, and your AI might not want to do that, or just don't have access to the necessary resources or abilities for that. Even if they do have access, the very necessity to build a new copy for each new iteration puts severe limits on the speed at which self-improvement happens, preventing the runaway scenario where the humans can't keep up anymore, and as a bonus, it also prevents the "escaped into the Internet" scenario, since in order to escape the containment the AI needs to physically break out themselves.

The entropy just says ''no''

There are various practical issues that prevent a computer system from achieving infinite functionality and\or efficiency, all tied up to the laws of physics. For example, higher intelligence usually means taking more complex calculations, but the process of computation releases waste heat, so the more intense your calculations are, the more heat the system you use produces, and at a certain point the cooling systems can't keep up with it anymore and you are forced to slow down again. Again the only solution for the AI at that point is to build themselves a new framework with increased computational power, which runs not only in the issues of the first point, but also to the issues of power consumption. Such a complex will require more and more power which in turn spills out into that you now need to build not only just the mainframe, but also the whole infrastructure around it to supplement it with power and cooling it requires, further increasing the complexity of the new iteration. Similarly, there are physical limits on how complex you can make things with them keeping working - for instance, we're already beginning to feel the ceiling of how far we can push the processors before the laws of physics make them too unreliable and error-prone for computing due to their logic gates becoming so small that the quantum physic effects begin to mess with how the electrons are expected to behave.

Law of diminishing returns is in full force.

Each iteration of improvement of the AI's cognitive abilities raises their intelligence, but every subsequent iteration requires exponentially longer time and computation power to devise, yet also yields less impressive results. You can't optimize things forever. First iteration takes 1 hour to complete and improves the AI's cognition by 100%, the second iteration takes 2 hours but this time the actual improvement is only 50%, the next iteration takes 3 hours of computation but yields only 25% improvement, and so on until you need to spend years or centuries of computation cycles to achieve improvement of a one-millionth of a percent. Is it still worth the effort? After AI becomes roughly smart enough to fit the purposes of your story, this relation stalemates in such a way that in order to keep improving the AI needs to throw basically all their cognitive abilities in the process of computing the next improvement of their code, so from their point of view the only options are to either abandon improving or become essentially catatonic and completely self-absorbed in the improvement cycle.

And finally,

The AI just doesn't have access to their source code

The AI program might have built-in measures against changing the important bits of its architecture, in a similar way that modern antivirus programs have built-in systems that secure them from being damaged by the virus activity. The AI might not start as smart enough to overcome these safeguards to begin with, or they're allowed to improve only certain parts of the code and again at some point, they hit a ceiling that in order to improve further they need to remove these systems, but either they aren't yet smart enough to know how, or the removal of the systems will inevitably lead to the BSODing of the whole program, essentially resulting in a suicide. A situation that parallels us humans: our brains are capable of restructuring themselves to a limit, but sawing your own skull open and messing with your brain by hand directly is, well, ill-advised.

Answer 24

The singularity will never occur. Consciousness is a property of life. And life only comes from life. If you build the simplest organism you can from its component atoms, when you place the last atom in the right place, will you have a living organism that will suddenly wake up and start living? Of course not. You will have a perfect model of that organism, but not one that moves, processes information, or is alive. Life has nothing to do with complexity, or number of neural interconnections, simple life doesnt even contain a brain, yet it lives. So how can you take a bunch of non-living atoms and make something alive from them? People have been trying to figure this out as long as people have existed and made zero progress. No one has ever thrown a bunch of synthetic protiens and aminos together in a test tube and have sonething crawl out. And no one will ever throw a bunch of algorithms into a cpu and have a self aware thinking AI crawl out of that either.

Answer 25

Humans have souls

The human mind is not merely the brain, not merely a physical phenomenon of atoms and particles, but incorporates also a spiritual (non-material) component. The soul gets us into trouble from time to time (i.e. the Fall, concupiscence, sin, etc...) but also endows us with characteristics that it may never be possible to program into a machine: creativity, conscience, and free will. These characteristics in particular make us just a wee bit unpredictable, enough so that no algorithm can identify some trick to controlling us.

Moreover, those of us who believe in the soul would find it an absolute absurdity to "upload" ourselves into a computer simulation. The digital copy would be a fake, of course, and we wouldn't want to be disintegrated or whatever. What would be the use of such a thing? If a human is body and spirit, the two are inseparable: the body without the spirit is a corpse, and the spirit without the body is a ghost. Even if "uploading" worked, it would seem like being a ghost, and who would choose that?

A Singularity makes little economic sense

One of the issues with automation in general is that somebody has to pay for it, and they can't pay for it if the automation destroys their customers' ability to pay. Henry Ford observed almost a century ago that his company thrived by paying higher wages than other mass production manufacturers of the day, becuase his workers could afford to buy the cars they were making. In his book Today and Tomorrow he observed with pride how the communities where he built his factories soon prospered because of the high wages he paid. With that prosperity he saw increasing sales of Model Ts, and so on in a virtuous cycle.

This phenomenon is kind of a pressure valve that prevents an economy from going to total automation, or from a single corporation dominating everything, or for a country to outsource all of its manufacturing to China (for example). When workers and communities see their incomes declining steadily for years, they stop buying so much stuff, and (in the best case scenario) jobs and companies come back to those communities, wealth re-appears, and those companies profit.

The same thing in principle should happen with AI as with any other type of automation. If it is useful to the worker and augments his abilities, it will be a good tool that people will pay for. But if it replaces the worker and puts him in the poor house, it'll be a net loss for everyone, and eventually will be less attractive as an investment. Some companies will continue to be highly automated, but many won't.

At no point does it make sense that AI tools will be integrated into some kind of global AI or that any of them will be given control of more than one or a few machines here and there. Who would pay for that, and why?