I disagree very strongly with most of the opinions here, regardless of which way they stand.
It is my opinion that the AI threat is massively overstated, and completely orthogonal to the true structural threats that technology has to society. I understand my opinion is not the common one, nor does it align with where those who seem most educated on the topic seem to align, but I cannot in good faith claim this to have swayed my opinion.
For what it's worth, I don't expect other people to agree with me. I hope at least I can encourage scepticism to some of the more extreme claims.
Let's start with Draco18s' ideas. Firstly his claim that
This is a trope that arises from Real World AI research
This is not something I can at all endorse. Perhaps this is from real-world philosophy, but it is certainly not an inevitability borne out in concrete research of any kind.
Current AI research is astonishingly successful, far more than some give it credit. But it is entirely naïve; its power comes from the surprising simplicity of tasks we'd previously thought to be hard. For example, with a lot of data and thought thrown at the problem, we can get neural networks to describe scenes from images. I can't begin to express how cool that is. But the semantic model behind it, the actual intelligent mind underlying this classifier is still really, really dumb. These advances don't show an intelligent mind making high-level inferences, they show that the space of crude image descriptions can devolve into very trivial and uninteresting functions of their inputs. Because of this, they're supremely easy to game.
If you look for thoughtful decision making in current AI research, you will not find anything that suggests any imminent threat. It is just not a reality of today's research.
At the end of the day, this AI is not AI as you probably intend it to mean. People are still deciding the cost functions and the nuances. Draco18s is right in saying
No amount of predictive software is going to make those hard decisions easier
but he is wrong in the next part. Giving the AI the final say to an AI like this cannot produce evil any more than cruise control would be evil for going over the speed limit. It's problematic, sure, but there's no evil and any threat comes from insufficient testing and analysis of the product, a flaw solely to blame on the suppliers. This is by nature a reasonably manageable thread.
But who cares? This is about the future, and the future is bright... or grim, depending on perspective.
He linked probably my least favourite AI story: the stamp collecting machine that (spoiler alert) ends up inadvertently turning the planet, and the people on it, into stamps.
This is the kind of idea that grabs attention, and gets people to think about things. It's also a story that will very easily go viral, as it's gone again and again. And, on the surface, it's not even that dumb. I know Draco18s called this "hyperbole", but I'm going to adress this fully as it's so often taken literally.
After all, the story goes, the AI only needs to be as smart to people as people are to chimps. We have indeed surpassed those.
But this kind of claim of a parallel forgets that the practicalities are so supremely different in both cases. People didn't beat out chimps (and, for that matter, every other large mammal) overnight, and we certainly didn't do it with the game rigged against us. Let's play a mind game, since those are a staple in this kind of discussion. Instead of people inventing an AI that is to people as people are to chimps, let's say the chimps "invent" a person which, naturally, is to chimps as people are to chimps.
You're that person, and you're born anew into a chimp world. You are immediately born into your adult intelligence. Sadly, you have no knowledge of modern human discoveries, since you're the first human, but the chimps did somehow imbue you with knowledge of every fact known to chimp-kind.
The chimps want you to collect bananas for them, and will keep you around as long as you do. Of course, you're an evil AI and actually want to take over, destroy or otherwise globally disrupt the chimp world.
Could you do it?
No, of course not. You might be able to escape, maybe. You might even make yourself better off than an average chimp, putting your renegade intelligence to good use gathering food and finding shelter. But none of the actual advances in human civilization exist in the head. Spoken language was not invented in a day. Reading and writing were not invented in a day. Schooling was not invented in a day. Nor mathematics nor science, nor guns nor factories or cars, planes, governments, spears, bows, medicines, psychoanalysis or anything that could help you take over the world.
And that's only counting the ideas.
Does this apply to AI? Yes. Intelligence is not enough, and history has proven it. You need the structural institutions and the millenia of research. This isn't just because humans aren't that smart after all; even the smartest man born into an amazonian tribe couldn't invent a jet plane.
There are also physical limitations that are ignored that are especially troublesome for AIs. Power, of electrical kind, is one; if an AI wished to make humans subservient to them, they would fail for lack of power. Nobody will power oppressive AIs willingly. There are no fully autonomous (and especially not self-replicating) anythings the AI could use for replication or repair; the entirety of human civilization is built under the assumption of human presence. Even the most automated factories have physical requirements for people to be present. Even autonomous fuel trucks need people at the end points to offload the fuel.
An AI clever enough to hijack arbitrary internet-connected devices is also clever enough to work this out for itself. It does have alternatives, mind you; it could go in for the long-con and subtly wait until it has affected the majority of media outlets and can subvert the news. Or it could play into people's emotional side in a way better than the most shifty of politicians. But these options are not only far less practical than was first envisioned, they're much, much slower. And that gives you time.
There is (at least) one other obvious option I'm overlooking for now, and won't mention, but should be countered by later points.
Time is one thing these hypotheticals so often assumed we don't have. But it's a little bit silly to hypothesize like this. Much to the dismay of AI reasearchers, intelligent minds are not something easy to stuble across. If there is only one claim on this whole website that was not an exaggeration, that would be it.
There is no switching point, some infinitesimal period of time, between the time someone thinks to use AI for a mundane task and the time AI becomes so competent for that to become a good idea.
By the time AI is good enough to contribute creatively to research in its own field, we'll already know that AI is not to be played with. Society will have to come to terms with another political entity, whether or not they have the vote. Ethics will be rewritten - not the study, perhaps, but certainly the ethical codes passed through generations. Science and research will reshape around such beings.
There can't be such a jump because if there was we'll already have taken it. Most people can't contribute significantly to AI research, and people will be well aware when AI is approaching young child levels of intelligence. Bargaining on computational power skyrocketing ignores recent trends in computational power.
There's no feasible way for AI to sneak up on us. We don't need to prepare as if there was.
And then perhaps one of the more painful misunderstandings. Here I'm quoting Uriel, but could have chosen elsewhere.
Happiness is not well-defined.
AI have no sense of scale. Their objective goes beyond any other concern and that's why it has nefarious consequences.
This is a kind of ethics-privilege which supposes that emotional reasoning or ethical judgements are a special kind of intelligence and that humans or perhaps other animals are more gifted in this department.
They're not. Teaching an intelligence the meaning of happiness is easy relative to teaching it general reasoning. You feed it the masses and masses of data we have on happiness, the inference cues it can use to reason about it and the best learnings that our social sciences have discovered on it.
And the AI will destroy us at emotional reasoning.
Because all it needs to do is make a bunch of stochastic judgements better than us. Which is trivial given that it doesn't have evolutionary pressure to purposefully do badly, it has far more data than us on this and has far more deliberation spent on its ethical opinions than the vast majority of us are to ever see.
Stochastic reasoning is the core of AI. If we don't have enough of that to make an emotionally sensitive AI, we don't have enough for general AI period. This is especially true given a good, general AI needs to be able to reason about people.
Scale is no different. Of course AIs have a sense of scale; it's needed to reason about things. A superintelligent AI isn't going to mistake a candle for a sun. That would make it a superdumb AI. AIs will clearly be able to reason about the cost-benefit of a scenario, else they wouldn't be able to reason. An AI that is so selectively blind as to not see the obvious is not smart enough to take over the world, and a AI-ready society that is so selectively blind as to give hyperintelligent AIs unequivocal commands is not an society capable of creating AI.
You can't just presume a superintelligent AI is producible but that we have no idea how to shape it. AI is not going to be produced by accident.
I would be much, much, much more willing to trust the ethical judgements of a superintelligent AI than even my closest friends. And if this wasn't so, the creators will have noticed long before things got to the point the AI had power. Heck, I might even be willing to trust its ethical judgements over my own.