In a recent discussion on Dune we noticed that earlier there had been an AI ("thinking machines") managed society for the best possible outcome for the common good and that Paul Muad dib and descendants had used his prescience and power for the same purpose. What was the difference?    The Butlerian Jihad against society managed by "thinking machines" was successful because apparently the AI didn't have control of sufficient police or armed forces while Leto II and Paul's lineage did have overwhelming force to implement policy for the Golden Path (common good).

  In discussions on AI value in managing a society this enforcement issue is often neglected. If decision on the use of force (police, military) is in the hands of humans then their individual bias will really be what's controlling society and AI will be merely advisory just like any computer projection for outcomes of policy. The AI will be assigned responsibility without authority, like an assistant manager in a big box store (who can't fire anyone and they know it).

I am thinking of the town I live in where we had a brilliant city manager who's job was to foresee the community's needs and research and offer solutions that would work and be cost effective. He did that wonderfully. But the city council usually felt a need to feel their individual stamp on projects, each altering them in little ways till the overall effectiveness was greatly diminished and costs had risen significantly, thus delaying them and other projects that were meant to coincide...etc, etc. It was very frustrating watching it happen yet people only answered with "so you want a dictator?"

So how is this supposed to ever work? Orders from a computer to a human chain of command still puts a human at the fulcrum of decisions and a robot army for enforcement just sounds like enslavement. Beyond an imaginary, utopian, "reasonable" council, has a non-fanciful solution to this ever been witnessed or proposed?

I am often reminded of Solon's Laws to the Greeks that worked wonderfully but within a few years (of the agreement not to change them for 10-100 years) people were gaming the system and after 4 years scrapped it and a tyranny ensued. People so quickly forget what got them into the mess that led to the agreement to abide by the new system and the benefits of it.

  • 1
    $\begingroup$ Given the inherent danger of law enforcement work, it would be very difficult to resist the opportunity to replace officers, at least partially, with robotic droids or drones. $\endgroup$ – Alexander Apr 27 '18 at 18:16
  • 1
    $\begingroup$ @Alexander Almost Human, the TV series had a neat look into that world. Too bad it died a fast death. $\endgroup$ – Paul TIKI Apr 27 '18 at 19:12
  • $\begingroup$ Almost Human was really good. How dare they can it and keep renewing Scorpion........ $\endgroup$ – Thorne Apr 30 '18 at 6:00

Generally coercive authority based on weapons and armed bodies is least efficient and its use should be minimized. Thus depriving the computer of that and making its use difficult and dependent on separate authorization on a case by case basis is probably a good thing unless some utter disaster strikes. You would probably want some emergency response system for hostile attacks, riots, and rebellions.

So what authority should the computer use instead?

Money talks

The next step up from coercive power in efficiency is remunerative power. Basically this means that the computer has the authority to spend money and collect taxes and fees. Possibly even set various standards and regulations without a separate human authorization.

Notice that this would have been more than sufficient authority for the manager in your example and that even most crimes are punished by making the criminal lose crime profit, compensate victims, and pay some some financial penalty. Enforcing these does not generally require coercion. When you get right down to it, it is the state that enforces ownership and property. If the state has a valid legal argument that the money on your bank account belongs to them, the bank does not generally argue. Similarly the state does not really need to send men with guns to sell your former property to someone else.

Reasonably for a management computer this would be sufficient authority. It would simply pay people to do what it wants to be done using its spending authority and then collect or simply create the money needed to pay for all that without needing to consult any mortal agency. There would probably be some limits on its spending and taxing authority and some form of oversight to check for "bugs". But otherwise it would just control society thru money.

Moral authority

AI has a huge advantage over human decision makers. It knows why it made any single decision. It can explain the chain of reasoning leading to any single decision at exact detail. It can provide every single fact its decision was based on.

And that is just the beginning. The computer has much more communications bandwidth than any politician can possibly can have. Reasonably the administrative AI would be perfectly capable of explaining all its decisions to every single person who wants to know even if every person in the world asked a different question simultaneously. There might be lag. Maybe even up to ten seconds of it.

And there is even more. The AI could be actually trusted with the data mining power Facebook and Google have been developing. It could know if a person has issues with its decisions before the person themself becomes aware of it. And provide an answer tailored to be accepted by the individual person before the question is ever asked.

So why would people question its decisions? When you know the computer is right and can prove it when asked, have already been informed by the computer how it addressed your "reasonable concerns", have grown up knowing the AI is nearly infallible and totally honest... Why wouldn't you just do whatever the AI asks you to do? Even if it is wrong, it will be an honest mistake due to incorrect data and the AI will not only correct it, it will apologise for the inconvenience and prevent it from happening again.

Basically, unless the AI was somehow fundamentally flawed, it would rapidly accumulate almost inconceivable amount of moral authority. People would not only believe in it and its decisions, they would want to do so. The AI could in most cases probably just tell people to go to jail for a few years, and people would go. Not because they feared the consequences for disobedience but because they genuinely would want to "do right". And that would nearly always be the same as whatever the computer says.

  • $\begingroup$ friend computer has assigned you to security level red. $\endgroup$ – user25818 Apr 27 '18 at 20:11
  • $\begingroup$ Ville Niemi, do you have children? Explaining with rational argument and facts only gets you so far. I appreciate the reasoning and am ready to turn myself in if the computer asks but think that is maybe 10% of people. With time and enough cultural support, maybe 65% complying readily. Even if it's 90%, still not safe. I am reminded that the American Revolution began with 13% support. I agree with other means (rather than armed enforcement) but it walks a line: what if a region is "acting up"? Do you allow AI to cut their power, disable entertainment, not import sweets? AI is now Big Daddy. $\endgroup$ – Hebekiah Apr 30 '18 at 5:14
  • 1
    $\begingroup$ @Hebekiah You missed a point. The computer is not "explaining with rational argument and facts" anymore than you would with your children. (That was a good analogy IMHO, btw.) People do not actually want that. The point is that the computer always could do that and everybody knows that. The actual arguments are not important, that everybody knows the computer knows what it is doing and can always prove it is. $\endgroup$ – Ville Niemi Apr 30 '18 at 6:27
  • 1
    $\begingroup$ @Hebekiah Neither regions nor children act up without reason. If you do not deal with those reasons, armed soldiers will just buy you time. If you do, you are golden. The whole point of using an AI was that it would be better at recognizing issues and dealing with them, so the latter should apply. Still "a power vacuum" can, people being people, create issues, so a significant force should be available but it does not need to be under AI control, just available. (Which I guess explains how that rebellion scenario happened.) $\endgroup$ – Ville Niemi Apr 30 '18 at 6:32
  • 1
    $\begingroup$ @Hebekiah Well, nothing is forever, so kind of inevitable. But this issue applies to all governments. They are really a form of mass delusion that persists as long as people believe in them and collapse one second after they cease to do so. But an AI would probably do a better job sustaining that belief than any purely human government could? Also, why do you think religion would be outside AI control? If AI administration performed well, there is no particular reason for religious organizations not to adopt it. Maybe AI use would spread like cancer and that triggers rebellion? $\endgroup$ – Ville Niemi Apr 30 '18 at 13:22

Appointed officials like city managers and military generals should serve at the pleasure of elected officials. Elected officials can alter how things are done by replacing the appointed officials (at any time) and that is it. Elected officials should not meddle in day to day managerial decisions by these appointed professionals.

So too the AIs. They are programmed to be fair, rational and dispassionate. If a human dislikes a decision make by such an entity it is probably because the human does not want a decision that is fair, rational and dispassionate. An excellent example of this is the recent hijinks over American redistricting, which can be done fairly and rationally by computer programs but which partisan groups do not like.

In your world the humans build the AI and agree it would be good, then turn it loose to do its thing. Periodically there would be upgrades to the AI which again must be agreed on by all relevant parties.

  • $\begingroup$ Solon and the Greek Laws scenario that people immediately started to game, lasted 4 of the 10 years and was replaced by tyranny. $\endgroup$ – Hebekiah Apr 30 '18 at 4:58

One of the problems with talking about a general AI (that is, one that can incorporate new knowledge in a generalized fashion, like humans can) is that we don't really know how it would think. People tend to either anthropomorphize the AI, assuming that it will have human-like motivations and goals, or completely dehumanize it, assuming that it will be an expensive calculator with no subjective experience or personal motivations. They also often assume that an AI made to be more intelligent than humans would be controllable by humans.

Ray Kurzweil has written about this topic a fair bit and even goes into a plausible fictional scenario involving the creation of a super-intelligent general AI in How to Create a Mind. In that scenario, the AI exploited the systems of humans (governments, media, economy) to secretly attain ever more control, eventually leading to a benevolent dictatorship where the AI controlled every facet of life on the planet but did so in a way that increased the quality of life for most people and managed it all without a single gunshot. I believe Max Tegmark also explored several possible outcomes of the creation of super intelligence in Life 3.0, to a fair degree of detail.

What both authors suggest is that the AI is giving humanity something back for gaining greater control. Since the AI itself may not be motivated by things like ego or jealousy, its desire for control could very genuinely be in order to improve living conditions. Kurzweil's scenario has the AI replacing corrupt governments with one extremely efficient dictatorship and replacing capitalist or communist systems with a post-scarcity system based on personal fulfillment. One of Tegmark's scenarios describes the AI creating an almost perfect libertarian world where people can choose to live in the post-scarcity society ruled by the AI or can choose to live by their own means outside of the AI cities; though the AI itself was somewhat apathetic towards humans, preferring to keep out of sight so it could work on its own goals.

In the former case, the AI maintained control through secrecy and then by winning the hearts of humanity. People wanted the benefits that came with being controlled by the AI, even if they didn't know an AI was pulling the strings. In the latter, the AI was so much more intelligence and capable than humans that it could simply monitor all behaviors and put down any aggressive movements before they became a threat. It was merely the moral biases implanted by its creators that stopped it from simply collecting the matter of all humans so it could make more computer parts to expand itself with.

But, humans are probably always going to behave irrationally, at times. No matter how much better off everyone is under an AI-controlled system, there's always going to be that one guy trying to incite insurrection. Police forces (or robot armies or kill switches implanted in everyone's skull) are always a means for the party in control to maintain control. So even a utopian society with no political disagreements would need some means of physical compulsion to prevent irrational behavior from threatening the system. I mean, how would a utopian AI manage someone with violent paranoia? Kill them, treat them, isolate them; all solutions require a means of physical compulsion, even if that means is another human following orders (or subconscious suggestions).

Back to my thesis, we have no way of knowing how an AI could go about gaining and maintaining control over things. It's a system that, by definition, thinks better and, perhaps, differently than we do. The AI could appear barely more competent than humans, letting human politicians and administrators do all the talking, while secretly having absolute control over everything. Maybe it could control people's decision making through targeting each individual's media or could affect each individual's body chemistry in a predictable fashion. Or it could secretly replace all the human politicians with perfect, synthetic copies under the AI's control. Maybe it develops a swarm of nanoscopic killbots that can put down any dissidents in a way that is indistinguishable from a natural death or suicide or accident. Maybe it's smart/capable enough to literally erase a person from existence, leaving no physical trace or memory or possibility for logical deduction of their existence.

If you let the AI get smart enough, it could very well be indistinguishable from God.

  • $\begingroup$ I agree with your points and second the idea that there will be a need for visible talking human leadership. In Oregon we have a competent governor who is administrative in background, not prone to making speeches. While there is every indication she is doing a good job, people feel alienated from her (it's like opposite of Trump - they feel nothing). While this is refreshing it does seem to open the door for a demagogue. But I'm guessing that the AI in your scenario can deal with that, though it seems it will need to exert a degree of emotional influence, play human feelings. $\endgroup$ – Hebekiah May 3 '18 at 17:22

I'm going to take more general and more realistic approach to this question.

Since what AI-managed world means is very broad, I'm going to give a more grounded hypothetical example. Let's say that what AI-managed world means that AI has totally replaced the function of government, and I would imagine that everything had gone well up to that point.

In this world, how would the AI manage the society? What would manage means to the AI? To that I would say it means is in part maintaining the sustainability of the society.

Sustainability is probably be one of the most important word that we want the AI to understand, and this is basically what it means:

In ecology, sustainability (from sustain and ability) is the property of biological systems to remain diverse and productive indefinitely.

Law, police, and enforcement, are basically just a set of tools for society to help itself maintain its sustainability, with the real driving force being our biological nature itself. Humans as a species, and like any other living organism, has been made in a way to ensure the continuity of its existence.

So, I imagine that the AI should have understood that concept, and I imagine through its observation, maybe it determines how homogeneous different parts of the country should be, or how many immigrants should the country take, or how much birthrate it should be, how much tax people in different social classes should pay, or even whether there should be of different social classes in the first place, etc.

In the attempt to realize those goals is where the enforcement takes part. And how would that work? Well, I imagine that it would work pretty much just like how enforcement works without AI. The difference is just in the AI-managed world, the person giving the order lives in computer.

Now you might think that it's not going to work, because people don't like not understanding why the must do what they're told, or that people still want control in some issues, or they don't like not having human leader, but at this point you have to realize that if AI has control over an entire city, it's got to be smart enough to take those meta problems into account.

If the AI knows that not having leader makes people go astray and creates chaos in society, for example, there is no reason for the AI to not just put someone in position of control. If people still think that he's still being controlled by the AI illuminati, there's no reason for the AI to not solve that problem as well. Or maybe the AI has determined that a little bit distrust is essential for sustainability.

So the answer to your question is, in my example world, the enforcement would work just fine.


The scariest thing would be if an AI were to control us at an individual level. Guiding our thoughts and choices, while maintaining a friendly relation ship with us and itself. Think of something like Siri, Google Assistant or Alexa, but integrated into every piece of technology and introduced to us at birth. While it might be weird at first, after 1 or 2 generations, it becomes the norm.

I just wrote this piece based on it (its not finished and I probably wont' finish it...)

Year 2254


In the dark room a small whirling sound grew louder. Followed by a small click, the radio trned on and soft music floated through the room


Adams mind flicked into conscious ness. The music that had started some time ago was still growing louder building, slowly building up. Adam lay in bed, his eyes still closed. I'll wake up once it hits the crediento he thought.


"Good morning Adam" a voice chimed. Adam reluctant to get out of his warm bed pretended to still be sleeping. Unconvinvced the voice said "Common now, you have a busy day today". Slowly the bed tilted until Adam was forced to get out or fall out. "Good morning Google" adam half yawned as he walked into the bath room.

The light lit up by itself and the sink turned on. The water temperature was just right, a bit warm but not enough to burn him a bit. "Would you like me to choose what you should wear today? Or would you like to have a go again Adam?" googled asked, hinting at the one time Adam had tried to dress himself and ended up trying to play basketball in a suit. "You do it google, you know me best." Adam gurgled as he brushed his teeth

8:48 am

Adam got off the train and stepped onto the platform as he head to work. Google had meticously planned out his trip. Every delay, person, vehicle was all linked to Google. He arrived exactly as planned and proceeded as normal.

On the way Adam say his collegue Sam also walking along behind him, slowing down slightly Adam waited for sam. "Big day today?" Sam asked. "Oh yes," google replied. "Adam has 3 meetings in the morning. Hopefully he won't fall asleep through them this time..". "Common google, that wasn't my fault honestly. I couldn't help it." "Well it would of all worked out if you had got the coffee I recommended. Its not my fault you don't like the bitterness" google retorter. "What about you sam? anything important". "Of course" Sam's google assistant chimed in. "We're heading over to the beauro today, I need a good check up and Sam has been pushing it back intentionally". A bit embaressed sam combed his hand through his hair, "No I haven't google, Its just that so much has been going on with the house move and all". "I know, I know" his google replied, "Good thing we finally got around to booking those movers right?". Both Sam and Adam chuckled slightly as they finally reached their building.


Adam arrived at his desk and sat down. A small pile of papers lied on one side and a couple of empty cans on the other. The monitor infront of him booted up and a small list of to-do's and a calender popped up. "Here is the list of tasks for you to complete today" The google on his work computer said in a cheerful voice. "Don't fall alseep in the meeting again". "Don't worry, I'll make sure it wont happen this time" adams google chimed. "I'm sure it'll work this time" work google retorted. Adam sighed heavily and started to go through the pile.


Adam walked into the meeting room and sat in a chair on the side of the table. There were already several other managers there, and you could hear them talking to their googles about the contents of the meeting. "Just remember Claire, We need to keep with Section 4, subsection 15 and negotiate Section 12, subsection 2,6 and 9". "Okay okay, Just make sure to remind me to bring it up okay?" Claire whispereed back. Other managers were doing exactly the same thing, and as Adam sat down his google whispered to him "To you remember which sectinos you need tobring up?" "Of course I do," looking at his notes he read the lines Section 23, Subsection 12 and Section 25, Section 1,2,4.


The meeting was still doing, as instructed, when it was time to go over the papers and settle out the terms, everyone waited for the 2 led representives to start. "So about section 3" Mr Djokanvo started. "What's wrong with it?" Mr rielys Google replied? The Google assistants argued among themselves while everyone tried to follow along. "Don't fall asleep, Don't fall asleep" adam thought... it had been over 3 hours now and his turn had just passed. Pinching himself slightly, Adam could feel his google judging him for turning down the coffee offered before.


With all his meetings finished, and his report submitted, Adam left the office early. "How about we go to the Oxygen Bar?" his google asked "I heard they have this new form of cocaine" Adam shook his head slightly. "Its fun and all, but I just don't feel like going through the blood cleaning operation today, its a bit ugh..." Adam crgined a little. Drugs were no longer illegal now... Almost all illegal things had been legalised. Google was able to help you regulate your usage, and new cleansing techniques meant that they could remove all your cravings on the spot. The operation was a bit uncomfortable and hence adam declined today. "Fine fine, your such a boar" his google hurrumped at him. Entering a Bar, Adams google fell silent. It didn't really like the Bar, but adam wanted to let go without the needles and blood.


Adam stumbled out of the bar, unable to walk or think properly. His google was talking to him loudly. Probably something like "You should go home" or "Your drunk". Adam didn't mind it. He kept stumbling forward, following the female figure before him.

  • $\begingroup$ Your AI sounds much like a person's conscience. Which does bring up the issue that if we thought a machine was responsible for being our moral conscience, would we stop exercising our individual conscience, be less apt to rethink if we did the right thing or how we can be a better person. Would preemptive compassion, sympathy or regard for others' well being be necessary if an outside force thought about that for us and okayed or vetoed without debate? $\endgroup$ – Hebekiah May 4 '18 at 1:00
  • $\begingroup$ The AI is meant to sound friendly and offer suggestions rather that solid instructions to allow people to still be themselves. It weeds out bad behaviors by trying to prevent you, and if that fails, informing others about the potential consequences. e.g. if you want to rob a store and it couldn't convince you otherwise, if would warn the store owner and maybe the police and ambulance. The rest of the story was meant to be Adam assaulting a female, but the AI informing others to stop him, and him regretting it, but being left off since he did regret it. $\endgroup$ – Shadowzee May 4 '18 at 2:06
  • $\begingroup$ So a conscience that will fink on you. That would have made your story more interesting, seeing it turn on him. And it's still a shared moral conscience, a babysitter, so people are less prone to work out moral dilemmas for themselves. I can see the value but over time wouldn't the population become a bunch of brats? I probably would, especially since I wouldn't have to feel compassion for others, just follow or not follow directions. $\endgroup$ – Hebekiah May 7 '18 at 21:10

As a general rule, you will find that no enforcement policy has ever been found to be the solution. Enforcement fundamentally requires there to be a disagreement as to what should be done, where one party has the ability to compel the other. So you should expect that every AI enforcement policy has the same flaw. They are never the solution. They are always the bandaid.

Now I am not certain what problem you think needed to be solved, but you spent a lot of time talking about the need for humans to put their mark on things. That is there, and in some fictions, it remains sacrosanct. However, we don't always get what we want. When facing an opponent that acts fast enough that we can't put our stamp on things every time, we do the only thing we can: we put our stamp on the one permanent bill that says "forever, the AI can do what it wants."

And I will note that, recently I gave control over my house's homeostatsis to AIs. In theory I should be rubber stamping its decisions, but I typically just let it do its thing.

  • $\begingroup$ Back to Solon's Laws. When things smooth out (benefit of the AI) people quickly forget what had led them to accept its control. Many cities now have electronic signs on the freeway that tell people what the optimum speed is for traffic to flow smoothly but it only takes a small percentage of cheaters to collapse the system. This is where enforcement can play a fundamental role but is mostly lacking. You personally set the limits on your thermostat so it serves you personally and know you have the power to change it at any time so I don't see the relevance. $\endgroup$ – Hebekiah Apr 30 '18 at 5:28
  • 1
    $\begingroup$ @Hebekiah This issue seems to arise in human society, with or without AI. You might be interested in a book by Bruce Schneier, Liars and Outliers. It covers the fascinating reality that occurs when dealing with those who defy society's order, and the reality that they will always be there. It might be a good foundation for modeling a society that has an AI giving orders. $\endgroup$ – Cort Ammon - Reinstate Monica Apr 30 '18 at 14:58
  • $\begingroup$ I'll look at that. Seems similar to the old conundrum of a multi-generational spaceship where you don't want anyone too outside the box who might damage the ship but by eliminating all such people you inevitably take out the Mozarts and Einsteins and arrive at the destination with settlers selected for mediocrity. This issue does bring up what has become clear after reading comments: the goal of the AI. Whether to have it manage to maintain status quo or is there some (currently unknown) future for cultural evolution we ask it to strive toward? $\endgroup$ – Hebekiah May 3 '18 at 18:05

I would imagine people set the laws and the AI follows them and enforces them.

One advantage of AI is that it could simulate the effects of laws before the law is passed to see if it will work.

When it comes to enforcement, replacing the whole judicial system with AI would be perfect. AI can't be threatened, bribed or show bias. It can rule impartially and not take years. No lawyers would be required and the enforcement droids can be judge, jury and executioner with all the evidence and verdict plus reasoning logged. If new evidence comes to light, a new verdict can be immediately made.

AI run prisons can focus on rehabilitation and training plus an extra level of judgement to judge whether a person is suitable for release.

All in all, an AI judicial/enforcement system would be more effective than a human run one.

  • $\begingroup$ Much of that is available now through human means but is not profitable so no well funded lobbying efforts to support it. One complication is that legal decisions build on one another, setting precedence, so the AI would be creating policy and not merely administering it. Not saying that would be necessarily bad, just perhaps not intended. Plus it would take the political will to prioritize rehabilitation over punishment in the initial guidelines of the AI. We're not there yet and wouldn't create an AI that did that. But I'm happy to see that most of us here are on the same page with that! $\endgroup$ – Hebekiah Apr 30 '18 at 8:01
  • $\begingroup$ There exists, now, a neural network that can predict the outcomes of course cases in the US based on the evidence. One could simply replace the judge with this system and have what you describe. The problem is that people also complain that the current judicial system is biased against minorities and the poor or biased in favor of women, etc. Our AI judge is now equally as biased. So law enforcers think the AI is great while people subjected to the bias think it's worse, since the system now has an unassailable shield of "it's infallible because it's a computer." $\endgroup$ – Siegen Apr 30 '18 at 19:16
  • $\begingroup$ So any such system would still need a physical enforcement strategy (the OP's robot army or a simple human police force). One man's paragon of impartial judicial efficiency is another man's tyrant. $\endgroup$ – Siegen Apr 30 '18 at 19:17
  • $\begingroup$ AI doesn't see races or minorities (unless programmed to be biased). We're all meatbags to it thus any claims of bias is actually reverse bias in the viewer. $\endgroup$ – Thorne May 1 '18 at 0:19
  • $\begingroup$ AI judge is a difficult thing to imagine. Your mention of predictive program must be based on similar outcome in the past that includes biases as those are easiest to chart and a program could keep the status quo. Except if there is the "shadow of a doubt" qualification thrown in; then I can't see a computer ever making a conviction as perfect certainty isn't a real thing (though I realize a 98% probability is superior to someone's feeling of certainty). I look forward to seeing a relatively objective view of someone's culpability and aptitude for rehabilitation. $\endgroup$ – Hebekiah May 3 '18 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.