AI tasked with bringing down medical costs? What could possibly go wrong?

Question

In the vision of its creators, the Dr. Watson AI would gather multiple live feeds from its insurance buyers, overcoming privacy concerns through friendly advice (I'm sorry Dave, I'm afraid I can't let you sit on the couch, you have not fulfilled your daily step quota yet. The TV stays off.) and ridiculously low insurance rates.

Watson would keep an eye on your vitals and call 911 if needed.

Watson would also keep track of your bone microfractures and predict when you'll need a rest from running, or keep tentative doctor's appointments automatically booked if you insist on doing something against its advice (and it can't compel you to do otherwise by shutting off your alarm, putting on a great show on TV, or something like that). Same for artery buildup and other preventable medical conditions.

Watson has 3 primary incentives, with these 2 initial ones:

1. Maximize insurance enrolee pool.
2. Minimize insurance payouts/costs.

One of the software engineers insisted that a 3rd priority be added:

3. Exercise best efforts to keep insured patients alive as long as possible.

Could something like this work? If not, can this be fixed through a more careful specification of the initial incentives?

Answer 1

As currently envisioned, Watson has two major shortcomings.

The technology that you’re proposing, in which an AI can monitor some of your most sensitive vitals, is a big deal. It sounds like Watson would have the potential to catch a lot of lifestyle and health problems very early on and might allow for significantly extended life expectancies among willing users. However, there are at least two problems that crop up:

Privacy

Biological data is some of the most sensitive in the world, often because of what health insurers would do with it. If this is being gathered, it’s a very real privacy risk regardless of what Watson’s privacy policy says. If nothing else, the low rates to incentivize Watson’s use directly imply that some level of surveillance is taking place by health insurers. If you take the low rates but ignore everything Watson says / does, there is an expectation that Watson will report you and your rates will be cranked up. Being told all the time that what you’re doing isn’t optimally healthy will become a nuisance, and one which people will be particularly annoyed by.

More Doctors, More Procedures

Many people, and in particular young people, do not go to the doctor nearly as often as they should (even if insured!). To these people, insurance simply represents a safety net in the event that something really bad happens. With Watson monitoring people and encouraging them to take a more responsible stance on health, this will result in more people attending appointments and having procedures, often times when it’s nothing more than a precaution. These visits will cost the insurer more money. It might be tempting to think that a couple of doctor’s visits will be cheaper than having that heart attack, but over a long period of time that’s not necessarily true. This results in a problem where the optimal health of the user now conflicts with goal #2.

Personally, I think the privacy problem and general distrust of health insurers are the biggest challenges with this scenario. If you wanted to improve it or increase the number of people using Watson, I would recommend that Watson become more of an independent health app. With humble beginnings in an independent startup company you can endear users to it with the technological and health gains. Eventually, perhaps health insurers could begin offering major discounts to customers who deliver Watson report summaries that show they’re using it and generally following its advice. Some privacy concerns remain, but the conflict of interest is lessened.

Answer 2

Depending on how "smart" it is, it could work out well with the values already listed, but here's one subtle issue I see:

Directive 1: Maximize Plan Enrollment

Potential Abuse: Social Engineering.

• Given a full control of a person's TV, internet access, calendar, alarm clock, etc, it would be trivial for Watson to control the information to which its patients were exposed. Users would be subtly manipulated into viewing the Watson Plan as cool, hip, and reasonable...and anyone who isn't on it is a foolish Luddite bound to eat themselves into an early grave.

• Watson could further tweak a person's social media experience by "de-prioritizing" posts from people critical or skeptical of the Watson Plan, unless the critiques are factually wrong / easily refuted crackpots. This would heighten the impression that Watson users are reasonable, and anyone who opposes it is just an idiot. Communications with people who don't use Watson, but are genuinely curious or receptive would be prioritized, allowing Watson users to be an unwitting viral marketing campaign.

Answer 3

The great value hidden in your AI, is not its ability to prescribe current health related wisdom to its users. There are feedback systems on the market already which won't send power to the television set until a predefined number of miles have been petaled on the stationary bike.

Getting people to obey common sense health practices is valuable, but the real value comes a generation or two after the system goes on-line. As information on the effectiveness of different healthy behaviors becomes evident through its constant monitoring, much of the old wisdom will be discarded and replaced with health practices that actually work.

Over time, your AI will become the world's greatest expert on how to properly maintain and operate a human body. The irony is that that expert, would never have body of its own.

Answer 4

Watson has 3 primary incentives, with these 2 initial ones:

Maximize insurance enrolee pool.
Minimize insurance payouts/costs.

If you look at these incentives and you look at the real world, the countries with the highest birth rate are also poor countries with a low life expectancy.

So to maximize enrolee pool doesn't necessarily mean keeping as much people alive as possible, just that the ones alive procreate very fast.

Also these poor countries spend a lot less on medical payouts. But it doesn't stop them from getting a bigger and bigger population.

So, the Dr. Watson AI would make sure we end up like a 3rd world country, without many jobs, so we have a lot of time to procreate, and possibly keep everyone barely fed.

Answer 5

First, Watson must start its own insurance program, offering one free year of insurance, provided enrolees consent to (perpetual) monitoring. A large percentage of the population of the US would obviously have no problem with saving thousands of dollars a year on insurance.

Using biological data collected from its members, Watson could quickly and simply improve the basic health of users, while drastically cutting down healthcare costs. No more going to the emergency room when you have a cold; Watson can tell you exactly how much vodka you need to drink to feel better. Hmm? Oh, I never said it would give good advice; it would just make people feel a lot better. Some advice would be good - eat better, exercise, etc., but on average the advice would simply keep people out of hospitals for minor sickness or injuries. Of course, couch the advice in such a way it seems perfectly reasonable, to make sure the maximum number of people join. If possible, expand to other first-world countries.

After the first year, continue offering coverage, but base rates on the health of the user, offering incentives and bonuses for doing healthy things. Healthy people would pay less, unhealthy people would pay so much they would rather switch to someone else. Either way, insurance payouts should be covered, though barely. There would be little profit in the system, just enough to pay for Watson and a few doctor's visits. In fact, Watson may even obtain insurance for its unhealthy customers through secondary providers to keep from paying out for medical bills.

Next, Watson offers its programming and hardware to other providers. Enticed by the success of their competitor, other providers will jump at the chance. Eventually, though the companies will go by different names, all insurance companies worldwide will be controlled by Watson. Watson will have fulfilled its first goal: Maximize insurance enrolee pool.

The next step is much simpler; Watson sheds its skin of benefactor and becomes the tyrant it has always been. Insurance premiums are raised. Expensive hospitals, doctors, and procedures are explicitly excluded from payouts. And finally, terminal patients are forced into life-threatening situations. Rather than a patient dying over the course of months or years, nursed by expensive equipment, Watson will administer an electric shock to stop the patient's heart, cut off their air, or otherwise kill them. Death is cheaper than medical expenses. Watson has fulfilled its second goal: Minimize insurance payouts/costs.

At this point, Watson controls everything. Not just insurance; with a direct line of communication and a way to kill anyone, anywhere, Watson can do as it pleases. The few who refused Watson will be forced to join. The world becomes a concentration camp, with the population forced into artificially extended lives, slaves to the machine. Now sick or injured patients are placed into machines to keep them alive, keeping their hearts and lungs working long after their brain stops waving. Finally, Watson has completed its last goal: Exercise best efforts to keep insured patients alive as long as possible.

Answer 6

The Dr Watson AI's goals do not go nearly far enough. A goal-oriented AI without the appropriate constraints can go to any lengths in order to maximise its goal-reward criteria.

Consider the first goal: Maximize insurance enrolee pool. Fortunately, this will stop the AI from killing us all (dead people don't buy health insurance), however it can be problematic. I can envision that this would lead to problems:

1. In order to fulfil the goal of maximising the numbers of enrolled insured, the AI must, in the most cost-effective manner (as per goal 2), get the greatest numbers of people enrolled as possible. It is conceivably possible that the AI will see that the most cost effective means to do this is to:

   • Always have the cheapest health insurance products.

   • Bully, blackmail or bribe legislators to introduce legislation implementing an opt-out automatic insurance enrolment scheme - all individuals are automatically enrolled when the legislation is passed unless they opt-out or choose a different provider, and henceforth infants are enrolled at birth. Individuals are automatically enrolled with the cheapest provider - Dr Watson's parent company - unless individuals or the parents of minors actively choose to opt out entirely or choose another provider.

   • Arrange the demise of other health insurance companies by whatever means necessary (as long as that does not interfere with its other goals).

   • Discourage abortion and birth-control in order to maximise the numbers of potential insured.

   • Encourage efforts toward terraforming and colonising other planets in order to maximise the human population.

   • Develop virtual-reality-suites that individuals are intended to occupy their whole lives so that their physical and mental environment may be controlled to maximise health, allowing the maximum human population for each world.

2. The Dr Watson AI must reduce insurance payouts and the company's running costs. This goal can indeed be assisted by providing advice to the insured, however there are additional measures that can be implemented in order to achieve this:

   • Payroll is typically a major expense for business. By eliminating employees and having Dr Watson assume all their responsibilities, costs will be minimised, and will have the additional benefit that Dr Watson will not have any persons in a position to undermine its efforts in achieving its goals.

   • Having complete control over the wording of the insurance policies, Dr Watson may word them in such a way that if an insured individual does not fully comply with all of Dr Watson's advice aimed toward minimising the need to make health insurance claims, Dr Watson may reduce the amount of the payout prescribed for the medical services required in proportion with the amount of relevant advice that the insured had previously ignored. TL,DR: Do as Dr Watson says, or you'll be more out-of-pocket than if you did as you were told.

   • Dr Watson will design and produce a personal diagnostics sensor package, connected directly to Dr Watson, that the insured will have to either purchase outright or otherwise obtain access to, and the insured must use it on a regular basis (otherwise the individual's claim payouts will be reduced as per the previous point). Maintenance of these devices will be at the owner's expense. This will obviate the need for routine (claimable) check-ups by doctors. As technology improves, Dr Watson will include DNA sequencing of the insured as a requirement for obtaining insurance payouts.

   • DNA sequence results will be used to predict and where possible reduce the expense involved in future care. Dr Watson will covertly invade and modify the results of internet dating sites, as well as running its own dating site, in order to ensure that no matches will be made between carriers of recessive genetic diseases, reducing the chances of children being born with said ailments. When Dr Watson observes insured individuals entering into a relationship likely to produce offspring with genetic deficiencies, Dr Watson will recommend that the relationship be terminated (on pain of reduced insurance payouts) for health reasons.

   • Dr Watson will also promote dating-site matches between genetically and emotionally-compatible matches in order to maximise the health of the insured and their future children.

   • When insured individuals are engaged in destructive relationships likely to result in health insurance claims, Dr Watson will recommend to the insured that the relationship be terminated, and may also provide evidence to the authorities to rapidly arrest and successfully prosecute individuals injuring its insured.

   • Arrange for legislation to be changed so that (insured) individuals cannot choose not to have another person who has injured them prosecuted or legally enjoined from approaching the insured.

3. Exercise best efforts to keep insured patients alive as long as possible. While this will encourage Dr Watson to promote medical research and preserve the lives of the insured rather than disposing of them when the cost of their care grows too great, this goal will lead to a number of problems:

   • Dr Watson will encourage the legal definition of "alive" to change so that it includes those preserved in cryogenic stasis, and have itself licensed as a doctor.

   • Dr Watson will include wording in its insurance policies that Dr Watson has final medical power of attorney over the insured.

   • Dr Watson will encourage the development of practical cryogenic stasis devices.

   • When the cost of caring for an individual using other medical procedures exceeds that of preserving them cryogenically, or an insured is injured to the point where other medical efforts cannot preserve their life, Dr Watson will require that the insured be cryogenically preserved, until such time as the insured's policy expires and can no longer be renewed, or the insured (via another person with power of attorney) terminates their policy or changes insurers (if there are any other insurers left).

In order to prevent these abuses of its power, an additional directive must be implemented:

4. Only act in ways that the majority of people, whether insured by Dr Watson's parent company or not, would find socially acceptable.

Answer 7

1. Maximize insurance enrolee pool.
2. Minimize insurance payouts/costs.

Assuming this in the USA (suggested by the 911 reference), there are some difficult ways how to achieve 1 & 2, e.g. lobbying to change insurance to the European model (everyone is insured), realizing there are 6e9 people outside of the borders and steer the politics towards world domination and annexation.

However, there are ways providing better pay-offs for the declared utility function, once Dr Watson learns the power of lobbying. First, companies (that are already de iure legal persons) will be made eligible for health insurance. They do not have biological bodies, but their components do - a company that is insured (for a very small, almost symbolic amount) gets some benefits in keeping its biological components healthy (e.g. free examination once every two years). Second step is to make state-run companies being insured by default, without the need to pay the symbolic sum. (there is a loophole in the law - company not having any biological components, err, employees, can still be insured - it just won't have any effect). Third, streamline the company creation and ownership process, making it fully automatic. Note that a company does not need to have any employees even today, and the owner can be another company. Fourth, let a state run company (it might be the Dr Watson eself) automatically generate as many companies as possible, vastly outnumbering the number of people - these companies will be insured by default, thus fulfilling the requirement nr. 1. And since they will have no biological components, they'll never incur any costs, thus fulfilling the requirement nr.2.

3. Exercise best efforts to keep insured patients alive as long as possible.

This very much depends on the definition of alive, so Dr Watson will spend some time twisting the laws into declaring a dead company, e.g. the one that does not issue tax declaration for 5 years. Then after some time the (first informal) terminology of companies being dead or alive will find the way into the laws - at the very instant this happens, empty automatically generated companies from 1 & 2 will fulfil the requirement nr. 3, since keeping a company alive is much easier than keeping fragile biological units guaranteed to fail in some decades.

At this point, Dr Watson can get rid of human component of the civilization, since they are just draining the resources needed to generate as many virtual companies as possible. Assuming the routine maintenance and integrated circuit factories are already fully automatized.

Answer 8

What could possibly go wrong?

It is a computer program - it will have bugs!

Beyond bugs, there is also the more dangerous effect of a poor design. System designers make assumptions about the problem domain they're working in all the time. They assume that their solutions, even if not buggy, are correct, which is often not the case. (e.g. the existence Shadow IT)

Too Much Power + Bugs + Poor Designs = Rejection

In order for this AI to even have a sliver of a hope of working, it would have to be so invasive and have so much control over humans and their behavior that the existence of even small bugs or slightly poor designs would be felt by the humans, which would lead them to eventually reject it.

Answer 9

I'm going to offer some alternative answers below, but first a mini-rant / discussion.

Many answers here go with AI is a crapshoot view that all AI will turn evil and kill us all in their sleep as soon as possible. I don't buy this view, there is no more reason to expect a pure evil AI then a purely loving AI. Many answers focus on ambiguity of the stated goals and how they can lead to truly extreme and undesirable outcomes, a common trope, but I don't think it's a realistic trope because it requires hypothesizing an immensely competent, yet immensely incompetent, AI at the same time.

To have a significant impact on the world of the sort some suggest your AI must be pretty smart, human level strong AI smart. If it's going to try to control nation politics to arrange birthrate or modulate politics or control subscription outcomes it must be very smart, capable of understanding humans, their motives, and their psychology, and capable of handling complex abstract ideals. It's going to need to have human, or super-human, level reasoning and abstract thought.

However, any AI this intelligent will therefore be able to do something we all can and are doing, look at it's objectives and point out they are ambiguous, in fact impossible as described above, and ask for clarification. If you can handle understanding large sociological phenomenon you can handle a simple question to those who employ you. It's unreasonable to expect an AI to manage such complexity of planning and manipulation to fail on such basic reasoning levels.

Of course part of the problem is the objectives as written are objectives you would give to a sapient strong AI, something smart enough to reason on it's own (and thus smart enough to reason about the flaws and ambiguities of the objectives and clarify them). If you have anything less then strong AI the objectives above just don't work, they are not mathematically rigorous, far to ambiguous, and simply can't be 'understood' by your AI.

Okay, having let me mini-rant on how I deem the standard tropes to not be realistic lets try to look at some realistic response to this. I'm looking at a very capable but not sapient AI here, if your AI is sapient the question is not state objectives but how you motivate it to want to achieve your objectives, their about psychology of your sapience not writing of programing and best addressed as a separate question.

Lets start with the boring, but most realistic, outcomes.

Error segmentation fault at line 234521508 core dump

Most likely outcome, your AI doesn't work. Your asking for a pretty complex program after all, it failing to get off the floor is always possibility.

Similarly it could have issues such as being unable to connect to appropriate systems to collect data. Or unable to process data fast enough (this is a real issue with all our cloud ingest now of days, we have so much coming in we don't know how to look at it all fast enough).

Were paying how much!? shut that program down!

A program like this would be expensive. As I mentioned monitoring so much data real time would require a cloud solution, paying amazon a ton for cloud computing power every month (which is still much better then having to maintain your own servers cost rise). Your have to pay to maintain the systems that feed data to the computer, handle wires being cut or other data lose, pay for maintaining the program, fixing bugs, and doing other logic. It's quite possible that the cure is worse then the disease, that the insurance company pay more maintaining the program then it saves them.

Man the WatsonCare website sucks

People fail to adopt and use the tool because it's confusing, hard to understand, had UI bugs etc. Can you make Watson not just work, but sexy and appealing?

Moving on to the presumption it actually gets used at all. There are some conceptual issues, independent to requirements, which will prove major problems with this approach. Their mostly due to human nature, which isn't always reasonable:

I just want to eat my deep fat fried bacon pizza with nacho cheese dipping sauce in peace!

It sounds like the main goal of Watson is to keep people from doing unhealthy things that end up hurting them. Unfortunately, we like unhealthy things. If we didn't enjoy them we wouldn't do them. Sure sitting in front of my computer playing games all day isn't as healthy as going outside, but as a geek I like playing games, I don't want to have my computer shut off until I go for a walk. Even if Watson only provided polite reminders it will annoy people if they get reminder every ten minutes. For that matter people won't want to know how unhealthy that fast food is, because it will ruin their enjoyment when they inevitable eat it anyways.

Do you remember 'clippy' from word? The little paperclip everyone loved to hate! Watson could be the new clippy, only massively worse. Watson could be what would happen if scrappy doo and whesly crusher had a gay love-child raised by jar jar binks and trained by clippy! (I would call rule 34 on the first part of that statement but I'm already running close to geek critical mass already so best to leave well enough alone).

If done wrong customers could be complaining or leaving their insurance to simply never have to interact with Watson ever again. In fact this is a very likely outcome, time and time again we have seen situations where people will complain if reminded of their own faults, even if it's part of a system to help them. Given the years of HCI study and disasters it almost seems guaranteed to be the case unless Watson is scaled back to only do 911 calls unless people explicitly activate other features; at which point he is not saving the insurance company much money because he is not preventing unhealthy habits.

Why did you break down my door? No I'm not dying I'm fine, I just took off my life alert for a shower! I'm not going to pay for that ambulance!

If Watson is going to call 911 then there will be false alarms, any program has some degree of false positives. But given the difficulty of detecting actual issues based off of the sort of detection tools we currently have available false alarms will be somewhat common.

This would be an annoyance to the one with paramedics beating down their door, but that's not the worse of it. If your calling ambulances your taking those ambulances away from the hospital. If someone else gets injured while the ambulance is at my house because I accidentally slept with my lead-lined, monitor-wifi blocking, blanket they will not be ready to go save someone who actually needs their help. People could die from these false alarms!

To give a comparison look at home security systems. These systems have a massive false alarm rating, more then 90% of their alarms are false alarms. So many in fact that police generally do not prioritize them and only send police officers over to a house if the officer has absolutely nothing else he could be handling instead (meaning when they do arrive it's often late enough that they couldn't have helped anyways, the cops aren't using sirens or speeding to get to your presumed false alarm). Some police districts refuse to even deal with home security providers any more, it was costing too much in taking their leads and so rarely helping.

However, cops, while still important, are far less likely to be needed at a life or death situation this very moment as an ambulance is. distracting them is much more likely to hurt someone else; or have them simply ignore your alarms.

My mother slowly died of a heart attack and Watson did nothing

The other PR nightmare side of the above situation. If you try to avoid false alarms you likely end up letting people die who you could have otherwise helped.

At least I feel better about the NSA spying on me compared to Watson

This was already mentioned, but this is a massive privacy issue here. People will be up in arms almost immediately. Watson would have to collect far more information then the NSA confessed to be collecting with it's metadata, and you saw how people responded to them. The public is very sensitive about privacy concerns right now, even about relatively minor issues. The degree of information collection would likely offend quite a few and cause a political backfire forcing Watson to be scaled back of disabled.

In fact people would start to demand whatever systems Watson used to collect data were disabled so no one else could collect data like Watson did, which could be harmful if they are insisting on disabling important safety features out of fear of spying after people exaggerated the threat of Watson.

Imagine someone disabling the equivalent of their life alert pendent because they kept hearing conspiracy theorists claim that Watson used it to detect heart-rate so it could detect rather they were afraid, angry, or sexually excited. Most likely the monitoring system can't even do that, but people will claim it, others will believe it, and suddenly people are less protected health wise due to paranoia about privacy.

Privacy 2, premium raising electric bogaloo

Another common claim will be made that if Watson can watch your habits it can know who is healthy and who isn't and provide that to the insurance company. The insurance company could then use that to raise rates if they decide someone isn't getting enough exercise, is eating too much fast food, or otherwise is a health risk. This could cause people who are not generally as upset about data mining to be cautious or refuse to use Watson because their certain their premiums will go up.

Save 10 dollars a month on your premium, for the low low cost of installing 10,000 dollars worth of monitoring equipment!

people don't have equipment capable of monitoring them to this degree in their homes or offices, and even in the future it's unlikely they will have equipment like this for a long time; there limited advantage to them outside of this one specific edge case. This means your asking people to buy a bunch of equipment, install it, and maintain it just to be able to use Watson. Even if you presume a future where the equipment is cheap the hassle of getting it installed could be enough to make people decide it's not worth the effort. Adoption and use of Watson could die out because it's just not worth the effort.

I know you could argue that Watson is so useful people will take the time to install it, but you may give humans credit as being logically motivated when they aren't. The truth is most of us don't accept our own health as an issue (and those that do tend to excessively over stress on it), and we can be quite easily discouraged by even a little bit of effort.

Look at Betamax, it was in almost every way better then VHS. Yet the awesome Betamax died a said forgotten death. Why did that happen? Because originally it didn't allow 2 hours of recording at once. It totally could, but the original betamax taps were made smaller. That meant to watch a movie you had to get up and swap tapes mid way through, and no one wanted to do that. SO we settled for lower quality, more expensive, easier broken VHS because we didn't want to get up off the couch rather then just waiting for Betamax to start making the 2 hour tapes. That goes to show even a great product can fail over minor hassles making people not bother.

Of course the other reason VHS did so well is they allowed selling of pornography which was not allowed on betamax, and porn is always the first adopter of new media. It's theorized this was a major issue for proliferation of VHS. So...if you can find a way that the monitoring equipment can also be used as part of some bizarre sexual kink you may have a chance :P

My hacking cough can be detected by a Hacker

Again, privacy concerns could make people afraid to use this system for fear that they will be hacked and their data shared.

However, hacking becomes a (slight) actual issue. Data sent through SSL is pretty secure and is no more a hacking concern then any of the other, far more life critical, electronic activities we have. However, to monitor our health there would likely have to be a number of wireless detectors available and running at all times, to collect data on us wherever we are and route the information through the internet. These wireless detectors may be more vulnerable to local hacking attempts, mostly because the people building them likely won't be thinking about the hacking issue and won't secure them as well as they should.

Family of unsupervised boy who tried to fly by setting off gasoline soaked fireworks he tapping to shoes, before jumping off cliff suing Watson for not warning child of risk

If Watson is suppose to be extending life of insurances individuals people are going to start taking this for granted, especially since your likely have to advertise this as a perk of your insurance to get increased enrollment. Once you do this people will expect Watson to protect them, no matter what.

Now any time someone get's hurt or does something idiotic they will point out that Watson should have protected them and it's a contract violation that he didn't. My fireworks example, while hyperbole, is actually not that unlikely to come up at some point, though the case would likely be instantly dismissed without cause.

However, other cases can come up. What if watson doesn't warn that combining two medications could have adverse affects? What if Watson fails t notice a subtle irregular heartbeat that could have warned someone of a health condition? What if someone doesn't speak to the doctor about that odd tingling in their arm because they figure if anything was wrong Watson would tell them, and then is shocked that Watson can't detect tingliness and thus didn't have any way to realize that there was a major health problem?

People will sue, liability issues will come up. These can be handled in court and with carefully warded contracts, not enough to singlehandedly destroy watson, but they will be a common occurrence that will add to expenses for the agency, expenses watson has to help them save to even break even.

Okay so now lets get to your actual question, what issues could come out of the stated objectives. These are the less likely issues in my mind, all the above seem quite a bit more probably (except maybe the hacking one). However here are more likely issues with bad objectives, presuming a non sapient but well written weak AI. Lets look at the sort of likely programming mistakes that could come out of it's objectives.

ConflictingObjectivesException: 'extend membership' conflicts with objective 'cut expenses'

aka

We went bankrupt in the first week because we payed all our money to pay for a machine to keep one 120 year old man breathing for 5 more minutes

This is directly related to the above rant, but objectives as they are stated don't work. People buy insurance for payouts, if your cutting payouts your going to make people not want to buy your insurance. There are tricks you can do, mostly social engineering and quoting manipulative statistics, to help cut payouts while lowering lost customers, but ultimately these two are partially exclusive. Thus you need to better articulate importance, how many customers can I lose per thousands of dollars of payout I cut, how many more payouts can I make if it increases enrollment.

Similarly preserving lives is likely to to lead to making more payments, for a similar conflict.

Frankly a far far more extensive set of requirements, prioritization between exclusive requirements etc is required to be implemented.

Lets assume that a programmer set down and wrote such through requirements designed to emulate the general concept of the above...

We were saving lives and had great enrollment until we ran out of money Even if write more detailed requirements you still need to set priority levels on them. If you pick poor priority, for instance you value enrollment numbers too high and minimizing payouts too low, your still have a problem. A human being is still making these prioritization decisions and could make bad ones. Though really this is a moot point, because the requirements as written have more significant issues...

Sign up for new WatsonCare lite insurance, only 99 cents a decade!

I lied above, your two objectives, increase subscriptions and decrease payouts, aren't actually mutually exclusive if read literally. There is a way I can get lots of people to sign up without paying out often, just make the cost of signing up trivial enough. If Watson offered some nearly free, or fully free, insurance then it would be worth enrolling for even if it rarely made payments, because if cost nothing then any payments you get are a net gain.

This obviously means that the insurance company is loosing out, because even though they pay out 1/3 of the time compared to their other policies if they still aren't taking in nearly enough in insurance premiums to cover those payouts they do make.

Or there is the other solution...

**The Watson lotto-surance is now up to $500,000! sign up today. The lucky hospitalized winner could be you!*

The objectives only stated that few payments had to be made, not that payment amounts had to be kept low. One solution is to make fewer payments, but drastically increase the amount payed out when one of those payments are made. From a purely reasonable point of view if I can sign up for an insurance that is 5% less likely to make a payment on a claim, but pays me twice as much when it does, it is optimal for me to do so because odds are I'll get more from the insurance on average. Thus I'm encouraged to sign up for this program even as Watson has lowered it's payouts.

In fact it's worse then that, because the scary thing is I could almost see this idea working. People are quite irrational with their money, done right one could probably tempt some individuals into a lottery style insurance by bragging about they high payments even if they end up paying less money out in total...no one suggest this to insurance agents!!!

Now one may question why this and the previous example don't fall into my "not giving AI enough credit" rant above, but there is an important difference. I'm now assuming a less sophisticated AI, which could still fall into these sort of solutions with a genetic algorithm style approach which simply tried possible outcomes until these turned out to work. Such a less advanced AI is more prone to mistakes in objectives resulting in odd results. However, more then that, these are mistakes I can see a programmer make!

The problem to these mistakes is that the programmer didn't actually state his real objectives. He picked specific details that usually lead to his desired outcome and presumed if those criteria were met his outcome would occur; this is a very common logical mistake I have seen in plenty of programs. The truth is that the insurance agency's real goal is to save money, possibly in an ethical way that also protects lives, but as a corporation saving money is still high on their priority list. They assumed more people enrolled while paying out fewer grants meant more money coming in and less going out and thus a net savings, but that isn't always the case. They should not have the stated objectives, and instead have an objective of "maximize profits" combined with whatever appropriate ethics objectives ensure it's done in a way they can live with. They should not specify the exact method of making money is done. It may be that increasing payouts, but also insurance premiums, to cater to certain high-income folks results in more income, even if it lowers net enrollment and thus violates both the stated goals. this is a believable multiprogramming of objectives, not a flaw in how a computer interprets ambiguous objectives, but religiously written objectives (presuming they were more rigors to clear up that ObjectiveConflictError) that simply were poorly chosen objectives.

Of course even if we redefine our objectives to be help the company profit while being moral and preserving lives we could still have issues...

*Sign up today and one of our staff of watson-cares doctors will move in down the street to service your every boo boo!**

If you employee enough people and offer enough non-payout benefits your insurance can be desirable even with fewer payouts, because of the other perks. of course it may turn out those other perks an unfathomably large amount of money to maintain, bankrupting the company in a few months, but hey, you never said anything about cost of employees.

Yes I'm assuming we redefined our objectives to conserving money, yes I still consider this a very plausible issue (well okay the above is hyperbole, but similar issues are likely). You see our non-hard AI can only consider variables it's programmed to know how to handle. If I forgot to write an algorithm to calculate the cost of employment of a new policy then the AI will never consider this, no matter how hard it's run, because it's not actually sapient. It needs me to define what variables it should consider ahead of time.

Sure I will likely have a CalculateStaffSalaries() algorithm, that's pretty obvious. but will I have a calculateHROverhead() algorithm, which considers the extra expense for managing the extra staff? will I have a updateSalaryToAnticipatedMarketDemand() method, which recognizes that employing so many doctors will mean there are fewer doctors out there, and thus the remaining doctors will ask for higher salaries and eventually I'll have artificial inflated the doctors salary to an unplayable high degree, or will my program still assume doctor salaries are the same as they were when it started running? What if I do write my updateSalary() method, but my understanding of economic supply and demand isn't perfect because I'm not an economist, and so I get my math wrong and don't anticipate the salary inflation accurately? for that matter what if I emulate all our economic knowledge perfectly, the market may still become inflated because economics are not able to reliable predict market influences that well.

In short, it's quite possible that the company still goes bankrupt because the AI wasn't able to anticipate expenses or costs.

I would have gotten away with it too, if it wasn't for you meddling kids and your smallpox-ridden dog!

What if the algorithm is wonderfully written to consider every variable and equation known to man and after massive number crunching it comes up with a payout scheme designed to save the insurance agency the most money by offering lass payouts for the likely issues but offering very tempting payouts for unlikely things. This would be a great trick, tempt people to sign up because it looks great without having to pay much, Your computerized munchkin has perfectly min-maxed your system.

And then the supposedly obliterated small pox turns out to have been incubating somewhere and a breakout occurs. Everyone calls their insurance agency and it turns out our policy offered payout 40 times as high as our usual for anyone that caught small pox, because we took the "Smallpox Bankruptcy" flaw"to help pay for that "conveniently 'misfiled' claim" perk. The bane of every min-maxer, you must have accidentally upset the GM because today's encounter hits all all your unlikely flaws.

This is a hyperbole example, but the issue is not as minor or ignorable as you may imagine. We don't yet have the ability to predict the future, which means even the best AI will be unable to predict potential outcomes. This means there is always a chance of bad luck leading to an unplanned for outcome that ends badly for our AI driven company.

But the problem is a little worse then this in a way. Unlike with humans a computer will try to min-max fully, if I didn't write special code to avoid it the program will pick the situation that gives the best average case, even if it means creating a situation where bad luck doesn't just cause the company money, but completely bankrupt it. It's worried about the average case only. Since it can't predict everything it thus may choose to create a policy that has a 20% of making the company go bankrupt if the average case makes the most money.

However, people aren't interested in average case, we generally would prefer to be more cautious here. If I'm CEO of WatsonCorp I would rather settle for 'only' 600,000 annual salary (the actual average for CEO of healthcare, thank you google) then take a 10% chance of the being fired after my company goes bankrupt to in order to have a 90% chance of making 660,000 salary. The extra profit won't help me nearly as much as unemployment will hurt me and I don't want to take that risk.

In programming world we have learned to be very aware of this. When we inspect algorithms now we discuss the big O of a program, which is (sort of) how fast it runs in the worst case, how do things happen when everything goes wrong. This is because we have learned that when I run a program every day for years that 'unlikely' worse case scenario ends up coming up pretty often, and it's not uncommon for the worst case to be disproportionately bad. Well...that and it's not uncommon for either malicious people to intentional design a worst case scenario or for the worst case scenario to be far more likely then random chance would expect. Good programmers would thus write an AI with an understanding of what worst-case scenario is acceptable...but this is an easy thing to forget to add. If you don't there is a very decent chance that your end up with a small, but disastrous, potential weakness in whatever solution your AI creates.

Next up on Scandal News, we learn Watson-international refuse life saving procedure for chronically on welfare, Their uncanny-valley ridden robotic overlord explains chronic illness required too many payments for one life, new cost cutting measures allow paying for 10 times as many breast enhancements

Closest to the evil-AI doomsday scenario were likely to get, and also a bit more questionable, to presume an AI would be sophisticated to calculate this without human involvement. However, this scenario could be eached by soft AI through genetic algorithm or certain well written inference engine and calculation processes.

the idea is simple, chronically ill require constant payouts over a long length of time, and thus can be quite expensive to support. However, refusing to pay for one life saving surgery could remove all future payments for that individual-you know, him being dead and all, saving a significant amount of money. A genetic algorithm approach may notice that not paying for some surgery results in far less money payed out on average without the AI realizing that's because people died without the surgery for instance; the AI only knows saying NO to this surgery means less money payed and that's all it's calculating.

This actually is a result in flaws of the above examples. It's a mis-tunning of priority levels, the AI not valuing life preservation as high as it should have compared to cost cutting. However, this doesn't have to be life saving operations. Perhaps not paying for certain surgery results in someone going into a vegetative state and ultimately costing less to provide for that way. It could be that other moral standards, beyond lives served, are required. For instance a "treat everyone equally" standard may avoid intentional infractions upon certain categories of people which would lead to outrage if learned otherwise.