Assume any law or policy was subjected to A/B testing after being enacted.

A/B testing: "is a randomized experiment with two variants".

Which consequences would it have on long term on the say Human Development Index?

Margin conditions:

  • The A/B test is configurable, i.e. a set of critera is applied to determine whether to use A/B test at all, for how long and on which scale.

  • The granularity is decided (city, region, state..)

  • In the modeled world there are about 250 states so you could group them to subject entities to give the experiment more sense.

  • To avoid overlap, states/regions subject to the A/B test are locked for other policy changes.

An over-simplified example with a taxation policy.

Income tax should be increased from x to y %.

One region gets this, another one (x-y)/2%.

After K years (to be decided as well individually) they analyze data and decide which tax increase would be good for the whole nation.

A more complex example, in two variants.

  • "A" test: more strict patent laws / either support abortion.
  • "B" test: disable patent laws for the test time / either invest to inhibit abortion .

closed as too broad by Renan, Erin Thursby, Aric, Rekesoft, Ash Aug 21 '18 at 12:49

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ You should explain what you mean with A/B testing for those not as familiar with the term, especially given that you are trying to establish it in a context where it's normally not used. Basically you want to have some people that need to adhere to the new law and some people that don't as a test period with the goal of seeing which group has the lowest crime-rate in regards to the specific law, right? What do you do about one person from A interacting with one person from B? Or are they split? This is currently unclear and also very broad. As such I am voting to temporarily put this on hold. $\endgroup$ – Secespitus Aug 16 '18 at 12:42
  • $\begingroup$ This sort of exists now between the US states. $\endgroup$ – GrandmasterB Aug 16 '18 at 18:53
  • $\begingroup$ @GrandmasterB - cool! are there examples? $\endgroup$ – J. Doe Aug 16 '18 at 18:55
  • $\begingroup$ @J.Doe Each US state has its own state tax rates, gun laws, drug laws (to some extent), etc. People and businesses migrate from state to state, when one state becomes more attractive to others. Interpreting the exact reasons for that can be complex, of course. $\endgroup$ – GrandmasterB Aug 20 '18 at 3:10

Something like this is going on -- government often try a "pilot" of a new policy, limited to one region, or a subset of population. The rest of regions/population keep the old policy, so they become the B group.

In addition to timing and strategic response of population, a major problem with "confounding" differences between regions. A policy might work well in one region, but it is only due to unique characteristics of that region, or even the state of ever-changing economic conditions at the start of experiment:

  • Policies that work in urban areas will not work in rural ones.
  • Younger population will respond to policies differently than the older one.
  • Cultural differences, e.g. trust in government or viability of "honor system".
  • Economic stimulus (like low interest rates) is beneficial during or right after a recession, but using it in boom years it overheats the economy and causes the recession (e.g. the 2008 crash).
  • 1
    $\begingroup$ Yes! For example, state A has death penalty, state B doesn't. Turns out, effect on the crime is never clear-cut, and support groups can argue without an end in sight. $\endgroup$ – Alexander Aug 16 '18 at 17:02

One could argue that this happens in the real world.

Capitalism has defeated communism, right? Lower corporate taxes in Ireland managed to create a sort of economic boom, to give a more recent and less Earth-shattering example. But there are a couple of problems:

  • If the customers are aware that there is an A/B test going on, and if they can influence if they are part of group A or group B, they will act on it. Especially if the stakes are significant.
    • Say city A decides to subsidize museums, theaters, public transport, and so on to see if it leads to a happier citizenry. Suburban district B decides to test if one can avoid all these expenses by building a parking lot just outside A's city limits, near the subway station.
    • Say country A funds the national budget with a tax on income, and country B funds it with a tax on wealth. Then retirees living on their savings will flock to A and younger working-age people, especially those who can work remotely, will flock to B.
    • Say country A decides on low taxes and bad public healthcare, and country B decides on high taxes and good public healthcare, then elderly or those with young children will go to B and healthy people will go to A.
  • An A/B test must make sure that the differences are clearly defined. Red button to green? A bigger picture of the product or more space for the text? That's very suitable for an A/B test.
    • Say you want to find out if the death penalty deters possible murderers. The way the test was defined, you go with official crime statistics. (No altering the criteria after the fact, that's cheating.) Except that a serial killer is caught and gets entered into the statistics for that year.

Of course you could ban the evasion strategies I've outlined above, but then you need to raise walls between the countries. Sooner or later the cultures will diverge so much that comparison is meaningless.


It depends on the society in question, and also comes with a few obvious and very much severe ethical issues.

A/B testing is acceptable among marketing departments, and some fields of science because very often the results of the change in the conditions has very little long-term effect, other than their immediate actions.

For a simple practical issue, for many types of policy, the timescale would have to be extremely long, and such long timescales both open it to manipulation (a bad actor could notice a flaw in the new tax policy during the A/B phase, then wait until the full implementation of the policy to exploit that flaw, making a huge amount of money in the process) While such long timescales may be acceptable in societies with very long lives, among a society with human lifetimes, many policies would take many generations to fully test.

There is another obvious ethical issue here, and that is the scale of the policies. In marketing and web design and so on, they might check user response to a change of button color, or position, or fonts etc. Such an action as clicking a button usually requires no commitment or monetary gain/loss. In comparison, to use the tax example, imagine how people would respond to being randomly selected for a tax policy that makes the unequivocally poorer. While the selection was 'fair', it was not ethical.

To give an even more extreme example, health-care policy. I doubt many people would respond well to being deprived of state health-care because they were selected as part of the B group in an A/B. While from a political perspective there are people against government healthcare, but if you are a poor individual, and you or a loved one is being refused treatment for an easily treatable disease, as a result of a random selection, it transcends politics.

Even if it was highly effective in finding the correct policy, it is far too slow and far too costly in terms of both money and lives, for any government to be able to implement without retaliation both by its own people and by whatever international community exists.

  • $\begingroup$ A/B tests are also legitimate in clinical studies of new medication, where the life of the subject may be at stake. It just requires stronger processes and safeguards. $\endgroup$ – o.m. Aug 16 '18 at 14:09
  • 1
    $\begingroup$ Considering the extreme degree of pre-trial, and other work, and the ethical standards that go into clinical trials before that point, I doubt any government would actually reach similar standards. Similarly, the subject, as per double-blind, would not know which group they are in. This is simply impossible for trials of policy. $\endgroup$ – Direwolf202 Aug 16 '18 at 14:12
  • $\begingroup$ Those are certainly important differences, but my point is that A/B-tests are done on life-and-death matters. I don't think A/B-tests for laws would work, but this is not the reason. $\endgroup$ – o.m. Aug 16 '18 at 15:23

You have to take significantly longer time scales than a year for the tests

A lot of measures bring a boom for a few years and result in collapse a few years later. You would have to monitor everything for decades. Which would mean you have a lot of overlap of experiments.

That means this method is ultimately useless as you can not reliably identify the results of the laws. There are way too many being passed per year.

  • $\begingroup$ what if we take 20 years and lock A/B test subjects? Also, in the world building afaik also modeling useless methods is allowed. $\endgroup$ – J. Doe Aug 16 '18 at 12:56
  • 1
    $\begingroup$ @J.Doe Then it would work, but it means every single country could only pass A SINGLE LAW PER 20 YEARS. That is far beyond possible. You might get great data on what works and what does not, but your countries have no possibilities to react to new problems, etc.. $\endgroup$ – ArtificialSoul Aug 16 '18 at 13:24
  • $\begingroup$ You are a pessimistic soul. A single law per two decades, indeed! Testing different versions of laws in different states doesn't need to be restricted to a few laws. Tax law won't interfere with the criminal law, for example. You're overdoing the methodology. It is only necessary to obtain sufficient data. Social systems have a lot more tolerance. $\endgroup$ – a4android Aug 16 '18 at 13:44
  • 1
    $\begingroup$ @a4android Yes, I am a pessimist, but that does not mean I am wrong. Saying different laws don't interfere with the results is absurd. Lets say you legalise some drug while also raising taxes. Does that mean a decrease in happiness is caused by the first, the latter or both? This concept simply can not work in the real world. Your data is completely useless if you can not exclude interferences. $\endgroup$ – ArtificialSoul Aug 16 '18 at 13:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.