The search for the best statistical model for A/B testing

It’s been almost ten years since I published the white paper “Efficient A/B Testing in Conversion Rate Optimization: The AGILE Statistical Method” (2017). It argues that Group-Sequential Tests (GSTs) are ideally-suited for online A/B testing. I figure it is a good time to reflect back on my search for the best statistical model and see if with my current knowledge I still think GSTs are the best.

Development of mainstream statistical models in A/B testing

In the middle of 2016 I began to seriously explore the question “What is the best statistical model that fits the use-case of online A/B testing”. At that point I have been developing tools for planning and analysis of A/B tests with simple fixed-sample statistical significance tests for a few years. Their limitations were becoming apparent and the biggest one was peeking with intent to stop.

At that time, tools like mine were standard in most A/B testing software offered on the market. The way many of them dealt with the issue of peeking is to not even make customers aware of it. No sample size calculation = no target sample size or duration to abide by = no problem, right?

Obviously not, and the industry was fast becoming aware of this reality. One of the leading platforms adopted a sequential testing model just a year before in 2015. It was soon followed by Google Optimize which was launched in closed beta in late 2016, and fully opened in the early months of 2017. It, too, featured a Bayesian statistical model, except it used a median instead of a mean as a metric of interest (don’t get me started…).

The issue with the above developments is that they bought heavily into the false narrative that Bayesian methods allow the practitioner to simply not care about stopping rules. Well, they don’t, not without sacrificing error-control. If lack of error control was desired, one could just apply a fixed-sample calculation to a continuously monitored metric and get the same output as these Bayesian tools, which was what misinformed practitioners have been doing anyways.

So here I was, looking for a frequentist statistical approach which would best fit the typical A/B testing use-case.

Key issues and requirements for a statistical model

In the process of digging deep into what models were available and comparing their strengths and weaknesses, I’ve identified five major issues which lead to five requirements the ideal model had to fulfill.

The first issue a model has to address is human nature and basic business logic, both of which are not going anywhere. These express themselves as pressure to stop a test early if the outcomes look either too good or too bad . The intuition that stopping early can capture more benefits and limit unnecessary exposure to losing variants is in fact, correct. So here comes requirement #1:

1) The ideal model needed to allow for and account for peeking in some way, adjusting statistical estimates so as to preserve error-control.

Many such models exist and they are collectively known as “sequential testing” methods. Some are very rigid, requiring strict adherence to a timeline fixed beforehand (fixed analysis points) while others are fully-sequential, meaning they allow or even require the evaluation of the data after every observation. Yet others took a middle-ground and allowed significant flexibility on the timing and number of analyses.

In the day-to-day reality of online A/B testing there is typically a schedule (of typically weekly) analyses would mostly work, but there should be room for adjustments, especially in the number of analyses. So comes requirement #2:

2) Ideally, a model should allow flexibility in how many analyses are performed, and how often one can perform them.

Then comes statistical power. A fixed-sample test is known to be a universally-most-powerful (UMP) from which it follows that all sequential methods sacrifice some statistical power (a.k.a. sensitivity) in order to allow for peeking. The ability to stop early and maximize gains as well as limit losses comes at the cost of lower probability to detect true effects, especially smaller ones. This leads to #3:

3) A good balance between how much power is sacrificed versus how early one can stop, on average, is needed.

The above is especially true given how small most true effects are estimated to be. Given how half of true effects are zero or below, it is also important for the chosen method to be able to stop just as early or even earlier if the effect is negative or so close to zero as to have a very low probability of being statistically significantly better than zero. Requirement #4 is:

4) An efficient model should stop futile tests before they demonstrate harm with a statistically significant outcome

Some models work equally on both sides of zero by design, so they will only stop if a result is proven to be negative with statistical significance which is way too late. It wastes time stopping a test which is losing money since typically it is clear much earlier that the test has a very low probability of demonstrating a statistically significant improvement. A model which does that has a separate rule for stopping for futility which does not mirror the one for stopping for efficacy.

Finally, but not last by importance, a statistical model should allow or even encourage practitioners to run A/B tests whose outcomes generalize well post-test. This external validity is achieved by obtaining a representative sample, meaning one which addresses known threats to generalizability. If an estimated effect does not hold after the change is released, all the internal validity of the statistical machinery is good for nothing. So comes requirement #5:

5) A good statistical model should align itself with practices for obtaining representative samples to produce results which generalize well post-test

After going through all of the above issues and the model requirements necessary to address them, I found that only

Group-sequential tests check all the boxes

Here is how:

1. Group-sequential tests account for peeking in a statistically robust manner, and there are good adjustments to statistical estimates such as confidence intervals and maximum-likelihood estimates
2. A GST using alpha-spending and beta-spending allows practitioners to treat the sampling plan as a suggestion, rather than something fixed in stone. Analyses can be added, moved, or removed, as necessary, within reason, without impacting the overall properties of the statistical model.
3. A Group-sequential test offers the best trade-off between stopping early (20-80% earlier) and statistical power, when compared to other sequential tests used in the industry.
4. A GST with beta-spending results in a futility boundary which stops tests typically well before they prove a negative effect is present.
5. Group-sequential tests are uniquely aligned with good practices for obtaining test results that generalize well. Want weekly analyzes to avoid day-of-the-week effects? Done. Need spending functions which put a higher bar for very early stopping in order to balance stopping early with evading novelty biases and learning effects? A GST has your back.

I found all other available methods to be lacking, typically in more than one regard.

Where other sequential methods trip up

Fully sequential tests are still preferred by some vendors and appeal to certain practitioners mainly due to their extreme flexibility. What is less often discussed are the outsized costs of this flexibility:

1. Losing a lot more statistical power than necessary. Anyone who has seen a distribution of estimated effect sizes from online A/B tests should understand how bad that is.
2. Lack of inherent guardrails against practices which result in poor generalizability.

As per the reference shared above, fully sequential methods trade early stopping with power way too aggressively and as result have a lower probability to detect true effects as statistically significant. This also applies to methods advertised as having a power of one (100%).

To add to the above, many implementations lack a futility stopping rule so they wait too long when the results are non-promising enough to actually move on to the next test and stop potentially losing money from the tested variants.

So, fully sequential methods fail on three of the five requirements I identified during my search for the best statistical model to use in A/B testing. It is probably no coincidence that when advertised, fully sequential methods are never compared to a GST counterpart, but only to fixed-sample tests. It is certainly no coincidence that I’ve left all other sequential tests aside and have recommended the use of Group-sequential tests of the type described in my white paper ever since.

Curious to learn more?

My original white paper is available for free here. An updated and improved version I’ve produced recently is available for free on the ABsmartly website. My book, “Statistical Methods in Online A/B Testing” has a good deal on the topic as well.

Get hands on with GSTs

The easiest way to get a taste is through a free, non-obligatory trial at Analytics-toolkit.com – the software suite I’ve been developing since 2012. It specializes in statistical planning and analysis of A/B tests, and not much else.

I’ve also assisted the talented team at ABsmartly.com with their GST implementation. If you are a larger enterprise in need of a comprehensive solution for your experimentation program I’d highly recommend you get in touch with them.

About the author