Articles on Statistics

Articles on statistics written for marketers, user experience specialists, and web analysists in mind. Digging deep into statistics for online experiments (a.k.a. online A/B tests) and the methodological issues and approaches for solving them.

How to Run Shorter A/B Tests?

Shorter A/B Tests

Running shorter tests is key to improving the efficiency of experimentation as it translates to smaller direct losses from testing inferior experiences and also less unrealized revenue due to late implementation of superior ones. Despite this, many practitioners are yet to start conducting tests at the frontier of efficiency. This article presents ways to shorten […] Read more…

Also posted in A/B testing | Tagged , , , ,

Comparison of the statistical power of sequential tests: SPRT, AGILE, and Always Valid Inference

Power and Average Sample Size of Sequential Tests

In A/B testing sequential tests are gradually becoming the norm due to the increased efficiency and flexibility that they grant practitioners. In most practical scenarios sequential tests offer a balance of risks and rewards superior to that of an equivalent fixed sample test. Sequential monitoring achieves this superiority by trading statistical power for the ability […] Read more…

Also posted in A/B testing, AGILE A/B testing | Tagged , , , , , ,

Statistical Power, MDE, and Designing Statistical Tests

Statistical Power and MDE Demystified

One topic has surfaced in my ten years of developing statistical tools, consulting, and participating in discussions and conversations with CRO & A/B testing practitioners as causing the most confusion and that is statistical power and the related concept of minimum detectable effect (MDE). Some myths were previously dispelled in “Underpowered A/B Tests – Confusions, […] Read more…

Also posted in A/B testing | Tagged , , , ,

Fully Sequential vs Group Sequential Tests

Sequential Testing Compared

What is the best design for a statistical test with sequential evaluation of the data at multiple points in time? This is a question anyone who has realized that unaccounted for peeking with intent to stop is the bane of A/B testing eventually comes to ask. So how does one go about answering that? This […] Read more…

Also posted in AGILE A/B testing | Tagged , , , , ,

A/B Testing Statistics – A Concise Guide for Non-Statisticians

AB Testing Statistics

Navigating the maze of A/B testing statistics can be challenging. This is especially true for those new to statistics and probability. One reason is the obscure terminology popping up in every other sentence. Another is that the writings can be vague, conflicting, incomplete, or simply wrong, depending on the source. Articles sprinkled with advanced math, […] Read more…

Also posted in A/B testing | Tagged , , , , ,

P-values and Confidence Intervals Explained

P-values and confidence intervals explained

Hundreds if not thousands of books have been written about both p-values and confidence intervals (CIs) – two of the most widely used statistics in online controlled experiments. Yet, these concepts remain elusive to many otherwise well-trained researchers, including A/B testing practitioners. Misconceptions and misinterpretations abound despite great efforts from statistics educators and experimentation evangelists. […] Read more…

Also posted in A/B testing, Statistical significance | Tagged , , , , , ,

Improve your A/B tests with 9 lessons from the COVID-19 vaccine trials

Lessons for Online AB Testing from the COVID Vaccine Trials

A full year has passed since results from the first clinical trials of COVID-19 vaccines became available. Vast swaths of follow-up observational data are now in circulation which enables some conclusions about the strengths and weaknesses of those trials. Since modern medical trials are among the most rigorous and scrutinized experiments they can serve as […] Read more…

Also posted in A/B testing | Tagged , , , , ,

The new standard for planning and analyzing A/B tests is here

A/B testing statistics done right

The first major overhaul of Analytics Toolkit since its release in early 2014 has finally arrived and it brings with it solutions to many of the hard questions facing practitioners when planning and analyzing A/B tests. Conducting statistically rigorous tests while achieving the best return on investment go hand in hand in the new Toolkit. […] Read more…

Also posted in A/B testing, AGILE A/B testing,, Conversion optimization | Tagged , , , ,

Bayesian Probability and Nonsensical Bayesian Statistics in A/B Testing

Bayesian Probability and Bayesian Statistics in AB Testing

Many adherents of Bayesian methods put forth claims of superiority of Bayesian statistics and inference over the established frequentist approach based mainly on the supposedly intuitive nature of the Bayesian approach. Rational thinking or even human reasoning in general is Bayesian by nature according to some of them. Others argue that proper decision-making is inherently […] Read more…

Also posted in A/B testing, Conversion optimization | Tagged , , , , ,