Articles on Statistical significance

Articles on the topic of statistical significance in online A/B tests. What is significance, how to use it in various advanced scenarios and how to avoid the many misinterpretations, misuses, and abuses of p-values and confidence intervals.

P-values and Confidence Intervals Explained

P-values and confidence intervals explained

Hundreds if not thousands of books have been written about both p-values and confidence intervals (CIs) – two of the most widely used statistics in online controlled experiments. Yet, these concepts remain elusive to many otherwise well-trained researchers, including A/B testing practitioners. Misconceptions and misinterpretations abound despite great efforts from statistics educators and experimentation evangelists. […] Read more…

Also posted in A/B testing, Statistics | Tagged , , , , , ,

Designing successful A/B tests in Email Marketing

The process of A/B testing (a.k.a. online controlled experiments) is well-established in conversion rate optimization for all kinds of online properties and is widely used by e-commerce websites. On this blog I have already written in depth about the statistics involved as well as the ROI calculations in terms of balancing risk and reward for […] Read more…

Also posted in A/B testing, Conversion optimization, Statistics | Tagged , , , , , , , , , ,

Confidence Intervals & P-values for Percent Change / Relative Difference

In many controlled experiments, including online controlled experiments (a.k.a. A/B tests) the result of interest and hence the inference made is about the relative difference between the control and treatment group. In A/B testing as part of conversion rate optimization and in marketing experiments in general we use the term “percent lift” (“percentage lift”) while in […] Read more…

Also posted in A/B testing, Conversion optimization, Statistics | Tagged , , , , , , , , ,

Risk vs. Reward in A/B Tests: A/B testing as Risk Management

Risks vs Rewards in AB Testing

What is the goal of A/B testing? How long should I run a test for? Is it better to run many quick tests, or one long one? How do I know when is a good time to stop testing? How do I choose the significance threshold for a test? Is there something special about 95%? […] Read more…

Also posted in A/B testing, Conversion optimization, Statistics | Tagged , , , , , , ,

Statistical Significance for Non-Binomial Metrics – Revenue per User, AOV, etc.

Non-Binomial Significance - Revenue, Per User Metrics

In this article I cover the method required to calculate statistical significance for non-binomial metrics such as average revenue per user, average order value, average sessions per user, average session duration, average pages per session, and others. The focus is on A/B testing in the context of conversion rate optimization, landing page optimization and e-mail […] Read more…

Also posted in A/B testing, Conversion optimization, Statistics | Tagged , , , , , , , ,

One-tailed vs Two-tailed Tests of Significance in A/B Testing

Two-tailed vs one-tailed test

The question of whether one should run A/B tests (a.k.a online controlled experiments) using one-tailed versus two-tailed tests of significance was something I didn’t even consider important, as I thought the answer (one-tailed) was so self-evident that no discussion was necessary. However, while preparing for my course on “Statistics in A/B Testing” for the ConversionXL […] Read more…

Also posted in A/B testing, Conversion optimization, Statistics | Tagged , , , , , , , ,

The Case for Non-Inferiority A/B Tests

The Case for Non-Inferiority Testing

In this article, I explore the concept of non-inferiority A/B tests and contrast it to the broadly accepted practice of running superiority tests. I explain where non-inferiority tests are necessary and how a CRO/LPO/UX testing specialist can make use of this new approach to A/B testing to run much faster tests, and to ultimately achieve […] Read more…

Also posted in A/B testing, Conversion optimization, Statistics | Tagged , , , , ,

Statistical Significance in A/B Testing – a Complete Guide

Statistical Significance P Value

The concept of statistical significance is central to planning, executing and evaluating A/B (and multivariate) tests, but at the same time it is the most misunderstood and misused statistical tool in internet marketing, conversion optimization, landing page optimization, and user testing. This article attempts to lay it out in as plain English as possible: covering […] Read more…

Also posted in A/B testing, Conversion optimization, Multiple variations testing, Statistics | Tagged , , , , , , , , ,

Multivariate Testing – Best Practices & Tools for MVT (A/B/n) Tests

Multivariate AB Tests / MVT Testing

Let’s get this out of the way from the very beginning: most “A/B tests” are in fact multivariate (MVT) tests, a.k.a. A/B/n tests. That is, most of the time when you read about “A/B testing” the term also encompasses multivariate testing. The only reason to specifically differentiate between A/B and MVT is when someone wants to […] Read more…

Also posted in A/B testing, AGILE A/B testing, Multiple variations testing, Statistics | Tagged , , , , ,

The Importance of Statistical Power in Online A/B Testing

Statistical Power and Test Sensitivity

What is Statistical Power? In null-hypothesis statistical testing (NHST) – the procedure most commonly applied in A/B tests, there are two types of errors that practitioners should care about, type I and type II errors. Type I is the probability of the test procedure to falsely reject a true null hypothesis. Type II error is […] Read more…

Also posted in A/B testing, AGILE A/B testing, Conversion optimization, Statistics | Tagged , , , , ,

5 Reasons to Go Bayesian in AB Testing – Debunked

Frequentist vs Bayesian A/B Testing

As someone who spent a good deal of time on trying to figure out how to run A/B tests properly and efficiently, I was intrigued to find a slide from a presentation by VWO®’s data scientist Chris Stucchio, where he goes over the main reasons that caused him and the VWO® team to consider and […] Read more…

Also posted in A/B testing, Bayesian A/B testing, Conversion optimization, Statistics | Tagged , , , , , ,

The Bane of AB Testing: Reaching Statistical Significance

Illusory Results AB Testing

Now, this is not a new topic on this blog. I’ve discussed the issue of optional stopping based on “achieving” or “reaching” statistical significance in my Why Every Internet Marketer Should be a Statistician post more than two years ago and others have touched on it as well – both before and after me. However, the issue […] Read more…

Also posted in A/B testing, Conversion optimization, Statistics | Tagged , , , , , , , , , ,