# Category Archives: Statistical Significance

## Statistical Significance in A/B Testing – a Complete Guide

The concept of statistical significance is central to planning, executing and evaluating A/B (and multivariate) tests, but at the same time it is the most misunderstood and misused statistical tool in internet marketing, conversion optimization, landing page optimization, and user testing. This is not my first take on the topic, but it is my best […] Read More…

## Multivariate Testing – Best Practices & Tools for MVT (A/B/n) Tests

Let’s get this out of the way from the very beginning: most “A/B tests” are in fact multivariate (MVT) tests, a.k.a. A/B/n tests. That is, most of the time when you read about “A/B testing” the term also encompasses multivariate testing. The only reason to specifically differentiate between A/B and MVT is when someone wants to […] Read More…

Also posted in A/B Testing, AGILE A/B Testing, Multiple Variations Testing, Statistics | | Comments closed

## The Importance of Statistical Power in Online A/B Testing

What is Statistical Power? In null-hypothesis statistical testing (NHST) – the procedure most commonly applied in A/B tests, there are two types of errors that practitioners should care about, type I and type II errors. Type I is the probability of the test procedure to falsely reject a true null hypothesis. Type II error is […] Read More…

Also posted in A/B Testing, AGILE A/B Testing, Conversion Optimization, Statistics | | Comments closed

## 5 Reasons to Go Bayesian in AB Testing – Debunked

As someone who spent a good deal of time on trying to figure out how to run A/B tests properly and efficiently, I was intrigued to find a slide from a presentation by VWO®’s data scientist Chris Stucchio, where he goes over the main reasons that caused him and the VWO® team to consider and […] Read More…

Also posted in A/B Testing, Conversion Optimization, Statistics | | Comments closed

## The Bane of AB Testing: Reaching Statistical Significance

Now, this is not a new topic on this blog. I’ve discussed the issue of optional stopping based on “achieving” or “reaching” statistical significance in my Why Every Internet Marketer Should be a Statistician post more than two years ago and others have touched on it as well – both before and after me. However, the issue […] Read More…

Also posted in A/B Testing, Conversion Optimization, Statistics | | Comments closed

## Bayesian AB Testing is Not Immune to Optional Stopping Issues

Fantasy vs the Real World: Naive Bayesian AB Testing vs Proper Statistical Inference This post is addressed at a certain camp of proponents and practitioners of A/B testing based on Bayesian statistical methods, who claim that outcome-based optional stopping, often called data peeking or data-driven stopping, has no effect on the statistics and thus inferences […] Read More…

Also posted in A/B Testing, Conversion Optimization, Statistics | | Comments closed