Tag Archives: sample size

Multivariate Testing – Best Practices & Tools for MVT (A/B/n) Tests

Multivariate AB Tests / MVT Testing

Let’s get this out of the way from the very beginning: most “A/B tests” are in fact multivariate (MVT) tests, a.k.a. A/B/n tests.┬áThat is, most of the time when you read about “A/B testing” the term also encompasses multivariate testing. The only reason to specifically differentiate between A/B and MVT is when someone wants to […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Multiple Variations Testing, Statistical Significance, Statistics | Also tagged , , , , | Leave a comment

The Importance of Statistical Power in Online A/B Testing

Statistical Power and Test Sensitivity

What is Statistical Power? In null-hypothesis statistical testing (NHST) – the procedure most commonly applied in A/B tests, there are two types of errors that practitioners should care about, type I and type II errors. Type I is the probability of the test procedure to falsely reject a true null hypothesis. Type II error is […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Conversion Optimization, Statistical Significance, Statistics | Also tagged , , , , | Leave a comment