Category Archives: Statistics

Frequentist vs Bayesian Inference

Frequentist vs Bayesian Inference

In this article I’m revisiting* the topic of frequentist vs Bayesian inference with specific focus on online A/B testing as usual. The present discussion easily generalizes to any area where we need to measure uncertainty while using data to guide decision-making and/or business risk management. In particular, I will discuss each of the following five […] Read More…

Also posted in A/B Testing | Tagged , , , , , , , | Leave a comment

Underpowered A/B Tests – Confusions, Myths, and Reality

Underpowered A/B Tests

In recent years a lot more CRO & A/B testing practitioners have started paying more attention to the statistical power of their online experiments, at least based on my observations. While this a positive development for which I hope I had contributed somewhat, it comes with the inevitable confusions and misunderstandings surrounding a complex concept […] Read More…

Also posted in A/B Testing | Tagged , , , , , , | Leave a comment

The Perils of Poor Data Visualization in CRO & A/B Testing

As any UX & CRO expert should now, the way we present information matters a lot both in terms of how well it is understood and in terms of the probability that it will lead to the desired action. A/B testing calculators and other tools of the trade are no exception and here I will […] Read More…

Also posted in A/B Testing, Conversion Optimization | Tagged , , , , , , , | Leave a comment

Does Your A/B Test Pass the Sample Ratio Mismatch Check?

Sample Ratio Mismatch AB Testing

Most, if not all successful online businesses nowadays rely on one or more systems for conducting A/B tests in order to inform business decisions ranging from simple website or advertising campaign interventions to complex product and business model changes. While testing might have become a prerequisite for releasing the tiniest of changes, one type of […] Read More…

Also posted in A/B Testing | Tagged , , , , , | Leave a comment

Statistical Methods in Online A/B Testing – the book

Book Cover Statistical Methods Online A/B Testing

The long wait is finally over! “Statistical Methods in Online A/B Testing” can now be found as a paperback and an e-book on your preferred Amazon store. Note that the Kindle edition is available for $2.99 or equivalent if you’ve already purchased the paperback (through Kindle Matchbook). The book is a comprehensive guide to statistics […] Read More…

Also posted in A/B Testing, Conversion Optimization | Tagged , , , , | Leave a comment

A/B Testing with a Small Sample Size

AB Testing Small Business

The question “How to test if my website has a small number of users” comes up frequently when I chat to people about statistics in A/B testing, online and offline alike. There are different opinions on the topic ranging from altering the significance threshold, statistical power or the minimum effect of interest all the way […] Read More…

Also posted in A/B Testing, Conversion Optimization | Tagged , , , , , , | Comments closed