Articles from 2019

What Can You Learn From Running an A/B Test for 2 Years?

AB Test Case Study

We just concluded an A/B test on Analytics-Toolkit.com that has been left to run for just over 2 years. And it failed, as in failing to demonstrate a statistically significant effect based on the significance threshold it was designed for. Has it been a waste of time, though, or can we actually learn something from […] Read more…

Posted in A/B testing, Conversion optimization | Tagged , , , ,

Does Your A/B Test Pass the Sample Ratio Mismatch Check?

Sample Ratio Mismatch AB Testing

Most, if not all successful online businesses nowadays rely on one or more systems for conducting A/B tests in order to inform business decisions ranging from simple website or advertising campaign interventions to complex product and business model changes. While testing might have become a prerequisite for releasing the tiniest of changes, one type of […] Read more…

Posted in A/B testing, Statistics | Tagged , , , , ,

AGILE A/B Testing Update: Custom API & Support for Non-Binomial Data

AGILE A/B Testing Tool Updates

I’m happy to announce the release of two long-awaited features for our A/B Testing Calculator: Support for non-binomial metrics like average revenue per user A new custom API for sending experiment data to the calculator Below is an explanation of each of these new features in some detail. Support for Non-Binomial Data While our more […] Read more…

Posted in A/B testing, AGILE A/B testing, Analytics-Toolkit.com | Tagged , , , , , , ,

Statistical Methods in Online A/B Testing – the book

Book Cover Statistical Methods Online A/B Testing

The long wait is finally over! “Statistical Methods in Online A/B Testing” can now be found as a paperback and an e-book on your preferred Amazon store. The book is a comprehensive guide to statistics in online controlled experiments, a.k.a. A/B tests, and tackles the difficult matter of statistical inference in a way accessible to […] Read more…

Posted in A/B testing, Conversion optimization, Statistics | Tagged , , , ,

A/B Testing with a Small Sample Size

AB Testing Small Business

The question “How to test if my website has a small number of users” comes up frequently when I chat to people about statistics in A/B testing, online and offline alike. There are different opinions on the topic ranging from altering the significance threshold, statistical power or the minimum effect of interest all the way […] Read more…

Posted in A/B testing, Conversion optimization, Statistics | Tagged , , , , , ,