Statistical Methods in Online A/B Testing – the book

Book Cover Statistical Methods Online A/B Testing

The long wait is finally over! “Statistical Methods in Online A/B Testing” can now be found as a paperback and an e-book on your preferred Amazon store. Note that the Kindle edition is available for $2.99 or equivalent if you’ve already purchased the paperback (through Kindle Matchbook). The book is a comprehensive guide to statistics […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistics | Tagged , , , , | Leave a comment

A/B Testing with a Small Sample Size

AB Testing Small Business

The question “How to test if my website has a small number of users” comes up frequently when I chat to people about statistics in A/B testing, online and offline alike. There are different opinions on the topic ranging from altering the significance threshold, statistical power or the minimum effect of interest all the way […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistics | Tagged , , , , , , | 3 Responses

Online glossary of A/B testing terms and abbreviations

AB testing glossary

We are happy to present a brand new addition to our website: a comprehensive A/B testing glossary containing terms and abbreviations used testing as part of conversion rate optimization (CRO).  Definitions start from very basic things such as “A/B test“, “mean“, “conversion rate” and “revenue per user“, go through “hypothesis“, “null hypothesis“, “standard deviation“, “p-value” […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistics | Tagged , , , | Comments closed

The A/B Testing Guide to Surviving on a Deserted Island

AB Testing Survival Guide Desert Island

The secluded and isolated deserted island setting has been used as the stage for many hypothetical explanations in economics and philosophy with the scarcity of things that can be developed as resources being a central feature. Scarcity and the need to keep risk low while aiming to improve one’s situation is what make it a […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistics | Tagged , , , , , , , , , , , , , , | Comments closed

Inherent costs of A/B testing: limited risk results in limited gains

Costs

I’ve already done a detailed breakdown of costs & benefits in A/B testing as well as the risks and rewards and how A/B testing is essentially a risk management solution. In this short installment I’d like to focus on the trade-off between limiting the downside and restricting the upside which is present in all risk management […] Read More…

Posted in A/B Testing, Conversion Optimization | Tagged , , , , | Comments closed

Representative samples and generalizability of A/B testing results

Representative samples and generalizability of AB testing results

I see a nice trend in recent discussions on A/B testing: more and more people realize the need for proper statistical design and analysis which is a topic I hold dear as I’ve written dozens of articles and a few white-papers on. However, there are cases in which statistical validity is discussed without consideration for […] Read More…

Posted in A/B Testing, Conversion Optimization | Tagged , , , , , , | Comments closed