Does Your A/B Test Pass the Sample Ratio Mismatch Check?

Sample Ratio Mismatch AB Testing

Most, if not all successful online businesses nowadays rely on one or more systems for conducting A/B tests in order to inform business decisions ranging from simple website or advertising campaign interventions to complex product and business model changes. While testing might have become a prerequisite for releasing the tiniest of changes, one type of […] Read More…

Posted in A/B Testing, Statistics | Tagged , , , , , | Leave a comment

AGILE A/B Testing Update: Custom API & Support for Non-Binomial Data

AGILE A/B Testing Tool Updates

I’m happy to announce the release of two long-awaited features for our AGILE A/B Testing Calculator: Support for non-binomial metrics like average revenue per user A new custom API for sending experiment data to the calculator Below is an explanation of each of these new features in some detail. Support for Non-Binomial Data While our […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Analytics-Toolkit.com | Tagged , , , , , , , | Leave a comment

Statistical Methods in Online A/B Testing – the book

Book Cover Statistical Methods Online A/B Testing

The long wait is finally over! “Statistical Methods in Online A/B Testing” can now be found as a paperback and an e-book on your preferred Amazon store. Note that the Kindle edition is available for $2.99 or equivalent if you’ve already purchased the paperback (through Kindle Matchbook). The book is a comprehensive guide to statistics […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistics | Tagged , , , , | Leave a comment

A/B Testing with a Small Sample Size

AB Testing Small Business

The question “How to test if my website has a small number of users” comes up frequently when I chat to people about statistics in A/B testing, online and offline alike. There are different opinions on the topic ranging from altering the significance threshold, statistical power or the minimum effect of interest all the way […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistics | Tagged , , , , , , | Comments closed

Online glossary of A/B testing terms and abbreviations

AB testing glossary

We are happy to present a brand new addition to our website: a comprehensive A/B testing glossary containing terms and abbreviations used testing as part of conversion rate optimization (CRO).  Definitions start from very basic things such as “A/B test“, “mean“, “conversion rate” and “revenue per user“, go through “hypothesis“, “null hypothesis“, “standard deviation“, “p-value” […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistics | Tagged , , , | Comments closed

The A/B Testing Guide to Surviving on a Deserted Island

AB Testing Survival Guide Desert Island

The secluded and isolated deserted island setting has been used as the stage for many hypothetical explanations in economics and philosophy with the scarcity of things that can be developed as resources being a central feature. Scarcity and the need to keep risk low while aiming to improve one’s situation is what make it a […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistics | Tagged , , , , , , , , , , , , , , | Comments closed