Articles on ab testing

How to Run Shorter A/B Tests?

Shorter A/B Tests

Running shorter tests is key to improving the efficiency of experimentation as it translates to smaller direct losses from testing inferior experiences and also less unrealized revenue due to late implementation of superior ones. Despite this, many practitioners are yet to start conducting tests at the frontier of efficiency. This article presents ways to shorten […] Read more…

Posted in A/B testing, Statistics | Also tagged , , ,

Fully Sequential vs Group Sequential Tests

Sequential Testing Compared

What is the best design for a statistical test with sequential evaluation of the data at multiple points in time? This is a question anyone who has realized that unaccounted for peeking with intent to stop is the bane of A/B testing eventually comes to ask. So how does one go about answering that? This […] Read more…

Posted in AGILE A/B testing, Statistics | Also tagged , , , ,

When Session-Based Metrics Lie

Per Session Metrics In AB Testing

In online A/B testing it is not uncommon to see session-based metrics being used as the primary performance indicator. Session-based conversion rates and session-based averages (like average revenue per session, in likeness to ARPU) are often reported by default in software by prominent vendors, including Google Optimize and Google Analytics. This widespread availability makes session-based […] Read more…

Posted in A/B testing, Conversion optimization | Also tagged , ,

A/B Testing Statistics – A Concise Guide for Non-Statisticians

AB Testing Statistics

Navigating the maze of A/B testing statistics can be challenging. This is especially true for those new to statistics and probability. One reason is the obscure terminology popping up in every other sentence. Another is that the writings can be vague, conflicting, incomplete, or simply wrong, depending on the source. Articles sprinkled with advanced math, […] Read more…

Posted in A/B testing, Statistics | Also tagged , , , ,

P-values and Confidence Intervals Explained

P-values and confidence intervals explained

Hundreds if not thousands of books have been written about both p-values and confidence intervals (CIs) – two of the most widely used statistics in online controlled experiments. Yet, these concepts remain elusive to many otherwise well-trained researchers, including A/B testing practitioners. Misconceptions and misinterpretations abound despite great efforts from statistics educators and experimentation evangelists. […] Read more…

Posted in A/B testing, Statistical significance, Statistics | Also tagged , , , , ,

Top Misconceptions About Scientific Rigor in A/B Testing

Have you ever thought that statistically rigorous A/B tests are impractical? Or do you have trouble selling the need for rigor in testing to your clients, coworkers, or boss? This article debunks the top five myths about the necessity and difficulties of applying scientific method in online A/B testing. Read more…

Posted in A/B testing, Conversion optimization | Also tagged , , , , ,

The new standard for planning and analyzing A/B tests is here

A/B testing statistics done right

The first major overhaul of Analytics Toolkit since its release in early 2014 has finally arrived and it brings with it solutions to many of the hard questions facing practitioners when planning and analyzing A/B tests. Conducting statistically rigorous tests while achieving the best return on investment go hand in hand in the new Toolkit. […] Read more…

Posted in A/B testing, AGILE A/B testing, Analytics-Toolkit.com, Conversion optimization, Statistics | Also tagged , , ,

The Perils of Using Google Analytics User Counts in A/B Testing

Google Analytics User Data in AB Testing

(Updated Sep 2022 & Jan 2023 to include updated information about Google Analytics 4) Many analysts, marketers, product managers, UX and CRO professionals nowadays rely on user counts provided by Google Analytics, Adobe Analytics, or similar tools, in order to perform various statistical analyses. Such analyses may involve the statistical hypothesis tests and estimations part […] Read more…

Posted in A/B testing, Google Analytics, Statistics | Also tagged , , , , , , ,

AGILE A/B Testing Update: Custom API & Support for Non-Binomial Data

AGILE A/B Testing Tool Updates

I’m happy to announce the release of two long-awaited features for our A/B Testing Calculator: Support for non-binomial metrics like average revenue per user A new custom API for sending experiment data to the calculator Below is an explanation of each of these new features in some detail. Support for Non-Binomial Data While our more […] Read more…

Posted in A/B testing, AGILE A/B testing, Analytics-Toolkit.com | Also tagged , , , , , ,

Statistical Methods in Online A/B Testing – the book

Book Cover Statistical Methods Online A/B Testing

The long wait is finally over! “Statistical Methods in Online A/B Testing” can now be found as a paperback and an e-book on your preferred Amazon store. The book is a comprehensive guide to statistics in online controlled experiments, a.k.a. A/B tests, and tackles the difficult matter of statistical inference in a way accessible to […] Read more…

Posted in A/B testing, Conversion optimization, Statistics | Also tagged , , ,

Online glossary of A/B testing terms and abbreviations

AB testing glossary

We are happy to present a brand new addition to our website: a comprehensive A/B testing glossary containing terms and abbreviations used testing as part of conversion rate optimization (CRO).  Definitions start from very basic things such as “A/B test“, “mean“, “conversion rate” and “revenue per user“, go through “hypothesis“, “null hypothesis“, “standard deviation“, “p-value” […] Read more…

Posted in A/B testing, Conversion optimization, Statistics | Also tagged , ,

The A/B Testing Guide to Surviving on a Deserted Island

AB Testing Survival Guide Desert Island

The secluded and isolated deserted island setting has been used as the stage for many hypothetical explanations in economics and philosophy with the scarcity of things that can be developed as resources being a central feature. Scarcity and the need to keep risk low while aiming to improve one’s situation is what make it a […] Read more…

Posted in A/B testing, Conversion optimization, Statistics | Also tagged , , , , , , , , , , , , ,