Articles on ab testing

Representative samples and generalizability of A/B testing results

Representative samples and generalizability of AB testing results

I see a nice trend in recent discussions on A/B testing: more and more people realize the need for proper statistical design and analysis which is a topic I hold dear as I’ve written dozens of articles and a few white-papers on. However, there are cases in which statistical validity is discussed without consideration for […] Read more…

Posted in A/B testing, Conversion optimization | Also tagged , , , , ,

Analysis of 115 A/B Tests: Average Lift is 4%, Most Lack Statistical Power

Observed Percent Change Significant

Oct 2022 update: A newer, much larger and likely less biased meta analysis of 1,001 tests is now available! What can you learn from 115 publicly available A/B tests? Usually, not much, since in most cases you would be looking at case studies with very basic data about what was tested and the outcome of […] Read more…

Posted in A/B testing, Conversion optimization | Also tagged , , , , , , , , , ,

Affordable A/B Tests: Google Optimize & AGILE A/B Testing

The problem most-often faced by owners of websites who want to take a scientific approach to improving them by using A/B testing is that they might have relatively small revenue. Thus, when the ROI calculation for the A/B test is done it might turn out that it is economically unfeasible to test. In some cases, […] Read more…

Posted in A/B testing, AGILE A/B testing, Analytics-Toolkit.com, Conversion optimization | Also tagged , , , , ,

The Google Optimize Statistical Engine and Approach

Frequentist vs Bayesian A/B testing - Google Optimize

Updated Sep 17, 2018: Minor spelling and language corrections, updates related to role of randomization and external validity / generalizability. Google Optimize is the latest attempt from Google to deliver an A/B testing product. Previously we had “Google Website Optimizer”, then we had “Content Experiments” within Google Analytics, and now we have the latest iteration: […] Read more…

Posted in A/B testing, Bayesian A/B testing, Conversion optimization, Statistics | Also tagged , , , , , , , , , , ,

20-80% Faster A/B Tests? Is it real?

Percent Runs and Stopping Stage 1Delta

I got a question today about our AGILE A/B testing calculator and the statistics behind it and realized that I’m yet to write a dedicated post explaining the efficiency gains from using the method in more detail. This despite the fact that these speed gains are clearly communicated and verified through simulation results presented in our AGILE […] Read more…

Posted in A/B testing, AGILE A/B testing, Statistics | Also tagged , , , , , , ,

Risk vs. Reward in A/B Tests: A/B testing as Risk Management

Risks vs Rewards in AB Testing

What is the goal of A/B testing? How long should I run a test for? Is it better to run many quick tests, or one long one? How do I know when is a good time to stop testing? How do I choose the significance threshold for a test? Is there something special about 95%? […] Read more…

Posted in A/B testing, Conversion optimization, Statistical significance, Statistics | Also tagged , , , , , ,

Costs and Benefits of A/B Testing: A Comprehensive Guide

Costs and Benefits in AB Testing

This is a comprehensive guide to the different types of costs and benefits, risks and rewards related to A/B testing. Understanding them in detail should be valuable to A/B testers and businesses considering whether to engage in A/B testing or not, what to A/B test and what not to test, etc. As far as I […] Read more…

Posted in A/B testing, Conversion optimization | Also tagged , , , , , ,

Statistical Significance for Non-Binomial Metrics – Revenue per User, AOV, etc.

Non-Binomial Significance - Revenue, Per User Metrics

In this article I cover the method required to calculate statistical significance for non-binomial metrics such as average revenue per user, average order value, average sessions per user, average session duration, average pages per session, and others. The focus is on A/B testing in the context of conversion rate optimization, landing page optimization and e-mail […] Read more…

Posted in A/B testing, Conversion optimization, Statistical significance, Statistics | Also tagged , , , , , , ,

The Case for Non-Inferiority A/B Tests

The Case for Non-Inferiority Testing

In this article, I explore the concept of non-inferiority A/B tests and contrast it to the broadly accepted practice of running superiority tests. I explain where non-inferiority tests are necessary and how a CRO/LPO/UX testing specialist can make use of this new approach to A/B testing to run much faster tests, and to ultimately achieve […] Read more…

Posted in A/B testing, Conversion optimization, Statistical significance, Statistics | Also tagged , , , ,

Multivariate Testing – Best Practices & Tools for MVT (A/B/n) Tests

Multivariate AB Tests / MVT Testing

Let’s get this out of the way from the very beginning: most “A/B tests” are in fact multivariate (MVT) tests, a.k.a. A/B/n tests. That is, most of the time when you read about “A/B testing” the term also encompasses multivariate testing. The only reason to specifically differentiate between A/B and MVT is when someone wants to […] Read more…

Posted in A/B testing, AGILE A/B testing, Multiple variations testing, Statistical significance, Statistics | Also tagged , , , ,

Running Multiple A/B Tests at The Same Time: Do’s and Don’ts

Concurrent AB Tests

Can running multiple A/B tests at the same time lead to interferences that result in choosing inferior variants? Does running each A/B test in a silo improve or worsen the situation? If there is any danger, how great is it and how much should we be concerned about it? In this post, I’ll try to answer […] Read more…

Posted in A/B testing | Also tagged , ,

Efficient AB Testing with the AGILE Statistical Method

AGILE AB Testing

Don’t we all want to run tests as quickly as possible, reaching results as conclusive and as certain as possible? Don’t we all want to minimize the number of users we send to an inferior variant and to implement a variant with positive lift as quickly as possible? Don’t we all want to get rid of […] Read more…

Posted in A/B testing, AGILE A/B testing, Conversion optimization, Statistics | Also tagged , , , , , , , , , , ,