Articles on AGILE A/B testing

Posts on the topic of a/b testing using the AGILE statistical method.

Sequential Testing is About Improving Business Returns

Sequential Testing Efficiency

A central feature of sequential testing is the idea of stopping “early”, as in “earlier compared to an equivalent fixed-sample size test”. This allows running A/B tests with fewer users and in a shorter amount of time while adhering to the targeted error guarantees. For example, a test may be planned with a maximum duration […] Read more…

Also posted in A/B testing | Tagged , , , ,

Comparison of the statistical power of sequential tests: SPRT, AGILE, and Always Valid Inference

Power and Average Sample Size of Sequential Tests

In A/B testing sequential tests are gradually becoming the norm due to the increased efficiency and flexibility that they grant practitioners. In most practical scenarios sequential tests offer a balance of risks and rewards superior to that of an equivalent fixed sample test. Sequential monitoring achieves this superiority by trading statistical power for the ability […] Read more…

Also posted in A/B testing, Statistics | Tagged , , , , , ,

What Can Be Learned From 1,001 A/B Tests?

Meta Analysis

How long does a typical A/B test run for? What percentage of A/B tests result in a ‘winner’? What is the average lift achieved in online controlled experiments? How good are top conversion rate optimization specialists at coming up with impactful interventions for websites and mobile apps? This meta-analysis of 1,001 A/B tests analyzed using […] Read more…

Also posted in A/B testing, Conversion optimization | Tagged , , , , , , , , ,

Fully Sequential vs Group Sequential Tests

Sequential Testing Compared

What is the best design for a statistical test with sequential evaluation of the data at multiple points in time? This is a question anyone who has realized that unaccounted for peeking with intent to stop is the bane of A/B testing eventually comes to ask. So how does one go about answering that? This […] Read more…

Also posted in Statistics | Tagged , , , , ,

The new standard for planning and analyzing A/B tests is here

A/B testing statistics done right

The first major overhaul of Analytics Toolkit since its release in early 2014 has finally arrived and it brings with it solutions to many of the hard questions facing practitioners when planning and analyzing A/B tests. Conducting statistically rigorous tests while achieving the best return on investment go hand in hand in the new Toolkit. […] Read more…

Also posted in A/B testing, Analytics-Toolkit.com, Conversion optimization, Statistics | Tagged , , , ,

Error Spending in Sequential Testing Explained

Sequential Hypothesis Testing with Efficacy and Futility Bound

Sequential analysis of experimental data from A/B tests has been quite prominent in recent years due to the myriad of Bayesian solutions offered by big industry players. However, this type of sequential analysis is not sequential testing proper as these solutions have generally abandoned the idea of testing and therefore error control, substituting it for […] Read more…

Also posted in A/B testing, Statistics | Tagged , , , , , ,

AGILE A/B Testing Update: Custom API & Support for Non-Binomial Data

AGILE A/B Testing Tool Updates

I’m happy to announce the release of two long-awaited features for our A/B Testing Calculator: Support for non-binomial metrics like average revenue per user A new custom API for sending experiment data to the calculator Below is an explanation of each of these new features in some detail. Support for Non-Binomial Data While our more […] Read more…

Also posted in A/B testing, Analytics-Toolkit.com | Tagged , , , , , , ,

Affordable A/B Tests: Google Optimize & AGILE A/B Testing

The problem most-often faced by owners of websites who want to take a scientific approach to improving them by using A/B testing is that they might have relatively small revenue. Thus, when the ROI calculation for the A/B test is done it might turn out that it is economically unfeasible to test. In some cases, […] Read more…

Also posted in A/B testing, Analytics-Toolkit.com, Conversion optimization | Tagged , , , , , ,

20-80% Faster A/B Tests? Is it real?

Percent Runs and Stopping Stage 1Delta

I got a question today about our AGILE A/B testing calculator and the statistics behind it and realized that I’m yet to write a dedicated post explaining the efficiency gains from using the method in more detail. This despite the fact that these speed gains are clearly communicated and verified through simulation results presented in our AGILE […] Read more…

Also posted in A/B testing, Statistics | Tagged , , , , , , , ,

Multivariate Testing – Best Practices & Tools for MVT (A/B/n) Tests

Multivariate AB Tests / MVT Testing

Let’s get this out of the way from the very beginning: most “A/B tests” are in fact multivariate (MVT) tests, a.k.a. A/B/n tests. That is, most of the time when you read about “A/B testing” the term also encompasses multivariate testing. The only reason to specifically differentiate between A/B and MVT is when someone wants to […] Read more…

Also posted in A/B testing, Multiple variations testing, Statistical significance, Statistics | Tagged , , , , ,

Futility Stopping Rules in AGILE A/B Testing

Stopping for Lack of Effect (Futility)

In this article we continue our examination of the AGILE statistical approach to AB testing with a more in-depth look into futility stopping, or stopping early for lack of positive effect (lack of superiority). We’ll cover why such rules are helpful and how they help boost the ROI of A/B testing, why a rigorous statistical rule […] Read more…

Also posted in A/B testing, Conversion optimization, Statistics | Tagged , , , , , ,

Efficient AB Testing with the AGILE Statistical Method

AGILE AB Testing

Don’t we all want to run tests as quickly as possible, reaching results as conclusive and as certain as possible? Don’t we all want to minimize the number of users we send to an inferior variant and to implement a variant with positive lift as quickly as possible? Don’t we all want to get rid of […] Read more…

Also posted in A/B testing, Conversion optimization, Statistics | Tagged , , , , , , , , , , , ,