Articles on conversion rate optimization

What Can Be Learned From 1,001 A/B Tests?

Meta Analysis

How long does a typical A/B test run for? What percentage of A/B tests result in a ‘winner’? What is the average lift achieved in online controlled experiments? How good are top conversion rate optimization specialists at coming up with impactful interventions for websites and mobile apps? This meta-analysis of 1,001 A/B tests analyzed using […] Read more…

Posted in A/B testing, AGILE A/B testing, Conversion optimization | Also tagged , , , , , , , ,

Fully Sequential vs Group Sequential Tests

Sequential Testing Compared

What is the best design for a statistical test with sequential evaluation of the data at multiple points in time? This is a question anyone who has realized that unaccounted for peeking with intent to stop is the bane of A/B testing eventually comes to ask. So how does one go about answering that? This […] Read more…

Posted in AGILE A/B testing, Statistics | Also tagged , , , ,

When Session-Based Metrics Lie

Per Session Metrics In AB Testing

In online A/B testing it is not uncommon to see session-based metrics being used as the primary performance indicator. Session-based conversion rates and session-based averages (like average revenue per session, in likeness to ARPU) are often reported by default in software by prominent vendors, including Google Optimize and Google Analytics. This widespread availability makes session-based […] Read more…

Posted in A/B testing, Conversion optimization | Also tagged , ,

Top Misconceptions About Scientific Rigor in A/B Testing

Have you ever thought that statistically rigorous A/B tests are impractical? Or do you have trouble selling the need for rigor in testing to your clients, coworkers, or boss? This article debunks the top five myths about the necessity and difficulties of applying scientific method in online A/B testing. Read more…

Posted in A/B testing, Conversion optimization | Also tagged , , , , ,

The A/B Testing Guide to Surviving on a Deserted Island

AB Testing Survival Guide Desert Island

The secluded and isolated deserted island setting has been used as the stage for many hypothetical explanations in economics and philosophy with the scarcity of things that can be developed as resources being a central feature. Scarcity and the need to keep risk low while aiming to improve one’s situation is what make it a […] Read more…

Posted in A/B testing, Conversion optimization, Statistics | Also tagged , , , , , , , , , , , , ,

The Google Optimize Statistical Engine and Approach

Frequentist vs Bayesian A/B testing - Google Optimize

Updated Sep 17, 2018: Minor spelling and language corrections, updates related to role of randomization and external validity / generalizability. Google Optimize is the latest attempt from Google to deliver an A/B testing product. Previously we had “Google Website Optimizer”, then we had “Content Experiments” within Google Analytics, and now we have the latest iteration: […] Read more…

Posted in A/B testing, Conversion optimization, Statistics | Also tagged , , , , , , , , , , ,

Costs and Benefits of A/B Testing: A Comprehensive Guide

Costs and Benefits in AB Testing

This is a comprehensive guide to the different types of costs and benefits, risks and rewards related to A/B testing. Understanding them in detail should be valuable to A/B testers and businesses considering whether to engage in A/B testing or not, what to A/B test and what not to test, etc. As far as I […] Read more…

Posted in A/B testing, Conversion optimization | Also tagged , , , , , ,

Efficient AB Testing with the AGILE Statistical Method

AGILE AB Testing

Don’t we all want to run tests as quickly as possible, reaching results as conclusive and as certain as possible? Don’t we all want to minimize the number of users we send to an inferior variant and to implement a variant with positive lift as quickly as possible? Don’t we all want to get rid of […] Read more…

Posted in A/B testing, AGILE A/B testing, Conversion optimization, Statistics | Also tagged , , , , , , , , , , ,

Improving ROI in A/B Testing: the AGILE AB Testing Approach

After many months of statistical research and development we are happy to announce two major releases that we believe have the potential to reshape statistical practice in the area of A/B testing by substantially increasing the accuracy, efficiency and ultimately return on investment of all kinds of A/B testing efforts in online marketing: a free white […] Read more…

Posted in A/B testing, AGILE A/B testing, Conversion optimization, Statistics | Also tagged , , , , , , , , ,