The A/B Testing Guide to Surviving on a Deserted Island

AB Testing Survival Guide Desert Island

The secluded and isolated deserted island setting has been used as the stage for many hypothetical explanations in economics and philosophy with the scarcity of things that can be developed as resources being a central feature. Scarcity and the need to keep risk low while aiming to improve one’s situation is what make it a […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistics | Tagged , , , , , , , , , , , , , , | Leave a comment

Inherent costs of A/B testing: limited risk results in limited gains

Costs

I’ve already done a detailed breakdown of costs & benefits in A/B testing as well as the risks and rewards and how A/B testing is essentially a risk management solution. In this short installment I’d like to focus on the trade-off between limiting the downside and restricting the upside which is present in all risk management […] Read More…

Posted in A/B Testing, Conversion Optimization | Tagged , , , , | Leave a comment

Representative samples and generalizability of A/B testing results

Representative samples and generalizability of AB testing results

I see a nice trend in recent discussions on A/B testing: more and more people realize the need for proper statistical design and analysis which is a topic I hold dear as I’ve written dozens of articles and a few white-papers on. However, there are cases in which statistical validity is discussed without consideration for […] Read More…

Posted in A/B Testing, Conversion Optimization | Tagged , , , , , , | Leave a comment

Designing successful A/B tests in Email Marketing

The process of A/B testing (a.k.a. online controlled experiments) is well-established in conversion rate optimization for all kinds of online properties and is widely used by e-commerce websites. On this blog I have already written in depth about the statistics involved as well as the ROI calculations in terms of balancing risk and reward for […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistical Significance, Statistics | Tagged , , , , , , , , , , | Leave a comment

Analysis of 115 A/B Tests: Average Lift is 4%, Most Lack Statistical Power

Observed Percent Change Significant

What can you learn from 115 publicly available A/B tests? Usually, not much, since in most cases you would be looking at case studies with very basic data about what was tested and the outcome of the A/B test. Confidence intervals, p-values and other measurements of uncertainty will often be missing, and when present they […] Read More…

Posted in A/B Testing, Conversion Optimization | Tagged , , , , , , , , , , , | 2 Responses

Confidence Intervals & P-values for Percent Change / Relative Difference

In many controlled experiments, including online controlled experiments (a.k.a. A/B tests) the result of interest and hence the inference made is about the relative difference between the control and treatment group. In A/B testing as part of conversion rate optimization and in marketing experiments in general we use the term “percent lift” (“percentage lift”) while in […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistical Significance, Statistics | Tagged , , , , , , , , , | Leave a comment