Articles on Conversion optimization

Articles on conversion rate optimization a.k.a. CRO with primary focus on e-commerce and online businesses and e-mail marketing optimization.

What Can Be Learned From 1,001 A/B Tests?

Meta Analysis

How long does a typical A/B test run for? What percentage of A/B tests result in a ‘winner’? What is the average lift achieved in online controlled experiments? How good are top conversion rate optimization specialists at coming up with impactful interventions for websites and mobile apps? This meta-analysis of 1,001 A/B tests analyzed using […] Read more…

Also posted in A/B testing, AGILE A/B testing | Tagged , , , , , , , , ,

When Session-Based Metrics Lie

Per Session Metrics In AB Testing

In online A/B testing it is not uncommon to see session-based metrics being used as the primary performance indicator. Session-based conversion rates and session-based averages (like average revenue per session, in likeness to ARPU) are often reported by default in software by prominent vendors, including Google Optimize and Google Analytics. This widespread availability makes session-based […] Read more…

Also posted in A/B testing | Tagged , , ,

Top Misconceptions About Scientific Rigor in A/B Testing

Have you ever thought that statistically rigorous A/B tests are impractical? Or do you have trouble selling the need for rigor in testing to your clients, coworkers, or boss? This article debunks the top five myths about the necessity and difficulties of applying scientific method in online A/B testing. Read more…

Also posted in A/B testing | Tagged , , , , , ,

The new standard for planning and analyzing A/B tests is here

A/B testing statistics done right

The first major overhaul of Analytics Toolkit since its release in early 2014 has finally arrived and it brings with it solutions to many of the hard questions facing practitioners when planning and analyzing A/B tests. Conducting statistically rigorous tests while achieving the best return on investment go hand in hand in the new Toolkit. […] Read more…

Also posted in A/B testing, AGILE A/B testing, Analytics-Toolkit.com, Statistics | Tagged , , , ,

Bayesian Probability and Nonsensical Bayesian Statistics in A/B Testing

Bayesian Probability and Bayesian Statistics in AB Testing

Many adherents of Bayesian methods put forth claims of superiority of Bayesian statistics and inference over the established frequentist approach based mainly on the supposedly intuitive nature of the Bayesian approach. Rational thinking or even human reasoning in general is Bayesian by nature according to some of them. Others argue that proper decision-making is inherently […] Read more…

Also posted in A/B testing, Bayesian A/B testing, Statistics | Tagged , , , , ,

The Perils of Poor Data Visualization in CRO & A/B Testing

As any UX & CRO expert should now, the way we present information matters a lot both in terms of how well it is understood and in terms of the probability that it will lead to the desired action. A/B testing calculators and other tools of the trade are no exception and here I will […] Read more…

Also posted in A/B testing, Statistics | Tagged , , , , , , ,

What Can You Learn From Running an A/B Test for 2 Years?

AB Test Case Study

We just concluded an A/B test on Analytics-Toolkit.com that has been left to run for just over 2 years. And it failed, as in failing to demonstrate a statistically significant effect based on the significance threshold it was designed for. Has it been a waste of time, though, or can we actually learn something from […] Read more…

Also posted in A/B testing | Tagged , , , ,

Statistical Methods in Online A/B Testing – the book

Book Cover Statistical Methods Online A/B Testing

The long wait is finally over! “Statistical Methods in Online A/B Testing” can now be found as a paperback and an e-book on your preferred Amazon store. The book is a comprehensive guide to statistics in online controlled experiments, a.k.a. A/B tests, and tackles the difficult matter of statistical inference in a way accessible to […] Read more…

Also posted in A/B testing, Statistics | Tagged , , , ,

A/B Testing with a Small Sample Size

AB Testing Small Business

The question “How to test if my website has a small number of users” comes up frequently when I chat to people about statistics in A/B testing, online and offline alike. There are different opinions on the topic ranging from altering the significance threshold, statistical power or the minimum effect of interest all the way […] Read more…

Also posted in A/B testing, Statistics | Tagged , , , , , ,

Online glossary of A/B testing terms and abbreviations

AB testing glossary

We are happy to present a brand new addition to our website: a comprehensive A/B testing glossary containing terms and abbreviations used testing as part of conversion rate optimization (CRO).  Definitions start from very basic things such as “A/B test“, “mean“, “conversion rate” and “revenue per user“, go through “hypothesis“, “null hypothesis“, “standard deviation“, “p-value” […] Read more…

Also posted in A/B testing, Statistics | Tagged , , ,

The A/B Testing Guide to Surviving on a Deserted Island

AB Testing Survival Guide Desert Island

The secluded and isolated deserted island setting has been used as the stage for many hypothetical explanations in economics and philosophy with the scarcity of things that can be developed as resources being a central feature. Scarcity and the need to keep risk low while aiming to improve one’s situation is what make it a […] Read more…

Also posted in A/B testing, Statistics | Tagged , , , , , , , , , , , , , ,

Inherent costs of A/B testing: limited risk results in limited gains

Costs

I’ve already done a detailed breakdown of costs & benefits in A/B testing as well as the risks and rewards and how A/B testing is essentially a risk management solution. In this short installment I’d like to focus on the trade-off between limiting the downside and restricting the upside which is present in all risk management […] Read more…

Also posted in A/B testing | Tagged , , , ,