Category Archives: Conversion Optimization

Statistical Significance for Non-Binomial Metrics – Revenue per User, AOV, etc.

Non-Binomial Significance - Revenue, Per User Metrics

In this article I cover the method required to calculate statistical significance for non-binomial metrics such as average revenue per user, average order value, average sessions per user, average session duration, average pages per session, and others. The focus is on A/B testing in the context of conversion rate optimization, landing page optimization and e-mail […] Read More…

Also posted in A/B Testing, Statistical Significance, Statistics | Tagged , , , , , , , , | Comments closed

One-tailed vs Two-tailed Tests of Significance in A/B Testing

Two-tailed vs one-tailed test

The question of whether one should run A/B tests (a.k.a online controlled experiments) using one-tailed versus two-tailed tests of significance was something I didn’t even consider important, as I thought the answer (one-tailed) was so self-evident that no discussion was necessary. However, while preparing for my course on “Statistics in A/B Testing” for the ConversionXL […] Read More…

Also posted in A/B Testing, Statistical Significance, Statistics | Tagged , , , , , , , , | Comments closed

The Case for Non-Inferiority A/B Tests

The Case for Non-Inferiority Testing

In this article, I explore the concept of non-inferiority A/B tests and contrast it to the broadly accepted practice of running superiority tests. I explain where non-inferiority tests are necessary and how a CRO/LPO/UX testing specialist can make use of this new approach to A/B testing to run much faster tests, and to ultimately achieve […] Read More…

Also posted in A/B Testing, Statistical Significance, Statistics | Tagged , , , , , | Comments closed

Statistical Significance in A/B Testing – a Complete Guide

Statistical Significance P Value

The concept of statistical significance is central to planning, executing and evaluating A/B (and multivariate) tests, but at the same time it is the most misunderstood and misused statistical tool in internet marketing, conversion optimization, landing page optimization, and user testing. This is not my first take on the topic, but it is my best […] Read More…

Also posted in A/B Testing, Multiple Variations Testing, Statistical Significance, Statistics | Tagged , , , , , , , , , | Comments closed

Futility Stopping Rules in AGILE A/B Testing

Stopping for Lack of Effect (Futility)

In this article we continue our examination of the AGILE statistical approach to AB testing with a more in-depth look into futility stopping, or stopping early for lack of positive effect (lack of superiority). We’ll cover why such rules are helpful and how they help boost the ROI of A/B testing, why a rigorous statistical rule […] Read More…

Also posted in A/B Testing, AGILE A/B Testing, Statistics | Tagged , , , , , , | Comments closed

Efficient AB Testing with the AGILE Statistical Method

AGILE AB Testing

Don’t we all want to run tests as quickly as possible, reaching results as conclusive and as certain as possible? Don’t we all want to minimize the number of users we send to an inferior variant and to implement a variant with positive lift as quickly as possible? Don’t we all want to get rid of […] Read More…

Also posted in A/B Testing, AGILE A/B Testing, Statistics | Tagged , , , , , , , , , , , , | Comments closed