Author Archives: Georgi Georgiev

Statistical Significance in A/B Testing – a Complete Guide

Statistical Significance P Value

The concept of statistical significance is central to planning, executing and evaluating A/B (and multivariate) tests, but at the same time it is the most misunderstood and misused statistical tool in internet marketing, conversion optimization, landing page optimization, and user testing. This is not my first take on the topic, but it is my best […] Read More…

Posted in A/B Testing, Conversion Optimization, Multiple Variations Testing, Statistical Significance, Statistics | Tagged , , , , , , , , , | 4 Responses

Multivariate Testing – Best Practices & Tools for MVT (A/B/n) Tests

Multivariate AB Tests / MVT Testing

Let’s get this out of the way from the very beginning: most “A/B tests” are in fact multivariate (MVT) tests, a.k.a. A/B/n tests. That is, most of the time when you read about “A/B testing” the term also encompasses multivariate testing. The only reason to specifically differentiate between A/B and MVT is when someone wants to […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Multiple Variations Testing, Statistical Significance, Statistics | Tagged , , , , , | Leave a comment

Running Multiple A/B Tests at The Same Time: Do’s and Don’ts

Concurrent AB Tests

Can running multiple A/B tests at the same time lead to interferences that result in choosing inferior variants? Does running each A/B test in a silo improve or worsen the situation? If there is any danger, how great is it and how much should we be concerned about it? In this post, I’ll try to answer […] Read More…

Posted in A/B Testing | Tagged , , , | Leave a comment

Top 10 ways to ruin your Google Analytics data and how to avoid them

No Pulse - No Data

A lot of businesses rely on Google Analytics to assess the performance of their online efforts with regards to online sales, marketing, support, or just providing users with information about a brand or a product. Any measurements and conclusions based on them, however, are only as good as the accuracy and reliability of the data […] Read More…

Posted in Google Analytics | Tagged , , , , , , | 2 Responses

Futility Stopping Rules in AGILE A/B Testing

Stopping for Lack of Effect (Futility)

In this article we continue our examination of the AGILE statistical approach to AB testing with a more in-depth look into futility stopping, or stopping early for lack of positive effect (lack of superiority). We’ll cover why such rules are helpful and how they help boost the ROI of A/B testing, why a rigorous statistical rule […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Conversion Optimization, Statistics | Tagged , , , , , , | Leave a comment

Efficient AB Testing with the AGILE Statistical Method

AGILE AB Testing

Don’t we all want to run tests as quickly as possible, reaching results as conclusive and as certain as possible? Don’t we all want to minimize the number of users we send to an inferior variant and to implement a variant with positive lift as quickly as possible? Don’t we all want to get rid of […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Conversion Optimization, Statistics | Tagged , , , , , , , , , , , , | 6 Responses