Author Archives: Georgi Georgiev

20-80% Faster A/B Tests? Is it real?

Percent Runs and Stopping Stage 1Delta

I got a question today about our AGILEĀ A/B testing calculatorĀ and the statistics behind it and realized that I’m yet to write a dedicated post explaining the efficiency gains from using the method in more detail. This despite the fact that these speed gains are clearly communicated and verified through simulation results presented in our AGILE […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Statistics | Tagged , , , , , , , , | Leave a comment

Risk vs. Reward in A/B Tests: A/B testing as Risk Management

Risks vs Rewards in AB Testing

What is the goal of A/B testing? How long should I run a test for? Is it better to run many quick tests, or one long one? How do I know when is a good time to stop testing? How do I choose the significance threshold for a test? Is there something special about 95%? […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistical Significance, Statistics | Tagged , , , , , , , | Leave a comment

Costs and Benefits of A/B Testing: A Comprehensive Guide

Costs and Benefits in AB Testing

This is a comprehensive guide to the different types of costs and benefits, risks and rewards related to A/B testing. Understanding them in detail should be valuable to A/B testers and businesses considering whether to engage in A/B testing or not, what to A/B test and what not to test, etc. As far as I […] Read More…

Posted in A/B Testing, Conversion Optimization | Tagged , , , , , , , | Leave a comment

Statistical Significance for Non-Binomial Metrics – Revenue per User, AOV, etc.

Non-Binomial Significance - Revenue, Per User Metrics

In this article I cover the method required to calculate statistical significance for non-binomial metrics such as average revenue per user, average order value, average sessions per user, average session duration, average pages per session, and others. The focus is on A/B testing in the context of conversion rate optimization, landing page optimization and e-mail […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistical Significance, Statistics | Tagged , , , , , , , , | Leave a comment

One-tailed vs Two-tailed Tests of Significance in A/B Testing

Two-tailed vs one-tailed test

The question of whether one should run A/B tests (a.k.a online controlled experiments) using one-tailed versus two-tailed tests of significance was something I didn’t even consider important, as I thought the answer (one-tailed) was so self-evident that no discussion was necessary. However, while preparing for my course on “Statistics in A/B Testing” for the ConversionXL […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistical Significance, Statistics | Tagged , , , , , , , , | Leave a comment

The Case for Non-Inferiority A/B Tests

The Case for Non-Inferiority Testing

In this article, I explore the concept of non-inferiority A/B tests and contrast it to the broadly accepted practice of running superiority tests. I explain where non-inferiority tests are necessary and how a CRO/LPO/UX testing specialist can make use of this new approach to A/B testing to run much faster tests, and to ultimately achieve […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistical Significance, Statistics | Tagged , , , , , | Leave a comment