Tag Archives: agile ab testing

20-80% Faster A/B Tests? Is it real?

Percent Runs and Stopping Stage 1Delta

I got a question today about our AGILE A/B testing calculator and the statistics behind it and realized that I’m yet to write a dedicated post explaining the efficiency gains from using the method in more detail. This despite the fact that these speed gains are clearly communicated and verified through simulation results presented in our AGILE […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Statistics | Also tagged , , , , , , , | Leave a comment

Futility Stopping Rules in AGILE A/B Testing

Stopping for Lack of Effect (Futility)

In this article we continue our examination of the AGILE statistical approach to AB testing with a more in-depth look into futility stopping, or stopping early for lack of positive effect (lack of superiority). We’ll cover why such rules are helpful and how they help boost the ROI of A/B testing, why a rigorous statistical rule […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Conversion Optimization, Statistics | Also tagged , , , , , | Leave a comment

Efficient AB Testing with the AGILE Statistical Method

AGILE AB Testing

Don’t we all want to run tests as quickly as possible, reaching results as conclusive and as certain as possible? Don’t we all want to minimize the number of users we send to an inferior variant and to implement a variant with positive lift as quickly as possible? Don’t we all want to get rid of […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Conversion Optimization, Statistics | Also tagged , , , , , , , , , , , | 6 Responses

Improving ROI in A/B Testing: the AGILE AB Testing Approach

After many months of statistical research and development we are happy to announce two major releases that we believe have the potential to reshape statistical practice in the area of A/B testing by substantially increasing the accuracy, efficiency and ultimately return on investment of all kinds of A/B testing efforts in online marketing: a free white […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Conversion Optimization, Statistics | Also tagged , , , , , , , , , | Leave a comment

The Importance of Statistical Power in Online A/B Testing

Statistical Power and Test Sensitivity

What is Statistical Power? In null-hypothesis statistical testing (NHST) – the procedure most commonly applied in A/B tests, there are two types of errors that practitioners should care about, type I and type II errors. Type I is the probability of the test procedure to falsely reject a true null hypothesis. Type II error is […] Read More…

Posted in A/B Testing, AGILE A/B Testing, Conversion Optimization, Statistical Significance, Statistics | Also tagged , , , , | Leave a comment