Articles on alpha spending

The search for the best statistical model for A/B testing

The best statistical model for A/B testing

It’s been almost ten years since I published the white paper “Efficient A/B Testing in Conversion Rate Optimization: The AGILE Statistical Method” (2017). It argues that Group-Sequential Tests (GSTs) are ideally-suited for online A/B testing. I figure it is a good time to reflect back on my search for the best statistical model and see […] Read more…

Posted in A/B testing, AGILE A/B testing, Statistics | Also tagged , , , ,

Error Spending in Sequential Testing Explained

Sequential Hypothesis Testing with Efficacy and Futility Bound

Sequential analysis of experimental data from A/B tests has been quite prominent in recent years due to the myriad of Bayesian solutions offered by big industry players. However, this type of sequential analysis is not sequential testing proper as these solutions have generally abandoned the idea of testing and therefore error control, substituting it for […] Read more…

Posted in A/B testing, AGILE A/B testing, Statistics | Also tagged , , , , ,