Articles from 2023

Q&A on Sequential Statistics in A/B Testing

Sequential testing QA

Sequential statistics are gathering interest and there are more and more questions posed by CROs looking into the matter. For this article I teamed up with Lucia van den Brink, a distinguished CRO consultant who recently started using Analytics Toolkit and integrated frequentist sequential testing into her client workflow. In this short interview she asks […] Read more…

Posted in A/B testing | Tagged , ,

Sequential Testing is About Improving Business Returns

Sequential Testing Efficiency

A central feature of sequential testing is the idea of stopping “early”, as in “earlier compared to an equivalent fixed-sample size test”. This allows running A/B tests with fewer users and in a shorter amount of time while adhering to the targeted error guarantees. For example, a test may be planned with a maximum duration […] Read more…

Posted in A/B testing, AGILE A/B testing | Tagged , , , ,

False Positive Risk in A/B Testing

False positive risk in A/B testing

Have you heard how there is a much greater probability than generally expected that a statistically significant test outcome is in fact a false positive? In industry jargon: that a variant has been identified as a “winner” when it is not. In demonstrating the above the terms “False Positive Risk” (FPR), “False Findings Rate” (FFR), […] Read more…

Posted in A/B testing, Bayesian A/B testing, Statistics | Tagged , , , , , , , , ,

A lightweight Google Analytics 4 integration

Lightweight Google Analytics 4

Google Analytics 4 has been a let down in many aspects based on every discussion I’ve seen and had with professionals of all stripes – marketers, advertising specialists, CROs, GA professionals, online experimentation experts, etc. One of the less discussed issues it brought with it is the default heavyweight GTAG library integration it comes with. […] Read more…

Posted in Google Analytics | Tagged , , , ,

Analytics Toolkit to discontinue Google Analytics-related functionalities

Discontinuing Google Analytics Functionalities

Analytics Toolkit was conceived in 2012 as a set of tools that automate essential Google Analytics-related tasks and augment the GA functionalities in various ways. This goal was achieved in the years since with the release of over a dozen tools utilizing the Google Analytics API. These were accompanied by dozens of in-depth technical articles […] Read more…

Posted in A/B testing,, Google Analytics | Tagged , , , ,