Articles from 2022

When Session-Based Metrics Lie

Per Session Metrics In AB Testing

In online A/B testing it is not uncommon to see session-based metrics being used as the primary performance indicator. Session-based conversion rates and session-based averages (like average revenue per session, in likeness to ARPU) are often reported by default in software by prominent vendors, including Google Optimize and Google Analytics. This widespread availability makes session-based […] Read more…

Posted in A/B testing, Conversion optimization | Tagged , , ,

A/B Testing Statistics – A Concise Guide for Non-Statisticians

AB Testing Statistics

Navigating the maze of A/B testing statistics can be challenging. This is especially true for those new to statistics and probability. One reason is the obscure terminology popping up in every other sentence. Another is that the writings can be vague, conflicting, incomplete, or simply wrong, depending on the source. Articles sprinkled with advanced math, […] Read more…

Posted in A/B testing, Statistics | Tagged , , , , ,

P-values and Confidence Intervals Explained

P-values and confidence intervals explained

Hundreds if not thousands of books have been written about both p-values and confidence intervals (CIs) – two of the most widely used statistics in online controlled experiments. Yet, these concepts remain elusive to many otherwise well-trained researchers, including A/B testing practitioners. Misconceptions and misinterpretations abound despite great efforts from statistics educators and experimentation evangelists. […] Read more…

Posted in A/B testing, Statistical significance, Statistics | Tagged , , , , , ,

Top Misconceptions About Scientific Rigor in A/B Testing

Have you ever thought that statistically rigorous A/B tests are impractical? Or do you have trouble selling the need for rigor in testing to your clients, coworkers, or boss? This article debunks the top five myths about the necessity and difficulties of applying scientific method in online A/B testing. Read more…

Posted in A/B testing, Conversion optimization | Tagged , , , , , ,

Improve your A/B tests with 9 lessons from the COVID-19 vaccine trials

Lessons for Online AB Testing from the COVID Vaccine Trials

A full year has passed since results from the first clinical trials of COVID-19 vaccines became available. Vast swaths of follow-up observational data are now in circulation which enables some conclusions about the strengths and weaknesses of those trials. Since modern medical trials are among the most rigorous and scrutinized experiments they can serve as […] Read more…

Posted in A/B testing, Statistics | Tagged , , , , ,