Tag Archives: peeking

Frequentist vs Bayesian Inference

Frequentist vs Bayesian Inference

In this article I’m revisiting* the topic of frequentist vs Bayesian inference with specific focus on online A/B testing as usual. The present discussion easily generalizes to any area where we need to measure uncertainty while using data to guide decision-making and/or business risk management. In particular, I will discuss each of the following five […] Read More…

Posted in A/B Testing, Statistics | Also tagged , , , , , , | Leave a comment

Analysis of 115 A/B Tests: Average Lift is 4%, Most Lack Statistical Power

Observed Percent Change Significant

What can you learn from 115 publicly available A/B tests? Usually, not much, since in most cases you would be looking at case studies with very basic data about what was tested and the outcome of the A/B test. Confidence intervals, p-values and other measurements of uncertainty will often be missing, and when present they […] Read More…

Posted in A/B Testing, Conversion Optimization | Also tagged , , , , , , , , , , | Comments closed