Category Archives: A/B Testing

The Importance of Statistical Power in Online A/B Testing

Statistical Power and Test Sensitivity

What is Statistical Power? In null-hypothesis statistical testing (NHST) – the procedure most commonly applied in A/B tests, there are two types of errors that practitioners should care about, type I and type II errors. Type I is the probability of the test procedure to falsely reject a true null hypothesis. Type II error is […] Read More…

Also posted in AGILE A/B Testing, Statistical Significance, Statistics | Tagged , , , , , | Leave a comment

5 Reasons to Go Bayesian in AB Testing – Debunked

Frequentist vs Bayesian A/B Testing

As someone who spent a good deal of time on trying to figure out how to run A/B tests properly and efficiently, I was intrigued to find a slide from a presentation by VWO®’s data scientist Chris Stucchio, where he goes over the main reasons that caused him and the VWO® team to consider and […] Read More…

Also posted in Conversion Optimization, Statistical Significance, Statistics | Tagged , , , , , , | Leave a comment

The Bane of AB Testing: Reaching Statistical Significance

Illusory Results AB Testing

Now, this is not a new topic on this blog. I’ve discussed the issue of optional stopping based on “achieving” or “reaching” statistical significance in my Why Every Internet Marketer Should be a Statistician post more than two years ago and others have touched on it as well – both before and after me. However, the issue […] Read More…

Also posted in Conversion Optimization, Statistical Significance, Statistics | Tagged , , , , , , , , , , | Leave a comment

Bayesian AB Testing is Not Immune to Optional Stopping Issues

Fantasy vs real world in bayesian ab testing

Fantasy vs the Real World: Naive Bayesian AB Testing vs Proper Statistical Inference This post is addressed at a certain camp of proponents and practitioners of A/B testing based on Bayesian statistical methods who claim that outcome-based optional stopping, often called data peeking or data-driven stopping, has no effect on the statistics and thus inferences […] Read More…

Also posted in Conversion Optimization, Statistical Significance, Statistics | Tagged , , , , , , , | Leave a comment

Using Content Groupings for User Flow Analysis in Google Analytics

Order Chaos Content Groupings

In the first part of this article we talked about how to set up content groupings in Google Analytics and how to make sure it’s working as expected. In this part we’ll discuss the 4 main benefits of having content groupings defined. If I have to sum it up in one word, I would go with […] Read More…

Also posted in Google Analytics | Tagged , , , , | Leave a comment

Setting up Content Groupings in Google Analytics

Planning

It’s been 2 years since content groupings first made it into Google Analytics, but it seems like this incredibly beneficial feature is still quite underused and poorly understood even among people otherwise familiar with the software. That’s why I decided to write a two-part guide of which you’re reading the first one – how to setup […] Read More…

Also posted in Google Analytics | Tagged , , , , | 2 Responses