Can running multiple A/B tests at the same time lead to interferences that result in choosing inferior variants? Does running each A/B test in a silo improve or worsen the situation? If there is any danger, how great is it and how much should we be concerned about it? In this post, I’ll try to answer […] Read more…
A lot of businesses rely on Google Analytics to assess the performance of their online efforts with regards to online sales, marketing, support, or just providing users with information about a brand or a product. Any measurements and conclusions based on them, however, are only as good as the accuracy and reliability of the data […] Read more…
In this article we continue our examination of the AGILE statistical approach to AB testing with a more in-depth look into futility stopping, or stopping early for lack of positive effect (lack of superiority). We’ll cover why such rules are helpful and how they help boost the ROI of A/B testing, why a rigorous statistical rule […] Read more…
Don’t we all want to run tests as quickly as possible, reaching results as conclusive and as certain as possible? Don’t we all want to minimize the number of users we send to an inferior variant and to implement a variant with positive lift as quickly as possible? Don’t we all want to get rid of […] Read more…
After many months of statistical research and development we are happy to announce two major releases that we believe have the potential to reshape statistical practice in the area of A/B testing by substantially increasing the accuracy, efficiency and ultimately return on investment of all kinds of A/B testing efforts in online marketing: a free white […] Read more…
What is Statistical Power? In null-hypothesis statistical testing (NHST) – the procedure most commonly applied in A/B tests, there are two types of errors that practitioners should care about, type I and type II errors. Type I is the probability of the test procedure to falsely reject a true null hypothesis. Type II error is […] Read more…
As someone who spent a good deal of time on trying to figure out how to run A/B tests properly and efficiently, I was intrigued to find a slide from a presentation by VWO®’s data scientist Chris Stucchio, where he goes over the main reasons that caused him and the VWO® team to consider and […] Read more…
Now, this is not a new topic on this blog. I’ve discussed the issue of optional stopping based on “achieving” or “reaching” statistical significance in my Why Every Internet Marketer Should be a Statistician post more than two years ago and others have touched on it as well – both before and after me. However, the issue […] Read more…
Fantasy vs the Real World: Naive Bayesian AB Testing vs Proper Statistical Inference This post is addressed at a certain camp of proponents and practitioners of A/B testing based on Bayesian statistical methods, who claim that outcome-based optional stopping, often called data peeking or data-driven stopping, has no effect on the statistics and thus inferences […] Read more…
Some two years after it gained prominence Google Analytics spam is all the rage again with one or several notorious spammers finding their way into hundreds of thousands of Google Analytics accounts in order to bring attention to stuff the spammer or his/her associates are making money off. The new wave started around Nov 8, […] Read more…
In the past 2 weeks we are witnessing a new wave of Google Analytics Spam – language spam and in this post I’ll outline what it is, how it happens and how you can protect your GA accounts from it (to some extent). We first started seeing it on Nov 8, just as the 2016 […] Read more…
OK, so you’ve decided that you’d start tracking your site with Google Analytics. Or you were given the task to install Google Analytics tracking on a site. In both cases you need to start by logging into a Google account (not necessarily a Gmail one), accepting the Google Analytics terms of service and then creating […] Read more…
Take your A/B testing program to the next level
Want to take your A/B tests to the next level?
If you enjoyed this article and want to read more great content like it make sure to check out the book “Statistical Methods in Online A/B Testing” by the author, Georgi Georgiev, and take your experimentation program to the next level.