Don’t we all want to run tests as quickly as possible, reaching results as conclusive and as certain as possible? Don’t we all want to minimize the number of users we send to an inferior variant and to implement a variant with positive lift as quickly as possible? Don’t we all want to get rid of […] Read more…
After many months of statistical research and development we are happy to announce two major releases that we believe have the potential to reshape statistical practice in the area of A/B testing by substantially increasing the accuracy, efficiency and ultimately return on investment of all kinds of A/B testing efforts in online marketing: a free white […] Read more…
What is Statistical Power? In null-hypothesis statistical testing (NHST) – the procedure most commonly applied in A/B tests, there are two types of errors that practitioners should care about, type I and type II errors. Type I is the probability of the test procedure to falsely reject a true null hypothesis. Type II error is […] Read more…
As someone who spent a good deal of time on trying to figure out how to run A/B tests properly and efficiently, I was intrigued to find a slide from a presentation by VWO®’s data scientist Chris Stucchio, where he goes over the main reasons that caused him and the VWO® team to consider and […] Read more…
Now, this is not a new topic on this blog. I’ve discussed the issue of optional stopping based on “achieving” or “reaching” statistical significance in my Why Every Internet Marketer Should be a Statistician post more than two years ago and others have touched on it as well – both before and after me. However, the issue […] Read more…
Fantasy vs the Real World: Naive Bayesian AB Testing vs Proper Statistical Inference This post is addressed at a certain camp of proponents and practitioners of A/B testing based on Bayesian statistical methods, who claim that outcome-based optional stopping, often called data peeking or data-driven stopping, has no effect on the statistics and thus inferences […] Read more…
This overview is a little different than most overeviews out there. It will focus on three specific basic statistical concepts and would show if and how well they are explored in 18 of the top books on Conversion Optimization, A/B testing, web analytics, SEO & PPC/AdWords. The three concepts this review focuses on are statistical […] Read more…
This question pops up often in more “advanced” discussion forums and blogs on Conversion Optimization and usually one (or several) of the following advices are given: Do an A/A test first in order to test your split testing framework. If the difference between the two is statistically significant at the decided level, then your framework is […] Read more…
I’ll start this post right with the punch line: Internet marketing at its current state is dominated by voodoo magic. There, I’ve said it, and I’ve said it in bold italic so you know it must be true. Internet marketing is not as data-driven as most of us have been led to believe, it is […] Read more…
I am happy to announce the official launch of Analytics-Toolkit.com – the toolkit that makes analytics easier. I’m excited to present the results of over 2 years of development because I believe many web analytics specialists, SEO/SEM specialists, and internet marketers will find the tools in the toolkit valuable in their daily work. Most of […] Read more…
After over 2 years of development, Analytics-Toolkit.com is almost ready for prime time. I’m really excited about what’s to come both in terms of Analytics Toolkit improving the way people use the Google Analytics platform and in terms of the blog posts that I’ll do here. Well, I have to get back to working on […] Read more…
Take your A/B testing program to the next level
Want to take your A/B tests to the next level?
If you enjoyed this article and want to read more great content like it make sure to check out the book “Statistical Methods in Online A/B Testing” by the author, Georgi Georgiev, and take your experimentation program to the next level.