Most of the time when discussing A/B testing, regardless of context, we discuss costs such as the expense of running an experimentation program, of shipping ‘winners’ to production. Only rarely do I see references to the less obvious, but usually more important costs in terms of opportunity cost (incurred during testing) and the cost of […] Read more…
Updated Sep 17, 2018: Minor spelling and language corrections, updates related to role of randomization and external validity / generalizability. Google Optimize is the latest attempt from Google to deliver an A/B testing product. Previously we had “Google Website Optimizer”, then we had “Content Experiments” within Google Analytics, and now we have the latest iteration: […] Read more…
This is a comprehensive guide to the different types of costs and benefits, risks and rewards related to A/B testing. Understanding them in detail should be valuable to A/B testers and businesses considering whether to engage in A/B testing or not, what to A/B test and what not to test, etc. As far as I […] Read more…
Don’t we all want to run tests as quickly as possible, reaching results as conclusive and as certain as possible? Don’t we all want to minimize the number of users we send to an inferior variant and to implement a variant with positive lift as quickly as possible? Don’t we all want to get rid of […] Read more…
After many months of statistical research and development we are happy to announce two major releases that we believe have the potential to reshape statistical practice in the area of A/B testing by substantially increasing the accuracy, efficiency and ultimately return on investment of all kinds of A/B testing efforts in online marketing: a free white […] Read more…
Now, this is not a new topic on this blog. I’ve discussed the issue of optional stopping based on “achieving” or “reaching” statistical significance in my Why Every Internet Marketer Should be a Statistician post more than two years ago and others have touched on it as well – both before and after me. However, the issue […] Read more…
This overview is a little different than most overeviews out there. It will focus on three specific basic statistical concepts and would show if and how well they are explored in 18 of the top books on Conversion Optimization, A/B testing, web analytics, SEO & PPC/AdWords. The three concepts this review focuses on are statistical […] Read more…
This question pops up often in more “advanced” discussion forums and blogs on Conversion Optimization and usually one (or several) of the following advices are given: Do an A/A test first in order to test your split testing framework. If the difference between the two is statistically significant at the decided level, then your framework is […] Read more…
Take your A/B testing program to the next level
Want to take your A/B tests to the next level?
If you enjoyed this article and want to read more great content like it make sure to check out the book “Statistical Methods in Online A/B Testing” by the author, Georgi Georgiev, and take your experimentation program to the next level.