Category Archives: Conversion Optimization

Bayesian Probability and Nonsensical Bayesian Statistics in A/B Testing

Bayesian Probability and Bayesian Statistics in AB Testing

Many adherents of Bayesian methods put forth claims of superiority of Bayesian statistics and inference over the established frequentist approach based mainly on the supposedly intuitive nature of the Bayesian approach. Rational thinking or even human reasoning in general is Bayesian by nature according to some of them. Others argue that proper decision-making is inherently […] Read More…

Also posted in A/B Testing, Statistics | Tagged , , , , , | Leave a comment

The Perils of Poor Data Visualization in CRO & A/B Testing

As any UX & CRO expert should now, the way we present information matters a lot both in terms of how well it is understood and in terms of the probability that it will lead to the desired action. A/B testing calculators and other tools of the trade are no exception and here I will […] Read More…

Also posted in A/B Testing, Statistics | Tagged , , , , , , , | Leave a comment

What Can You Learn From Running an A/B Test for 2 Years?

AB Test Case Study

We just concluded an A/B test on Analytics-Toolkit.com that has been left to run for just over 2 years. And it failed, as in failing to demonstrate a statistically significant effect based on the significance threshold it was designed for. Has it been a waste of time, though, or can we actually learn something from […] Read More…

Also posted in A/B Testing | Tagged , , , , | Leave a comment

Statistical Methods in Online A/B Testing – the book

Book Cover Statistical Methods Online A/B Testing

The long wait is finally over! “Statistical Methods in Online A/B Testing” can now be found as a paperback and an e-book on your preferred Amazon store. Note that the Kindle edition is available for $2.99 or equivalent if you’ve already purchased the paperback (through Kindle Matchbook). The book is a comprehensive guide to statistics […] Read More…

Also posted in A/B Testing, Statistics | Tagged , , , , | Comments closed

A/B Testing with a Small Sample Size

AB Testing Small Business

The question “How to test if my website has a small number of users” comes up frequently when I chat to people about statistics in A/B testing, online and offline alike. There are different opinions on the topic ranging from altering the significance threshold, statistical power or the minimum effect of interest all the way […] Read More…

Also posted in A/B Testing, Statistics | Tagged , , , , , , | Comments closed

Online glossary of A/B testing terms and abbreviations

AB testing glossary

We are happy to present a brand new addition to our website: a comprehensive A/B testing glossary containing terms and abbreviations used testing as part of conversion rate optimization (CRO).  Definitions start from very basic things such as “A/B test“, “mean“, “conversion rate” and “revenue per user“, go through “hypothesis“, “null hypothesis“, “standard deviation“, “p-value” […] Read More…

Also posted in A/B Testing, Statistics | Tagged , , , | Comments closed