Articles from 2020

Bayesian Probability and Nonsensical Bayesian Statistics in A/B Testing

Bayesian Probability and Bayesian Statistics in AB Testing

Many adherents of Bayesian methods put forth claims of superiority of Bayesian statistics and inference over the established frequentist approach based mainly on the supposedly intuitive nature of the Bayesian approach. Rational thinking or even human reasoning in general is Bayesian by nature according to some of them. Others argue that proper decision-making is inherently […] Read more…

Posted in A/B testing, Bayesian A/B testing, Conversion optimization, Statistics | Tagged , , , , ,

The Perils of Using Google Analytics User Counts in A/B Testing

Google Analytics User Data in AB Testing

(Updated Sep 2022 & Jan 2023 to include updated information about Google Analytics 4) Many analysts, marketers, product managers, UX and CRO professionals nowadays rely on user counts provided by Google Analytics, Adobe Analytics, or similar tools, in order to perform various statistical analyses. Such analyses may involve the statistical hypothesis tests and estimations part […] Read more…

Posted in A/B testing, Google Analytics, Statistics | Tagged , , , , , , , ,

The Effect of Using Cardinality Estimates Like HyperLogLog in Statistical Analyses

This article will examine the effects of using the HyperLogLog++ (HLL++) cardinality estimation algorithm in applications where its output serves as input for statistical calculations. A prominent example of such a scenario can be found in online controlled experiments (online A/B tests) where key performance measures are often based on the number of unique users, […] Read more…

Posted in A/B testing, Google Analytics, Statistics | Tagged , , , , , ,

Error Spending in Sequential Testing Explained

Sequential Hypothesis Testing with Efficacy and Futility Bound

Sequential analysis of experimental data from A/B tests has been quite prominent in recent years due to the myriad of Bayesian solutions offered by big industry players. However, this type of sequential analysis is not sequential testing proper as these solutions have generally abandoned the idea of testing and therefore error control, substituting it for […] Read more…

Posted in A/B testing, AGILE A/B testing, Statistics | Tagged , , , , , ,

Updates to Our A/B Testing Statistical Calculators

Meta Analysis Statistical Calculator

This short post presents several changes, updates, and new tool functionalities released in the past couple of months – some very recent, some less so. The updates concern all statistical calculators for design and analysis of A/B tests at Analytics-Toolkit.com and consist of the following: New tool: Statistical Calculator for Meta Analysis Quality of life […] Read more…

Posted in Analytics-Toolkit.com | Tagged , , , ,

Frequentist vs Bayesian Inference

Frequentist vs Bayesian Inference

In this article I’m revisiting* the topic of frequentist vs Bayesian inference with specific focus on online A/B testing as usual. The present discussion easily generalizes to any area where we need to measure uncertainty while using data to guide decision-making and/or business risk management. In particular, I will discuss each of the following five […] Read more…

Posted in A/B testing, Bayesian A/B testing, Statistics | Tagged , , , , , , ,

Underpowered A/B Tests – Confusions, Myths, and Reality

Underpowered A/B Tests

In recent years a lot more CRO & A/B testing practitioners have started paying more attention to the statistical power of their online experiments, at least based on my observations. While this a positive development for which I hope I had contributed somewhat, it comes with the inevitable confusions and misunderstandings surrounding a complex concept […] Read more…

Posted in A/B testing, Statistics | Tagged , , , , , ,

The Perils of Poor Data Visualization in CRO & A/B Testing

As any UX & CRO expert should now, the way we present information matters a lot both in terms of how well it is understood and in terms of the probability that it will lead to the desired action. A/B testing calculators and other tools of the trade are no exception and here I will […] Read more…

Posted in A/B testing, Conversion optimization, Statistics | Tagged , , , , , , ,

The Cost of Not A/B Testing – a Case Study

The cost of not a/b testing

Most of the time when discussing A/B testing, regardless of context, we discuss costs such as the expense of running an experimentation program, of shipping ‘winners’ to production. Only rarely do I see references to the less obvious, but usually more important costs in terms of opportunity cost (incurred during testing) and the cost of […] Read more…

Posted in A/B testing | Tagged , , , ,