Tag Archives: p-value

Confidence Intervals & P-values for Percent Change / Relative Difference

In many controlled experiments, including online controlled experiments (a.k.a. A/B tests) the result of interest and hence the inference made is about the relative difference between the control and treatment group. In A/B testing as part of conversion rate optimization and in marketing experiments in general we use the termĀ “percent lift” (“percentage lift”) while in […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistical Significance, Statistics | Also tagged , , , , , , , , | Leave a comment

The Google Optimize Statistical Engine and Approach

Frequentist vs Bayesian A/B testing - Google Optimize

Google Optimize is the latest attempt from Google to deliver an A/B testing product. Previously we had “Google Website Optimizer”, then we had “Content Experiments” within Google Analytics, and now we have the latest iteration: Google Optimize. While working on the integration of our A/B Testing Calculator with Google Optimize I was curious to see […] Read More…

Posted in A/B Testing, Conversion Optimization, Statistics | Also tagged , , , , , , , , , , , | Leave a comment

Statistical Significance in A/B Testing – a Complete Guide

Statistical Significance P Value

The concept of statistical significance is central to planning, executing and evaluating A/B (and multivariate) tests, but at the same time it is the most misunderstood and misused statistical tool in internet marketing, conversion optimization, landing page optimization, and user testing. This is not my first take on the topic, but it is my best […] Read More…

Posted in A/B Testing, Conversion Optimization, Multiple Variations Testing, Statistical Significance, Statistics | Also tagged , , , , , , , , | 6 Responses