Georgi GeorgievThis overview is a little different than most overeviews out there. It will focus on three specific basic statistical concepts and would show if and how well they are explored in 18 of the top books on Conversion Optimization, A/B testing, web analytics, SEO & PPC/AdWords.
The three concepts this review focuses on are statistical significance, statistical power and the multiple comparisons problem. Why? Because these are the three basic statistical concepts that every marketer should be familiar with, as I argued in this post: Why Every Marketer Should be a Statistician. In short: without a grasp of those concepts, no proper analysis of data is possible. No cost/benefit analysis can really be based on data without them. And we all know that bad analysis results in suboptimal decisions, at best.
The books were chosen from the top results on Amazon.com’s search engine. The analysis is based on text searches in the full text of the books using the “Look Inside” feature.
Books on Conversion Optimization / A/B Testing
Let’s take a look at the CRO-oriented books first. With subtitles like “…The Art and Science of Optimized Marketing”, “The Art and Science of Converting Prospects to Customers”, “…Holistic Conversion Rate Optimization” and “The Definitive Guide to Testing and Tuning for Conversions” these should at the very least mention the basic statistical concepts that provide the basis for a proper analysis. Let’s see how they fare in this:
| Book Title | Author(s) | Statistical Significance Mentions | Statistical Power / Fixed Sample Size Mentions | Multiple Comparisons Problem Mentions |
| You Should Test That! (2013) | Chris Goward | 19 | 0* | 0 |
| Landing Page Optimization (2014) | Tim Ash, Maura Ginty, Rich Page | 8 | 0 | 0 |
| Experiment! (2012) | Colin McFarland | 4 | 0 | 0 |
| A/B Testing (2013) | Dan Siroker, Pete Koomen | 3 | 0* | 0 |
| Conversion Optimization (2010) | Khalid Saleh, Ayat Shukairy | 4 | 0* | 0 |
| Always Be Testing (2008) | Bryan Eisenberg, John Quatro-vonTivadar | 3 | 1* | 0 |
| Convert Every Click (2013) | Benji Rabhan | 5 | 0* | 0 |
| Sum Total: | 46 | 1* | 0 |
Surprisingly, almost all of them fail to mention 2 of the 3 concepts! “Always Be Testing” is the only one to mention “statistical power” and “sample sizes” in a proper context, but it’s only found in one of the reviews for the book, part of its “Advance Praise” section. The book indeed has a section on “statistical significance and sample sizes” but it’s a mere one page long, so I highly doubt the concepts were explored in any reasonable depth.
“Statistical significance” is mentioned in all of the books, with Chris Goward’s “You Should Test That!” scoring #1 with 19 mentions. However, there is a problem. In 4 of the 7 books statistical significance is mentioned in ways that imply that “reaching statistical significance” is somehow telling you something, i.e. it’s a good time to stop your test and declare results. You should already know that this is a bad advice, and if you don’t read more about it here. In that same article I showed how Goward is not following his own advice and gives examples that are far from statistically significant, so more mentions doesn’t necessarily mean “well thought”.
So, it seems most of the literature on CRO is lacking badly on information about the most basic of statistical concepts without which “data-based” decisions are not “data-based” at all, rather “data-guessed” or “data-wished”. While many of the books provide good advice on different aspects of conversion rate optimization, they fail to educate on a most crucial aspect of the CRO process and thus receive a very low mark in our overview.
Books on Web Analytics / Digital Marketing Analytics
Several books on web analytics are up next. These have even less “excuses” for not tackling basic statistics than the books on conversion optimization. They are supposed to be all about data analysis, after all. Let’s see how well they fare under our little test:
| Book Title | Author(s) | Statistical Significance Mentions | Statistical Power / Fixed Sample Size Mentions | Multiple Comparisons Problem Mentions |
| Web Analytics 2.0 (2009) | Avinash Kaushik | 5 | 0 | 0 |
| Digital Marketing Analytics (2013) | Chuck Hemann, Ken Burbary | 0 | 0 | 0 |
| Advanced Web Metrics with Google Analytics (2012) | Brian Clifton | 1 | 0 | 0 |
| Web Analytics: An Hour a Day (2007) | Avinash Kaushik | 8 | 0 | 0 |
| Web Analytics For Dummies (2007) | Pedro Sostre, Jennifer LeClaire | 0 | 0 | 0 |
| Practical Web Analytics for User Experience (2013) | Michael Beasley | 4 | 0* | 0 |
| Sum Total: | 18 | 0 | 0 |
Two of the books fail to mention “statistical significance” even once! All of them fail to mention anything about sample size calculations, fixing the sample size or statistical power. None of the mentions multiple comparisons issues either. No doubt, if you rely on those books to teach you how to analyze web traffic your work will be lacking severely.
Books on SEO / PPC / Google AdWords
With words in the titles like “ultimate guide” and “advanced” we expect at least some decent mentions for the three concepts that are the basis for most optimization that is done in the fields of SEO and especially PPC. Let’s look at the results:
| Book Title | Author(s) | Statistical Significance Mentions | Statistical Power / Fixed Sample Size Mentions | Multiple Comparisons Problem Mentions |
| Ultimate Guide to Google AdWords (2012) | Perrry Marshall, Bryan Todd | 1 | 0 | 0 |
| Ultimate Guide to Pay-Per-Click Advertising (2014) | Richard Stokes, Perry Marshall | 0 | 0 | 0 |
| The Art of SEO (2012) | Eric Enge, Stephan Spencer, Jessue Stricchiolla, Rand Fishkin | 0 | 0 | 0 |
| Inbound Marketing & SEO (2013) | Rand Fishkin, Thomas Høgenhaven | 2 | 0 | 0 |
| Advanced Google AdWords (2014) | Brad Geddes | 7 | 0* | 0 |
| Sum Total: | 10 | 0 | 0 |
Almost all fail our test miserably, with “Advanced Google AdWords” standing out with 10 mentions of “statistical significance”. However, none of the books mention statistical power or multiple comparison issues with statistical significance testing. The latter is especially hurtful in those areas since we regularly measure the performance of several ad creatives, multiple keywords and landing pages.
Why such results?
Given the general under-appreciation of these topics in the industry, especially the last two, maybe it shouldn’t be surprising that pretty much all books fail to address these statistical concepts properly. I think this is due to the fact that the internet marketing industry is still in its infancy so failure to use data properly is not punished by the market so hard, yet. However, in my opinion, this situation will be changing fast in the next several years as more and more marketers adopt a more a scientific, data-driven approach to online advertising and conversion optimization, simply because it yields much better results.
2021 update
Since writing the above overview it became obviously clear that a proper book on statistics in A/B testing needed to be written and so that is exactly what I did in 2019, nearly 5 years after posting the above overview. Find more about the book, which has, as of the time of writing this in late 2021, become the go-to reference for statistical methods in online experimentation.
About the author
Managing owner of Web Focus and creator of Analytics-toolkit.com, Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments for hundreds of websites.
He is the author of the book "Statistical Methods in Online A/B Testing", of white papers on statistical analysis of A/B tests, and has been a speaker at conferences, seminars, and courses. Georgi has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.
