Overview of Books on Conversion Optimization, Web Analytics, SEO & PPC

This overview is a little different than most overeviews out there. It will focus on three specific basic statistical concepts and would show if and how well they are explored in 18 of the top books on Conversion Optimization, A/B testing, web analytics, SEO & PPC/AdWords.

The three concepts this review focuses on are statistical significance, statistical power and the multiple comparisons problem. Why? Because these are the three basic statistical concepts that every marketer should be familiar with, as I argued in this post: Why Every Marketer Should be a Statistician. In short: without a grasp of those concepts, no proper analysis of data is possible. No cost/benefit analysis can really be based on data without them. And we all know that bad analysis results in suboptimal decisions, at best.

The books were chosen from the top results on Amazon.com’s search engine. The analysis is based on text searches in the full text of the books using the “Look Inside” feature.

Books on Conversion Optimization / A/B Testing

Let’s take a look at the CRO-oriented books first. With subtitles like “…The Art and Science of Optimized Marketing”, “The Art and Science of Converting Prospects to Customers”, “…Holistic Conversion Rate Optimization” and “The Definitive Guide to Testing and Tuning for Conversions” these should at the very least mention the basic statistical concepts that provide the basis for a proper analysis. Let’s see how they fare in this:

Book TitleAuthor(s)Statistical Significance MentionsStatistical Power / Fixed Sample Size MentionsMultiple Comparisons Problem Mentions
You Should Test That! (2013)Chris Goward190*0
Landing Page Optimization (2014)Tim Ash, Maura Ginty, Rich Page800
Experiment! (2012)Colin McFarland400
A/B Testing (2013)Dan Siroker, Pete Koomen30*0
Conversion Optimization (2010)Khalid Saleh, Ayat Shukairy40*0
Always Be Testing (2008)Bryan Eisenberg, John Quatro-vonTivadar31*0
Convert Every Click (2013)Benji Rabhan50*0
 Sum Total:461*0

Surprisingly, almost all of them fail to mention 2 of the 3 concepts! “Always Be Testing” is the only one to mention “statistical power” and “sample sizes” in a proper context, but it’s only found in one of the reviews for the book, part of its “Advance Praise” section. The book indeed has a section on “statistical significance and sample sizes” but it’s a mere  one page long, so I highly doubt the concepts were explored in any reasonable depth.

“Statistical significance” is mentioned in all of the books, with Chris Goward’s “You Should Test That!” scoring #1 with 19 mentions. However, there is a problem. In 4 of the 7 books statistical significance is mentioned in ways that imply that “reaching statistical significance” is somehow telling you something, i.e. it’s a good time to stop your test and declare results. You should already know that this is a bad advice, and if you don’t read more about it here. In that same article I showed how Goward is not following his own advice and gives examples that are far from statistically significant, so more mentions doesn’t necessarily mean “well thought”.

So, it seems most of the literature on CRO is lacking badly on information about the most basic of statistical concepts without which “data-based” decisions are not “data-based” at all, rather “data-guessed” or “data-wished”. While many of the books provide good advice on different aspects of conversion rate optimization, they fail to educate on a most crucial aspect of the CRO process and thus receive a very low mark in our overview.

Books on Web Analytics / Digital Marketing Analytics

Several books on web analytics are up next. These have even less “excuses” for not tackling basic statistics than the books on conversion optimization. They are supposed to be all about data analysis, after all. Let’s see how well they fare under our little test:

Book TitleAuthor(s)Statistical Significance MentionsStatistical Power / Fixed Sample Size MentionsMultiple Comparisons Problem Mentions
Web Analytics 2.0 (2009)Avinash Kaushik500
Digital Marketing Analytics (2013)Chuck Hemann, Ken Burbary000
Advanced Web Metrics with Google Analytics (2012)Brian Clifton100
Web Analytics: An Hour a Day (2007)Avinash Kaushik800
Web Analytics For Dummies (2007)Pedro Sostre, Jennifer LeClaire000
Practical Web Analytics for User Experience (2013)Michael Beasley40*0
 Sum Total:1800

Two of the books fail to mention “statistical significance” even once! All of them fail to mention anything about sample size calculations, fixing the sample size or statistical power. None of the mentions multiple comparisons issues either. No doubt, if you rely on those books to teach you how to analyze web traffic your work will be lacking severely.

Books on SEO / PPC / Google AdWords

With words in the titles like “ultimate guide” and “advanced” we expect at least some decent mentions for the three concepts that are the basis for most optimization that is done in the fields of SEO and especially PPC. Let’s look at the results:

Book TitleAuthor(s)Statistical Significance MentionsStatistical Power / Fixed Sample Size MentionsMultiple Comparisons Problem Mentions
Ultimate Guide to Google AdWords (2012)Perrry Marshall, Bryan Todd100
Ultimate Guide to Pay-Per-Click Advertising (2014)Richard Stokes, Perry Marshall000
The Art of SEO (2012)Eric Enge, Stephan Spencer, Jessue Stricchiolla, Rand Fishkin000
Inbound Marketing & SEO (2013)Rand Fishkin, Thomas Høgenhaven200
Advanced Google AdWords (2014)Brad Geddes70*0
 Sum Total:1000

Almost all fail our test miserably, with “Advanced Google AdWords” standing out with 10 mentions of “statistical significance”. However, none of the books mention statistical power or multiple comparison issues with statistical significance testing.  The latter is especially hurtful in those areas since we regularly measure the performance of several ad creatives, multiple keywords and landing pages.

Why such results?

Given the general under-appreciation of these topics in the industry, especially the last two, maybe it shouldn’t be surprising that pretty much all books fail to address these statistical concepts properly. I think this is due to the fact that the internet marketing industry is still in its infancy so failure to use data properly is not punished by the market so hard, yet. However, in my opinion, this situation will be changing fast in the next several years as more and more marketers adopt a more a scientific, data-driven approach to online advertising and conversion optimization, simply because it yields much better results.

2021 update

Since writing the above overview it became obviously clear that a proper book on statistics in A/B testing needed to be written and so that is exactly what I did in 2019, nearly 5 years after posting the above overview. Find more about the book, which has, as of the time of writing this in late 2021, become the go-to reference for statistical methods in online experimentation.

About the author

Georgi Georgiev

Managing owner of Web Focus and creator of Analytics-toolkit.com, Georgi has over eighteen years of experience in online marketing, web analytics, statistics, and design of business experiments for hundreds of websites.

He is the author of the book "Statistical Methods in Online A/B Testing" and white papers on statistical analysis of A/B tests, as well as a lecturer on dozens of conferences, seminars, and courses, including as a Google Regional Trainer.

This entry was posted in A/B testing, Conversion optimization, Google Analytics, Internet marketing, Multiple variations testing, Statistical significance, Statistics and tagged , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

Take your user testing program to the next level with the most comprehensive book on A/B testing statistics.

Learn more