Statistical Methods in Online A/B Testing – the book

The long wait is finally over! “Statistical Methods in Online A/B Testing” can now be found as a paperback and an e-book on your preferred Amazon store. Note that the Kindle edition is available for $2.99 or equivalent if you’ve already purchased the paperback (through Kindle Matchbook).

Book Cover Statistical Methods Online A/B Testing

The book is a comprehensive guide to statistics in online controlled experiments, a.k.a. A/B tests, and tackles the difficult matter of statistical inference in a way accessible to readers with little to no prior experience with it. Each concept is built from the ground up, explained thoroughly, and illustrated with practical examples from website testing. The presentation is straight to the point and practically oriented so you can apply the takeaways in your daily work.

I’ve basically written the book I wish I had years ago when I was first attempting to understand the role and methods of applying statistics to business problems. If I had it then, it would have saved me literally years of daunting and laborious research, trial and error, running simulations, etc. As such, I’m certain “Statistical Methods in Online A/B Testing” will become your “go to” reference when designing, running, analyzing, and communicating the results of A/B tests.

Yet another book on statistics?

If you’ve read a dozen books on statistics, I understand how you might fear this is yet another book that is either too technical, or lacking depth, or just right in this regard, but written only with scientists in mind and therefore detached from business realities. These fears can be put aside since unlike other books and courses on statistics you might have stumbled upon, “Statistical Methods in Online A/B Testing” is firmly grounded in practical applications. Taking business actions while measuring and controlling risk through online experiments is the overarching idea which penetrates all parts of the exposition.

The language is accessible even for readers with limited exposure to statistics and the math is thoroughly explained so you find it illuminating rather than confusing or obscure. Most of the illustrative examples are lifted right out of everyday conversion rate optimization practice – artificial or non-business related examples are at a sanitary minimum.

Other than laying out statistical methods in various practical scenarios, “Statistical Methods in Online A/B Testing” also points out frequent mistakes, misuses, and misunderstandings of said methods that I’ve seen committed and/or promoted over the years. Avoiding them will help you run tests faster, interpret their results more accurately, as well as align your usage of statistics and experimentation with any business goals you might want to pursue.

Who is this book for?

The book is best suited for professionals working in:

  • Conversion Rate Optimization
  • Analysis of experimental data
  • Website & mobile app A/B testing
  • User Experience testing
  • Digital marketing testing

This includes data analysts, growth experts, product managers, and marketing managers.

Experienced statisticians should also find the book interesting due to the new research presented and the fresh perspective on how to apply statistics and experimentation to achieve business goals.

Book reviews

Here is what reviewers of advance reading copies of “Statistical Methods in Online A/B Testing” had to say about the book:

“This is simultaneously the most comprehensive and the most accessible book on A/B testing statistics I’ve encountered. Most companies are running A/B tests now, but the level of statistical malpractice is surprisingly high. If you run tests, then, this book should be required reading. Read it, and then gift it to the rest of your team to read, and watch your program’s maturity rise in front of your eyes.”

Alex Birkett, senior growth marketing manager at Hubspot

“If you’re running online experiments, you absolutely need this book. It will help you avoid costly testing mistakes stemming from misuse and misunderstanding of statistics.”

Peep Laja, founder and principal of CXL Institute

“Finally, a comprehensive reference on statistics for my own industry! No longer do I need 8 different books on statistics that were each bought for only one or two chapters each, because Georgi has delivered a text whereby each and every chapter is applicable to success in my daily work.”

Philip Cross, CEO and conversion rate optimizer at Premeni Services Company

More editorial reviews are available at

Contents – sneak preview

Here are the main chapter titles, which should give you an idea of the overall layout of the book:

  1. Using statistics in business (p.1-24)
  2. Estimating uncertainty. Statistical significance, p-values, other estimates (p.25-58)
  3. Statistical assumptions. Assessing model adequacy (p.59-70)
  4. Statistical power and sample size calculations (p.71-96)
  5. Types of statistical hypotheses (p.97-110)
  6. Tests with more than one variant (p.111-130)
  7. Segmentation and multiple performance indicators (p.131-138)
  8. Working with continuous data (p.139-148)
  9. Percentage Change (p.149-156)
  10. Sequential testing: continuous monitoring of data (p.157-186)
  11. Optimal significance thresholds and sample sizes (p.187-216)
  12. External validity a.k.a. generalizability of A/B test results (p.217-232)
  13. Miscellaneous topics (p.233-252)
  14. Communicating statistical results (p.253-268)

Further information and a link for a more detailed preview can be found on the book website at

Georgi Georgiev is a managing owner of digital consultancy agency Web Focus and the creator of His 16 years of experience with online marketing, data analysis & website measurement, statistics and design of business experiments include owning and operating over a dozen websites and hundreds of consulting clients. Georgi is also the author of the book "Statistical Methods in Online A/B Testing" as well as several white papers on statistical analysis of A/B tests. He’s been a lecturer on dozens of conferences, seminars, and courses, including as Google Regional Trainer for Bulgaria and the region.

Facebook Twitter LinkedIn 

Statistical Methods in Online A/B Testing - the book by
This entry was posted in A/B Testing, Conversion Optimization, Statistics and tagged , , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.