Georgi GeorgievThe long wait is finally over! “Statistical Methods in Online A/B Testing” can now be found as a paperback and an e-book on your preferred Amazon store.
The book is a comprehensive guide to statistics in online controlled experiments, a.k.a. A/B tests, and tackles the difficult matter of statistical inference in a way accessible to readers with little to no prior experience with it. Each concept is built from the ground up, explained thoroughly, and illustrated with practical examples from website testing. The presentation is straight to the point and practically oriented so you can apply the takeaways in your daily work.
I’ve basically written the book I wish I had years ago when I was first attempting to understand the role and methods of applying statistics to business problems. If I had it then, it would have saved me literally years of daunting and laborious research, trial and error, running simulations, etc. As such, I’m certain “Statistical Methods in Online A/B Testing” will become your “go to” reference when designing, running, analyzing, and communicating the results of A/B tests.
Yet another book on statistics?
If you’ve read a dozen books on statistics, I understand how you might fear this is yet another book that is either too technical, or lacking depth, or just right in this regard, but written only with scientists in mind and therefore detached from business realities. These fears can be put aside since unlike other books and courses on statistics you might have stumbled upon, “Statistical Methods in Online A/B Testing” is firmly grounded in practical applications. Taking business actions while measuring and controlling risk through online experiments is the overarching idea which penetrates all parts of the exposition.
The language is accessible even for readers with limited exposure to statistics and the math is thoroughly explained so you find it illuminating rather than confusing or obscure. Most of the illustrative examples are lifted right out of everyday conversion rate optimization practice – artificial or non-business related examples are at a sanitary minimum.
Other than laying out statistical methods in various practical scenarios, “Statistical Methods in Online A/B Testing” also points out frequent mistakes, misuses, and misunderstandings of said methods that I’ve seen committed and/or promoted over the years. Avoiding them will help you run tests faster, interpret their results more accurately, as well as align your usage of statistics and experimentation with any business goals you might want to pursue.
Who is this book for?
The book is best suited for professionals working in:
- Conversion Rate Optimization
- Analysis of experimental data
- Website & mobile app A/B testing
- User Experience testing
- Digital marketing testing
This includes data analysts, growth experts, product managers, and marketing managers.
Experienced statisticians should also find the book interesting due to the new research presented and the fresh perspective on how to apply statistics and experimentation to achieve business goals.
Book reviews
Here is what reviewers of advance reading copies of “Statistical Methods in Online A/B Testing” had to say about the book:
“This is simultaneously the most comprehensive and the most accessible book on A/B testing statistics I’ve encountered. Most companies are running A/B tests now, but the level of statistical malpractice is surprisingly high. If you run tests, then, this book should be required reading. Read it, and then gift it to the rest of your team to read, and watch your program’s maturity rise in front of your eyes.”
Alex Birkett, senior growth marketing manager at Hubspot
“If you’re running online experiments, you absolutely need this book. It will help you avoid costly testing mistakes stemming from misuse and misunderstanding of statistics.”
Peep Laja, founder and principal of CXL Institute
“Finally, a comprehensive reference on statistics for my own industry! No longer do I need 8 different books on statistics that were each bought for only one or two chapters each, because Georgi has delivered a text whereby each and every chapter is applicable to success in my daily work.”
Philip Cross, CEO and conversion rate optimizer at Premeni Services Company
More editorial reviews are available at www.abtestingstats.com.
Contents – sneak preview
Here are the main chapter titles, which should give you an idea of the overall layout of the book:
- Using statistics in business (p.1-24)
- Estimating uncertainty. Statistical significance, p-values, other estimates (p.25-58)
- Statistical assumptions. Assessing model adequacy (p.59-70)
- Statistical power and sample size calculations (p.71-96)
- Types of statistical hypotheses (p.97-110)
- Tests with more than one variant (p.111-130)
- Segmentation and multiple performance indicators (p.131-138)
- Working with continuous data (p.139-148)
- Percentage Change (p.149-156)
- Sequential testing: continuous monitoring of data (p.157-186)
- Optimal significance thresholds and sample sizes (p.187-216)
- External validity a.k.a. generalizability of A/B test results (p.217-232)
- Miscellaneous topics (p.233-252)
- Communicating statistical results (p.253-268)
Further information and a link for a more detailed preview (PDF download) can be found on the book website at www.abtestingstats.com. You can also directly download a pdf preview of “Statistical Methods in Online A/B Testing” for the full table of contents and some of the introductory pages.
About the author
Managing owner of Web Focus and creator of Analytics-toolkit.com, Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments for hundreds of websites.
He is the author of the book "Statistical Methods in Online A/B Testing", of white papers on statistical analysis of A/B tests, and has been a speaker at conferences, seminars, and courses. Georgi has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.
