Georgi GeorgievWe are happy to present a brand new addition to our website: a comprehensive A/B testing glossary containing terms and abbreviations used testing as part of conversion rate optimization (CRO). Definitions start from very basic things such as “A/B test“, “mean“, “conversion rate” and “revenue per user“, go through “hypothesis“, “null hypothesis“, “standard deviation“, “p-value” and “confidence interval” and go into depth with “generalizability“, “alpha-spending“, “statistical model“, “statistical design” and so on. All term definitions and explanations are lavishly cross-referenced so one can start anywhere and learn on the go. Of course, there is also an alphabetic index.
The aim of the glossary is not only to explain each concept, but to put it in the context of A/B testing and to even provide some best-practice advice and warnings against common misuses, sort of as a guide. Naturally, the goal of covering all these aspects in full depth is far too ambitions for a simple glossary, but I hope it delivers to a satisfactory extent. If you have any suggestions for improving certain glossary entries or want to suggest an important one we’ve missed, feel free to drop us an e-mail or find me on LinkedIn.
Basic Statistics
Since the glossary is all about stats, it makes sense to present some rudimentary statistics related to the project. At release (Dec 3, 2018) it contains 148 unique definitions which have a total of 65 aliases and abbreviations. The total length of the definitions is about 228,000 symbols, which is about 32,000 words or about a 100-page book in A-5 format.
The mean entry length is 1544 while 50% of entries are over 1428 symbols long. The length distribution is:
We see sort of a power-curve where most terms require significantly less words to explain while some need multiple times more words (graph is for symbols, not words, but an average word count of symbols / 7 should be a good approximation). I believe the current distribution offers a good balance between offering concise and on-point descriptions and advice while not going overboard and becoming a full encyclopedia entry.
Another curious statistic is the number of terms starting with different letters:
“S” is a natural leader due to all terms that start “statistical” and “significant/significance”. Curiously, none of the current entries start with “J”, “Q”, “W”, “X” and “Y”.
We always love to hear from our readers, so let us know if you found value in the glossary. Also, share with us if a term entry surprised you by presenting a new angle to a concept you thought you are pretty familiar with or by any of the auxiliary info presented (@AnalyticsToolki on Twitter).
About the author
Managing owner of Web Focus and creator of Analytics-toolkit.com, Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments for hundreds of websites.
He is the author of the book "Statistical Methods in Online A/B Testing", of white papers on statistical analysis of A/B tests, and has been a speaker at conferences, seminars, and courses. Georgi has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.
