The new standard for planning and analyzing A/B tests is here

The first major overhaul of Analytics Toolkit since its release in early 2014 has finally arrived and it brings with it solutions to many of the hard questions facing practitioners when planning and analyzing A/B tests. Conducting statistically rigorous tests while achieving the best return on investment go hand in hand in the new Toolkit.

If you care about trustworthy A/B testing statistics you might have noticed I have been pointing out issues in the ways statistics are employed in online experimentation for many years while in parallel providing solutions in the form of statistical tools. However, since my knowledge and understanding were developing actively over a long timespan Analytics Toolkit gradually came to feature a diverse set of instruments with lacking integration between them. Rough edges appeared in the overall experience and in particular workflows as the boundaries of the original platform were being pushed further and further.

In contrast, the new version of Analytics Toolkit comes as the result of a master plan to integrate all of my know-how into an accessible package of state-of-the-art statistical tools. Its main highlight – the A/B Testing Hub – implements the expertise that I’ve accumulated over the years in an easy-to-use experience thus setting a new standard for planning and analyzing online experiments.

Enjoy unrivaled ease-of-use combined with statistical rigor

The two main ingredients of this overhaul are simplicity and integration. Adding these features to our statistically rigorous tools means it has never been easier to conduct A/B tests with confidence in your results.

It means one can now navigate the difficult A/B testing statistics through an improved user experience, more visuals, and an ever-expanding combo of contextual help and well-organized in-depth documentation regarding all aspects of the underlying statistical methods. Fans of AGILE tests will LOVE the improved AGILE engine while the seamless integration of planning ROI-optimal tests should be a welcome addition for anyone who has struggled to justify a test’s duration or confidence threshold.

The multiple benefits introduced with this overhaul are summed up in the following paragraphs, but there is only so much one can convey with words and screenshots. I highly encourage you to try the new Analytics Toolkit for yourself.

Analytics Toolkit now enables you to achieve more

Level up your A/B testing with the new Analytics Toolkit as you:

• Plan and analyze tests with ease with the help of the reimagined A/B test creation wizard which now integrates our unique optimal test planner tool, previously called “A/B Test ROI calculator / planner”. Under that ugly name and difficult to use interface was hidden a true must-have tool. its use is now streamlined with a much more accessible interface.
• Arrive at ROI-optimal sample sizes and confidence thresholds and avoid the guesswork by using the Advanced mode in the test creation wizard. Switch to Basic mode for a more straightforward but potentially suboptimal way to specify a test’s statistical plan (close to the old experience). In both cases setting up a new A/B test in a statistically rigorous manner should be a breeze.
• Organize all of your A/B tests through the A/B testing hub (both single evaluation and sequential evaluation ones). Meta-analyses on groups of tests can be done easily, for example when one needs to present a summary of all tests for a client or all tests performed on a given section of a website. The hub replaces the old AGILE A/B testing calculator fully and completely.
• Conduct tests on metrics such as Average revenue per user much more easily due to the multiple improvements for tests of non-binomial metrics. These include easy estimation of the baseline mean and standard deviation and better intervals and point estimates in sequential tests of non-binomial metrics.
• Get notifications about test status updates in the new Activity Feed (previously sent only via email)
• Organize tests via Projects that they belong to. A “Project” can be anything you want: a client, a website area such as PDPs or Checkout, a type of test intervention, etc. Each A/B test belongs to a single project.
• Accomplish all tasks effortlessly due to the improved user experience of all tools. The site is also faster and more responsive except where heavy lifting is required (e.g. complex statistical calculations still take time).
• Navigatе the advanced calculators in Analytics Toolkit with greater ease due to the much more detailed help information and contextual help. It includes a brand-new Learning center which consolidates the blog articles, the A/B testing glossary, the whitepapers, and features some brand-new “How to” articles.
• Better present your test results with nicer-looking exports of the different reports and calculation results via the print to PDF function of your browser & OS. Further improvements to come on this front.

Also included in the release:

• New ways to enter test data, including support for Excel spreadsheets.
• New and easy way to share calculation results in all calculators (with others logged into the toolkit) and simplified the public sharing of A/B test results.
• Easier to read graphs and useful new graphs that make it easier to understand and communicate your test results.
• Better insights into your (AGILE) sequential tests at interim analyses as they report confidence intervals at each analysis. Previously intervals were only available after a test has ended one way or the other.
• Better visibility of the diagnostic tests performed automatically on your data with sample ratio mismatch results being displayed always in the report page. Previously they would be included only when there was an indication of an SRM.
• A standalone Sample Ratio Mismatch calculator for the stat geeks (SRM checks have been an integrated feature in all relevant calculators for a few years now).
• Fixes to the typical assortment of bugs.

Expected future developments:

• Dozens of new “how to” articles with step-by-step instructions on how to employ each tool
• Dozens of video lessons to replace and improve on the old ones.
• Adding a strong superiority alternative hypothesis option to all statistical tools (simple superiority and non-inferiority would remain supported as before).
• Even more ways to connect your data.

It should be said that all tools which were available in the old Analytics Toolkit are still available in the new version. This includes the Google Analytics automation tools, the analysis tools such as the Analytics Dashboard (now incorporated into the Dashboard) and the Smart Data Explorer. All Statistical Calculators are also here but are now broken down into a half-dozen separate tools which should make them much easier to use than before.

New pricing plans

As part of the overhaul some changes to Analytics Toolkit’s pricing structure were called for – see the Pricing page for details. The new structure should be more aligned with the different types of uses we’ve seen and expect and should make it easier to choose your subscription level. The old plans were Starter, Medium, and Agency, as well as a highly customizable Enterprise plan for clients with specific needs. These are being replaced with Freelancer, Agency, and Enterprise plans, details on which can be seen on the pricing page.

Existing users should see no difference in their experience of the platform as the new A/B test limits should compare favorably to the old ones for all tiers.

Enjoy the new Analytics Toolkit

I hope you like what I and the super talented designer and UX pro Martin Bonov were able to achieve and that you would give the overhauled Analytics Toolkit a try. I hope you are just as excited to see what it enables you to accomplish as I am thrilled to present it after so many months of development. Feedback would be most appreciated on all changes, additions, and future plans.

About the author