Statistical Significance

A brief overview of how we calculate statistical significance in AB Testing and MVT at Webtrends Optimize

AB Tests

In AB testing statistical significance represents the likelihood that an observed difference in conversion rates between a variation and the control is not purely due to chance. For AB tests (and ABn) Webtrends Optimize uses a students t-test to calculate statistical significance.

Specifically this is the more robust two tailed approach, meaning that both positive and negative directions are considered. Further calculations are made at a 95% level of confidence meaning that the risk of a false positive is limited.

In addition, we report on the the probability of an experiment beating control. Effectively this is the likelihood that the variation outperforms the live site.

road sign
landing page

Multivariate Tests

Webtrends Optimize offers both Full and Fractional MVT approaches. Our Fractional Factorial approach leverages a highly tuned Design of Experiments approach coupled with a standard predictive analytics engine to deliver actionable results in a fraction of the time required for Full Factorial tests.

This approach is intended to overcome the limitations of the Taguchi Fractional technique by making over 200 additional test designs available for use in campaigns. All of the test designs offered are constructed for balance and orthogonal testing with the goal of enabling us to calculate statistical significance within the results as accurately as possible.

Non-Binomial Calculations

To calculate statistical significance for non-binomial metrics (such as order value, time on page, units per transaction) requires specific methodologies. Comparing just the variation averages can be misleading as they do not consider the impact of outliers.

For this reason Webtrends Optimize will calculate statistical significance of non-binomial KPI using both the Wilcoxon and Kruskal-Wallis non-parametric methods. This again ensures that improvements in average order value are not down to chance.

Get in touch

If you have any more questions about how we calculate statistical significance for AB Tests or MVT feel free to send us a message, or call us on 0333 444 5502.

Thank you!

Thanks for submitting your enquiry with us.
A member of the team will be in touch shortly to follow up with you.

Thank you!

Please check your inbox and validate your email address so that we can create your free trial account.

Training Videos

While we're setting up your account, feel free to check out our training videos. You'll find them at:


Or, take a look through our blog to review industry insights, wins and all things Experimentation.