AB testing is widely used by organisations to make informed, data-driven decisions. However, ‘testing’ is such a broad term, and many people can fall into the trap of investing their time and money in the wrong places, not maximising their ROI.
Kevin Satti, Lead Software Engineer at Webtrends Optimize, explores what makes an AB test truly great? And what would “best” even mean in this context?
Is it the one that made the most money? Or the one that taught us something new? It all depends on how we define success.
Measuring Success
We could choose to judge success based on a single KPI. Or we could track 80 metrics and analyse them all. (Some of our clients really do this.)
Most of the time, there’s a sweet spot: a focused set of core metrics that tell us whether the test is a winner or a loser. But what if the goal is to learn? Would we gain more insight by tracking everything imaginable, say 500 metrics? Could we miss something important by tracking too little?
In my view, no. A small, well-chosen set of metrics is usually enough. Too much data creates noise, and then it’s easy to miss what matters. It also takes more time to analyse. At least for now… once AI can reliably analyse hundreds of metrics, that trade-off might shift.
Profit ≠ The Only Goal
Tests that drive profit are the home runs that everyone wants to present to their bosses on a Monday.
But that doesn’t mean that a losing test is a bad one. Negative results clearly show what doesn’t work and knowing what your customers don’t like is valuable. In fact, by testing you’ve already avoided the costly mistake of making the change permanent.
A negative result might also push you to reevaluate your ideas and change strategy. The least useful tests are often the ones with no clear result at all. The best AB tests drive learning and decisions.
Keep It Simple
Complexity is another factor. A complex test might be exciting for the developer building it, but if it’s hard to scope, harder to QA, and likely to introduce bugs, is it worth it?
We’ve seen some of our most impactful results come from the simplest tests. So, the best AB test is only as complex as it needs to be to test the hypothesis, no more.
We have lots of success with this at Webtrends Optimize. For example, after redesigning their website, Sussex Cricket Club wanted to improve their users experience and increase the conversion rate. We tested small changes, such as making the ‘Book Tickets’ icon more prominent and changing the colour to blue, which increased clicks on the button by 70%. We also utilised hero banners and a dynamic promo-banner which increased the number of inquiries submitted.
These small changes delivered the results the client wanted without creating complex tests that invite problems or bugs.
What About Repeatability?
Imagine a test that gives a big uplift on Black Friday but has no effect at other times. Is that a great test? Or a test that delivers wildly different results each time it runs - can you trust it?
While some variation is expected, the best tests tend to produce strong, repeatable signals. Consistency matters.
So, What Is the Best AB Test?
If we define “best” as “most valuable,” then the best AB tests tend to have a few key traits:
- They deliver clear, actionable insights.
- They produce significant results - positive or negative.
- They’re built as simply as the problem allows.
- They produce consistent, repeatable signals.
That’s what makes an AB test great: it teaches you something, and those lessons fuel long-term growth.
