WHAT TO DO WHEN YOUR WEBSITE TEST IS INCONCLUSIVE

The foundation of any good testing and optimization program are your hypotheses. A good hypothesis creates a strong framework for analyzing the interaction between your website visitors and your site’s user experience. Your hypothesis delineates your beliefs about your audience, your beliefs about your user experience and your beliefs about change.

A test hypothesis should encompass several elements:

• An explanation why you believe an alternative will produce better results than your current user experience.1
• A solid plan for measuring results so that clear comparisons can be made (and this measurement plan should mirror your current testing and optimization efforts).

Once you have your hypothesis, you can create a test and gather your results. If the statistical analysis indicates that your experimental change was responsible for better results against your business goals, then your go-forward path is clear. Likewise, if your statistical analysis indicates that your experimental change was responsible for worse results, then your go-forward path is also clear.

But what do you do if the results show that your experimental change did not make a difference or it performs less than your current design? What should you do when your hypothesis for improvement does not lead to actual improvement? Especially when the experimental change was neither better nor worse – it was just the same as your current experience. What then? There are three possibilities that may have led to those results.

1. The way your team makes decisions about what will make “a difference.” Groupthink or decision by committee does not always drive the best results because you often remove the champion who is willing to take chances, go bold and do something different. From a statistical perspective, we would call this regression to the mean and it could lead to weak results.

When you are working in a group, the standard process is to come to consensus and a final decision. This is generally fine, but in testing, that approach can be counterproductive. When you are working toward a single idea, a group will also tend to come to a consensus that is close to whatever exists; even when the goal is to do “something different.” The problem is that any truly unique idea from one group member can often seem like a bad idea to another member who will veto it, and the group will work to “make it better” in a way that everyone accepts. Generally this leads to the drastically different idea morphing back into something that isn’t that different at all.

The fix for this is pretty simple. When using a group to create test ideas, don’t limit the number of ideas. You can literally take every idea generated and test them. You do not need a single challenger – you can have a dozen. If you are not sure how this would work, then contact your testing specialist, or contact Webtrends Optimize if you need a testing specialist.

2. The elements you changed do not elicit the desired behaviour. The elements of your experience that elicit the most reaction from your audience can usually be determined through a quick MVT test of all the elements on the page. Sometimes your hypothesis is wrong and you need to adjust.

When you change something on the page, but you do not see any improved performance, one possibility is that the item changed is not that important to the desired behaviour. If you changed a button and you did not get more clicks on the button, or if you added a video and did not get more purchases, maybe the button or the video are not as important as you thought.

There is a quick fix for this too. This is the perfect situation for an MVT test where you can determine the relative importance of your different page elements on the desired behaviour. The importance of an MVT test is that it statistically calculates the interaction between your page elements relative to your behavioural goal. From this analysis you can determine the element(s) on the page on which you should focus.

3. You didn’t dig deep enough into your data. Deeper evaluation will help you understand if you are attacking the right issue or if you might have bigger issues other than optimizing current results.

When you make changes on your page, but it does not move the needle on your primary KPI, you might need to dig into your site analytics. One common mistake is trying to adjust something that is not working currently. As an example, if you make modest changes to your hero area, but do not get better results on your primary call-to-action, maybe you need to step back and look at the overall picture.

The fix for this is to look deeper in your analytics. Sometimes we work on the hero area because we assume it must be the most important real estate on the site. However, your site analytics can tell you specifically which part of your site or page is the most visited or effective in leading to conversions. This information may lead to radically rethinking your hero area.

If you would like to talk through some of your testing and targeting optimization goals or challenges, feel free to get in touch!

1 – In this sense user experience is the broadest possible definition; it would include messaging, graphical design, etc.; everything.