Here I’ll address how intuitive experiences can make the most complex testing much easier. Does this seem too obvious? It should. Why, then, is it that so many website optimisation companies only tout making simple testing easier with phrases like “simple A/B testing” or “A/B testing you will use”? They offer more complex multivariate testing, but do their best to steer users away (“MVT takes a lot of traffic.” “Most of the time, multiple A/B tests will accomplish the same thing.”). What a bunch of hooey! Sure, full-factorial MVT takes a lot of traffic, but that’s why fractional factorial methods exist – they allow multivariate testing with far less traffic. And, multivariate testing allows you to discover not only winning presentations and experiences, it also lets you know which of the elements tested had the most impact on both positive and negative influence on the testing goals. Insight like that is nearly impossible to gain with any confidence from a series of A/B tests.
So if multivariate testing if is so beneficial, why do many vendors avoid promoting it? They do this because testing done well is not easy, and complex testing done well is just plain hard. It requires a solution that has sound math behind it, as well as the discipline and knowledge from the users to understand how to interpret and present the results. The goal of any optimization solution should be to build learnings into the system either to remove barriers or to provide guardrails that keep online testing professionals on track based on the ability to automate processes or inform users. Done well, this will expose greater value to clients. Done poorly, it will confuse the heck out of them.
Automating Processes
One of the benefits of having a team of optimisation experts within Webtrends Optimize is that our solution “learns” from interactions with clients. In other words, our team of experts feeds our pipeline of projects and calls us out when we start going down the wrong path. This in-house expertise also helps us develop intuitive workflows that reduce the number of steps it takes to accomplish common tasks, and makes life easier for both themselves and for our clients to achieve their goals. One way of automating experiences is through recognising and enabling familiar user patterns.
As an example, think about how visual editors work today. One of the first things you do when you set up a test is to tell the system what kind of test you will be running: baseline (A/A), A/B/n, split, or multivariate (full or fractional). So, users have to understand what each of these test types is and when they should be selected. However, you can define all of this by the test setup. For example, if a user selects:
- One or more factors, each with only a single variation of content, then run an A/B/n test.
- One or more factors, where one or more of the factors has multiple variations, then run a multivariate test.
- Anything, then run a baseline analysis alongside the test to detect effects of testing on the system.
While you would obviously want users to be able to manually determine test types, in most cases, the recommendations would be valid and this approach gets users more quickly into the design of experiments.
Informing Users
Optimisation professionals must get answers to a large number of questions that surround a test, but are not directly part of the hypotheses or post-test analyses. This information comes about as a result of heavy use of simple web analytics combined with advanced data-mining. Again, a superior website optimisation company can proactively inform users since it knows from experience what those questions are. For example, the data surrounding the setup of a test can do the following:
- Instantly calculate test duration based on historical traffic to test pages so users know when they can expect to see results.
- Monitor tests/targets after they run to see when predicted results decay and suggest it is time for a change.
- Discover interesting segments of visitors that the test did not identify but surfaced through the results.
Protecting Users
Successful optimisation programs gain momentum and grow over time. Eventually, every department in an organisation wants to use testing to improve key metrics. This proliferation of practitioners means more chefs in the kitchen who risk running into each other without careful assignment of stations and swim lanes. Optimisation solutions should have the ability to manage all the elements of a test and proactively alert users when they may be making changes that could potentially affect other tests or targets. Possible points of conflict to consider are:
- Locations for a test when multiple tests need to execute on the same page.
- Content in use that could conflict with changes imposed by another test or target.
- Segmentation rules that could affect traffic allocation.
- Custom Javascript or CSS that you can use to control layout or dynamic actions.
What Does It All Mean?
The idea of an “intuitive experience” within an optimisation solution moves beyond simplifying the ability to build and run simple tests to automating processes, informing users about visitor dynamics, and keeping testers from stumbling all over themselves as their program grows. Marketers and marketing departments are growing more sophisticated in their understanding of the value of testing and they naturally want to do more. In addition, as they understand the mechanics of testing, they want to do more on their own. It is the challenge of optimisation solutions to allow marketers to do more, do it more easily, and ensure they realise the full value an effective optimisation program.
The challenge for marketers is to understand their own level of maturity when it comes to optimisation. They need to assess the in-house expertise they have to test, evaluate results, and act on the insights. They also need to assess their ability to staff the program to meet future needs when more campaigns and more departments want to take advantage of the benefits demonstrated. Choose vendors who can help you grow your program through training and expertise, not merely through a tool.
To find out more about how we can help you achieve your CRO goals please get in touch!
