WHAT SHOULD YOU EXPECT FROM A TESTING TOOL?

 

Today we will look at what you should expect from good testing tools in your marketing programs and campaigns. In my previous blogs, I showed that:

1. Decision making requires causal relationships.
2. The causal relationships can be found only through carefully performed experimentation, the process that is not that different from the scientific method.

I also mentioned at the end of my last blog that proliferation of testing tools made it possible to introduce the scientific method to the marketing practice. Continuing along the same lines, I will now address the question of what you should expect from good testing tools.

In order to address this question, let’s first look at the goals of testing in Digital Marketing. The most common scenario is finding out if a specific piece of content causes a change in conversion behavior of the visitors to your website. We would like to accomplish this task in a minimum amount of time and with the highest confidence in the predictive value of the results, in order to maximize the return on testing. Notice that the time and confidence are competing objectives – you can increase the confidence by running the test longer, and vice versa, you can stop the test earlier, but the confidence in the predictive value of the results will be lower. The only way to solve this dichotomy is to apply rigorous statistical techniques.

So, here are my suggestions on what you should be looking for when choosing a testing tool.

1. Ease of setting up a test. The tool must provide an intuitive workflow.
2. Data collection must be reliable. If there are problems with collecting data during a test, then, most likely, you will have to restart the test and throw the data collected thus far away.
3. A reporting system that employs a rigorous statistical approach to analysis. It requires additional efforts to develop such a system (that is why many vendors take various shortcuts), but it will save you a lot of testing time while providing high-confidence predictions.

Incidentally, while I was writing this blog, a new Forrester Wave report was released [1]. In this report, the analyst states, when referring to Webtrends Optimize, that “Webtrends applies a rigorous approach to experiment design, execution, and analysis, with the intention of providing the most statistically trustworthy approach possible.”