This is a guest blog from one of our strategic partners, Journey Further.
When Harvard Business School analysed the data of 100,000 website experiments, they found that only 1 in 10 of those experiments were able to statistically validate their hypotheses.
That's hundreds of thousands of changes which people thought seemed like a good idea and would shift customer behaviour... and yet 9 out of 10 of them didn't work.
The fact that these companies were using experimentation tools to validate their ideas suggests that they are mature businesses who likely also use research and analysis to come up with their ideas in the first place. This, in turn, suggests that for the average business, 1 in 10 is probably generous.
Technically speaking, the 9 out of 10 experiments that failed to prove their hypotheses didn't necessarily 'fail' or not work in the sense that they were bad ideas or caused a negative impact, it just means the effect was too subtle to detect and is therefore unknown.
Nevertheless, as someone who has conducted countless experiments on websites for many years, I can tell you that this study validates what everyone has always known: it's very difficult to come up with ideas that work and have a significant impact.
The process and experience of running experiments will very quickly humble a person to the absolute fallibility and futility of their own opinions and convictions about customer behaviour. Time and time again, we test things that we are utterly convinced will be huge and amazing, and then the opposite happens or, more commonly, nothing happens.
There is no idea so big or radical that it will circumnavigate this law. I've seen it many, many times.
But the question I want to explore is: Why? Why is it that our opinions, ideas and even data insights so often fail to make the dramatic impacts we want them to? Why do our 'no-brainers' so often end up very firmly being huge surprises?
I believe there are three fundamental reasons behind this…
1. Unconscious and illogical behaviour
When we come up with ideas for our website, we are using rational, logical thinking and pattern recognition in order to devise solutions to problems.
We observe that customers are spending a long time on the shipping page and then abandoning here at a high rate. We also observe that people who have previously been to the delivery page have higher conversion rates. We deduce that people must not have enough information about some aspect of delivery; either how it works or the fees.
We then analyse the experience on these pages and try to understand how to make it clearer. We come up with a solution and test it.
We're using reason and logic in order to: understand the likely customer experience; identify issues; uncover likely causes; and create solutions.
Herein lies the problem: customers are not using reason and logic when they're shopping on your website. Mostly they're not even thinking at all, or are thinking about something else entirely. Their behaviour is mostly unconscious and also largely irrational and illogical, in the sense that is driven by bias and forces beyond their comprehension.
John Hopkins University conducted a wide-ranging survey study that concluded two-thirds of the US population are actively eating less meat. Conversely, in the same time frame, actual per-capita meat consumption is at a record high. Americans are saying (and probably believing) they eat less meat than they used to but are actually eating more than they ever have in their history.
What's going on here? Well, nobody knows exactly and that's the point. People are eating more meat, regardless of the fact that they know it to be bad for them and the environment. Why do they do that? We don't know and neither do they!
The field of behavioural economics was meant to be a scientific approach to uncovering the laws that govern behaviour in such a way that it can be predicted and 'nudged'. However, as many famous experiments simply cannot be repeated, this discipline is fast becoming a kind of pseudo-science and rapidly losing academic respect.
Our own unconscious mind works in ways we can never really know, but that we post-rationalise after the fact. More importantly, if we don't know how our own minds work, how can we possibly know how other people's minds work?
Consequently, you cannot rationalise the behaviour of consumers!
2. Personal subjectivity and cognitive bias
Most people intuitively think that data analysis is an incredibly objective endeavour - we're just looking at data and understanding the pure fact of what it tells us!
However, it is impossible to process data or information in the brain without contextualising that data using our own experiences, beliefs, knowledge, relationships, goals and more.
Data is just data; the meaning of that data is what we add subjectively. This is called cognitive bias and it is the reason that we can and do become utterly convinced of wrong decisions.
If you learn that twice as many women as men bought product X, your own mind then answers the question about why that happened and what to do about it.
This is based on a huge number of different, entirely subjective, processes going on in the brain. The way you understand and interpret this 'data fact' boils down to your life experiences within a particular gender identity, plus your perception of other gender identities.
In turn, this is entirely driven by the context, surroundings and other people you have come into contact with. There's no way anyone could begin to explain this or come up with laws and logic about how it works for everyone, let alone one person.
At the end of the day, there is far more of someone's personal opinion in any 'insight' than there is actual data, and you can never really pick that apart to understand how it's driven.
When you think you 'understand your customers' and how they behave, you're only really understanding yourself and the way only you perceive that behaviour.
On a very simple level, if you're a middle-class 40-year-old man living in London and working in an advertising agency, how can you ever hope to understand the conscious experience (let alone unconscious experience) of an 18-year-old single mother living in Sunderland on universal credit? You don't even understand your own unconscious experience.
You do not know your customers, no matter how much you relate to them or think you are similar to them. You have an opinion about them, which is entirely your own subjective opinion.
3. The impossibility of knowing a mass
When you try to understand your customers and predict how they will behave or solve their problems, you are simultaneously trying to apprehend the consciousness of thousands or maybe even millions of people.
The human brain is simply not capable of doing this. Try it now. Think about Wembley stadium full of 90,000 people at a concert. Try and simultaneously imagine all 90,000 of those separate, highly individual people. Not even remotely possible, is it?
But this is what happens when we say 'I know my customers' - you are making the assumption that every single one of your customers is the same and behaves the same, and that you know exactly who they are.
Even if you have some clever segmentation, it's not really any different. Segmentation models typically carve up a customer base using very broad and simple factors such as gender and age, and then use highly creative license to describe those segments or even give them names and 'personas' - but this is all complete nonsense really.
How can you take a segment of people which is, literally, just women aged 18-34 and give them a name and say they're all 'fun loving, outgoing sassy girls who love to shop' - what, every single young woman fits that profile?
You cannot know a mass of people. You absolutely can and do make incorrect, entirely subjective, generalisations about a mass of people. You're effectively picking a single person out of a crowd and using them as an example of everyone else in the crowd, who you thereby believe to be identical to that person.
So, even if you could design a solution that is right for that one person (which you couldn't, based on the above two points), then you have ignored every single other person in the group. If, conversely, you base your solution on a deliberate generalisation, then you will have an incredibly bland and useless understanding of the masses (for example, people who live in the UK).
So what do we do?
This all sounds like there’s no point in customer insight or even trying to understand your customers. Wrong! What else could you do? You cannot escape subjectivity and interpretation. It’s simply a part of being human.
You need to come up with ideas and innovate. Despite the issues described and the fallibility of the insight, going through the rigor of proper insight is still somewhat better than not bothering or just guessing.
The problem is when you take your findings as fact and just ‘do them’.
Instead, what you can do with every single one of those ideas is experiment and validate!
Experimentation doesn’t care what you think; that’s the whole point of it. You might believe that your customers will respond positively to better information about delivery fees, but if you deploy that as a controlled experiment you are going to learn whether they do or not, through the way in which they actually act.
Most importantly, when you learn that customers behaved a certain way given a real binary option, this is a learning you can build on.
If they responded that way to this, maybe they’ll respond this way to something bigger? It doesn’t mean they definitely will, but it’s more likely.
Nike famously said ‘Just Do It’ - well, you should absolutely not just do it, because I guarantee you are wasting time and money, or even damaging the commercial performance of your business.
Nobody's opinion is good enough that it can accurately design solutions to consumer problems that don’t need testing.
About the author
Jonny Longden, Conversion Director, Journey Further
Jonny has helped the likes of Sky, Visa, Nike, O2, Mvideo, Principal Hotels and Nokia put analytics and experimentation at the heart of their digital innovation and engineering.
“Doing a 'bit of CRO' is not enough - every decision you make about development without experimentation has a 90% chance of wasting you money. The time is right to embrace experimentation.”
