Virgin Wines Dynamic Approach to Testing

Most brands measure the success of a test by whether it wins. This is something that Virgin Wines strive towards, but what sets them apart is also their desire to learn.

As one of the UK's largest direct-to-consumer wine retailers, Virgin Wines have built their reputation not just on the quality of what's in the bottle, but on the quality of the experience around it. Understanding their customers, removing friction, continuously improving the journey from browse to basket, and of course the quality of their product, is what keeps people coming back. Webtrends Optimize have been lucky enough to work with the Virgin Wines team for more than 8 years and have seen their dedication and desire to consistently improve and evolve.

The two tests in this case study both delivered strong results. But the insights that will shape Virgin Wines' next moves weren't only in the headline numbers — they were in what the data revealed about how their customers actually behave. That's the difference between testing to win and testing to learn.

Virgin Wines knew their customers had preferences. Research into buying patterns showed that visitors were more likely to purchases from a winery or grape grower they'd bought from before/ they already had in their basket. Therefore, an opportunity was identified that showing the right product at the right time would make the path to purchase even easier for customers. This would provide support to shoppers at a key stage of their journey as well as encourage them to increase their average order value.

Virgin Wines therefore tested a Product Recommendations carousel on Product Display Pages which would recommend other wines by the same grower. This was an AB test that ran for 58 days, with half of visitors seeing the control version with no changes and the other half seeing Product Recommendations.

This AB test had some great uplifts, with a 9% increase in visitors converting. It is great to see a hypothesis backed up with results, and on this occasion, we predicted that better product recommendations would ultimately boost conversion rates.

However, in our opinion, this is not the best thing about this test.

After analysing users' journey when experiencing the test, Virgin Wines learnt that due to the average scroll of mobile visitors, approximately 52% of the page was missed. Users needed to scroll right to the end of the page to see the test, and the majority did not. This means that users who were in the test environment were not actually seeing the test they should be.

This means the learnings from this test were in abundance. The uplift proved the idea of product recommendations on the PDP was a worthwhile investment, and still a good winner. However, we also learnt that the pattern of behaviour from users is key and making sure they can access the test they are in and what barriers they may face is important.

Knowing Me Knowing You is the name of this test, something that was labelled before any findings and results, and it has become more reflective than we predicted. Knowing your users is key for providing content and experiences that work for them.

This is a special test as it had a fantastic uplift in conversion, which is always great for a brand, as they know they are improving their sales. But also learning things about the way your users utilise your website is just as important for future experiments, which this experience also delivers.

On another occasion, Virgin Wines identified that many visitors struggle to complete a case minimum for Mix Your Own Cases. Therefore, another version of Product Recommendations was tested to see if offering a feature that allows visitors to complete their case based on Virgin Wine's recommendations reduces the level of effort needed and improves conversion to sale.

The original test was a pop-up message that said, 'Would you like some help choosing?' in bold and a message below that said 'We get it, there's a lot to choose from. In just one click, we can help you fill your basket'.

This test ran for approximately 2 weeks before the Virgin Wines team determined that the messaging was too wordy and overwhelming for visitors and made it more likely they would close the pop-up before reading the copy and clicking on the CTA. Therefore, new copy was introduced to improve the visitor's experience. The new copy read as 'Would you like some help?' followed by 'In just one click, we'll fill your case based on your current choices' with a clear CTA to Fill Up My Case.

This goes beyond just waiting to see if a test is a winner or a loser and demonstrates the goal of giving customers the most straightforward route to buying wines they will enjoy and return for.

This simple copy changes increased interactions by 1.38%, which suggests visitors resonated more to the simplified version of the test. The results were compelling, with 40% of customers in the test converting.

The pivotal point of this test was the barrier that Virgin Wines identified to users interacting with the Pop-Up. This was hindering the real results of the test as ultimately; they were trying to identify whether users would benefit from recommended products based on what was known about them and their preferences.

Moreover, based on the results and findings of this test, Webtrends Optimize recommended expanding the test further. The same logic could be applied to more parameters, such as a gifts or spirits, where a pop-up could suggest add-ons or other products that would go well with products already in their basket.

This test is another strong example of how testing features, like product recommendations, can increase conversions and also ensure customers are buying products they will be happy with. The improvement in conversion after the copy change affirms the fact that pop-ups should not too complicated or wordy. Another fantastic learning for future experimentation strategies.

Both tests delivered results worth celebrating. A 9% uplift in conversion and 40% of customers completing their case are numbers any brand would be pleased with. But the most valuable outputs from this work weren't in the win column — they were in what Virgin Wines chose to do with the findings.

It would be easy to look at a 9% conversion uplift and call it a success. And it is. But if Virgin Wines had stopped there, they would have missed something far more useful — the knowledge that most of their mobile users never even saw the test they were in.

Finding out that over half your mobile users never scroll far enough to see a test is not ideal. Discovering that your pop-up copy is pushing people away before they even read it isn't what anyone wants to hear or what you set out to achieve. But Virgin Wines' reaction to these things is what makes the difference; they acted on them, iterated, and came back with something better.

That's the mindset that separates brands who test from brands who learn. And iterating off the back of both good and bad results is an extremely underrated way to maintain high standards, keep customers engaged and continue to improve conversion rates.

After 8 years working together, it's what continues to make Virgin Wines one of the most rewarding partnerships we have — not because every test wins, but because every result is an opportunity.