Fake Door tests
Fake door experiments, also known as Painted Door / Phantom Button / Pretend to open tests, are creation of a button or link that allows users into a potential journey without actually making it. Instead, you inform users that it doesn’t exist yet, but gather valuable data on user behaviour and potential popularity.
There is a great need to understand user demand before investing resources. You have a lot of new features that you could build and deliver for your product/website. But there’s no telling which of these are going to be widely used by customers. Or exactly where the demand is. So what do you do?
This is where the fake door experiment is invaluable. For a tiny amount of dev work to create an entry point and measure behaviour, you can gauge potential popularity and prioritise future dev roadmaps accordingly.
In this blog, we will cover:
- What are fake door tests?
- How fake door tests work
- The benefits and advantages of fake door tests
- Factors that influence the effectiveness of fake door tests
- Real-world uses and success stories
- Best practices when conducting fake door tests
- A conclusion on using fake door tests
(yes, we’re trying to get our SEO fix).
What are Fake door tests?
Fake Door tests, also known as Painted Door, Pretend-to-Open or Phantom Button tests, are a technique used in UX research to measure interest in a feature or journey that doesn't exist yet. We see this happen quite commonly in prototype walkthroughs that users conduct on tools like UserTesting.com, where they want to gauge clicks without an existing onward journey.
Fake Door tests in the wild consider taking this approach, and applying it to a live journey such that your real users will be able to provide feedback at a much higher volume than some UX research may provide – thousands of decisions instead of tens.
To make this happen, you strategically place dead-end buttons or links into a journey, and on interaction inform the user that they’ve entered a journey that you’re not yet ready to service.
This approach allows organisations to make informed decisions about their product development roadmap, optimising allocation of effort and resources to where the demand is, which allows for optimal rewards for your effort.
Ref: https://medium.com/agileinsider/fake-door-the-mvp-before-the-mvp-32bffcec0ca2
How fake door tests work
The process of creating a fake door test is thankfully fairly straightforward. You should:
- Identify the feature to test: If a feature is easy to build or in an area of the website that receives very little traffic, or has low value, it’s perhaps not worth investing in. If on the other hand it’s difficult to build, it’s worth gauging risk beforehand.
- Create variations of the user interface: In Webtrends Optimize, you can then build your variation with the new button/link in it. Once clicked, you should also arrange for some messaging to the user to explain that they’ve reached a dead-end journey.
- Implementing behaviour tracking: This is perhaps the most important part of the whole process – you want to accurately capture user behaviour so you can then analyse potential popularity. This is useful when coupled with a threshold for success/failure that you’ve determined in advance, rather than making a decision once you see the data.
We recommend using our conversion tracking or passing data into your web analytics to measure this.
Be sure to capture KPI metrics too, so you can be certain that the frustration of a dead-end journey isn’t hurting your overall conversion.
- Run the experience, analyse the data: Once built and tested, set it live. You can start at a low throttle e.g. 10%, and grow from there depending on your risk. Over time, you will be able to build a good understanding of popularity for the potential feature, and thus make a decision on whether or not to build it.
Benefits of using Fake Door tests
The primary advantage of Fake Door tests is the benefit of using it as a precursor to serious development.
You probably have a finite resource of developers, and a limitless pool of ideas for what you could build. The Fake Door allows you to better utilise your resources, by pointing them to high-value features and experiences and away from things that would waste their time.
This is therefore less about the mitigation of risk (beyond the cost of resources), and more about the ability to exploit opportunities better.
There are also potential long-tail benefits. For example, your Fake Door test could include a 1-click sign up for Beta Testers, in which case when your MVP goes live you have some users already committed to helping you.
Ref: https://userpilot.com/blog/fake-door-testing/
Factors which influence the effectiveness of fake door tests
Bug or fake door? The first factor to consider is the presentation of the painted door to your users. It should do well to blend into the experience or look as it would do when you launch the real feature, but the dead-end should not be so poor as to look like an error.
If someone moves something on your desk, would you notice? Your user base and their curiosity matters here. We operate in an industry where users are naturally curious, and so painted door experiments come with the bias of knowing users will click on the options just to see what’s there – not because they’re genuinely interested in the feature, but because it’s different to what they’re used to seeing.
The quantity of fake doors is also a consideration factor, mainly for the overall user experience. Too many will undoubtedly frustrate your users, at which point your more important metrics are likely to suffer.
The final one is success criteria. It’s easy to be excited about users interacting with your Fake Door, but do 5 interactions mean it’s worth building? Does 10? How about 1000? Everyone has a different threshold, and you should decide what your tolerance is before running the painted door – not after.
Real world uses and success stories
We’ve not written anything this far down the blog, but will track behaviour to understand if we should next time. If nobody gets this far, is it worth us building polished case-studies to explore?
The majority of the effort is here, and so the fake door is to build an area and see if anyone chooses to explore it.
See – it doesn’t just need to be a website or application – fake door experiments can be used in digital content too!
Best practices when conducting fake door tests
In a nutshell, make sure you consider:
- Clearly defining your objectives and thresholds beforehand. Not when you analyse the data.
- Build a realistic experience. Don’t influence interaction by making it stand out any more than it would do when built “for real”.
- Measure behaviour. The most important part isn’t building the fake door, it’s measuring interaction with conversion tracking in Webtrends Optimize or events in your web analytics.
- Combine with other methods. Perhaps the entry point isn’t great, but the feature would be incredibly valuable. Interview customers and really understand value alongside the quantitative data that the fake door test would give you.
Conclusion
The fake door test is a way for product owners to understand the potential popularity of features or journeys long before asking designers or developers to make anything. By building a small dead-end journey, you can gauge popularity and prioritise internal resources more effectively.
