Use Beta Tests to Create Customer-Validated Roadmaps

November 08, 2016

An effective product launch is important to product success, but equally important is having a solid product roadmap that lays the groundwork for your team and the future of your product.

Building that roadmap, however, is one of the biggest struggles product management professionals face. How do you decide what goes on the roadmap? How do you prioritize features? How do you know you aren’t missing something critical? These are the types of difficult questions you must answer to build a viable roadmap.

                                                      

At the same time, products are increasingly driven by the customer experience. In fact, according to Forrester, we’re in the “age of the customer,” where building an amazing customer experience is a key driver of revenue growth and product success. And the best way to develop a customer experience that delights your target market is to involve customers in the product development process. This means your product roadmap should be powered by feedback from real customers.

But, too often, roadmaps are created in a vacuum. Product teams develop plans based on internal priorities, anecdotal feedback from vocal customers or what they believe their customers need. The result: roadmaps that miss the mark or don’t prioritize what customers truly want.

So, how do you create a roadmap driven by customer feedback? The answer is beta testing.

Beta testing is the phase that occurs after the internal quality-assurance team has put your product through the wringer in the lab, but before your product launches. During beta, you put your new product (or the latest release of an existing product) into the hands of real users in real environments to see how the product performs. You collect bug reports, feature requests, journals, surveys and other feedback to find weak spots and improve the product quality and overall customer experience.

Since beta testing often occurs right before the launch of a product, it may seem that it can’t contribute to that product’s roadmap. (At this point in the product lifecycle, it’s often too late to make major changes based on the feedback collected during beta.) But the beta phase is the perfect time to gain customer validation of a product and collect the data needed to plan for future iterations.

Centercode’s research team has found that a large number of companies performing beta tests leverage customer feedback to inform their product roadmaps. In a survey of 250 respondents (including product managers and others involved with their beta phase) from 215 companies, 73 percent said that beta testing has a moderate-to-high impact on enhancing their roadmap. Only 7 percent of respondents said that beta testing has no impact on enhancing their roadmap (and we’d argue that they just aren’t fully leveraging it for this stage of product development).


What Beta Brings to the Table

Beta testing offers unique insight into the complete customer experience with your product over a period of weeks or months and uncovers the true usage of your product in customer hands. It demonstrates whether your existing feature set meets customer expectations, the first step in building a good product roadmap.

Next, beta testing can uncover unmet needs and allow you to dig into those situations to understand the exact scenarios where new features are needed. You’ll gain a sense of the frequency of that need and the percentage of your customer base that has it. All of which you can use to build detailed use cases to create a solid foundation on which to develop future features.

You can also survey beta testers about existing roadmap plans to see whether the intended additions interest them. This will help validate, prioritize and flesh out the existing roadmap. Finally, testers will provide a pile of new ideas to inspire future iterations or additions to your roadmap.

Real customer feedback can be a gold mine of useful product information about the current—and future—versions of your product. But to make this trove of data reliable and actionable, you’ll need to run a beta test that’s specifically designed to help reach these goals.

How to Run a Beta Test that Fuels Your Roadmap

Simply running a beta test isn’t enough to ensure you’re getting the feedback you need to power your product roadmap. The beta test has to have the right objectives and collect the right information from the right people.

Recruit the Right Testers

The first step to ensure the success of any beta test is recruiting the right testers. These testers should not only be members of your target market, but they should reflect the demographic breakdown of your customer base. This means beta testers should not be friends, family or employees. And if 30 percent of your market is made up of Mac users, 30 percent of your beta testers should be as well.

Much of the feedback that product teams receive from their customer base is either anecdotal or comes from a vocal minority that shares their opinions via support channels. It can be tempting to listen to these voices when making product decisions, but they often don’t reflect your complete target market. Having a beta tester team that closely resembles your market will provide a more complete picture of the customer experience with your product and provide data that helps you draw useful conclusions.

Collect Feature Requests
Many product teams are hesitant to collect feature requests during beta tests because they know they don’t have the runway to implement those requests now and don’t want to disappoint their testers. However, there are reasons why feature requests should be included in every beta test.

Collecting feature requests serves an important psychological purpose by allowing testers to share ideas and feel that they have been heard. If testers aren’t given a structured way to make suggestions, they’ll end up submitting them as bug reports or discussion topics, which will muddy the data. By creating a dedicated place where testers can submit feature ideas, you provide a centralized place to process, prioritize and respond to those ideas, even if you don’t have immediate plans to use them.

You can also capture more general needs by calling them “suggestions” rather than “feature requests.” This will encourage testers to submit ideas and unmet needs, instead of limiting themselves to requesting specific features. Then it becomes your job to analyze the feedback, identify the feature to fix it and add it to your roadmap for design and implementation.


Create a Scoring System

Include the ability for beta testers to vote or comment on ideas that other testers submit. Voting makes it easy for popular ideas to rise to the top, and commenting allows the tester team to flesh out the ideas for you, providing additional context or use cases. Testers may also point out potential risks or downsides of a new feature, helping you understand the current concerns of your target market.

At Centercode, one of our key successes has been the implementation of a scoring system for feedback. This feedback scoring system is based on popularity (which considers things like votes and comments), category (which refers to the part of the customer experience it touches) and other criteria that allow the strongest ideas to rise to the top. If you see a critical need that isn’t being met, this information can help you prioritize ideas and alter your roadmap. These are the ideas or features that, if implemented, could have the most impact on your product.

You can determine the criteria that make a new feature the most important to your product, and then develop a rubric to identify the most valuable ideas. This will ensure you collect the relevant information in your beta test to help prioritize potential product additions for the future.


Survey Testers on Planned Features

To validate features that are already part of your roadmap, directly survey your beta testers about them. Would they use them, and how would they use them?

While this may be speculative data, your beta testers will likely have the best insights about future feature ideas. You will be able to analyze whether different segments of your market have different opinions about the planned features and have quantifiable data on which features your target market would likely use, along with which features they have no interest in. Beyond just validating your roadmap, you could also use the results from this survey and compare it to the feature requests and ideas that were submitted during beta to gain a deeper understanding of any unmet needs or to find holes in your future feature plans.


Collect Daily Journals

Surveys, bug reports and feature requests capture snapshots of a customer’s product use. Journals, on the other hand, are a great way to collect details of the day-to-day customer experience. Journals are more free-form, allowing testers to talk about their experiences without feeling like they must address a specific bug or need.

While it may be more difficult to discern quantifiable data from journals, feedback in the entries can help uncover unmet needs or other small issues that users might not think are worthy of a bug report or feature request. Journal entries also provide invaluable insight into how customers use the product and showcase real-life use cases about features that testers may enjoy or find frustrating.

This information can be vital when analyzing your product’s customer experience and illustrating the value of certain features to key stakeholders within your company.


To Sum It Up

Using the feedback collected during beta testing to fuel your product roadmap ensures that you drive your product closer to the needs of your target market. You’ll gain detailed feedback about where your product hits the mark, what gaps exist and whether your plans will fill those gaps. With this intel, you can be confident that you’re releasing the best-possible product for your target market.

Looking for the latest in product management news, articles, webinars, podcasts and more?