A Model for Metrics-Driven Feature Prioritization

August 10, 2007

As a product manager, one of the things I rarely see in any software company is a true metrics-driven approach to product planning and enhancement prioritization. What do I mean by a metrics-driven approach? It is an approach that relies heavily on statistically-valid samples of broad-based customer data that is used to understand customer needs and prioritize product enhancements.

In most companies, feature prioritization does have a basis in customer needs and requirements, but this information is typically collected rather anecdotally from a small number of customers and then extrapolated to apply across the larger customer base. Along with this information though, a lot of what is prioritized is based on the “decibel factor” of various parties involved in the process. This could be a very loud customer demanding an enhancement be made, or a loud salesperson complaining that a big deal is possible if only certain features are added, or a loud development manager refusing to add something because it is not a good use of the engineers’ time.

The problem is that without good analytic information, it is very difficult for product managers to defend any given prioritization, and whatever work the product manager has done can be pushed aside to make way for other deciding factors (like soothing loud and annoying customers). The question is how to create a scalable, repeatable and efficient process for getting the analytic data to support feature prioritization.

Defining the formal process

Over the past three years at my company, we have worked to develop a process that delivers exactly that statistically-valid broad-based customer data described earlier. We call it PEP, short for Product Enhancement Process. OK, I agree, it could have a fancier name like “CustomerPulse” or “ProductVision,” but PEP is simple, easy to remember, and to-the-point. It also translates well across languages—an important point as you will see later.

Not only has PEP helped us make better decisions about what we will, and will not, add to our products, but the gathered data is used by groups in the company well beyond Product Management, including Marketing and Technical Support, to improve their knowledge of customer needs and problems.

In addition, many customers love PEP because it gives them a clear voice into product direction, and clear visibility into why and when enhancements will be delivered. And of course, the virtual “decibel factor” of hundreds of customers requesting particular functionality is much louder than the noisy few.

Before I describe the process, there is one caveat. The enterprise software company I work at has a large, global customer base. We also hold regular Regional User Group meetings across North America and Europe, and PEP leverages these User Groups extensively. I mention this because not every software company fits this profile, and the process as we have implemented it may not be possible for other companies. That said, by describing the process, the benefits it has provided, as well as sharing some of the lessons learned, hopefully others can benefit and implement their own versions of PEP as they see fit.

The model

The goals of PEP are to acquire broad-based statistically-valid data that help understand customer needs and prioritize product enhancements. The process must be scalable across the customer base, yet deliver consistent data that is easy to tabulate, analyze and report back, both internally and to the customers.

PEP is a user-centric and cooperative process. It is conducted annually and consists of the following five major steps:

  1. Soliciting enhancement requests from the customer base
  2. Create a standardized product enhancement survey that can be used worldwide, and that is based on consolidated customer input, corporate objectives, competitive issues and market trends
  3. Conducting the survey
  4. Analyzing the survey results and factoring this information into our overall product planning process to develop priorities for upcoming releases
  5. Responding back to the customer base

Soliciting enhancement requests

Product enhancement requests come in year-round from customers, partners, our professional services teams, sales consultants and technical support. Previously, we had used a bug tracking system to log and track feature requests, but to be perfectly honest, the system was somewhat old, and while adequate for tracking bugs, it was not suitable for managing feature requests.

We found the best system was to set up a form on our external website so anyone could submit feature requests whenever they wanted. As requests are submitted, the data is stored in a simple database, but also forwarded directly to Product Management, via email. We not only capture feature request details, but also the contact information and problem descriptions. Receiving this information via email enables us to forward requests internally to interested parties, and also to respond easily to the submitter to gather additional details if necessary.

Consolidating customer input

This is probably the most difficult, as well as most important step in the process. The success of the entire process rests on the relevance and accuracy of the information collected via the survey. Given the volume of feature requests that are received during the year, it is clear that not all of these requests can be included in a single customer survey. A large part of the process of survey creation is pouring through the enhancement requests and identifying those requests that have the following characteristics:

  • Are likely to be relevant to many customers
  • Are likely to be implemented if they rank highly amongst the customers
  • Will move the product forward in the intended direction it needs to be taken

These criteria are rather straightforward, but can be easily forgotten. It is very easy to get enamored by a “cool” feature submitted, but realistically is not something that you would implement. No point in asking for feedback on features such as this. The proposed features should all help improve the product in a meaningful way and fit in with the overall strategy and direction that is set for the product.

Aside from the specific features requests coming from customers, the survey creation phase allows product managers to add in additional features or enhancements that are already under consideration or in development for future releases. The survey is a means to identify customers who may be able to provide additional requirements or detail.

There are numerous ways to format the survey to collect the needed metrics. The format could be to rank individual features on a linear scale (e.g., 1-5 or 1-10) of importance. It could be to rank features relative to one another in importance (e.g., given a list of possible enhancements, which is the most important, the second most important, third most, etc.) Another format could use a “purchase the feature” model, where people are given a set amount of fictitious money, and they can allocate the amount they want to spend against individual feature requests. Anything unimportant, they can ignore, while anything of importance they can “spend” the money at a level they see fit, until their wallets are empty. Regardless of the method, the objective is to ensure the process is not too complex or onerous causing low response rates or invalid results.

Conducting the survey

The survey is conducted on paper at the User Group meetings. One may ask, given all the technology available to us, why conduct a paper survey? The main reasons are:

  • Response rate
  • Need for collaboration and discussion during the survey
  • Benefit to customers in attending the User Group meeting

The main reason for a paper-based survey is the high response rate. Our response rate for paper-based surveys averages 70%, whereas a response rate of 10%-15% for Web-based surveys is considered high. Keep in mind that this is a fairly detailed survey, and many people have neither the time nor the patience to fill out extensive forms via the Web. At the User Group meetings, time is provided so that people can complete the survey, discuss with one another ideas and questions about the survey, and have questions answered by a facilitator. The model of the survey is meant to be collaborative, and our experience has been that users find the group discussion a very valuable part of the process.

Analyzing the results

If the survey makes it easy for customers to fill out and rank their feature requests, tabulating the information is very straightforward. In our process, we set up a small database, enter the data, and then use a reporting tool to slice-and-dice the numbers as needed.

Aside from ranking the top N requests, data can be viewed in many different ways. It is very easy to slice the data geographically, whether by User Group, by region, continent (e.g., Europe vs. North America), or by job title, for example. One interesting pattern in our survey data was little variation in rankings across different geographies. While the exact ordering of the top feature requests was slightly different between Europe and North America, there was a significant overlap in their top-ranked requests. As a product manager, this was very satisfying because it meant we could address many global customer needs with a common set of product enhancements.

Responding back to customers

In my opinion, this is the second most important step, next to the actual survey creation itself. Without a feedback mechanism to the customer base, the process is a black hole and lacks credibility. Customers will refuse to participate in an extensive annual process if the information loop is left open. We made it a policy that every User Group that participated in the survey must receive a response at their next meeting providing them with their top ten feature requests, along with a comparison of the overall rankings and information on development decisions for the feature requests.

Benefits and lessons learned

Overall, the Product Enhancement Process is a real boon to our development efforts and to our customers. Our executive management has seen the benefits and Product Management has received kudos for implementing PEP. But, we have learned a few lessons along the way. For example:

Some customers brought preconceived notions based on their experience with product enhancement sessions from other vendors. Many of those experiences were either adversarial or perceived as fruitless exercises, so it a challenge to convince them that our method would be effective and positive.

There were cultural differences that had to be addressed when we went international with the process. Different countries have different perceptions that had to be taken into consideration. For example, some countries viewed the written survey as a test and preferred to participate via small group discussion. For some countries, we had to translate the surveys into native languages in order to solicit responses.

When responding back to the customer base with results, expectations varied quite a bit. Many were satisfied with high-level summary data, while others wanted more detailed information on a feature-by-feature basis. We worked with those users and found a reasonable middle ground where the vast majority of customers were satisfied with the level of feedback provided.

Finally, for Product Management, there have been many additional benefits from PEP. Aside from executive management recognition, PEP has helped us streamline our work and get necessary and valuable functionality to market faster than we could previously. It has created a true dialog with our customers and helped us build personal relationships with them that we did not have previously. And of course, we have statistically-valid data on which we can make analytic decisions regarding our product plans, and defend our decisions if needed. In short, PEP is a truly scalable and repeatable process that has become a strategic advantage for our company.

To find out more about market-driven prioritization, attend Pragmatic Marketing's courses.

Looking for the latest in product management news, articles, webinars, podcasts and more?