Monitoring Metrics Puts You in the Driver's Seat

By Barbara Nelson August 09, 2007

Edward Deming, the quality guru, says, 'You cannot manage what you do not measure.' Mature businesses rely on metrics to gauge product performance in the market as well as monitor the cost of building the product.

Product managers in technology companies typically don't spend enough time analyzing product metrics. The numbers are sometimes hard to obtain and the volume of tactical activities they are faced with is overwhelming. When they do manage to carve out the time to figure out what's next, they don't know where to begin. As companies downsize to bring expenses under control, more work piles up on the already overworked product managers who remain. And yet, executives expect product managers to run their products like a business.

In the fast-paced world of technology, product managers need to balance time-to-market pressures with risk mitigation. Market analysis, which takes time to do, is simply done to mitigate risk: risk that you're building something no one wants; risk that you can't deliver something to the market profitably.

In mathematical terms, there are infinite things to measure and analyze. But, in the world of practical product management, we can't measure everything. We need to focus on a few key performance indicators (KPI's) that give us feedback that we are on track or indication that we need to make changes before it is too late. The question is 'What should I measure and analyze?'

Think about your car. It has gauges and dials and displays. They provide measurements or metrics. But which gauges do you actually need to understand and monitor? For the novice driver, the speedometer is the first gauge you learn about. How fast are you going? You compare that to the speed limit. As you drive, you monitor your speed and either accelerate or decelerate based on external conditions. Another gauge a novice must understand is the gas gauge. How far can you travel before you run out of gas? You learn to plan a trip to the gas station to fill up the tank before it is empty. The odometer tells you how far you are traveling.

As you become more proficient, you start paying attention to other metrics such as oil level and tire pressure. A car buff or professional race car driver cares about oil pressure and revolutions per minute (RPM). When you take your car in for routine service, your mechanic will check the alignment, the timing, fluid levels, and will probably perform a 20-point computer-based analysis of other 'systems? in your car. But normal drivers don't need to know about these when driving around town.

Just as we need gauges and metrics for operating a car, we need similar metrics to run a product. This article highlights ten product metrics that product managers should monitor. Some gauges are harder to find or see than others, but all provide insight into either how well the product is performing in the market or what impact the product has on the organization. The focus of this article is on business-to-business technology products sold primarily through direct sales channels but can be applied to any technology product.

For any metric, you should first establish a baseline measurement. Then, keep track by period (weekly, monthly, quarterly, or annually) and observe the trends. Establish goals and measure how you are doing. If you are not meeting the goal, find out why. And remember, the answer to the question is probably not in the building. Leave the building and interview customers and potential customers to find out what is going on. Once you have figured out what is going on, you will need a plan or a program to improve or change the situation.

The first one you should look at is the customer gauge.

New/lost customers

How many new customers are you acquiring per month? How many customers have canceled maintenance or subscriptions per month?

These metrics tell you whether your customer base is growing or shrinking. You need both of these numbers. You might have high customer retention (few customers ever leave you), but if you are not also acquiring new customers, eventually you might not have enough business to sustain the company. Customer count is the speedometer of a product. Falling customer numbers indicate that the product will fail. This information should be easily attained from sales, customer service/technical support, or accounting. A more advanced customer metric is to calculate the lifetime value of a customer. How much profit does each customer generate? Go to accounting for this. It is an important metric, but more like the computer-generated analysis your mechanic does.

Product quality

Product quality should be monitored on an ongoing basis to make sure it doesn't become a serious problem. Set up a spreadsheet with either weekly or monthly statistics of the number of defects. A stacked bar chart can show you trends over time.

For released products, technical support typically keeps track of open defects. As soon as a product is released, it is common for the number of defects to go up as more people use the product. If the quality is high, it should be a manageable number and should settle down after awhile. For unreleased products, ask quality assurance for the statistics. If the number of open defects is not going down as the release date approaches, the release date is likely to slip. Obviously we want the Sev 1 and Sev 2 defects approaching zero as the release date approaches. (Releasing a product with poor quality will cost you money, time, and reputation--don't even go there!) Severity four 'defects' are typically enhancement requests and are often ignored in terms of product quality. Yet they indicate areas where the product fails to satisfy the customer. Product quality is the best indicator of internal health of the product; quality is the gas gauge of the product.

Technical support

Technical (or Customer) Support already measures the number of calls per product and the nature of the calls. Product management should analyze these numbers to identify areas of that product that can be improved for a better customer experience. Are there problems with installation of the product? Is the documentation too hard to understand? Is the product too hard to use? By streamlining a process, will you cut down the number of technical calls? Review the number and type of calls for your product to uncover hidden profit leaks.

Product life cycle

Although product managers usually have no control over the product development process, you should be tracking what is going on. How long does it take from the time you define requirements until you have a finished product in the market? Is the cycle predictable? Keep track of the percentage of original approved requirements that get delivered in the final product. How many 'out of scope' requirements were included? Do your developers have a clear idea of what problem to solve and for whom? The problem may be poor requirements definition, which you can control, or it may be scope creep which you need to communicate to your management. Report the facts and let management manage.

Sales support

How many non-sales people does it take to support the sales cycle? Every six months or so, do a poll of product managers, developers, and executives to find out how many hours per week they spend in direct sales efforts. This would include demo support, conference calls with prospective customers, onsite presentations, needs assessment, and other sales calls. Early in a company's life, this number is typically high. People wear many hats and everyone seems to support sales.

In order to grow the company, roles need to become more specialized, and you undoubtedly need sales engineers. Sales engineers (as part of the sales organization) are factored into the cost of the sale. Hopefully, the price point of your solution warrants the cost. (If not, you won't make it up in volume!)

Consider how to make the product easier to sell by creating sales tools that limit the amount of time non-sales people have to support sales. When non-sales people participate heavily in the sales process, they are not doing their day job. And the cost is rarely attributed to the sales activity; another hidden profit leak.

Now it's time to look at revenue and profit. Ultimately, we want to build and sell products profitably.

Product revenue/profit

Congratulations if you are already tracking both of these by product. If not, how do you know which products are worth investing in and which should be retired?

Sales should be able to tell you how much revenue is generated per product. In order to calculate profit, you need to subtract costs from revenue. The tricky part is getting the costs. Unless each discrete product (not product family) has a profit and loss statement tracked in accounting, product managers don't have access to the actual costs, even if they have access to product revenue. Finance may already have this number.

Here's a simple way to calculate costs: multiply the number of full time equivalents (FTE's) times $100,000 for an annual cost. If resources work on multiple products, estimate how much time is spent on your product to determine the FTE's. Include development, quality assurance, release control, technical support, sales, product management, and marketing. (You can use a flat percentage across all products for general and administrative costs. Ask accounting for the number.)

The last 2 metrics are not easy to get and yet management expects product managers to know them.

Market share

Everyone wants to know what market share you have. By tracking this metric over time (year over year), you can discover whether the market is saturated or ripe for expansion. If the market is growing and yet your market share is shrinking, it indicates that another competitor is growing at a rate faster than you are.

But, getting this metric is extremely difficult. Conducting your own survey to find this information is usually cost prohibitive. Research analysts typically have market share information by vendor (you will pay for it). However, you should question their research methods. Where did they get their numbers? What assumptions were made? Our experience is that all vendors 'stretch' their sales numbers and the analysts make corrections based on their own market research.

Other sources for market share estimates are technical publications and trade associations in vertical markets. They often survey their readers and members to find out who is using what products.

Marketing programs

Ineffective marketing programs might be another profit leak. Marketing communications must demonstrate a return on investment for marketing programs. Theoretically, this is good and makes a lot of sense. Practically, it is hard to do and not many companies do it very well (if at all). In order to do this well, you need a closed loop lead tracking system that allows you to track activity from the marketing program that generated the lead all the way through to a closed deal. Are the right market segments being targeted? Do the positioning and market messages resonate with the buyers? Do you have a compelling solution?

Product management typically doesn't own this, but should report the metrics (obtained from marketing).

What to do with the numbers

Start looking at your product gauges. You won't be able to collect all of the metrics at once, but determine the key performance indicators you need to start tracking. Report the baseline, begin tracking on a periodic basis, graph the trends, and drill down to find out what is going on.

Start with the metrics for customers and product quality. These are the speedometer and gas gauges of the product. When customer count and product quality are dropping, so is the product speed in the market.

Calculate the profitability of your product. If you have a loss leader, can you justify further investments? Do win/loss analysis to find out whether your product is driving business, even though it isn't profitable. If it should be retired, show some leadership and present the facts about why it should be retired and how you would redeploy the resources in a more profitable way.

To run your product like a business, act as the CEO of your product. Report the metrics at least annually, and use facts rather than opinions to help you make decisions. Publish the facts of your results to your product teams and senior management.

And don't forget to look up from the dashboard once in awhile to see where you're going.


Summary of 10 Product Metrics

For each metric, establish a baseline measurement and a target to track against. If you are not meeting your targets, drill down to find out why.



Metric

What to track

When to track it

Where to get it

What to look for

1. New customers

2. Lost customers

Plot trends of new and lost customers over multiple periods.

Calculate customer retention % (divide the number of customers this period by the number of customers in prior period)

Monthly or Quarterly

Finance or CRM system

A drop in the customer retention percentage is an indication that something is wrong. Do win/loss analysis to figure out why you are not acquiring enough new customers. Interview lost customers to find out why they are leaving.

3. Product quality

Stacked bar chart for severity 1-3 defects.

Trend graph of total defects.

Released product: weekly

Unreleased product: daily

Tech Support, Quality Assurance

Released product: is quality an issue?

Unreleased product: will we ship on time?

4. Technical support

Number of calls per product.

By category (defect, procedure, ease of use, missing features)

Monthly

Technical support

What changes to the product will reduce # of calls?

Can we improve profitability by addressing areas with high call volume?

5. Product life cycle

Time from requirements to finished product.

% of original requirements.

# of out of scope requirements

After every release cycle

Release team or track it yourself

Use facts to instill change.

Is the development cycle longer than the sales cycle?

What are the causes?

Work with development manager to improve the process.

6. Sales support

How many non-sales resources are supporting the channel:

# of resources time hours/week

Every 6 months

Survey Product Management, Development, Executives

Look for more efficient ways to utilize non-sales personnel.

Is the product too hard for the channel to sell?

Do we need more sales engineers? Sales tools?

7. Product revenue

Bar chart of actual versus planned revenue: by channel, region, market segment.

Monthly

Sales management

When forecasts aren't met:

How was the forecast developed?

Overly optimistic?

Channel not trained?

Product/market fit?

Address areas to improve

Improve future forecasting.

8. Product profit

Revenue & expense by product.

Net profit or contribution.

At least annually

Finance (if tracked), or estimate

To manage the product as a business.

To justify investment.

If the product isn't profitable, can you justify it (loss leaders)?

Are there other profit models you can use?

9. Market share

Graph trends over multiple years.

Market share by key players (3-5) in your segment.

Annually

Industry analysts, trade associations or publications, surveys

Look for changes from year to year.

Is the market shrinking or growing?

Is your share shrinking or growing?

Are there new competitors?

10. Marketing programs

Lead quality (leads to close ratio).

Return on Investment (ROI) for marketing programs.

Cost per lead closed.

Quarterly

Marketing Communications

Identify areas for improvement that Product Management can impact.

Are we targeting the right segments?

Do we have the right messages?

Do we know the buyers?

Do we have a compelling solution?


Categories: Leadership
Barbara Nelson

Barbara Nelson

Pragmatic Marketing, Inc. has continuously delivered thought leadership in technology product management and marketing since it was founded in 1993.Today, we provide training and present at industry events around the world, conduct the industry’s largest annual survey and produce respected publications that are read by more than 100,000 product management and marketing professionals. Our thought-leadership portfolio includes the Pragmatic Marketing Framework, e-books, blogs, webinars, podcasts, newsletters, The Pragmatic Marketer magazine and the bestseller “Tuned In.”

To learn more about our courses and join the growing international community of more than 80,000 product management and marketing professionals trained by Pragmatic Marketing, please click here.

Looking for the latest in product management news, articles, webinars, podcasts and more?