How Data and AI Are Revolutionizing Products and Product Management

By Michael Li February 27, 2019

DATA IS REVOLUTIONIZING THE WORLD. IBM estimates that the world is producing 2.5 exabytes of data each day. That’s enough hard disks to cover more than six NFL football fields when laid out flat. Gartner estimates that we will be spending $232 billion on data from 2011 to 2016. According to Statista, that’s the total Amazon ($136B) and Target ($79B) combined. And McKinsey estimates there’s a talent shortage of 1.5 million datasavvy product and business managers. That’s larger than the population of San Diego.

Companies are taking advantage of the data revolution, combining big data, machine learning (ML) and artificial intelligence (AI) to find predictive and valuable patterns in their data to lead them to new products and services that people haven’t even dreamt of before. In a world where data defines so many products, data literacy is increasingly necessary for product managers.


Data Products are Everywhere

In today’s data-rich world, our applications are becoming increasingly data-driven. Everything from executive business decisions to automated product recommendations have data at their heart. And we see them everywhere. Products like Alexa and Siri are leveraging AI to understand our voice commands and communicate (semi)-intelligently with users. Netflix uses its vast trove of user preferences and ML recommendation engines to provide personalized recommendations for TV shows based on what users with similar tastes and preferences like.

Facebook uses similar technology to populate its addictive newsfeed with stories from your friends that keep you hooked. Amazon employs its wealth of commercial data to offer product recommendations and targeted discounts for goods it knows you’ll want. Google uses all the data it’s harvesting from webpages, search histories and user surfing behavior to find everything from menus for local restaurants to scores for historical sports games.

"In a world where data defines so many products, data literacy is increasingly necessary for product managers."

tweet

And it’s not just digital-born companies based in Silicon Valley that are leveraging data. We’ve worked with many non-digital-born companies that are learning about the value of their data.

  • Banks use their data to better price loans. Their strategies go beyond traditional FICO scores and credit histories to using more sophisticated ML techniques and novel data sources.
  • Insurance companies leverage telematics data to identify dangerous-driver behavior when offering auto insurance and data from wearables or gym membership attendance records to understand exercise habits when offering life or health insurance.
  • Pharma companies comb through electronic medical records to identify potentially lucrative off-label uses of their drugs.
  • Telecom companies are merging store, call center, website, email and mobile data to improve digital confinement—identifying unclear digital interfaces or communications that drive users toward costly call-center or on-site interactions.

This is not just a passing fad. At its heart, the data revolution comes from three macroeconomic trends that show no signs of slowing:

1. The relentless drop in the cost of computation. This is not confined to processing (Moore’s law); it also applies to memory, hard disk and network bandwidth. This means that data is becoming increasingly cheaper to store, transfer and process, making the analysis of data feasible.

2. As technology becomes more ubiquitous, more and more data is being captured. We are collecting data when people surf the internet not only on their computers, but also on their mobiles, tablets and wearables. And with the growth of the industrial Internet of Things (IOT), we will be collecting data not just from humans, but also from machinery all over the world.

3. As more companies are embracing data and AI to be smarter and more personalized, consumers are beginning to demand it as table stakes. Data and AI are becoming competitive issues.

In a world where data defines so many products, data literacy is increasingly necessary for product managers.


Incorporating Data Into Products

Data has fundamentally altered the way product managers need to think about their products. When we consider how personalized and addictive Facebook’s newsfeed is or how Amazon is able to recommend the next product you want to buy, we can see the business value of embracing data.

At a superficial level, embracing data appears like a move toward the much-vaunted evidence-based decisionmaking—a move away from HiPPO (Highest Paid Person’s Opinion) decision-making toward relying on empirically proven results. But using data in products goes far beyond that. It subtly shifts to a very different way of thinking about products.

Since grade school, we’ve been taught to apply rules of logical deduction to our reasoning and decisionmaking. Unsurprisingly, such deductive reasoning has been codified into our business rules: Deliver the product to the customer once they’ve paid for the product. Grant existing customers a 10 percent loyalty discount for continuing with our product.

In the data-driven world, these business decisions become much more fluid. We don’t use data simply to confirm business rules, but to generate and optimize them. By leveraging purchase data, we might understand how to optimize customer discounts to increase sales and revenue. We can give selective discounts on products that lure customers back, bundle common goods to increase basket size, or offer time-sensitive discounts to induce customers to buy as soon as possible. None of these ideas is new in retail, but each has a myriad of accompanying decisions, such as: How deep is the discount? On what products? To whom should we offer a discount? When should we offer a discount? When should we stop offering a discount?

By leveraging data, your strategy can continuously optimize these parameters—and many more. It can track trends to adapt to seasonal changes, discover product groupings or personalize for customer preferences. And we leverage these ideas to constantly optimize for revenue.

Instead of thinking about products as a static collection of fixed business rules, we should think of them as dynamic, living products that are constantly adapting to changing business circumstances. Rather than starting with what a product should do, product managers define what the objectives are and where the boundaries are, and then let the underlying software optimize within those bounds.

This frees up product managers from thinking about the mundane day-to-day features and from interminable meetings to decide on small product features, instead allowing them to think more strategically about how to measure success, new types of data to incorporate into decision-making and the strategic direction of the industry.


A Change in Mindset Around Data

Much of today’s data gold comes from what was previously “data exhaust,” collected data that’s a byproduct of other business processes. For example, programmers used to maintain web logs. These logs were kept around for a few days and not well-structured—their original purpose was primarily for programmers to comb through to debug customer complaints of bad interactions so they could find mistakes in their code. Ingenious product managers began to realize that these logs were a potential gold mine of data about their customers, telling them things such as:

  • What pages are people visiting and in what order?
  • Are they using the product as we intended or clicking around in new or unexpected ways?
  • Can we use this data to improve our site layout, or even build a recommendation engine to personalize pages for customers?

This new way of thinking requires a massive mindset shift. Rather than seeking data to answer their questions, they’re looking for the questions to pose of their data.

These trailblazers began leveraging data for their products, turning what was previously considered trash into one of the most critical competitive advantages a company can have. Correctly identifying and harvesting value from data is at the heart of how product managers drive value using data.

This mindset shift isn’t happening only in Silicon Valley. Traditional legacy industries are following suit. The pharmaceutical industry is an excellent case in point. As an industry with long-established business practices and complex government regulation, it would seem like an unlikely place to find innovative approaches to data. And yet, we’ve worked with a number of pharma giants that are tackling this challenge. Traditionally, data and analytics in pharma have been geared toward passing multiple rounds of FDA clinical trials. The problem was clear: demonstrate the safety and efficacy of your treatment with statistical significance. The data was collected specifically for this purpose. Human test subjects were enrolled, given a treatment (or not) and then monitored for either improvement in the target condition or adverse side effects.

This data wasn’t cheap to collect—the cost can run into the billions of dollars. And once the data was collected, the analysis was relatively easy. The pharmaceutical companies who are leaders in data have flipped the paradigm on its head. They’re combing through their data exhaust and those of other medical services sniffing for cheap data that might be valuable. The questions they’re posing aren’t always clear or well directed. Instead, data scientists and product managers need to understand how to leverage data science techniques to answer valuable business questions based on the data they can find.

This new way of thinking requires a massive mindset shift. Rather than seeking data to answer their questions, they’re looking for the questions to pose of their data. And the analysis is much more creative. Instead of applying the same cut-and-dried analysis prescribed by the FDA for purpose-collected data, they’re exploring novel data sources and looking for the right questions to pose that can lead to valuable products and services.


Data-Literate Product Managers

For product managers, this mindset shift means there’s a new set of skills to master as they become better versed in data science, machine learning and artificial intelligence. As data starts to play a fundamental role in the product, product managers face an increasing need to become data literate.

  • Do you understand the difference between structured, semi-structured and unstructured data?
  • How are today’s data analytics different from their forebears?
  • How do product leaders go about cataloging their own data and thinking creatively about building products of their own?
  • Once product leaders understand what data they have, how do they envision potential products they can deliver to quickly establish traction in a pilot and then iterate?
  • And once pilots have confirmed value, how do they think about productionizing a product to make it robust?
  • How do business leaders scale the impact of data across an entire organization?

No product manager would ever try to manage a mobile product without having used a smartphone. Similarly, no product manager should manage a data-driven product without understanding the fundamentals of data.

Michael Li

Michael Li

Michael Li is the founder of The Data Incubator (now part of Pragmatic Institute), a data science training and placement firm. As a data scientist, he has worked at NASA, Foursquare and Andreessen Horowitz. Michael is a regular contributor to VentureBeat, The Next Web and Harvard Business Review. He earned a master’s degree from Cambridge and a Ph.D. from Princeton.

Looking for the latest in product management news, articles, webinars, podcasts and more?