Price Data, the Internet Arms Race and You

A reader sent in the following:

Hey Mark, hope all is well with you these days. Saw this article about Web price scrapers and how companies are using them to optimize their pricing (possibly instantaneously). I thought you’d be interested:

Yes, I’m interested. Fascinating article, but here is the gist.

Some companies use bots to surf competitor websites and automatically collect their prices. This makes a ton of sense. If you want to price competitively, you need to know where your competition is pricing. If you collect this data often, you can detect when your competition changes their prices. If you want to have the lowest price on other websites like PriceGrabber, then you need to know how much competitors charge.

Some retailers don’t want their competitors knowing their prices, so they use technology to make it very difficult to do web scraping. Then companies figure out how to get around it. Then companies get better at blocking bots. This data war will go on, maybe forever. Absolutely fascinating.

Unless you’re a high-volume retailer, this probably isn’t directly applicable to you, but there are some great lessons we can take away.

  • First, you should put together a program to systematically and periodically collect your competitors’ prices. This is not always easy, but understanding your competitors’ pricing is crucial to understanding your buyers’ purchase decisions.
  • Second, your competitors change prices once in a while. You get to respond, but you can only respond if you know that it happened. Watch for price changes.
  • Third, web scraping is awesome. Unless you’re in the highly data competitive space of retail, odds are very good that you will be able to use a bot to scrape your competitors’ prices. Look into this. Several years ago, my company (a B2B company) used a bot to scrape prices from a distributor that put prices on their website. This resulted in fabulous information for intelligent pricing decisions.

You probably won’t get into an Internet arms race with your competitors, trying to scrape or hide your prices, but this article reminds us of the power of knowing our competitors’ prices, and that we can likely find them relatively easily. Put together your competitor pricing data collection program.


Mark Stiving

Mark Stiving

Mark Stiving is an instructor at Pragmatic Institute with more than 20 years of experience in business startup, development, management, turnaround and sales and design engineering. He has helped companies create and implement new pricing strategies to capture more from the value they create, and has consulted with Cisco, Procter & Gamble, Grimes Aerospace, Rogers Corporation and many small businesses and entrepreneurial ventures. He has led pricing initiatives as director of pricing at Maxim Integrated and as a member of technical staff at National Semiconductor. Mark also has served as president of both Home Director Inc. and Destiny Networks Inc. and as an assistant professor of marketing at The Ohio State University. Mark also is the author of “Impact Pricing: Your Blueprint to Driving Profits” (Entrepreneur Press, 2011). He can be reached at

(0) Comments

Looking for the latest in product management news, articles, webinars, podcasts and more?