Beware False Precision In Your Analytics
- People are more comfortable acting on measurement that they believe to be precise
- End users rarely have insight into how those measurements were created and what assumptions they may have been based on
- Analytics professionals should always be mindful that the metrics they provide don’t appear more precise than the data warrants
When using data to understand our world, we’re often comforted by exactness in measurement. When we ask someone the time and they respond, “It’s about 3:30,” it’s not the same as “It’s 3:26.” “About 3:30” is an answer we’re not sure we can trust. Are they just guessing? Similarly, we’re unsure about a weekend weather forecast calling for temperatures “in the 70s,” or a political poll declaring our candidate “the favorite,” or the cost of an imported marshmelon at “around $10.”
We’re much happier with a forecasted high of 76 degrees Fahrenheit this Saturday, even though the average 5-day forecast for daily high temperatures is usually wrong by up to 5 degrees in either direction. It feels good to know our candidate is up by 3% in the polls, even though the margin of error is 4%. And hearing that fresh marshmelons are currently selling for $11.97 gives us confidence that someone actually looked that answer up, even though they’re just a joke from Star Trek V: The Final Frontier and don’t exist.
As analytics professionals, we need to understand this concept to keep from misleading our audiences to make bad decisions by providing measurement that appears more precise than the facts warrant. Let’s avoid specific industry examples and look at a classic American business: a child’s lemonade stand.
Annie opens her stand with a goal to earn enough profit to buy herself a paint set that costs $12.99. She estimates that each container of lemonade mix costs $1.00. The label says the container makes 8 quarts of lemonade, which she has learned is the same as 32 cups. She knows she’ll drink a couple of cups herself, so that’s 30 cups to sell. She figures out that at 50 cents per cup, she’ll have $15.00 when she sells out. She also figures out that if she can sell an average of one glass every 10 minutes on her busy street, she’ll be done in 5 hours.
Annie gets a bunch of disposable plastic glasses from her parents, sets up a table, and gets started. The weather is good, her neighbors are friendly, and entire families stop by. Mom hustles out to refill the pitcher when needed, and Annie gets her $15.00 before she even gets tired.
Well, success like this needs to be celebrated. On the school bus on Monday, Annie tells the whole story to her friend Dennis while showing off her new paints. She tells him that if he can sell one 50-cent glass of lemonade every 10 minutes, he can afford his own paints in 5 hours. So, the next weekend, Dennis gets to work.
The trouble is, Annie’s detailed calculations were all based on estimates, not facts. She’s 8 years old and has never had to buy groceries. She estimated $1.00 for the lemonade mix because that seemed like a large number to her. In reality, each container cost $2.97. Her plastic glasses were 12 ounces, so she was selling 1.5 cups of lemonade for 50 cents and her dad had to run out to buy a second container of mix before she was done. Finally, each disposable glass cost 17 cents, which she didn’t consider in her math at all since the glasses were already on hand. So, her $15.00 in revenue actually came with more than $11.00 in expenses. Her parents didn’t bother to correct her — because after a long workweek, they were so excited she had something to keep her busy on the weekend that they would’ve bought all the lemonade themselves. They would’ve even bought her the paint set!
Unfortunately for Dennis, his mom is an economist and sees this as a “life lesson.” His decision to act was based on a piece of analysis that felt precise: sell a 50-cent glass of lemonade every 10 minutes, have the money to buy a paint set in 5 hours. But his mom held him to the real math, and after a long first day at the lemonade stand, he was less than halfway to his goal. His second day saw the neighborhood market mostly dried up, and with few repeat customers, he finished a hard weekend with no new paints.
Now let’s bring this back to the businesses we work in every day. Are we delivering data points to leadership that appear to be precise but are actually built on a foundation of estimates or assumptions? Many statistical models produce values that can be misunderstood easily by readers who don’t know the mechanics of the model or its limitations. Are we performing extended calculations using estimated data, hiding the initial assumptions in layers of detail? End users can rarely see the formulas behind our dashboards, so we must ensure they don’t make decisions with false confidence.
At Forrester, we work with clients regularly to understand which forms of measurement are intended to show absolute values and which are best limited to providing directional guidance or relative comparisons. In many situations, reasonable estimates can provide insights while further manipulation to present them as “precise” values creates confusion. To learn more about how your analytics work can make the best impact, reach out to the Forrester SiriusDecisions Marketing Operations team for guidance and access our full library of published research on these topics. Not a client? Explore Forrester’s B2B research to see how we can help.