The Measurement Curmudgeon — Part 1: Detection
Perhaps you’ve heard him in meetings — he is the one questioning your results. Perhaps you’ve seen him at his desk surrounded by tombs and tables in an effort to lower incremental sales calculations — he calls it reducing bias. Perhaps you’ve hoped he will not be assigned to your project — he delivers lower lift estimates than his peers. He is the measurement curmudgeon.
How do you detect if a measurement curmudgeon resides in your office? Listen for the following clues/questions:
- Is that control group really comparable to the experimental group? Isn’t it biased toward less engaged customers and inflating your measured lift?
- Wasn’t that concurrent with our fall promotion? Isn’t that event likely accounting for most of your positive results?
- Haven’t sales been trending up? Did you incorporate that trend into your analysis?
Sometimes his questions take on a more academic tone, sprinkled with words that bring back vague memories of a long-forgotten analytics course:
- If you take into account customer level heterogeneity do you get the same results? (Is the average misleading here?)
- Isn’t that simply regression toward the mean? (Shouldn’t we naturally expect last year’s underperformers to partially rebound this year?)
- Doesn’t multicollinearity call your recommendations into question? (Does correlation between your predictors make it hard to figure out which one we should address first?)
When explaining his results, as opposed to questioning yours, he may sound a bit academic. While not meaning to be defensive, it can be hard to collaborate with the curmudgeon when you hear:
- Correlation simply shows co-occurrence and should not be thought of causally.
- Those counterintuitive results are a simple example of Simpson’s paradox.
- An adequate counterfactual cannot be created, so I cannot provide any insights on that topic.
Measurement curmudgeons are tremendous assets to Customer Insight teams, though a little behavioral modification is often necessary to increase their effectiveness. In later blogs we will further explore the marketing curmudgeon, his mindset, and how to increase his impact on the organization.
But first I would like to know your experience — Do you have a measurement curmudgeon in your office? Would you define the working relationship as helpful, frustratingly helpful, or simply frustrating?