Is The Other Shoe Dropping In Panel Quality?
Some recent events make me hopeful that major moves are afoot with enhancing panel quality.
Since the beginning of online surveys, there have been questions about how clean the online panels that enable them are. Questions abounded about representativeness, fraud, professional survey takers, inattentive survey takers and the like. The response from panel vendors has been that they have strong measures in place, and that the problems were overstated. Naysayers have claimed bad sample numbers that range from 20-30%. Buyer's of sample were largely in a "trust me" position, since most of the quality measures were in the hands of the panel vendor. Associations (such as ESOMAR and ARF), have come up with protocols that all good panels should follow, and many have.
But still, the sample quality debate has raged. Tempers have flared in major association meetings on the topic, as irritated major research customers have demanded that the industry do better. The problems were felt to be especially acute in B2B research, since the incentives were much higher, giving bad actors a greater reason to game the system. In response, the industry has created solutions, such as machine fingerprinting from Peanut Labs or RelevantID, or identity matching combined with machine fingerprinting and engagmement measuring from MarketTool's TrueSample. Mostly, however, there has not been a consensus about a best approach to the problem, in part because panel vendors are disinclined to adopt solutions from companies they consider to be competitors.
Major research buyers have started to flex their muscle. Microsoft recently announced that, after extensive testing, it would only buy research from companies that cleaned sample through TrueSample. General Mills and ARSgroup have made a similar commitment. P&G has demanded quality commitments that are equivalent to TrueSample's results from their major research vendors. The result? Kantar, IPSOS, GfK, and Synovate — normally fierce competitors — have formed "The Consortium" to address the issue with their private panels, as well as the sample they purchase from open panels. The expectation is that if they cannot demonstrate comparable quality to TrueSample, that P&G will demand they use it.
In the meantime, leading panel vendors have embraced machine fingerprinting. Some are also beefing up identity corroboration.
So, who will win? Eventually, all buyers of market research, who ride along with major companies who are finally demanding better sample.
What are your thoughts about the likely winner in the "panel quality wars?" Are Microsoft and P&G right to be concerned, or is the whole situation overblown?