Online research: Who are we talking about here?
There has been a lot of discussion and chatter around social market research (SMR) lately, fueled in part bythe social sessions at the MRA conference a couple of weeks ago. We’ve had social on our radar here at Forrester for awhile and my colleague Tamara Barber has done a great job looking at social market research and its opportunities and challenges. Some of the issues around social MR are hot topics for online research in general, representivity being one of the key ones. Just who are we gathering data on? Whether we’re talking about new social methods or tried and true online panels, the question is still relevant.
The topic of representivity in online panels surfaced a few years ago as MR professionals began to examine and question the data coming back from online surveys. Vendors began to address client concerns through a variety of approaches like MarketTools' TrueSampleand Peanut Labs’ Optimus initiatives. Some clients seemed appeased by the measures, but the debate has continued to rage within the MR industry — and rightfully so.
Research Magazine recently included an article interviewing Adrian Sanger, Tim Britton, Jeffrey Henning, and Terry Sweeney on the issue of online panels and the representative nature of the research being drawn from such studies. The interviews are very insightful and touch upon some highly relevant issues for market researchers dealing with online research. As Terry says:
“ …there’s a bias from the recruitment methodology all the way down to technology issues, bandwidth issues, things like that. We try to educate clients on the biases that could exist within the sample population.”
Just who are we reporting data on when we survey online panels? As market researchers, are we clearly communicating the issues about representivity and non-response so that our end clients know the true value of the data? As we move into the world of hybrid methodologies, where online panels are not only used for quantitative research, but also serve as sampling base for online communities, these issues will become even more important. It's our responsibility to know exactly what the data is representative of and so, as an industry, we need to continue these conversations about the quality of online panels. This also means understanding everything possible about the vendors you’re buying sample from. Where do they recruit from? What are the panel demographics? How often are respondents contacted? How much time do the respondents spend online? How are they incented? These initial questions will start to shed light on who your survey takers are and will help you choose a supplier that most closely matches the type of respondents you need to reach.
Moving forward I'm going to focus some of my research on issues like online panels, panel quality, and MR vendors. I look forward to having conversations (and debates!) with many of you on the current and future landscape of these issues. If you're an MR vendor that wants to discuss the specific steps you're taking to address the online panel issue, feel free to connect with me and we can set up a time for a briefing. In the meantime, happy researching!