Categories
March 20, 2023
Corrupted data can defeat the missions that drive survey efforts. Bad data can lead executives to invest in the wrong expansion areas, marketing teams to target irrelevant audiences, finance teams…
0
Corrupted data can defeat the missions that drive survey efforts. Bad data can lead executives to invest in the wrong expansion areas, marketing teams to target irrelevant audiences, finance teams to implement pricing models that lead to customer churn, and more. IntelliSurvey’s new ebook, “Overcoming the Biggest Threats to Market Research: Bad Data & Bad Actors,” explores the ways bad actors can infiltrate research efforts and recommends how forward-thinking research agencies can overcome these threats.
Four key factors have driven the prevalence of problematic data:
Professional survey cheating takes various forms: respondents speed-clicking through surveys to “earn” incentives, bots pretending to be real survey respondents, or criminals assembling networks to bypass survey controls.
Attackers work diligently to defeat fraud protection measures. As an example, many research groups run IP checks on survey arrival and completion. Professional attackers thus sometimes assemble in-country actors to commence a survey (when IP checks are made), and then transition to offshore machines (which are presumably less costly) for the preponderance of the interview, before handing it back to the in-country actor for the final few pages and panel passbacks. This isn’t frequent, but does illustrate the sophistication of attacker operations.
Minimizing the impact of such attacks requires vigilance, innovation, and holistic attention to the research process – and technology to concurrently identify cheating signals.
IntelliSurvey has been tracking sample quality algorithmically for more than a decade. Sample suppliers once showed consistent measures on content-independent cheating metrics. This is no longer the case – even the same panels have varying quality contours week by week. ost pressures have driven diminished rigor from the sampling industry.
As such, researchers must be mindful of the supplier sources, and project contours, in crafting data cleaning frameworks. Different sampling sources and target populations may have different cheating proclivities. Systematic multi-sourcing can provide powerful reference points to identify problematic sampling practices.
Today’s respondents live in a scattered and distracting media landscape. Many complete surveys on smartphones as they text with friends and watch other screens. The screen on which they complete surveys may be small. It is relatively easy for respondents to give less than full attention to a survey, which degrades response quality.
Researchers benefit from prioritizing audience engagement. This requires holistic efforts. Strong studies:
Bad actors grow more sophisticated every day. For example, bots can be programmed to take surveys at a human pace, defeating speeder checks. Respondents who are imperfectly engaged learn to spread battery responses across a scale instead of in a straight line. Dishonest respondents learn to look for, and avoid, improbable responses from qualification sets.
Innovative technologies and techniques enable researchers to identify and measure a broader set of problematic signals. These signals can be combined into probabilistic estimates of response quality, which can be used to concurrently clean data. New cheating signals can be integrated into the framework as they are identified, securing the ongoing quality of the responses.
Thoughtful survey design is the first step in collecting good data. For most target populations, it’s essential to incorporate sense checks, attention checks, knowledge checks, seeking checks, and frequency checks.
The second step is to employ sophisticated, modern data cleaning techniques to intercept bad data early. The wisest process is to identify a diverse set of cheating signals, be informed about the probabilistic value of each signal, and have a good framework for synthesizing observations into actionable metrics. Increased rigor and faster processing result.
When exceptional data collection professionals employ well-designed survey methodologies, stakeholders can trust the insights generated to make the most thoughtful and well-informed strategic decisions possible. Absent quality data, the entire research exercise becomes incomplete and perilous.
For more in-depth information, including a real-world example of the difference a small percentage makes in removing bad actors from survey data, download IntelliSurvey’s new ebook, “Overcoming the Biggest Threats to Market Research: Bad Data & Bad Actors.”
Disclaimer
The views, opinions, data, and methodologies expressed above are those of the contributor(s) and do not necessarily reflect or represent the official policies, positions, or beliefs of Greenbook.
Comments
Comments are moderated to ensure respect towards the author and to prevent spam or self-promotion. Your comment may be edited, rejected, or approved based on these criteria. By commenting, you accept these terms and take responsibility for your contributions.
Top in Quantitative Research
Why are we still measuring brand loyalty? It isn’t something that naturally comes up with consumers, who rarely think about brand first, if at all. Ma...
Sign Up for
Updates
Get content that matters, written by top insights industry experts, delivered right to your inbox.
67k+ subscribers