Overcoming the Biggest Threats to Market Research

Corrupted data can defeat the missions that drive survey efforts. Bad data can lead executives to invest in the wrong expansion areas, marketing teams to target irrelevant audiences, finance teams to implement pricing models that lead to customer churn, and more. IntelliSurvey’s new ebook, “Overcoming the Biggest Threats to Market Research: Bad Data & Bad Actors,” explores the ways bad actors can infiltrate research efforts and recommends how forward-thinking research agencies can overcome these threats.

Four key factors have driven the prevalence of problematic data:

1. Incentives have attracted bad actors with growing sophistication
2. Sample providers may recruit with less rigor and consistency
3. Respondents’ attention spans are shrinking
4. Imperfect execution of, and innovation within, data cleaning frameworks

Cheating

Professional survey cheating takes various forms: respondents speed-clicking through surveys to “earn” incentives, bots pretending to be real survey respondents, or criminals assembling networks to bypass survey controls.

Attackers work diligently to defeat fraud protection measures. As an example, many research groups run IP checks on survey arrival and completion. Professional attackers thus sometimes assemble in-country actors to commence a survey (when IP checks are made), and then transition to offshore machines (which are presumably less costly) for the preponderance of the interview, before handing it back to the in-country actor for the final few pages and panel passbacks. This isn’t frequent, but does illustrate the sophistication of attacker operations.

Minimizing the impact of such attacks requires vigilance, innovation, and holistic attention to the research process – and technology to concurrently identify cheating signals.

Compromised Sample Quality

IntelliSurvey has been tracking sample quality algorithmically for more than a decade. Sample suppliers once showed consistent measures on content-independent cheating metrics. This is no longer the case – even the same panels have varying quality contours week by week. ost pressures have driven diminished rigor from the sampling industry.

As such, researchers must be mindful of the supplier sources, and project contours, in crafting data cleaning frameworks. Different sampling sources and target populations may have different cheating proclivities. Systematic multi-sourcing can provide powerful reference points to identify problematic sampling practices.

Audience Engagement

Today’s respondents live in a scattered and distracting media landscape. Many complete surveys on smartphones as they text with friends and watch other screens. The screen on which they complete surveys may be small. It is relatively easy for respondents to give less than full attention to a survey, which degrades response quality.

Researchers benefit from prioritizing audience engagement. This requires holistic efforts. Strong studies:

• Deliver questions quickly, so that there is the slimmest moment between response submission and the next question
• Present questions in an engaging and intuitive manner, which substantially accelerates interviews and increases response quality
• Leverage engaging survey presentation frameworks, which have significantly higher levels of attention, decreasing the difficulty and amount of data cleaning required

Need for Innovation

Bad actors grow more sophisticated every day. For example, bots can be programmed to take surveys at a human pace, defeating speeder checks. Respondents who are imperfectly engaged learn to spread battery responses across a scale instead of in a straight line. Dishonest respondents learn to look for, and avoid, improbable responses from qualification sets.

Innovative technologies and techniques enable researchers to identify and measure a broader set of problematic signals. These signals can be combined into probabilistic estimates of response quality, which can be used to concurrently clean data. New cheating signals can be integrated into the framework as they are identified, securing the ongoing quality of the responses.

Overcome the Biggest Threats to Data Corruption

Thoughtful survey design is the first step in collecting good data. For most target populations, it’s essential to incorporate sense checks, attention checks, knowledge checks, seeking checks, and frequency checks.

The second step is to employ sophisticated, modern data cleaning techniques to intercept bad data early. The wisest process is to identify a diverse set of cheating signals, be informed about the probabilistic value of each signal, and have a good framework for synthesizing observations into actionable metrics. Increased rigor and faster processing result.

When exceptional data collection professionals employ well-designed survey methodologies, stakeholders can trust the insights generated to make the most thoughtful and well-informed strategic decisions possible. Absent quality data, the entire research exercise becomes incomplete and perilous.

For more in-depth information, including a real-world example of the difference a small percentage makes in removing bad actors from survey data, download IntelliSurvey’s new ebook, “Overcoming the Biggest Threats to Market Research: Bad Data & Bad Actors.”