Diagnosing Survey Response Quality

Short descriptionBest practices for evaluating data quality issues in surveys.
TagsData QualitySurvey Research
ForHandbook on Politics and Public Opinion


As non-probability samples become more commonplace in survey research, concerns about obtaining quality survey responses have increased tenfold. Issues of low-quality data or ‘satisficing’—arising from inattention, carelessness, or fraudulent behavior—pose significant threats to the reliability and validity of survey estimates. We emphasize the industry’s need for a systematic approach to diagnose and address response quality issues, and we explore the strengths and limitations of various metrics.

We emphasize proactive measures to address data quality in online nonprobability surveys. Researchers should prioritize survey design practices and incorporate response quality assessment into the study plan, including pre-registration. They should use multiple detection methods for identifying problematic respondents, avoiding reliance on a single metric. If exclusions are necessary, conservative criteria should be applied, and the effects of exclusion should be analyzed and reported. We must be transparent when assessing data quality and acknowledge the inevitability of errors. This is necessary to uphold scientific integrity and ensure the credibility of research findings.