Open Collections

Webster, Alexis Danielle

University of British Columbia

Data collection through online sources such as Amazon’s Mechanical Turk (MTurk) is valuable to social science research. However, online sampling is susceptible to obtaining high rates of low quality data that are provided by individuals who are careless, fraudulent, and/or bots. Failure to identify and remove these problematic respondents can distort research results and lead to inaccurate conclusions. Current quality control strategies have challenges and limitations, including respondents learning to circumvent detection. Open-ended items have been found to offer rich insights into data quality and thus are useful for data quality assessments. Yet, more research is needed to better understand the usefulness of different open-ended items for data quality assessment purposes. The present study was a secondary analysis of a dataset comprising 343 participants recruited from MTurk to investigate the prevalence of problematic characteristics in response to four distinct open-ended items. Findings indicate that 86% of participants provided one or more low quality responses. These low quality responses tended to have multiple problematic characteristics, including duplication with other participants, vagueness, irrelevance, linguistical errors, and even blatant plagiarism. Importantly, the rates of these problematic characteristics varied depending on the specific open-ended item. A strict exclusionary approach led to a 14% retention rate whereas two different balanced approaches that allowed for some error had retention rates between 40% and 44%. These findings underscore the need for rigorous, yet pragmatic, detection methods and standards for online data collection, which should incorporate open-ended items. Recommendations for the use of open-ended items, including data handling procedures are discussed.

Text

2023-09-01

Attribution-NonCommercial-NoDerivatives 4.0 International

http://hdl.handle.net/2429/85765

Measurement, Evaluation and Research Methodology

University of British Columbia

UBCV

http://creativecommons.org/licenses/by-nc-nd/4.0/