Open Collections

Impact of scoring and response format on inferences from electronic surveys for rating scales

University of British Columbia

In social sciences, the measurement of constructs requires decisions on how to quantify, record and analyse scores. The use of electronic surveys allows for additional measurement tools different from traditional paper surveys. However, the manner of how these choices affect the validity of scores has not been explored thoroughly. This dissertation uses the Center for Epidemiology Scale of Depression to explore the effects of scoring method and response formats. Study 1 compares four response formats commonly used in electronic surveys, and scores obtained using different electronic devices. Measurement invariance of the scale was examined using multi-group confirmatory factor analysis. Results showed that electronic response formats suggest response behaviours to participants. Furthermore, scores obtained using different response formats and electronic devices failed to show measurement invariance at a configural level. Study 2 compares changes in differential item functioning (DIF) using ordinal logistic regression based on the scoring method. Furthermore, DIF contrasts included binary sex, minority gender identity and sexual orientation. Results showed that changes in scoring methodology altered which items presented DIF. Moreover, contrasts with sexual and gender minority identity altered the presence of DIF. As such, treatment of scores and choice of focus for demographic characteristics impact the findings of psychometric properties of surveys. Overall, findings from both studies suggest that the structure of surveys can be fragile, and conclusions drawn from scores are sensitive to any changes to the measurement and analyses. Any such change, even if seemingly minor, may change the construct, and consequently affect the validity of scores. Researchers must be mindful of the specific needs of their target construct and use adequate forms of measurement and analyses to suit their studies, thus guiding more precise measurement. A fuller understanding of the factors that affect validity will result in more control to ensure accurate measurement and appropriate conclusions from obtained scores.

Text

2020-09-24

Attribution-ShareAlike 4.0 International

http://hdl.handle.net/2429/76151

Measurement, Evaluation and Research Methodology

University of British Columbia

UBCV

http://creativecommons.org/licenses/by-sa/4.0/