Open Collections

Hu, Shun-Fu

University of British Columbia

Motivated to integrate machine learning with quantitative methods in social science, this research consists of two studies, both using College Major Preference Assessment (CMPA) as data. Study-1 trained multilabel neural networks (MNNs), from machine learning, to predict the outcomes of the original CMPA, which consisted of two stages, with its short version, which consisted of only one of the stages. The findings showed the effectiveness of MNNs over the traditional simple sum scoring method: the MNN trained for accuracy had an accuracy of .94 (an improvement from .91 of simple sum), the one for recall had .93 recall (from .64), and for precision, .60 precision (from .34). The metrics, recall and precision, had been corrected to exclude chance guessing. Study-2 proposed a method for the intuitive diagnostics of the behaviour of trained MNNs by introducing Pratt’s measures, from quantitative methods in social science, to machine learning’s perturbation-based explainable AI methods. Two MNNs were chosen to demonstrate the explanatory method: The one trained to maximize the accuracy of CMPA short-version scoring, and the other maximizing the adjusted recall of it. To understand a single MNN, 5000 trials were conducted. In each trial, a new MNN with the same settings as the one to be studied was trained on a random subset of 50% of the originally used input variables. Then, the relative importance of each input variable was obtained by calculating the Pratt’s measures for a linear regression where the independent variables were the inclusion statuses of all input variables, and the dependent variable was the MNN’s performance on a major. The visualized patterns of the Pratt’s measures (the results) were shown diagnostic of the strength of validity evidence for using the two MNNs for the prediction of respondents’ preferences for certain college major assessed by CMPA, and the lack thereof, for other majors. The implications were discussed in the context of existing pioneering fields integrating machine learning with social science research, including computational psychometrics, educational data mining and psychoinformatics. Limitations and future research were addressed.

Text

2022-10-20

Attribution-NonCommercial-NoDerivatives 4.0 International

http://hdl.handle.net/2429/82941

Measurement, Evaluation and Research Methodology

University of British Columbia

UBCV

http://creativecommons.org/licenses/by-nc-nd/4.0/