Evaluating the correctness of IRT-based methods in computing classification consistency and accuracy indices in the presence of model misspecification
department or school: Education, Faculty of
department or school: Educational and Counselling Psychology, and Special Education (ECPS), Department of
data provider: DSpace
degree campus: UBCV
creator: Chau, Lok Heng
issued: 2018-08-29T15:46:53Z
issued: 2018
related degree: Master of Arts - MA
degree grantor: University of British Columbia
description: This thesis investigated the correctness of the classification consistency (CC) and classification accuracy (CA) indices computed using the Item Response Theory (IRT) -based Rudner, Guo, and Lee methods on tests with a single cut score for pass-or-fail decision. Specifically, simulation studies were conducted to evaluate the correctness of the classification indices under the IRT model misspecification conditions, in which a simpler, more restricted IRT model was misspecified for item calibrations. The correctness of the indices was also evaluated in the baseline conditions in which a correct IRT model was specified for item calibrations. The location of cut score on the theta scale was also manipulated to study its effects on the correctness of the indices. Overall, the results suggested that the classification indices computed from all three IRT-based methods were close to their “true” values in both model misspecification and baseline conditions when the cut score was set at or near where the test had the most information. However, it should be noted that under the model misspecification conditions, the proportions of examinees classified into either one of the binary categories (i.e. passing rates) were different from those when the correct models were specified. The finding suggests that although the computed classification indices were mostly correct when evaluated at the same theta cut score even when the IRT model was misspecified, the differences in the passing rates could have implications on the overall purpose and consequence of the classification decisions. Limitations of this thesis were also discussed.
aggregated CHO: https://circle.library.ubc.ca/rest/handle/2429/66984
note: EVALUATING THE CORRECTNESS OF IRT-BASED METHODS IN COMPUTING CLASSIFICATION CONSISTENCY AND ACCURACY INDICES IN THE PRESENCE OF MODEL MISSPECIFICATION by Lok Heng Chau A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Measurement, Evaluation, and Research Methodology) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) August 2018 © Lok Heng Chau, 2018 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled: Evaluating the Correctness of IRT-based Methods in Computing Classification Consistency and Accuracy in the Presence of Model Misspecification submitted by Lok Heng Chau in partial fulfillment of the requirements for the degree of Master of Arts in Measurement, Evaluation, and Research Methodology Examining Committee: Dr. Amery Wu Supervisor Dr. Bruno Zumbo Supervisory Committee Member Dr. Liu Yan Supervisory Committee Member iii Abstract This thesis investigated the correctness of the classification consistency (CC) and classification accuracy (CA) indices computed using the Item Response Theory (IRT) -based Rudner, Guo, and Lee methods on tests with a single cut score for pass-or-fail decision. Specifically, simulation studies were conducted to evaluate the correctness of the classification indices under the IRT model misspecification conditions, in which a simpler, more restricted IRT model was misspecified for item calibrations. The correctness of the indices was also evaluated in the baseline conditions in which a correct IRT model was specified for item calibrations. The location of cut score on the theta scale was also manipulated to study its effects on the correctness of the indices. Overall, the results suggested that the classification indices computed from all three IRT-based methods were close to their “true” values in both model misspecification and basel…
has type: Thesis/Dissertation
date issued: 2018-09
is shown at: 10.14288/1.0371604
language: eng
degree discipline: Measurement, Evaluation and Research Methodology
provider: Vancouver : University of British Columbia Library
publisher: University of British Columbia
rights: Attribution-NonCommercial-NoDerivatives 4.0 International
rights URI: http://creativecommons.org/licenses/by-nc-nd/4.0/
scholar level: Graduate
type: Text
identifier URI: http://hdl.handle.net/2429/66984