- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- UBC Theses and Dissertations /
- Evaluating the correctness of IRT-based methods in...
Open Collections
UBC Theses and Dissertations
UBC Theses and Dissertations
Evaluating the correctness of IRT-based methods in computing classification consistency and accuracy indices in the presence of model misspecification Chau, Lok Heng
Abstract
This thesis investigated the correctness of the classification consistency (CC) and classification accuracy (CA) indices computed using the Item Response Theory (IRT) -based Rudner, Guo, and Lee methods on tests with a single cut score for pass-or-fail decision. Specifically, simulation studies were conducted to evaluate the correctness of the classification indices under the IRT model misspecification conditions, in which a simpler, more restricted IRT model was misspecified for item calibrations. The correctness of the indices was also evaluated in the baseline conditions in which a correct IRT model was specified for item calibrations. The location of cut score on the theta scale was also manipulated to study its effects on the correctness of the indices. Overall, the results suggested that the classification indices computed from all three IRT-based methods were close to their “true” values in both model misspecification and baseline conditions when the cut score was set at or near where the test had the most information. However, it should be noted that under the model misspecification conditions, the proportions of examinees classified into either one of the binary categories (i.e. passing rates) were different from those when the correct models were specified. The finding suggests that although the computed classification indices were mostly correct when evaluated at the same theta cut score even when the IRT model was misspecified, the differences in the passing rates could have implications on the overall purpose and consequence of the classification decisions. Limitations of this thesis were also discussed.
Item Metadata
Title |
Evaluating the correctness of IRT-based methods in computing classification consistency and accuracy indices in the presence of model misspecification
|
Creator | |
Publisher |
University of British Columbia
|
Date Issued |
2018
|
Description |
This thesis investigated the correctness of the classification consistency (CC) and classification accuracy (CA) indices computed using the Item Response Theory (IRT) -based Rudner, Guo, and Lee methods on tests with a single cut score for pass-or-fail decision. Specifically, simulation studies were conducted to evaluate the correctness of the classification indices under the IRT model misspecification conditions, in which a simpler, more restricted IRT model was misspecified for item calibrations. The correctness of the indices was also evaluated in the baseline conditions in which a correct IRT model was specified for item calibrations. The location of cut score on the theta scale was also manipulated to study its effects on the correctness of the indices. Overall, the results suggested that the classification indices computed from all three IRT-based methods were close to their “true” values in both model misspecification and baseline conditions when the cut score was set at or near where the test had the most information. However, it should be noted that under the model misspecification conditions, the proportions of examinees classified into either one of the binary categories (i.e. passing rates) were different from those when the correct models were specified. The finding suggests that although the computed classification indices were mostly correct when evaluated at the same theta cut score even when the IRT model was misspecified, the differences in the passing rates could have implications on the overall purpose and consequence of the classification decisions. Limitations of this thesis were also discussed.
|
Genre | |
Type | |
Language |
eng
|
Date Available |
2018-08-29
|
Provider |
Vancouver : University of British Columbia Library
|
Rights |
Attribution-NonCommercial-NoDerivatives 4.0 International
|
DOI |
10.14288/1.0371604
|
URI | |
Degree | |
Program | |
Affiliation | |
Degree Grantor |
University of British Columbia
|
Graduation Date |
2018-09
|
Campus | |
Scholarly Level |
Graduate
|
Rights URI | |
Aggregated Source Repository |
DSpace
|
Item Media
Item Citations and Data
Rights
Attribution-NonCommercial-NoDerivatives 4.0 International