Open Collections

Relative performance of scoring designs for the assessment of constructed responses

University of British Columbia

This study used computer simulation to investigate the relative performance of three rater allocation designs and an objective scoring condition at two levels of comparisons: comparison to the examinee true ability, and comparison to the ability estimate obtained under the objective scoring condition. At both comparison levels, score level (theta estimate) differences across theta bins and decisions at four cut-off levels were investigated. At comparison level one, small differences across scoring designs were found in the accuracy of ability estimates obtained using the scoring designs investigated. However, large differences in percentage of accurate pass and fail decisions were found at the first comparison level. This pattern of findings was repeated at comparison level two; comparison to the objective scoring estimates and decisions. The decisions associated with the row allocation design in particular were somewhat erratic when compared to the other scoring methods at both comparison levels making it the least desirable of the rater allocation designs. The random and spiral allocation designs provided similar levels of accuracy in estimating examinee ability and provided similar levels of decision accuracy. These allocation designs showed similar levels of discrepancy and showed similar percentages of agreement with the decisions obtained under the objective scoring condition.

Text

2009-12-23

For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use.

http://hdl.handle.net/2429/17331

Measurement, Evaluation and Research Methodology

University of British Columbia

UBCV

DSpace