Open Collections

Application of a log linear diagnostic classification model to the 2011 Progress in International Reading Literacy Study

University of British Columbia

Assessment developers and educational researchers have been seeking ways to make assessments more informative and to advance understandings of students’ cognitive processes (Ercikan, 2006; Huff & Goodman, 2007; Leighton & Gierl, 2007). Demands for readily useful formative evidence have prompted educational researchers to evaluate the validity of applying multidimensional latent variable models to existing unidimensional large-scale international assessment (LSA) data (Huff & Goodman, 2007; Lee, Park & Taylan, 2011). One approach to the measurement of multiple dimensions is diagnostic classification models (DCMs). These models rely on categorical latent attributes to represent students’ mastery or non-mastery of underlying skills. This study investigates the utility of applying the log-linear cognitive diagnostic model (LCDM) framework to Progress In International Reading Literacy Study (PIRLS) 2011 data in Canada and the U.S and the psychometric impact of retrofitting a saturated DCM to LSA data. The application of DCMs to international LSA data provides a means to examine patterns of attribute classifications to gain diagnostic information at the group level. Such diagnostic information can help educators better understand pathways to mastery and provide evidence about general trends for students’ strength and weaknesses within and across countries. As a whole, evidence from this study indicates that diagnostic information can be gained by retrofitting DCMs to extant data; however, the quality of the information depends on the quality of the data with respect to the magnitude of parameters and the discrimination values. Overall mastery rates for reading literacy attributes were higher in the U.S. than in Canada. However, the detection of item bias due to construct irrelevant variance provides evidence of incomparability. Evidence also indicates that the attribute ‘making straightforward inferences’ was the most difficult to master across countries, which supports previous research (Oakhill, 2007). Further results also reveal important patterns about proficiency including that the attribute ‘retrieving explicitly stated information’ is not mastered in isolation. Also mastery of the attribute ‘interpreting and integrating ideas and information’ does not occur with mastery of ‘making straightforward inferences.’ Lastly, results indicate that there are statistical and estimation limitations to applying a saturated DCM to extant data.

Text

2018-07-31

Attribution-NonCommercial-NoDerivatives 4.0 International

http://hdl.handle.net/2429/66630

Measurement, Evaluation and Research Methodology

University of British Columbia

UBCV

http://creativecommons.org/licenses/by-nc-nd/4.0/