Open Collections

Zumbo, Bruno D.

Paragon Testing Enterprises/UBC Paragon Research Initiative, University of British Columbia

A novel multimethod research methodology and accompanying statistical methods for operational and validity research is described in response to the emergence of remote online proctored test administration. The multimethod strategy was designed to allow for a robust comparison of the test centre and online test performance that far exceeds conventional methods to investigate the comparability of tests- i.e., their concordance. The rigour and logic of our methodology are grounded in test validity and a framework based on four key principles. First, Angoff’s (1993) matching principle allows for the definition of optimal statistical psychometric methods that do not confound concordance with true differences in item performance (i.e., impact). Second, the equity principle states it should be a matter of indifference to a test taker or a test user about which test takers choose between two modes of test administration (test centre or online). Third, the test use principle states that the comparison across test administrations should focus on the scale on which scores are reported- for example, band scores on each of the (four) components of a language test rather than an item-by-item comparison. Fourth, there is an overall principle of multiple sources of evidence (multimethod methodology) that calls for more than one source of evidence supporting the concordance investigation to rule out rival plausible alternative interpretations and ferreting out multiple sources of potentially hidden invalidity. This novel methodology is applied to investigate the concordance of the CAEL delivered at a test centre and online. The concordance study used a sample of 1,455 CAEL test takers, 765 test takers who completed the CAEL at a test centre, and 690 who completed it remotely online between June and October 2020. The findings from the six statistical and psychometric methods are consistent. The sample of the test centre and online test takers were equivalent, and the test performance was found to be consistently concordant. Together, this is strong evidence that the conclusions from the CAEL band scores from the test centre and online versions are concordant, fully comparable band score performance of test takers at various levels of CAEL’s language domains. The results of this analysis will serve as evidence to support the interpretation and use of scores from tests administered online using a remote-proctored test-delivery platform or at a test centre. The multimethod approach introduced in this monograph is a general model for other concordance studies that provides a principled rationale for designing such studies to investigate any delivery modes; for example, a concordance study may investigate a test administered simultaneously at a test centre, remotely online, at pop-up administration centres, and in paper-and-pencil format. A rigorous test of the concordance is of importance to test users, test providers, and external stakeholders who rely on valid and comparable test performance and test use across different test administration modalities.

Text

2021-07-21

Attribution-NoDerivatives 4.0 International

http://hdl.handle.net/2429/79064

Zumbo, B. D. (2021). A Novel Multimethod Approach to Investigate Whether Tests Delivered at a Test Centre are Concordant with those Delivered Remotely Online: An Investigation of the Concordance of the CAEL [monograph]. Vancouver, BC: Paragon Testing Enterprises/UBC Paragon Research Initiative, University of British Columbia.

Faculty

http://creativecommons.org/licenses/by-nd/4.0/