Education, Faculty of
Educational and Counselling Psychology, and Special Education (ECPS), Department of
DSpace
UBCV
Chau, Lok Heng
Master of Arts - MA
University of British Columbia
This thesis investigated the correctness of the classification consistency (CC) and classification accuracy (CA) indices computed using the Item Response Theory (IRT) -based Rudner, Guo, and Lee methods on tests with a single cut score for pass-or-fail decision. Specifically, simulation studies were conducted to evaluate the correctness of the classification indices under the IRT model misspecification conditions, in which a simpler, more restricted IRT model was misspecified for item calibrations. The correctness of the indices was also evaluated in the baseline conditions in which a correct IRT model was specified for item calibrations. The location of cut score on the theta scale was also manipulated to study its effects on the correctness of the indices. Overall, the results suggested that the classification indices computed from all three IRT-based methods were close to their “true” values in both model misspecification and baseline conditions when the cut score was set at or near where the test had the most information. However, it should be noted that under the model misspecification conditions, the proportions of examinees classified into either one of the binary categories (i.e. passing rates) were different from those when the correct models were specified. The finding suggests that although the computed classification indices were mostly correct when evaluated at the same theta cut score even when the IRT model was misspecified, the differences in the passing rates could have implications on the overall purpose and consequence of the classification decisions. Limitations of this thesis were also discussed.
EVALUATING THE CORRECTNESS OF IRT-BASED METHODS IN COMPUTING CLASSIFICATION CONSISTENCY AND ACCURACY INDICES IN THE PRESENCE OF MODEL MISSPECIFICATION by Lok Heng Chau A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Measurement, Evaluation, and Research Methodology) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) August 2018 © Lok Heng Chau, 2018 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled: Evaluating the Correctness of IRT-based Methods in Computing Classification Consistency and Accuracy in the Presence of Model Misspecification submitted by Lok Heng Chau in partial fulfillment of the requirements for the degree of Master of Arts in Measurement, Evaluation, and Research Methodology Examining Committee: Dr. Amery Wu Supervisor Dr. Bruno Zumbo Supervisory Committee Member Dr. Liu Yan Supervisory Committee Member iii Abstract This thesis investigated the correctness of the classification consistency (CC) and classification accuracy (CA) indices computed using the Item Response Theory (IRT) -based Rudner, Guo, and Lee methods on tests with a single cut score for pass-or-fail decision. Specifically, simulation studies were conducted to evaluate the correctness of the classification indices under the IRT model misspecification conditions, in which a simpler, more restricted IRT model was misspecified for item calibrations. The correctness of the indices was also evaluated in the baseline conditions in which a correct IRT model was specified for item calibrations. The location of cut score on the theta scale was also manipulated to study its effects on the correctness of the indices. Overall, the results suggested that the classification indices computed from all three IRT-based methods were close to their “true” values in both model misspecification and baseline conditions when the cut score was set at or near where the test had the most information. However, it should be noted that under the model misspecification conditions, the proportions of examinees classified into either one of the binary categories (i.e. passing rates) were different from those when the correct models were specified. The finding suggests that although the computed classification indices were mostly correct when evaluated at the same theta cut score even when the IRT model was misspecified, the differences in the passing rates could have implications on the overall purpose and consequence of the classification decisions. Limitations of this thesis were also discussed. iv Lay Summary In many standardized tests, examinees are classified into mutually exclusive categories based on their performance in relations to the employed cut scores. Since many of these tests are of high-stakes in nature, indices in evaluating the accuracy and consistency of the classification decisions are often computed and reported. One of them is the classification consistency (CC) of decisions, which is the proportion of examinees classified in the same categories over repeated testing. Another one is the classification accuracy (CA) of decisions, which is the proportion of examinees classified in the true categories they belong. This thesis investigated the correctness of the classification indices given by three recently proposed Item Response Theory (IRT) –based methods. Since IRT is a model-based approach, there is also some degree of model misfit between the chosen IRT model and the observed item response data. Specifically, this thesis aimed to evaluate the correctness of the classification indices under the specification of both the correct or incorrect IRT model. In addition, the effects of the employed cut score location on the correctness of the classification indices were also evaluated. v Preface This thesis is the original work of the author, Lok Heng Chau, under the supervision of Dr. Amery Wu. vi Table of Contents Abstract ......................................................................................................................................... iii Lay Summary ............................................................................................................................... iv Preface .............................................................................................................................................v Table of Contents ......................................................................................................................... vi List of Tables ................................................................................................................................ ix List of Figures ............................................................................................................................... xi List of Abbreviations ................................................................................................................. xiii Chapter 1: Introduction ................................................................................................................1 1.1 Background ..................................................................................................................... 1 1.2 Statement and significance of problem ........................................................................... 3 1.3 Purpose of the Study ....................................................................................................... 4 Chapter 2: Literature Review .......................................................................................................6 2.1 Concepts of CC and CA .................................................................................................. 6 2.2 Overview of CTT-based Methods ................................................................................ 10 2.2.1 Huynh’s Method ....................................................................................................... 10 2.2.2 Subkoviak’s Method ................................................................................................. 11 2.2.3 Livingston and Lewis’ Method ................................................................................. 12 2.3 IRT-based Procedures in Estimating CA and CC ......................................................... 13 2.3.1 The Rudner Method .................................................................................................. 13 2.3.2 The Guo Method ....................................................................................................... 16 2.3.3 The Lee Method ........................................................................................................ 19 vii 2.4 Overall CC and CA indices........................................................................................... 21 2.5 Factors Affecting CC and CA ....................................................................................... 24 2.5.1 Location of the Cut Scores ........................................................................................ 24 2.5.2 Number of Cut Scores ............................................................................................... 25 2.5.3 Test Length ............................................................................................................... 25 2.5.4 Sample Size ............................................................................................................... 26 2.5.5 Estimator for Ability ................................................................................................. 26 2.6 Summary and Conclusion ............................................................................................. 26 Chapter 3: Method .......................................................................................................................29 3.1 Data Simulation ............................................................................................................ 29 3.2 True Classification Indices ........................................................................................... 31 3.2.1 True CA, FP, and FP ................................................................................................. 32 3.2.2 True CC and Kappa .................................................................................................. 34 3.3 Computing the Sample Classification Indices .............................................................. 34 3.4 Evaluation Criteria ........................................................................................................ 36 3.5 Summary ....................................................................................................................... 37 Chapter 4: Results........................................................................................................................38 4.1.1 Baseline Conditions .................................................................................................. 38 4.1.1.1 Test Information................................................................................................ 38 4.1.1.2 Average Differences Between Sample and “True” Indices .............................. 39 4.1.1.3 Passing Rates .................................................................................................... 42 4.1.2 Model Misspecification Conditions .......................................................................... 43 4.1.2.1 Test Information................................................................................................ 43 viii 4.1.2.2 Average Differences Between Sample and “True” Indices .............................. 45 4.1.2.3 Passing Rates .................................................................................................... 48 Chapter 5: Conclusions and Discussions ...................................................................................50 Bibliography .................................................................................................................................54 Appendices ....................................................................................................................................59 Appendix A Item Parameters of All Dichotomous Items in 2013 NAEP Grade 12 Mathematics Assessment ................................................................................................................................ 59 Appendix B Tables of Average Differences (and SD) Between the Sample and “True” CA and CC Indices ................................................................................................................................. 61 Appendix C - R Code with Description .................................................................................... 67 ix List of Tables Table 2-1 Classification Consistency Contingency Table .............................................................. 8 Table 2-2 Classification Accuracy Contingency Table .................................................................. 9 Table 3-1 Distributions of Item Parameters of NAEP Grade 12 Mathematic Assessment .......... 30 Table 3-2 Summary of the Three Baseline and Three Model Misspecification Conditions ........ 35 Table B-1 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 1PL. Data Calibration Model: 1PL) .............................................................. 61 Table B-2 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 1PL. Data Calibration Model: 1PL) .............................................................. 61 Table B-3 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 2PL. Data Calibration Model: 2PL) .............................................................. 62 Table B-4 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 2PL. Data Calibration Model: 2PL) .............................................................. 62 Table B-5 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 3PL. Data Calibration Model: 3PL) .............................................................. 63 Table B-6 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 3PL. Data Calibration Model: 3PL) .............................................................. 63 Table B-7 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 2PL. Data Calibration Model: 1PL) .............................................................. 64 Table B-8 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 2PL. Data Calibration Model: 1PL) .............................................................. 64 Table B-9 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 3PL. Data Calibration Model: 1PL) .............................................................. 65 x Table B-10 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 3PL. Data Calibration Model: 1PL) .............................................................. 65 Table B-11 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 3PL. Data Calibration Model: 2PL) .............................................................. 66 Table B-12 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 3PL. Data Calibration: 2PL) .......................................................................... 66 xi List of Figures Figure 2-1 Area under the normal curve representing conditional classification accuracy, false positive, and false negative rates at a particular estimate of theta. ............................................... 15 Figure 3-1. Steps in computing “true” classification indices for classification decisions made on theta scale (Rudner and Guo method). .......................................................................................... 33 Figure 3-2 Steps in computing 'true' classification indices for classification decisions made on raw score scale (Lee method). ...................................................................................................... 33 Figure 4-1 Test information functions in the baseline conditions. Solid curves represent the average test information functions across the 500 simulated sample test data. Dotted curves represent the 95% variability across the 500 simulated sample test data. .................................... 39 Figure 4-2 Average differences between the sample and the “true” CC and CA values in baseline conditions. ..................................................................................................................................... 40 Figure 4-3 Average differences between the sample and the “true” FP and FN values in baseline conditions. ..................................................................................................................................... 41 Figure 4-4. Average differences between the sample and the “true” kappa values in baseline conditions. ..................................................................................................................................... 42 Figure 4-5 Average “true” and actual passing rates in baseline conditions. ................................. 43 Figure 4-6 Average test information functions in model misspecification conditions. Solid curves represent the average test information functions across the 500 simulated sample test data. Dotted curves represent the 95% variability across the 500 simulated sample test data. ............. 44 Figure 4-7 Average differences between the sample and the “true” CC and CA values in model misspecification conditions. .......................................................................................................... 45 xii Figure 4-8 Average differences between the sample and the “true” FP and FN values in model misspecification conditions. .......................................................................................................... 47 Figure 4-9 Average differences between the sample and the “true” kappa values in model misspecification conditions. .......................................................................................................... 48 Figure 4-10 Average “true” and actual passing rates in model misspecification conditions........ 49 xiii List of Abbreviations CA classification accuracy CC classification consistency CTT classical test theory EAP expected a posteriori FN false negative FP false positive IRT item response theory MAP maximum a posteriori ML maximum likelihood TCC test characteristic curve 1PL one-parameter IRT logistic model 2PL two-parameter IRT logistic model 3PL three-parameter IRT logistic model 1 Chapter 1: Introduction 1.1 Background In many standardized tests in education and professional certification and licensure contexts, examinees are classified into mutually exclusive categories based on their performance and the predefined standards. These standards are determined in a standard setting process, and they define the levels of achievement or proficiency and the corresponding cut scores (Bejar, 2008). One example is the binary classification of pass/fail or master/non-mastery categories employed in many certification and licensure exams. Multiple proficiency categories can often be found in many educational and language assessments. Tests with classifications are often of high-stakes in nature and can have important consequences on the test takers, such as the eligibility to become a licensed professional, college admission, graduation, and immigration. However, it is unrealistic for any tests to have perfect classifications in the presence of measurement errors. Misclassifications into a lower category could mean a deprivation of a precious opportunity for well-deserved examinees, while misclassifications into a higher category could potentially lead to undesirable consequences both on the examinees and the society. An underprepared student who is erroneously classified to be qualified for an academic program may lag behind other well-prepared students and feel frustrated. Quality of healthcare and patients’ safety are at risk if an underqualified physician is mistakenly granted license to practice. Therefore, it is vital to assess the reliability and validity for the test scores designed to make classification decisions and ensure the classification decisions made are accurate and reliable. Establishing validity requires accumulating evidence to support the intended interpretations of test scores for the proposed purpose (Hubley & Zumbo, 2011), and the importance of computing and reporting the reliability of classification decisions 2 from such tests has been recognized in Standard 2.16 in the Standards for Educational and Psychological Testing (Joint Committee on Standards for Educational and Psychological Testing (U.S.), National Council on Measurement in Education, American Psychological Association, & American Educational Research Association., 2014). It states that when a test is used for classifying examinees, estimates of the percentage of test takers who would be classified in the same category on two replications of the test procedure should be provided. As noted by Livingston and Wingersky (1979), the traditional measures of reliability, such as the reliability coefficient and standard error of measurement, do not directly provide the information regarding the reliability of classification decisions of a test. Hambleton and Novick (1973) suggested two measures that would be most appropriate in evaluating the classification decisions made from test scores. The first measure, which is commonly referred to as classification consistency (CC), is defined as the extent of agreement in classification decisions from two administrations of the same test or parallel forms of the test. The second measure, classification accuracy (CA), is the measure of the agreement between the actual classifications based on observed scores and the true classifications based on the examinees’ true and unknown ability. The computation of CC is straightforward if data from repeated testing are available. However, in real testing, it is often that only a single form of the test is administered. Also, the true scores of the examinees are unknown and need to be estimated for the calculation of CA. For these reasons, procedures in computing CC and CA based on single administrations have been developed and proposed. Numerous procedures developed in the framework of Classical Test Theory (CTT) have been proposed in the literature (Hanson & Brennon, 1990; Huynh, 1976; Lee, Brennon & Wan, 3 2009; Livingston & Lewis, 1995; Subkoviak, 1976). These methods assume that the observed scores are reported and the cut scores are set on the raw score scale. They differ primarily in the assumed mathematic models underlying the observed raw score distribution (e.g., 2-parameter and 4-parameter beta binomial model, compound binomial model, and compound multinomial model) and whether dichotomous, polytomous, or a mix of both items are included in the tests. The computation of CC and CA under the framework of Item Response Theory (IRT) have also been developed in recent decades (Guo, 2006; Lee, 2010; Rudner, 2001, 2004). While these methods are all based on IRT models and the corresponding item parameters, they differ in terms of the underlying distributional assumptions and whether the classification decisions are based on the raw score scale or the latent ability scale. 1.2 Statement and significance of problem In both CTT and IRT based methods, classification indices are computed by imposing some mathematical models on the observed data, and some degree of model misspecification will most certainly arise as there are always discrepancies between the true model underlying the observed data and the model implied for the data. The concern of the effect of model misspecification, hence model misfit, on the computation of classification indices has been brought up by several researchers (Lee, 2010; Hanson and Brennon, 1990; Lathrop & Chen, 2013). IRT has been adopted by many testing institutions in test development, data calibration, and score reporting. In practice, the issues of model specification not only involve the degree of model fit, but also practical issues such as software availability, costs, and operation efficiency, along with other considerations such as sample size, instrument characteristics, model 4 assumption tenability, and political realities (de Ayala, 2009). It is not uncommon a simpler model is employed instead of a more sophisticated and fitting model. Despite of the differences in the underlying assumptions and procedures between the IRT-based methods in the computations of classification indices, the correctness of the computed indices may very well be adversely affected if a misspecified IRT model is employed in the calibration procedures. Although there have been several studies on evaluating and comparing the performance of the IRT-based methods in the computations of classification indices, to the best of the author’s knowledge, there is a lack of study on the impact of model misspecification on computing IRT-based classification indices. As the indices are used to evaluate the quality of tests that are designed to make classification decisions, the effects of model misspecification on computing the indices should not be ignored, and it deserves a systematic and thorough study. 1.3 Purpose of the Study The purpose of this thesis is to investigate the effects of IRT model misspecification on computing the classification indices. Specifically through a series of simulation studies, this thesis examines the performance of the Lee method, Rudner method, and Guo method in computing classification indices when a misspecified IRT model is employed in item and ability calibrations. Simulation studies were used because the “true” model that generated the test data could be specified by the researcher, and subsequently, the “true” values of classification indices could be computed. That way, by purposely calibrating the simulated test data with a misspecified model, the computed classification indices given by the three IRT-based procedures could be compared with the “true” classification indices. The effects of employing a misspecified 5 model on the computed classification indices by the three IRT-based methods could be investigated. 6 Chapter 2: Literature Review This chapter starts with a conceptual description of CC and CA, and the commonly used indices in reporting CC and CA in practice. Next, the development and the procedures of the CTT-based methods are briefly described. Then, a more complete description of the three IRT-based procedures, namely, the Rudner method, Guo method, and Lee method were explained. Factors that have been shown to influence the computation of CC and CA are discussed. Lastly, a summary is provided to inform how the literature review motivates this study, and the specific questions this study aims to address. 2.1 Concepts of CC and CA Test scores on achievement tests provide two different kinds of information for interpretation (Glaser, 1963). Depending on the intended use of the test scores, test score interpretation can be defined as either normed-referenced or criterion-referenced. Norm-referenced information provides the relative position or ranking of an examinee’s test score in relation to a reference sample in order to understand how the particular examinee compares with the others in the predefined population. Criterion-referenced information shows whether an examinee’s performance has exceeded some predetermined standards. In both normed-referenced and criterion-referenced tests, a desirable property of the test scores is their consistency and replicability if the examinees were repeatedly tested, assuming there was no change in their true ability. Traditional measures of reliability, such as parallel-test reliability, split-half reliability, and Cronbach’s alpha of internal consistency, provide the information of the stability and consistency of the examinees’ test scores and their relative ranking with one another. However, in criterion-referenced tests, the focus is usually on the 7 interpretations of the classification decisions, not necessarily on the actual test scores themselves. These traditional measures fail to provide the information of whether an examinee will be consistently classified in an alternative, parallel test form, nor whether an examinee is correctly classified in his or her true category. The concepts of CC and CA were first proposed by Hambleton and Novick (1973) as alternative measures for the reliability and validity for criterion-referenced tests, in which examinees are compared with some predefined standards, such as cut scores. They asserted that it would be more appropriate to evaluate reliability through CC, which is defined as the degree in which examinees will be consistently classified in the same categories in repeated testing with the same or parallel form of the test. As for validity, they believed that CA provides the information of the degree of accuracy in classification decisions. In other words, assuming the true classifications of the examinees are known, CA measures the degree in which the classifications of the examinees made based on the observed test data agree with their true classifications. The concept of CC can be conceptualized as a K by K contingency table, where K represents the number of classifications in a given test. For simplicity, with a test that only has two classification categories, pass or fail, the resulting contingency table can be represented as a 2 by 2 table (Table 2-1). It is also important to point out that in this conceptualization of CC, it is assumed that a parallel form of the test is administered, or the original test is administered for the second time. The most commonly used index for measuring CC is the overall CC rate, p, which is equal to the sum of the diagonal in the contingency table. Each cell in the diagonal represents the proportion of examinees who are consistently classified in the same category in both 8 administrations of the test, and p represents the overall proportion of examinees consistently classified across all categories in both administrations. The sum of the off diagonal represents the inconsistent classification rate, which is the overall proportion of examinees that are classified as pass in one administration but as fail in another, or vice versa. Generally, the inconsistent classification rate is not reported. Table 2-1 Classification Consistency Contingency Table Classification decisions from the 2nd administration Pass Fail Marginal Proportion Classification decisions from the 1st administration Pass Consistent Classification Inconsistent Classification pꞏ1 Fail Inconsistent Classification Consistent Classification pꞏ2 Marginal Proportion p1ꞏ p2ꞏ Another commonly reported index for quantifying CC is the Cohen’s kappa coefficient (Cohen, 1960), 𝜅, which is given by: 𝜅 =𝑝 − 𝑝𝑐1 − 𝑝𝑐 (1) where 𝑝𝑐 is the consistency agreement between two test forms expected by chance alone. Assuming the two test forms are independent, 𝑝𝑐 is given by: 𝑝𝑐 = ∑(𝑝·𝑘)(𝑝𝑘·)𝐾𝑘=1 (2) 9 where 𝑝·𝑘 and 𝑝𝑘· are the marginal rates of examinees classified into category k in the two test forms. By definition, the denominator in Equation (1) expresses the additional proportion of test takers over chance that would be consistently classified under perfectly consistent classifications, and the numerator expresses the additional proportion of consistent classifications computed based on the contingency table. In other words, kappa coefficient as defined in Equation (1) can be interpreted as the proportion of consistent classification exists, over and above that which can be expected by chance alone. Table 2-2 Classification Accuracy Contingency Table Classification decisions from a single administration Pass Fail True classification Pass Correct Classification False Negative Fail False Positive Correct Classification The concept of CA can be similarly presented as a K by K contingency table (Table 2-2). Instead of having two administrations of the same test or parallel forms of the test, only a single form of the test is administered, and it is assumed the ‘true’ classifications of the examinees are known. Each cell in the diagonal represents the proportion of examinees being correctly classified as his or her true classifications, and the overall CA rate, 𝛾, is equal to the sum of the diagonal. The cells in the off diagonal represent incorrect classifications, and they are usually separately reported for the two types of misclassification scenarios. Overall false positive (FP) rate, 𝛾+, is the proportion of examinees who are incorrectly classified into a category above their ‘true’ classifications (i.e. given a pass instead of the “true” fail category), and overall false 10 negative (FN) rate, 𝛾−, is the proportion of examinees who are incorrectly classified into a lower category (i.e. given a fail instead of the “true” pass category). Swaminathan, Hambleton, and Algina (1974) first demonstrated the computation of CC indices with sample test data from two administrations. However, it is not practical in reality to have the same examinees tested twice and data collected from two administrations. Moreover, the “true” scores or the “true” classifications of the examinees are unknown and must be estimated. It is of practical interests and needs to have some methods to compute classification indices using data from single administration. 2.2 Overview of CTT-based Methods 2.2.1 Huynh’s Method Huynh (1976) proposed a procedure in computing CC indices based on test scores from single administrations of tests consisting of only dichotomous items. In this approach, it is assumed that the examinees’ true scores follow a two-parameter beta distribution, and the observed scores follow a beta binomial distribution. The use of the binomial model automatically assumes all the items in the tests are perfectly exchangeable in the known universe of items. In other words, it implies the items are similar in terms of difficulty, and to some degree, implies content homogeneity. It further assumes that a parallel hypothetical test form, consisting of the same number of items drawn from the same specified universe, is administered to the same group of examinees. Under the assumption that test scores from different samples are locally independent, the test scores from the two parallel test forms, x and y, follow a bivariate two-parameter beta binomial distribution with joint density function: 11 𝑓(𝑥, 𝑦) =(𝐽𝑥) (𝐽𝑦)Β(𝛼, 𝛽)Β(𝛼 + 𝑥 + 𝑦, 2𝑗 + 𝛽 − 𝑥 − 𝑦) (3) where 𝛼 and 𝛽 are the fitted parameters of the beta distribution, B is the beta function, j is the total number of dichotomous items, and x and y are the summed-scores from the administrations of two parallel test forms. The overall CC rate is equal to the probability of examinees having both x and y scores belonging into the same category given by Equation (3). Hanson and Brennan (1990) extended Huynh’s approach by applying the four-parameter beta distribution for the examinees’ true scores, which includes a lower and an upper bound for the true score distribution. They also demonstrated the procedures in computing CA, FP and FN indices. They showed the four-parameter beta binomial model provided better model fit to the observed score distributions, and concluded the four-parameter beta binomial model was the preferred model in the computation classification indices. They further compared the four-parameter beta model with the four-parameter beta compound binomial model, but found indistinguishable results. 2.2.2 Subkoviak’s Method The procedure proposed by Subkoviak (1976) rests on the assumptions that scores from parallel test forms are independently distributed and follow a binomial distribution, which are similar to those assumed in Huynh’s method. One major difference between the two methods is that the distribution of the examinees’ true scores is not assumed in Subkoviak’s approach. An individual’s true score is estimated by using either a maximum likelihood estimation, or a linear regression method that includes collateral information, such as the mean, variance, and reliability 12 of the test scores. Since no distribution of the examinees’ true scores is assumed, the overall CC rate is obtained through averaging the computed conditional CC rate from each examinee. While the binomial model is favored for its simplicity because it assumes the probability of correct response remains constant across items, Subkoviak (1976) pointed out the compound binomial model would provide a better approximation to real testing situations by allowing the probability of correct response to vary between items. 2.2.3 Livingston and Lewis’ Method One limitation of Huynh’s and Subkoviak’s approach is that both methods are only applicable on test scores from tests consisting of only dichotomous items. Livingston and Lewis (1995) extended Huynh’s method to allow computations of classification indices for tests with mixed items. They introduced the concept of effective test length, which is defined as the number of discrete, dichotomously scored, locally independent, and equally difficult test items necessary to produce the total scores with the same precision as the actual observed scores given by the mixed items. For example, consider a test consisting of multiple polytomous questions, and each question has a possible score range from 0 to 3 points. If the test has the same reliability as a test with 20 dichotomous items, the effective test length for the test would be 20. The effective test length can be estimated by: 𝑗𝐿𝐿 =(𝜇𝑥 − 𝑋min)(𝑋max − 𝜇𝑥) − 𝑟𝜎𝑥2𝜎𝑥2(1 − 𝑟) (4) where 𝑋𝑚𝑖𝑛 and 𝑋𝑚𝑎𝑥 are the lowest and the highest score in the original score scale, 𝜇𝑥 and 𝜎𝑥2 are the mean and the variance of the observed scores, and r is the reliability coefficient. Once the effective test length is determined, the original observed scores are transformed to a new scale 13 ranging from 0 to jLL, and the rest of the procedures in computing classification indices are the same to those described by Hanson and Brennon (1990). 2.3 IRT-based Procedures in Estimating CA and CC In this section, the three recently proposed IRT-based methods, namely, the Rudner method, the Guo method, and the Lee method, are presented. Emphasis will first be placed on describing the procedures in computing the conditional CA and CC indices for each individual examinee. Procedures in obtaining the overall CA and CC indices are later presented in Section 2.4. 2.3.1 The Rudner Method Rudner (2001) proposed a procedure in computing CA indices for tests consisting of dichotomous items, and later extended the procedure for tests with polytomous items (Rudner, 2015). In this approach, it is assumed that the individual test scores and the employed cut scores are both mapped onto the continuous theta (𝜃) scale for the latent trait, and the estimated theta scores are compared directly to the cut scores set on the theta scale (hereafter referred to as theta cut score) in making classification decisions. Since the true theta score of any individual is unknown, the estimated theta score (𝜃) is taken as the best point estimate for his or her true score, 𝜃. When 𝜃 is obtained through maximum likelihood (ML) estimation, with a sufficiently large number of items and sample size, it is asymptotically normally distributed with the mean of 𝜃 and the standard error of estimate, se(𝜃), which is given by: 14 se(𝜃𝑖) =1√𝐼(𝜃𝑖) (5) where the subscript i indicates the ith examinee, and I(𝜃𝑖) is the test information function evaluated at 𝜃𝑖 for examinee i. The test information function is the sum of item information functions evaluated based on the given IRT model and the item parameters. When classifying individuals into any one of the K categories, K – 1 number of theta cut scores, 𝜆1 < 𝜆2 < … < 𝜆𝐾−1, are aligned on the continuous theta scale. Two additional theta scores, 𝜆0 = −∞ and 𝜆𝐾 = ∞, are included to represent the theoretical lower and upper bounds of the possible theta scores for computational purposes. An individual is classified into the kth category if the 𝜃𝑖 is greater than or equal to 𝜆𝑘−1 and less than 𝜆𝑘. The probability of examinee i being classified into the kth category, given 𝜃𝑖, is given by: 𝑃𝑟(𝐾 = 𝑘|𝜃𝑖) = 𝜙 (𝜆𝑘 − 𝜃𝑖 se(𝜃𝑖)) − 𝜙 (𝜆𝑘−1 − 𝜃𝑖 se(𝜃𝑖)) (6) where 𝜙() is the cumulative normal distribution function (CDF) with mean of 𝜃𝑖 and standard error of estimate of se(𝜃𝑖). Equation (6) represents the area under the normal distribution, N(𝜃𝑖 , se(𝜃𝑖)), between the two theta cut scores. The “true” classification of an examinee is unknown, but the classification decision made from the observed data can be taken as the best estimate of the examinee’s “true” classification. The conditional CA, 𝛾𝜃𝑖, which is the probability of an examinee being classified in his or her “true” classification, can be calculated from Equation (6) by using 𝜃𝑖 and the theta cut scores that define the “true” category. The computations of the conditional CA indices are graphically illustrated in Figure 2-1. The vertical dashed line represents the location of the examinee’s 𝜃𝑖, and the distribution of 𝜃𝑖 is asymptotically normal around 𝜃𝑖 with a standard error of estimate of 15 se(𝜃𝑖). The two vertical solid lines represent the two employed theta cut scores, 𝜆1 and 𝜆2. Since 𝜃𝑖 falls between the two theta cut scores, the “true” classification of the examinee is the middle category. The examinee is correctly classified in the middle category if his or her 𝜃𝑖 also falls between the two employed theta cut scores. The probability of being correctly classified for the examinee, or the conditional CA of the examinee, is equal to the area under the normal curve bounded by the two theta cut scores. Subsequently, the conditional FP and FN can be computed by calculating the area under the normal curve in the respective regions. Figure 2-1 Area under the normal curve representing conditional classification accuracy, false positive, and false negative rates at a particular estimate of theta. The original Rudner method was developed for the purpose of computing only the CA indices. The procedure for computing CC indices was proposed as an extension from the original method (Li, 2006; Wyse & Hao, 2012). In a test with K categories, the probabilities of an examinee with 𝜃𝑖 being classified into each of the category in one test form can be calculated by λ1 θi λ2 𝜃𝑖 16 using Equation (6), which is equal to the area under the normal curve between the respective theta cut scores. When a hypothetical parallel test form with the same employed theta cut scores and items with the same IRT parameters is administered, the probabilities of the same examinee with the same 𝜃𝑖being classified into each of the category can be calculated in the same way by using Equation (6). Under these conditions, the probabilities obtained for the hypothetical parallel test form will be identical to the probabilities obtained from the actual test form administered. With the assumption that the two test forms are independent, the probability of being consistently classified across two test forms for an examinee is equal to the sum of the squared probability of being classified into each category in the actual test form administered. The conditional CC, 𝑝𝜃𝑖, can be computed by: 𝑝𝜃𝑖 = ∑[𝑃𝑟(𝐾 = 𝑘|𝜃𝑖)]2𝐾𝑘=1 (7) where K is the total number of classification categories, 𝑃𝑟(𝐾 = 𝑘|𝜃𝑖) is the probability of being classified in category k given 𝜃𝑖, and the conditional CC is the summation of the squared probabilities of being classified across all categories. 2.3.2 The Guo Method One drawback for the Rudner method is the dependence on the normality assumption of the estimated thetas of the examinees, and the subsequent calculations of the probabilities in being classified into each category by calculating the area under the normal curve between the theta cut scores. In practice, the normality assumption only holds asymptotically and is never completely met (Wyse & Hao, 2012). Guo (2006) developed a procedure as an extension of 17 Rudner method that did not require the normality assumption for the estimated thetas, but based on the likelihood of the data. Similar to Rudner method, Guo method assumes that both the observed scores and the employed cut scores are mapped onto the theta scale. For a test with K categories, K – 1 number of theta cut scores, 𝜆1 < 𝜆2 < … < 𝜆𝐾−1, along with two theta scores representing the lower and upper bounds of possible theta scores, 𝜆0 = −∞ and 𝜆𝐾 = ∞, are aligned on the theta scale. An individual is classified into the kth category if 𝜃𝑖 is greater than or equal to 𝜆𝑘−1 and less than 𝜆𝑘. Under the usual IRT assumptions of unidimensionality and local independence, the likelihood of an examinee having any 𝜃, given the observed responses to the items and the known or calibrated item parameters, can be written as, 𝐿(𝑢𝑖1, 𝑢𝑖2, … , 𝑢𝑖𝐽|𝜃) = ∏ 𝑃𝑖𝑗𝑢𝑖𝑗(1 − 𝑃𝑖𝑗)𝑢𝑖𝑗𝐽𝑗=1 (8) where j represents the total number of items, uij represents the response to item j by examinee i, and Pij is the probability of getting a correct response for item j by examinee i. The likelihood of being classified into the kth category, which is bounded by the theta cut scores 𝜆𝑘−1 and 𝜆𝑘, is given by: 𝐿𝑘𝑖 = ∫ 𝐿(𝑢𝑖1, 𝑢𝑖2, … , 𝑢𝑖𝐽|𝜃)𝜆𝑘𝜃=𝜆𝑘−1𝑑𝜃 (9) To make the integration in Equation (9) computationally feasible, the minimum and the maximum theta cut scores are replaced by some arbitrary large values. For instance, in the procedures demonstrated in Guo’s (2006) original paper, the minimum and maximum theta cut scores were set equal to -3 and 3. In addition, the integration is replaced with summation by 18 creating a number of equally spaced theta points (i.e., 100 points) between the theta cut scores. Equation (9) can now be rewritten using the summation form: 𝐿𝑘𝑖∗ = ∑ 𝐿(𝑢𝑖1, 𝑢𝑖2, … , 𝑢𝑖𝐽|𝜃)𝜆𝑘𝜃=𝜆𝑘−1 (10) The values given by the likelihood function in Equation (10) cannot be directly taken as probabilities. A normalization step is needed so that the sum of the likelihoods across all categories for an examinee equals to 1. The expected probability of classifying in the kth category is given by: 𝑃𝑟(𝐾 = 𝑘|𝑢𝑖1, 𝑢𝑖2, … , 𝑢𝑖𝐽) =𝐿𝑘𝑖∗∑ 𝐿𝑘𝑖∗𝐾𝑘=1 (11) where 𝐿𝑘𝑖∗ in the numerator is obtained from Equation (10) with equally spaced theta points between the cut scores, and the denominator is the normalization constant that is sum of the likelihoods of classifying into each category. An examinee’s classification from the observed data, which is taken as the best estimate of an examinee’s “true” classification, can be determined by comparing his or her 𝜃𝑖 to the employed theta cut scores. Subsequently, the probability of an examinee being classified in the “true” classification, which is the conditional CA, can be obtained from Equation (11). As in Rudner method, Guo only proposed the procedure of computing CA in his original paper. Wyse and Hao (2012) extended Guo’s method and proposed a procedure in computing CC, and the rationale is similar to the one based on the extension to Rudner method they developed. Under the assumption that a hypothetical parallel test form, which has the same employed theta cut scores and items with the same IRT parameters, is administered to the examinee who provides the same responses to the items as in the original administered test, the 19 probabilities of being classified into each category in the hypothetical parallel test form will be identical to those in the original administered test. Then, assuming the two test forms are independent, the conditional CC, 𝑝𝑖, can be calculated by: 𝑝𝑖 = ∑[𝑃𝑟(𝐾 = 𝑘|𝑢𝑖1, 𝑢𝑖2, … , 𝑢𝑖𝐽)]2𝐾𝑘=1 (12) where 𝑃𝑟(𝐾 = 𝑘|𝑢𝑖1, 𝑢𝑖2, … , 𝑢𝑖𝐽) is the normalized probability of being classified into category k obtained from Equation (10). 2.3.3 The Lee Method The Lee (2010) method, or sometimes referred to as the IRT-recursive method, is different from Rudner and Guo method in which the classification decisions are made on the raw score scale. In other words, the cut scores are employed on the raw score metric, and the classification decisions are made by comparing the examinees’ raw scores with the raw cut scores. The Lee method is a generalization of the previously developed and proposed methods by Huynh (1990) and Schulz et al. (1999), who studied the use of dichotomous IRT models, and Wang et al. (2000), who extended the methods by considering polytomous IRT models. Assuming we have a test that classifies examinees into K categories, K – 1 number of cut scores, c1 < c2 < … cK-1, are employed onto the raw score scale, along with two raw scores, c0 = 0 and cK = Xmax, representing the possible minimum and maximum raw scores. If all the items are dichotomously scored, Xmax is simply equal to the number of items. An examinee is classified into the kth category if his or her raw score xi is greater than or equal to 𝑐𝑘−1 and less than 𝑐𝑘, or automatically classified into the highest category if the raw score is equal to the maximum score. 20 The first step in the Lee method is to calculate the probability of observing each of the possible raw score conditional to the examinee’s 𝜃𝑖 and the item parameters. Any possible raw score, x, is a function of the response pattern to the items, which can be denoted as a vector 𝒖𝒊 =[𝑢𝑖1, 𝑢𝑖2, … , 𝑢𝑖𝑗], where i represents the examinee, j represents the individual item, and uij represents the response to item j given by examinee i. In the simplest case when a test only consists of dichotomous items, 𝑢𝑖𝑗 only takes on the values of either 0 or 1. The realization of any raw score x is the sum of the elements in 𝒖𝒊, or 𝟏′𝒖𝒊. The probability of observing a raw score x can be expressed as: Pr(𝑋 = 𝑥|𝜃𝑖) = ∑ ∏ 𝑃𝑖𝑗𝑢𝑖𝑗(1 − 𝑃𝑖𝑗)𝑢𝑖𝑗𝐽𝑗=1∀ 𝑢𝑖 𝑤ℎ𝑒𝑟𝑒 𝟏′𝒖𝒊=𝑥 (13) where ∏ 𝑃𝑖𝑗𝑢𝑖𝑗(1 − 𝑃𝑖𝑗)𝑢𝑖𝑗𝐽𝑗=1 is the same as Equation (8) and has the same definition, and Equation (13) requires summing the probabilities of all possible item response patterns that would yield the raw score x. One can see that the computation for Equation (13) can become unmanageably cumbersome as the number of possible response patterns increase exponentially with the number of items. Nonetheless, a recursive formula was developed for dichotomous items by Lord and Wingersky (1984), and it was later extended to support polytomous items by (Thissen et al., 1995). The computation of CA indices is not as straightforward as in Rudner and Guo method because the “true” ability for an examinee is estimated on the theta scale, while the classification decisions are made on the raw score scale. To determine the “true” classification of an examinee, the 𝜃𝑖 is transformed to the expected, or “true”, raw score, 𝜏?̂?𝑖, through the test characteristic curve (TCC). Then, 𝜏?̂?𝑖 can be treated as the best point estimate of the “true” ability of the examinee as measured on the raw score scale and compared with the raw cut scores to determine the “true” 21 classification. Note that if the cut scores are originally employed on the theta scale, they need to be transformed to the raw score scale through the TCC, and then rounded to the nearest possible raw score. The probability of being classified in the “true” category k can then be calculated by: Pr(𝐾 = 𝑘|𝜃𝑖) = ∑ Pr(𝑋 = 𝑥|𝜃𝑖)𝑐𝑘−1𝑥=𝑐𝑘−1 (14) which requires summing over the probabilities of observing each possible raw score in category k from all possible response patterns, given the examinee’s 𝜃𝑖. The computation of CC indices follows the similar assumptions that a hypothetical parallel test form having the same cut scores and items with the same IRT parameters is administered, and the probabilities of falling into each category will be identical across both administrations. The conditional CC is then given by: 𝑝𝜃𝑖 = ∑[Pr (𝐾 = 𝑘|𝜃𝑖)]2𝐾𝑘=1 (15) where Pr (𝐾 = 𝑘|𝜃𝑖) is the probability of being classified into a category obtained from Equation (14). 2.4 Overall CC and CA indices The conditional indices in section 2.3.1 to 2.3.3 only provide information for a given examinee or theta level, and the conditional indices are usually not of interest as we want to interpret the overall CC and CA indices of the decisions made for a group of examinees. There are two methods to marginalize the conditional CC and CA indices, and they are referred to as the P-method and the D-method (Lee, 2010). 22 In the P-method, or the person-based method, the conditional indices for each examinee are first calculated using the procedures described in Section 2.3.1 to 2.3.3. The overall indices are computed by simply summing all the conditional indices and averaged over the total number of examinees: 𝛾 =1𝑁∑ 𝛾𝜃𝑖𝑁𝑖=1 (16) 𝑝 =1𝑁∑ 𝑝𝜃𝑖𝑁𝑖=1 (17) where N represents the total number of examinees. The D-method, or the distribution-based method, requires a distribution of the theta to be either estimated from the observed data or assumed to follow a certain distribution, such as the standard normal distribution. The overall indices are calculated by integrating the conditional indices over the applicable theta domain: 𝛾 = ∫ 𝛾𝜃𝑔(𝜃)𝑑𝜃∞𝜃=−∞ (18) 𝑝 = ∫ 𝑝𝜃𝑔(𝜃)𝑑𝜃∞𝜃=−∞ (19) where 𝑔(𝜃) is the density of 𝜃 from the estimated or assumed distribution. The D-method provides a convenient way for computing the overall CC and CA indices when the theta distribution can be reasonably assumed to follow a certain distribution. In this case, the estimations will then only require the IRT parameters and the cut score locations, and the 𝜃 of 23 each individual examinee does not need to be estimated. However, the results are only expected to hold when the assumed ability distribution is reasonably close to the target ability distribution. Both P- and D- methods can be applied on Rudner and Lee method, however, only the P-method can be used in computing CC and CA indices under the Guo method. The reason is that the computations of CC and CA indices in Guo method are driven by the likelihood function, which requires the IRT parameters, cut score locations, and more importantly, the observed item responses from the examinees. The estimated thetas play no parts in the calculations of the probabilities of being classified into each of the category, other than determining the observed classified categories of the examinees. Studies based on both real and simulated data showed that both P- and D- methods give very similar results when the observed thetas are normally distributed (Lathrop & Chen, 2013; Lee, 2010). Lee (2010) further commented that P-method is more suitable when the focus of a test is on each individual examinee, such as in computer adaptive testing, while D-method is preferred when the interest is on group-level statistics. Wyse and Hao (2012) showed that Guo method generally produced the highest computed overall CC and CA rates, followed by Rudner and then Lee method. In most cases, differences between the computed overall CC and CA rates given by the three IRT-based methods were trivial; however, at a shorter test length (e.g., 30 items), the differences between the recursive-based Lee method and the ML-based Rudner or Guo method could approach .04 to .05, suggesting the choice the computation method could have some impacts on the reported indices in some situations. 24 2.5 Factors Affecting CC and CA Following the realization of the importance of calculating and reporting CC and CA indices, and subsequently the proposal of different methods in computing the indices, numerous studies have been conducted in evaluating the impact of various factors on the computed indices and comparing the performance of the different methods. 2.5.1 Location of the Cut Scores The effects of cut score locations on the overall CC and CA are well known and have been discussed in the literature. The locations of the cut scores play dual and opposite roles. On one hand, it has been shown that as the locations of cut scores are set further away from the center of the ability distribution, the overall CC and CA increase (Hanson & Brennon, 1990; Huynh, 1976; Lathrop & Chen, 2013; Puhan & Gall, 2012). This can be explained by the idea that when the ability of an examinee, measured either by theta or summed-score, is close to a cut score, there is a greater chance that the examinee will be misclassified either into a higher or lower category. Subsequently, the chance of being inconsistently classified in repeated testing is greater compared to another examinee whose ability is further away from the cut score. In other words, the conditional CC and CA are expected to be lower. If the center of the ability distribution is close to any of the employed cut scores, the overall CC and CA will be dragged down due to the higher number of examinees with lower conditional CC and CA. On the other hand, when the cut scores are set at where the test information is high, the conditional standard error of estimates of the examinees’ thetas will be low. In this case, the examinees whose ability is close to the cut scores will benefit from having a high conditional CC and CA, which in turns will contribute to a higher overall CC and CA. 25 2.5.2 Number of Cut Scores For tests that employ more than one cut score and classify examinees into more than two categories (multiple proficiency levels vs. binary pass/fail categories), the overall CC and CA are expected to be lower (Ercikan & Julian, 2002; Lee, 2010; Martineau, 2007). The explanation is similar to the effects of cut score locations. As the number of cut scores increases, more examinees are expected to be located close to any of the cut scores, which will lead to lower conditional CC and CA for those examinees. As a result, the overall CC and CA are expected to be negatively affected by the number of cut scores. 2.5.3 Test Length The overall CC and CA have been shown to increase as the test length increases, assuming the added test items are of at least the same quality and measuring the same latent ability (Li, 2006; Wyse & Hao, 2012). As the amount of test information increases from the inclusion of more test items, it is expected that the classification decisions can be made with improved accuracy and consistency. In addition to the effects of test length on the overall CC and CA, Lathrop and Chen (2013) investigated the effects of test length on the correctness of overall CA given by Rudner and Lee method in a series of simulation studies. Under the assumption that the item parameters were known and the correct IRT-models were used in estimating thetas, the authors showed that the values of overall CA given by the Rudner method were on average less than the “true” values while those given by Lee method were on average larger than the “true” values at short test length (e.g., 10 items). The magnitude of the absolute differences between the computed and the 26 “true” values ranged from .01 to .03 at 10 items, but began to diminish as the test length increased. At 80 items, the differences were practically trivial, ranging from less than .01 to at most .05. 2.5.4 Sample Size In the same simulation studies, Lathrop and Chen (2013) also studied the effects of sample size on the correctness of overall CA. While the average absolute differences between the computed and the “true” CA values were unaffected by the sample size, the authors showed that the variations of the absolute difference decreased as the sample size increased. The results suggested that a large sample size has a stabilizing effect on the computed CA values. 2.5.5 Estimator for Ability Lastly, the effects of using different ability estimators, namely, ML estimator, expected a posteriori (EAP) estimator, and maximum a posteriori (MAP) estimator, in computing the overall CC and CA under the three IRT-based methods have been investigated in simulation studies by Wyse and Hao (2012). Their results showed that the choice of the ability estimator had minimal impact on the computed overall CC and CA. In most cases, the differences between the computed values using thetas given different estimators were less than 1 percent. 2.6 Summary and Conclusion This chapter covered the conceptualization of CC and CA, brief descriptions of the CTT-based procedures in computing the CC and CA indices using test data from single administrations, and complete delineations of the recently developed IRT-based procedures. The 27 review of literature also showed that the impacts of various factors on the computed indices, as well as the performance of the three IRT-based methods under various conditions, have been well studied. However, as stated in Chapter 1, there is a lack of study on the robustness of the IRT-based methods and the effects on the computed indices in consideration of model misspecifications. The concern of the impact of model misfit due to model misspecification on the computed CC and CA indices has been brought up by several researchers. When comparing the indices given by different beta-binomial distributions assumed for the observed data under the CTT-based procedure, Hanson and Brennon (1990) concluded that the 4-beta binomial distribution model should be used as it consistently showed close fit to the observed data. Lee (2010) showed that when fitting different IRT models to the observed data, they tended to produce more discrepant computed indices when the models did not show adequate data fit. The results suggested that the correct/incorrect specification of the IRT model could have an impact on the correctness of the computed CC and CA indices. In other simulation studies investigating the correctness of the computed CC and CA indices and the performance of the different IRT-based methods under various conditions (Lathrop & Chen, 2013; Wyse & Hao, 2012), the researchers assumed that the “true” item parameters were known and the correct IRT model was used to estimate the examinees’ abilities. In practice, however, the true item parameters are rarely known but are estimated through item calibrations. The results from those simulation studies did not fully reflect the potential source of errors from both from model misspecification and calibration errors. One possible reason for the lack of study in this topic is that the “true” indices are unknown in real test data. Therefore, simulation studies would be well suited in investigating the 28 impact and practical consequences of model misspecification as the “true” CC and CA indices can be calculated in the stipulated “true” models. In addition, previous simulation studies have only investigated the correctness of the overall CA given by Rudner and Lee method under the assumption that the “true” item parameters are known. Although it is not uncommon to find the reported classification indices computed using the CTT-based Livingston’s and Lewis’ method, as IRT has become widely used in test development, equating, and standard setting, IRT-based methods in computing classification indices should be considered for tests developed under the IRT framework (Zhang et al., 2017). It is hoped that this study could expand the current understanding of the performance of the IRT-based Rudner, Guo, and Lee method in computing both CC and CA indices. Specifically, the following research questions are addressed in this study: 1. How correct are the classification indices when either the correct or incorrect IRT model is specified in the calibration procedure? Specifically, how correct are the classification indices under the Rudner method, the Guo method, and the Lee method? 2. To what extent do the locations of cut score affect the correctness of the classification indices? 29 Chapter 3: Method This chapter provides the descriptions of the research design of this study. A series of simulation studies were conducted in order to answer the research questions. In simulation studies, the “true” models that generated the test data are known, and the values of the “true” classification indices can be calculated. Factors manipulated in the simulation studies include the model specification and the cut score location. The negative effects of mis-specifying an IRT model on the simulated test data, if any, could be investigated by comparing the values of the sample and the “true” indices. 3.1 Data Simulation Test data were simulated by following the Rasch model, which is a special case of the one-parameter logistic IRT model (1PL), the two-parameter logistic IRT model (2PL), and the three-parameter logistic IRT model (3PL). The 3PL model is given by: 𝑃𝑟(𝑋𝑗 = 1|𝜃𝑖) = 𝑐𝑗 + (1 − 𝑐𝑗)𝑒𝑎𝑗(𝜃𝑖−𝑏𝑗)1 + 𝑒𝑎𝑗(𝜃𝑖−𝑏𝑗) (21) where the subscript i and j represent examinee and item, a is the item discrimination parameter, b is the item difficulty parameter, c is the item pseudo-guessing parameter, and 𝑃𝑟(𝑋𝑗 = 1|𝜃𝑖) is the expected probability of observing a correct response for the item given the examinee’s theta and the parameters of the item. For the 2PL model, the equation is simplified in which the pseudo-guessing parameters of the items are fixed to be 0. For the 1PL model, the equation is further simplified so that all discrimination parameters are fixed to be the same across all items. In the case of the Rasch model, the discrimination parameters are fixed to be equal to one for all items. 30 The “true” item parameters in the simulation studies were randomly drawn from the distributions shown in Table 3-1. The distributions were in reference to those of the 146 dichotomous items in the 2013 NAEP Grade 12 Mathematic Assessments (see Appendix A). The rationale for simulating test data this way was to mimic real test situations by generating test items with parameters that are based on those from a real, operational test. By taking the item parameters as random variables rather than fixed values, each simulated data set represents the responses from a test that come from a family of tests with the assumed parameter distributions. Thus, the results of this study could be generalized to tests constructed with items having similar distributions of discrimination, difficulty, and pseudo-guessing parameters rather than a particular test with particular fixed item parameters. Table 3-1 Distributions of Item Parameters of NAEP Grade 12 Mathematic Assessment Parameters Distribution Mean Sd Min Max IRT-b normal .52 .92 -1.84 4.62 IRT-a log normal 1.2 .25 .13 3.85 IRT-c normal .17 .08 .00 .41 Note. a: item discrimination, b: item difficulty, c: pseudo-guessing The simulated data were first in a data matrix form of expected probabilities of having a correct response to every item by every examinee, calculated by inputting the generated examinee thetas and item parameters in Equation (21). The expected probabilities were transformed to the item responses (correct or incorrect) by assuming that the responses followed a Bernoulli distribution with the probability parameters equal to the simulated probabilities of correct responses. In the current simulation studies, the effect of the test length was to be controlled for and hence was fixed at 60. The test length of 60 items was chosen to reflect a realistic number of 31 items in some standardized mathematic tests (e.g., ACT). In addition, it has been shown to be a sufficient test length in minimizing the average absolute differences between the computed and “true” CA rates under the assumption that “true” item parameters are known (Lathrop & Chen, 2013). By fixing the test length at 60 items, any large differences observed between the computed and “true” indices cannot be attributed to an insufficient number of test items. The effect of number of examinees was also to be controlled for and fixed at 5,000. A large sample size would ensure stable estimations of IRT item parameters and minimize the variation of the absolute differences between the computed and the “true” indices. Examinees’ abilities were drawn from a standard normal distribution. Only the P-method in computing the classification indices was considered for two reasons. First, the difference between the P- and D- method has been shown to be trivial when the distribution of examinees’ ability is normal. Second, only the P-method could be applied to all three IRT-based procedures considered in this study. The number of cut score was fixed at 1 to isolate the effects of the relative locations of multiple cut scores to each other. A single cut score was set on the theta scale, ranging from -1.5 to 1.5 with an increment of 0.5, to classify the examinees into the two categories, pass or fail. This way, the effects of the cut score locations on the correctness of the classification indices could also be investigated. The cut scores on the raw score scale for the Lee approach were transformed from thetas through TCC. 3.2 True Classification Indices Since the true data generating model was known, the “true” classification indices could be calculated to assess the correctness of the sample indices given by the IRT-based methods. 32 Note that the steps in obtaining “true” indices under Rudner and Guo method are slightly different from those under the Lee method, as the classification decisions are made on different score scales. Figure 3-1 and 3-2 illustrate the flowcharts in computing the ‘true” CA and CC indices for classifications made on theta and raw score scale. 3.2.1 True CA, FP, and FP Recall that CA is the measure of the agreement between the “true” classifications based on the examinees’ “true” ability and the actual classifications based on the observed test data. Under the Rudner and Guo approach, the “true” classifications of the examinees were obtained simply by comparing their generated “true” thetas and the employed cut scores. To obtain the actual classifications, the theta of each examinee was estimated with the simulated test data described in Section 3.1 by using ML estimator with the “true” item parameters and the correct IRT model specification. The estimated thetas were compared with the employed theta cut scores to determine their actual classifications. The “true” CA was then derived by computing the proportion of agreement between the “true” classifications and the actual classifications based on the test data. The “true” classification error rates, FP and FN, were calculated as the respective proportions of disagreement between the “true” classifications and the classifications based on the test data. Under the Lee method, the “true” classifications were obtained by first transforming both the generated “true” thetas and the theta cut scores to raw score scale through TCC, with the generated “true” item parameters and the correct IRT model. The “true” expected raw scores were compared with “true” raw cut scores to determine the examinees’ “true” classifications. For the actual classifications, the simulated item responses were summed up to obtain the observed 33 raw scores, which were then compared with the raw cut scores to determine the actual classifications based on the test data. The “true” CA, FP and FN, were calculated as the respective proportions of agreement or disagreement between the “true” classifications and the classifications based on test data. Figure 3-1. Steps in computing “true” classification indices for classification decisions made on theta scale (Rudner and Guo method). Figure 3-2 Steps in computing 'true' classification indices for classification decisions made on raw score scale (Lee method). 34 3.2.2 True CC and Kappa For “true” CC, a second set of item responses were simulated by using the same generated thetas and item parameters. The rationale was to mimic a scenario where the same examinees have actually taken a strictly parallel form of the same test, and hence having two sets of simulated item responses. The classifications based on the second set of test data were obtained by comparing the estimated thetas and the theta cut scores under Rudner and Lee methods, or comparing the raw scores and the raw cut scores under Lee method. The “true” overall CC rate was derived by finding the proportion of agreement between the classification decisions made based on two sets of simulated item responses. The “true” Cohen’s kappa was obtained by using Equation (2) and the marginal proportions of examinees classified into the categories in both simulated test forms. 3.3 Computing the Sample Classification Indices The sample classification indices under the three IRT-based methods were calculated using the procedures described in Section 2.3.1 to 2.3.3. Instead of assuming the generated “true” item parameters were known, estimates of item parameters and examinees thetas were obtained by fitting certain IRT models to the simulated sample test data. Three IRT baseline conditions and three model misspecification conditions were examined in this study (Table 3-2). In each condition, 500 sets of sample test data were simulated by using either the 1PL (Rasch), 2PL, or 3PL model. Each of the sample test data were simulated with item parameters randomly drawn from the assumed distributions displayed in Table 3-1. 35 In the three baseline conditions, the test data were simulated by using the 1PL (Rasch), 2PL and 3PL model. Then, the correct IRT model was specified to fit the simulated test data to obtain the item parameters and the theta estimates. Since the correct model was used in the calibration process in each baseline condition, the degree of model-data misfit was expected to be minimized. Using the sample estimates of item parameters and the examinee thetas, the classification indices were computed under each IRT-based method. The correctness of the sample indices under each method was investigated by comparing them to the respective “true” indices. Table 3-2 Summary of the Three Baseline and Three Model Misspecification Conditions Conditions Data generating model Data calibration model Baseline 1PL (Rasch) 1PL (Rasch) 2PL 2PL 3PL 3PL Model Misspecification 2PL 1PL (Rasch) 3PL 1PL (Rasch) 3PL 2PL In the three model misspecification conditions, the sample test data were first simulated by using the known 2PL and 3PL model, however, a simpler and more restricted (less parameterized) misspecified IRT model was fitted to the simulated test data to compute the sample classification indices under each IRT-based method. That is, the 1PL (Rasch) model was fitted to obtain the item parameters and theta estimates for the sample test data simulated by the 2PL and 3PL model, and the 2PL model was fitted to the sample test data simulated by the 3PL model. Using the item parameters and thetas estimated from the misspecified model, the sample classification indices were computed under each IRT-based method. The correctness of the 36 sample indices under each method when the IRT model was misspecified was investigated by comparing them to the respective “true” indices. 3.4 Evaluation Criteria The correctness of the sample indices given by each of the IRT-based method under both baseline and model misspecification conditions were evaluated by calculating the average differences between the sample and the “true” indices over the 500 sets of sample test data. The signs of the average differences reflect whether the sample indices are systematically above or below the “true” indices, and the absolute values of the average differences reflect the magnitude of the differences of the sample indices from the “true” indices. The standard deviations of the differences across the 500 sets of sample test data were also calculated to reflect the variability of the differences between the sample and the “true’ indices across the simulated sample test data. The impacts of the cut score locations on the correctness of the computed indices cannot be interpreted alone without considering the ability distribution and the characteristics of the tests, such as the test information function. The “true” test information functions that are based on the “true” item parameters and the correct IRT models, and the test information functions that are based on the sample item parameters from the baseline (correctly specified) or misspecified IRT models were calculated. Lastly, while the correctness of the sample classification indices was the central focus of this thesis, model misspecifications could have an unintended consequences on the proportions of examinees classified into pass or fail (i.e., the passing rate). This was due to the possible shifts in the theta scales among different calibrations of the true and the misspecified models, i.e., the IRT models were not concurrently calibrated, and the thetas from different models were, in fact, 37 not on a common scale. In order to document the effect of the shift in the theta scale, the “true” passing rates under the “true” model, and the sample passing rates under either the baseline (correctly specified) or misspecified model were also calculated and reported. 3.5 Summary To summarize, the correctness of the sample classification indices given by the three IRT-based methods were evaluated for both baseline and model misspecification conditions against the corresponding true values. Factors manipulated include the model specification and the cut score location. Under each condition, 500 sets of sample test data were simulated. Differences between the computed and “true” classification indices were averaged over the 500 sets of sample test data, and the standard error of the differences were also calculated as the standard deviations over the 500 sets of sample test data. The average test information functions based on the “true” and the specified models, and the passing rates under the “true” and the specified models were also calculated and reported. All simulations of the data and the estimations of the item parameters, examinee thetas, and the computations of the classification indices were performed in R (R Core Team, 2017). Examinee thetas were estimated through maximum likelihood with the catR (Magis & Rachie, 2012) package. Item parameters were estimated with the irtoys package (Partchev & Maris, 2017) that utilizes the estimation engine built in the ltm package (Rizopoulos, 2006). The classification indices of the Rudner and Lee method were computed using the cacIRT package (Lathrop, 2014), and the calculation procedure of the Guo method was programmed in R by the author. The complete R codes with descriptions can be found in Appendix C. 38 Chapter 4: Results 4.1.1 Baseline Conditions 4.1.1.1 Test Information The test information functions based on the generated “true” item parameters and the calibrated item parameters by using the correct IRT model in the three baseline conditions are shown in Figure 4-1. The solid curves represent the test information functions averaged across the 500 simulated sample test data, and the dotted curves represent the 95% variability of the test information functions across the 500 simulated sample test data. As expected, since the IRT models were correctly specified in the item calibration procedures, the test information functions based on the estimated item parameters across the 500 sample test data closely resembled the “true” test information functions in terms of the shapes and the locations of where the tests had the most information. In all three baseline conditions, the test information functions peaked at around .52 on the theta scale, which was the mean of the difficulty parameter distribution. 39 Figure 4-1 Test information functions in the baseline conditions. Solid curves represent the average test information functions across the 500 simulated sample test data. Dotted curves represent the 95% variability across the 500 simulated sample test data. 4.1.1.2 Average Differences Between Sample and “True” Indices The average differences between the sample and “true” classification indices and the standard deviations of the differences for the three baseline conditions are summarized in Tables B-1 to B-6 in Appendix B. The average differences are displayed in Figures 4-2 to 4-4. CC and CA. Figure 4-2 shows the average differences between the sample and “true” CC and CA values in all baseline conditions. In the first two baseline conditions, as shown in the left and middle panels of Figure 4-2, the absolute differences between the sample and “true” CA and CC values under all methods were trivially small (i.e. less than .01) when the theta cut score was set at or near where the tests had the highest information. It is notable that the absolute differences under Rudner method remained trivially small at all theta cut score locations. From 40 the right panel of Figure 4-2, it appeared the sample CA and CC rates were relatively more “incorrect” under all methods when the 3PL models were both used as the data generation and the data calibration models. When the cut score was set at or near the highest test information, the average absolute differences under Guo and Lee methods still remained less than .01. However, in general, the magnitude of the differences under all three methods was larger than those from the first two baseline conditions. Figure 4-2 Average differences between the sample and the “true” CC and CA values in baseline conditions. FP and FN. Figure 4-3 shows that the differences between the sample and the “true” FP and FN values in all baseline conditions. The left and middle panel of Figure 4-3 show that the differences displayed different patterns for the three methods but remained similar in between the first two baseline conditions. Under the Lee method, the sample FP values were consistently less than the “true” FP values, while the sample FN values were consistently higher than the “true” FN values across all cut score locations. For Rudner and Guo method, the directions of the 41 differences depended on the location of the cut score. The sample FP and FN values under the Rudner method were very close to their “true” values at all cut score locations. While those computed under the Guo method were also very close to the “true” values at cut score locations close to the highest information, the absolute differences under the Lee method were more than .01 for most cut score locations. From the right panel of Figure 4-3, the magnitude of the differences appeared to be much larger, especially when the cut scores were set at the negative end of the theta scale. The sample FP and FN values under both Rudner and Guo method were closer to the “true” values at cut scores set at or near the highest test information, while those under the Lee methods were consistently greater than .01 at most cut score locations. Figure 4-3 Average differences between the sample and the “true” FP and FN values in baseline conditions. Kappa. Figure 4-4 shows the average differences between the sample and “true” kappa values in all baseline conditions. The results showed that the differences generally appeared to be 42 trivially small at cut score locations at or near where the test information peaked, except for the Rudner method in the condition when 3PL models were both used as the data generation and the data calibration model. When the cut scores were set at locations where there were less test information (i.e., -1.5), the impacts on the correctness of the computed kappa seemed to be more pronounced. Figure 4-4. Average differences between the sample and the “true” kappa values in baseline conditions. 4.1.1.3 Passing Rates Lastly, the “true” passing rates and the actual passing rates for classification decisions made on both the theta and raw score scales are displayed in Figure 4-5. The results showed that the two passing rates overlapped for all cut score locations in all baseline conditions. 43 Figure 4-5 Average “true” and actual passing rates in baseline conditions. 4.1.2 Model Misspecification Conditions 4.1.2.1 Test Information The test information functions based on the generated “true” item parameters and the item parameters calibrated by using the misspecified IRT models in the three model misspecification conditions are shown in Figure 4-6. The solid curves represent the test information functions averaged across the 500 simulated sample test data, and the dotted curves represent the 95% variability of the test information functions across the 500 simulated sample test data. In general, the test information functions based on the calibrated item parameters with the misspecified models did not show good “match” to the “true” test information functions. The left panel of Figure 4-6 shows that in the condition where the 1PL models were misspecified for the 2PL generated data, the tests under the misspecified model produced less information than the “true” test information functions due to the constraint on the discrimination parameters 44 across all items. In the conditions where the 1PL or the 2PL models were misspecified for the 3PL generated data, as shown in the middle and right panels of Figure 4-6, the test information functions under the misspecified models differed from the “true” test information functions in terms of both the shapes and the locations of where the tests had the most information. These findings suggest that the tests under the misspecified models could be interpreted as functioning or performing differently due to the test information functions being different from those based on the “true” item parameters and the correctly specified models. In addition, the tests under the misspecified models would have different, or in most cases, lower score precisions due to the differences in the test information functions with regard to the misspecified and the “true” models. Figure 4-6 Average test information functions in model misspecification conditions. Solid curves represent the average test information functions across the 500 simulated sample test data. Dotted curves represent the 95% variability across the 500 simulated sample test data. 45 4.1.2.2 Average Differences Between Sample and “True” Indices The differences between the sample and “true” classification indices and the standard deviations of the differences for the three model misspecification conditions are summarized in Tables B-7 to B-12 in Appendix B. The average differences are displayed in Figures 4-7 to 4-9. CC and CA. Figure 4-7 shows the average differences between the sample and “true” CC and CA values in the three model misspecification conditions. The left panel of Figure 4-7 shows that when the 1PL model was misspecified for the 2PL generated data, the overall patterns of the differences between the computed and the “true” indices were very similar to those observed from the baseline conditions. The average differences under all methods appeared to be trivially small (i.e. less than 0.01) when the cut score was set at or near the highest test information. Figure 4-7 Average differences between the sample and the “true” CC and CA values in model misspecification conditions. 46 In the conditions where the 1PL or the 2PL models were misspecified for the 3PL generated data, as shown in the middle and right panels of Figure 4-7, the patterns were apparently different from those in the previous conditions. It could be due to the reason that the comparisons were complicated by the different information the tests provided with regard to the misspecified and the “true” IRT models. In both conditions, the average differences under the Guo and Rudner method remained trivially small when the theta cut score was set at or above -.5, while the differences under the Lee method were above .01 at most cut score locations. At cut scores below -.5, the absolute differences could reach up to .08 under the Rudner method, and up to .06 under the Lee and Guo methods in the condition where the 1PL model was misspecified (middle panel of Figure 4-7). FP and FN. Figure 4-8 shows the average differences between the sample and “true” FP and FN values in the three model misspecification conditions. In the condition when the 1PL models were misspecified for the 2PL generated data (left panel of Figure 4-8), the overall patterns of the differences were very similar to those observed from the baseline conditions. Among the three methods, the sample FP and FN values under the Rudner method appeared to be closest to their “true” values at all cut score locations. While those computed under the Guo method were also very close to the “true” values at cut score locations close to the highest information based on the misspecified models, the absolute differences under the Lee method were more than .01 at most cut score locations. 47 Figure 4-8 Average differences between the sample and the “true” FP and FN values in model misspecification conditions. As shown in the middle and right panels of Figure 4-7, the patterns of the differences under the conditions when the 1PL or the 2PL models were misspecified for the 3PL generated data were largely similar to the previous conditions, with the exception that when the cut score was set at -1.5, the absolute differences under the Rudner could reach up to .04. When cut score was set at the highest test information, again, the absolute differences under Guo and Rudner method remained trivially small. Kappa. Figure 4-9 shows the average differences between the sample and “true” kappa values in the three model misspecification conditions. A common observation was that the average differences were trivially small (less than .01) when the cut score was set at or near the highest test information. Another similarity was that the absolute differences became larger at cut score locations with little test information (i.e. -1.5). 48 Figure 4-9 Average differences between the sample and the “true” kappa values in model misspecification conditions. 4.1.2.3 Passing Rates Figure 4-10 shows the “true” passing rates and the actual passing rates of the classification decisions made in the three model misspecification conditions. The results showed that the actual passing rates under the misspecified models were different from those would have been observed if the correct models were used. The findings were expected as the thetas estimated under different models were not on the same scales. The maximum absolute differences were .028 when the 1PL model was misspecified for the 2PL generated data, .089 when the 1PL model was misspecified for the 3PL generated data, and .048 when the 1PL model was misspecified for the 3PL generated data. 49 Figure 4-10 Average “true” and actual passing rates in model misspecification conditions. 50 Chapter 5: Conclusions and Discussions Simulation studies were conducted in this thesis to evaluate the correctness of the classification indices given by the three IRT-based methods, namely, the Rudner method, the Guo method, and the Lee method, on tests with a single cut score for pass-or-fail decision. The test data were simulated by taking the item parameters as random variables drawn from the assumed parameter distributions, and the results could be generalized to tests with items having similar distributions of parameters rather than a particular test with particular fixed item parameters. Specifically, the correctness of these indices was studied under the baseline conditions, where the calibration model was correctly specified and was the same as the model used in generating the data, as well as under the model misspecification conditions, where the calibration model was misspecified and was more restricted than the model used in generating the data. The effects of the cut score location on the correctness of the indices were also evaluated. Simulation designs were used as the method in answering the research questions posed in the thesis because the data could be conveniently generated by the “true” IRT model with the known item and examinee parameters, and the “true” classification indices could be obtained so that the correctness of the indices under the three IRT-based methods could be evaluated. The results from the baseline conditions revealed the following: (1) The overall CC and CA rates under the three IRT-based methods were mostly correct (absolute differences from the “true” values of less than .01), especially when the location of the cut score was set at or close to where the test had the most information. (2) In the conditions where the 1PL and 2PL models were the correct models, it is notable that the Rudner method was able to produce overall CC and CA rates that were closest to the “true” values at all cut score locations. (3) In the condition 51 where the 3PL model was the correct model, the Rudner method appeared to be outperformed by the other two methods. This finding largely agreed with those concluded by Lathrop and Chen (2013) in their simulation studies, in which their results suggested that under the assumption the “true” item parameters were known and the correct IRT model was used to estimate ability, the Rudner method under the 3PL model did not perform as well as it would otherwise under the 1PL and 2PL model. One possible explanation is that with the 3PL, the distribution of the estimated thetas converged to asymptotic normal slower (Lathrop and Chen, 2013). In other words, the required assumption of asymptotic normal distribution of estimated thetas was not fully met at the test length of 60 items in the simulation conditions. In this case, the Guo and the Lee methods provided more correct overall CC and CA rates when the cut score was set at or near the highest information, as these two methods do not require such assumption. (4) For FN and FP, the absolution differences under Guo and Rudner method remained trivially small when the cut score was set close to the highest test information in all conditions. However, it is notable that the Lee method produced the largest absolute differences (more than 0.01) between the computed and the “true” values in most conditions, even at cut score locations where the tests had high information in some cases. (5) All methods produced kappa values that were close to the “true” values at cut score location close to the highest test information, except for Rudner method when the data calibration model was the 3PL model. The effects of model misspecifications on the correctness of the computed indices were also evaluated. The results from the model misspecification conditions revealed the following: (1) The computed overall CC and CA rates given by the Rudner and the Guo methods were largely correct (absolute differences within .01) with the cut score set at or near the highest test information. (2) When the 1PL or 2PL model was misspecified for the 3PL data, the computed 52 classification indices under the Lee method had absolute differences of above .01 at most cut score locations. (3) The last observation, perhaps the most critical one, is that under all model misspecification conditions, although the indices under all three methods appeared to be mostly correct when evaluated against the “true” indices at cut score locations near the highest test information, there were discrepancies in the passing rates. In other words, although the quality of the classification decisions under model misspecified conditions was nearly as good as those when the correct model was specified, the classification decisions being evaluated are actually not the same as reflected by the different proportions of examinees classified in each category due to the shift of the theta scale from one calibration to another and also due to different functioning of the tests as reflected by the test information functions. For example, a licensure exam regularly passes around 20% of the examinees by using a cut score on the theta scale under the correct IRT model in operation. If an incorrect IRT model is misspecified in the computations of the classification indices, even if the computed indices would appear to be minimally affected, fundamentally, the classification decisions being evaluated are not the same as those made in operation. Two implications can be concluded from the findings of this thesis. (1) Since the classification indices are most accurate as long as the cut score location has high test information even when the model is misspecified, it is advised, in practice, to develop or assemble a test that has high test information at or near the cut score set by the standard setting panel. This finding also echoes with the common recommendations of designing tests with high test information at or near the cut score if one wants to maximize the overall CC and CA. Reversely, one could place the cut score at or near where the test has the highest information to obtain the maximized and most accurate overall CC and CA, but the validity of placing the cut score this way must first 53 be established. (2) Unlike in a simulation study, in practice, the consistency and the accuracy of the classification decisions are evaluated after the cuts are set and decisions are made accordingly. Although operationally, the IRT model could be misspecified for various practical reasons, it is recommended that the same IRT model be used in item calibrations, scoring, and computing the classification indices even if the model is misspecified so that the classification decisions being made and evaluated will remain consistent. There are several limitations in this thesis. First, only tests with dichotomous items and a single cut score were considered in the simulation conditions, whereas investigations of the correctness of the classification indices for tests with mixed items and multiple cut scores are also of practical importance. Second, the test data were simulated by taking the item parameters as random variables drawn from the assumed distributions, and examinee abilities drawn from a standard normal distribution. The results can only be generalized to tests with item parameters and examinee abilities having the same distributions as the ones used in the simulations. The effects of having a non-normal distribution of ability, and items with different parameter distributions on the correctness of the indices are also worth investigating. Third, it should be noted that the test data were generated by using either the 1PL, 2PL, or 3PL model, and in each simulation condition there was a clearly correct, “true” model. In reality, model fit can only be evaluated as a matter of degree in how well a model is relatively to the data at hand. Factors that can cause data-model misfit such as multidimensionality of the data and local dependence that cannot be handled by the 1PL, 2PL, and 3PL IRT models could be investigated in the future simulation studies. 54 Bibliography Bejar, I. I. (2008). Standard setting: What is it? Why is it important? R&D Connections, 7, 1-6. Retrieved from https://www.ets.org/Media/Research/pdf/RD_Connections7.pdf Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. doi:10.1007/s11336-003-1110-4 de Ayala, R. (2009). The Theroy and Practice of Item Response Theory. New York, NY: The Guilford Press. Ercikan, K., & Julian, M. (2010). Classification accuracy of assigning student performance to proficiency levels: Guidelines for assessment design. Applied Measurement in Education, 15(3), 269-294. doi:10.1207/S15324818AME1503_3 Guo, F. (2006). Expected classification accuracy using the latent distribution. Practical assessment, 11(6), 1-9. Retrieved from https://pareonline.net/getvn.asp?v=11&n=6 Hambleton, R. K., & Novick, M. R. (1973). Toward an integration of theory and method for criterion-referenced tests. Journal of Educational Measurement, 10(3), 159-170. Hanson, B. A., & Brennan, R. L. (1990). An investigation of classification consistency indexes estimated under alternative strongtrue score models. Journal of Educational Measurement, 27(4), 345-359. doi:10.1111/j.1745-3984.1990.tb00753.x Hubley, A. M., & Zumbo, B. D. (2011). Validity and the consequences of test interpretation and use. Social Indicators Research, 103(2), 219-230. doi:10.1007/s11205-011-9843-4 Huynh, H. (1976). On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement, 13(4), 253-264. doi:10.1111/j.1745-3984.1976.tb00016.x 55 Huynh, H. (1990). Computation and statistical inference for decision consistency indexes based on the Rasch Model. Journal of Education Statistics, 15(4), 353-368. doi:10.3102/10769986015004353 Joint Committee on Standards for Educational and Psychological Testing (U.S.), National Council on Measurement in Education, American Psychological Association, & American Educational Research Association. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Lathrop, Q. N. (2014). R package cacIRT: Estimation of classification accuracy and consistency under item response theory. Applied Psychological Measurement, 38(7), 581-582. doi:10.1177/0146621614536465 Lathrop, Q. N., & Cheng, Y. (2013). Two approaches to estimation of classification accuracy rate under item response theory. Applied Psychological Measurement, 37(3), 226-241. doi:10.1177/0146621612471888 Li, S. (2006). Evaluating the consistency and accuracy of proficiency classifications using item response theory. Unpublished doctoral dissertation. University of Massachusetts, Amherst, MA. Lee, W. (2010). Classification consistency and accuracy for complex assessments using item response theory. Journal of Educational Measurement, 47(1), 1-17. doi:10.1111/j.1745-3984.2009.00096.x Lee, W., Brennan, R. L., & Wan, L. (2009). Classification consistency and accuracy for complex assessments under the compound multinomial model. Applied Psychological Measurement, 33(5), 374-390. doi:10.1177/0146621608321759 56 Livingston, S. A., & Lewis, C. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32(2), 179-197. doi:10.1111/j.1745-3984.1995.tb00462.x Livingston, S. A., & Wingersky, M. S. (1979). Assessing the reliability of tests used to make Pass/Fail decisions. Journal of Educational Measurement, 16(4), 247-260. doi:10.1111/j.1745-3984.1979.tb00106.x Lord, F. M., & Wingersky, M. S. (1984). Comparison of IRT true-score and equipercentile observed-score "equatings". Applied Psychological Measurement, 8(4), 453-461. doi:10.1177/014662168400800409 Magis, D., & Raiche, G. (2012). Random generation of response patterns under computerized adaptive testing with the R package catR. Journal of Statistical Software, 48(8), 1-31. Martinaeu, J. A. (2007). An expansion and practical evaluation of expected classification accuracy. Applied Psychological Measurement, 31(3), 181-194. doi:10.1177/0146621606291557 Partchev, I., & Maris, G. (2017). irtoys: A collection of functions related to item response theory (IRT).R package version 0.2.1. Retrieved from https://CRAN.R-project.org/package=irtoys Puhan, G., & Gall, L. (2012). Reliability of pass and fail decision on tests employing cut scores. Psychological Studies, 57(3), 273-282. doi:10.1007/s12646-012-0147-9 R Core Team (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. Retrieved from http://www.R-project.org/ Rizopoulos, D. (2006). ltm: An R package for latent variable modelling and item response theory analyses. Journal of Statistical Software, 17(5), 1-25. 57 Rudner, L. M. (2001). Computing the expected proportions of misclassified examinees. Practical Assessment, 7(14), 1-5. Retrieved from https://pareonline.net/getvn.asp?v=7%26n=14 Rudner, L. M. (2005). Expected classification accuracy. Practical Assessment, 10(13), 1-4. Retrieved from https://pareonline.net/pdf/v10n13.pdf Schulz, E. M., Kolen, M. J., & Nicewander, W. A. (1999). A rationale for defining achievement levels using IRT-estimated domain scores. Applied Psychological Measurement, 23(4), 347-362. doi:10.1177/01466219922031464 Subkoviak, M. J. (1976). Estimating reliability from a single administration of a criterion-referenced test. Journal of Educational Measurement, 13(4), 265-276. doi:10.1111/j.1745-3984.1976.tb00017.x Swaminathan, H., Hambleton, R. K., & Algina, J. (1974). Reliability of criterion-referenced tests: a decision-theoretic formulation. Journal of Educational Measurement, 11(4), 263-267. doi:10.1111/j.1745-3984.1974.tb00998.x Thissen, D., Pommerich, M., Billeaud, K., & Williams, V. S. L. (1995). Item response theory for scores on tests including polytomous items with ordered responses. Applied Psychological Measurement, 19(1), 39-49. doi:10.1177/014662169501900105 Wang, T., Kolen, M. J., & Harris, D. J. (2000). Psychometric properties of scale scores and performance levels for performance assessments using polytomous IRT. Journal of Educational Measurement, 37(2), 141-162. doi:10.1111/j.1745-3984.2000.tb01080.x Wyse, A. E., & Hao, S. (2012). An evaluation of item response theory classification accuracy and consistency indices. Applied Psychological Measurement, 36(7), 602-624. doi:10.1177/0146621612451522 58 Zhang, S., Du, J., Chen, P., Xin, T., & Chen, F. (2017). Using procedure based on item response theory to evaluate classification consistency indices in the practice of large-scale assessment. Frontiers in Psychology, 8, 1676. doi:10.3389/fpsyg.2017.01676 59 Appendices Appendix A Item Parameters of All Dichotomous Items in 2013 NAEP Grade 12 Mathematics Assessment NAEP ID a b c NAEP ID a b c NAEP ID a b c M192801 0.81 -1.7 0.19 M185702 3.85 1.41 0.1 M198801 0.96 0.49 0.26 M188701 0.94 -0.22 0.25 M175001 0.9 -0.12 0.32 M193601 1.44 -0.42 0.1 M182401 2.41 1.26 0.18 M188101 1.15 0.12 0.26 M199401 1.01 0.52 0.33 M185901 1.8 1.73 0.17 M180301 1.12 2.02 0.05 M185501 1.52 0.51 0.28 M196601 0.84 -0.43 0.16 M182201 3.72 2.07 0.2 M199001 1.32 -1.03 0.2 M183901 1.33 0.19 - M182901 1.57 1.82 0.26 M189601 0.99 -0.13 0.22 M303801 0.65 -1.12 0.19 M198101 1.01 0.13 0.18 M194201 0.77 -0.24 0.16 M305601 0.92 -0.31 0.22 M198201 2.51 -0.33 0.41 M178401 1.14 -0.14 0.13 M304501 1.18 0.62 0.15 M198202 1.35 0.22 0.06 M199101 2.74 1.16 0.17 M305201 1.6 1.08 0.17 M198203 3.45 -0.16 0.35 M190901 0.62 0.82 0.08 M305301 0.47 -1.79 0.19 M176701 0.77 1.49 0.1 M199301 1.61 1.86 0.28 M304901 1.02 0.03 0.18 M177201 1.04 0.24 0.2 M184301 0.99 -1.22 0.21 M304201 0.96 1.01 0.3 M177202 0.93 1.13 - M199501 1.08 0.68 0.28 M301701 1.12 1.04 0.14 M174701 1.27 1.25 0.15 M180601 1.76 0.84 0.21 M305101 1.37 0.88 0.17 M309201 1.4 -0.33 0.22 M180701 1.29 -0.02 0.15 M197101 1.23 0.67 0.12 M310601 0.44 0.71 0.17 M181201 0.67 -0.73 - M181701 0.72 -1.84 0.16 M311001 0.5 0.25 0.23 M181203 0.49 3.35 - M197201 0.55 -0.45 0.18 M310401 1.34 -0.53 - M184401 1.25 0.48 0.16 M197301 0.6 -0.08 0.15 M309101 1.49 -0.19 0.15 M199601 0.92 0.18 0.22 M197302 0.76 -0.13 0.1 M307901 0.71 -0.76 0.19 M183801 0.97 1.95 0.16 M197303 1.29 1.69 - M307701 1.6 -0.26 0.19 M189201 1.07 0.83 0.21 M197304 1.37 1.36 - M311501 0.81 1.35 0.22 M183101 2.12 1.51 - M197401 1.29 -0.16 0.18 M308201 1.25 1.12 0.14 M176601 1.29 1.17 0.18 M179101 0.75 0.11 0.18 M309901 1.16 1.52 0.3 M199701 0.77 -1.42 0.17 M175101 0.35 4.62 0.14 M309601 0.78 1.56 0.3 M199801 0.92 0.89 0.17 M183301 1.31 1.64 0.18 M187601 1.27 -0.36 0.11 M186001 0.92 0.21 - M195401 0.13 -0.07 0.21 M193801 0.95 -0.67 0.27 M175301 0.85 -0.13 0.16 M184201 1.19 0.73 - M193101 0.85 1.15 - M199901 0.9 0.4 0.21 M177001 1.14 -0.2 0.29 M198601 2.09 1.26 0.1 M304101 0.91 -0.19 0.2 M190401 0.95 -0.52 0.15 M189301 1.61 0.25 0.29 M182101 1.01 1.77 0.32 M174901 1.8 1.4 0.13 M188401 2.33 1.16 0.15 M182102 1.32 1.63 0.07 M182601 1.11 0.54 0.22 M189501 0.61 0.76 0.2 M200001 0.49 -0.22 0.22 M197701 1.72 1.25 0.22 M187101 1.31 0.44 0.26 M181801 1.73 1.02 - M197801 1.12 0.72 0.22 M178601 1.33 0.27 0.09 M200101 0.88 0.26 0.13 M180101 2.4 1.87 0.14 M192701 0.86 0.5 0.19 M304701 0.87 1.58 0.16 M185201 0.77 -0.56 - M192901 1.11 -0.64 0.18 M177901 1.16 1.61 0.36 M197901 1.13 -0.02 0.05 M198701 0.75 0.19 - M185001 2.4 1.43 0.13 M198001 1.31 1.06 0.21 M177601 0.5 1.13 0.18 M302401 0.77 -0.03 0.14 M183001 1.09 -0.55 0.23 M190501 1.01 0.91 0.14 M306201 0.31 -0.21 0.24 M175701 1 0.64 0.23 M194101 1.02 0.54 0.14 M305801 1.24 -0.1 0.15 M194501 0.23 0.24 0.19 M181901 1.1 0.3 0.17 M301101 0.96 0.77 0.18 M183201 0.99 0.62 0.2 M192301 1.35 0.18 0.3 M305501 1.12 -0.3 - 60 NAEP ID a b c NAEP ID a b c NAEP ID a b c M301201 1.41 0.29 0.11 M308401 1.11 0.31 0.11 M176401 1.17 1.71 0.22 M306501 1.14 0.02 - M309801 0.94 -0.42 0.15 M311301 1.29 1.93 0.22 M301601 1.29 1.2 0.29 M308001 0.78 0.86 - M310901 0.8 2.08 0.23 M305701 1.2 0.85 0.2 M196501 0.81 0.58 0.35 M186101 2.31 0.79 0.24 M303001 1.18 1.23 0.14 M310201 1.21 1.6 0.04 M308901 2.47 1.35 0.16 M307801 0.56 -0.84 0.2 M308601 1.38 -0.16 0.15 M185801 2.01 1.68 0.12 M308801 0.91 0.31 0.2 M310301 1.05 0.27 0.12 Note: Items with missing c parameters were originally calibrated with the 2PL model. 61 Appendix B Tables of Average Differences (and SD) Between the Sample and “True” CA and CC Indices Table B-1 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 1PL. Data Calibration Model: 1PL) Rudner Guo Lee Cut γ γ+ γ- γ γ+ γ- γ γ+ γ- -1.5 0.003 0.004 -0.007 -0.012 -0.007 0.019 -0.011 -0.007 0.018 (0.004) (0.006) (0.009) (0.006) (0.004) (0.01) (0.005) (0.004) (0.008) -1 0.004 0.004 -0.008 -0.004 -0.010 0.013 -0.004 -0.013 0.017 (0.004) (0.008) (0.01) (0.006) (0.007) (0.011) (0.005) (0.006) (0.010) -0.5 0.003 0.003 -0.006 0.003 -0.009 0.006 0.001 -0.018 0.016 (0.004) (0.009) (0.009) (0.004) (0.008) (0.010) (0.005) (0.008) (0.010) 0 0.002 0.002 -0.004 0.004 -0.002 -0.002 0.004 -0.018 0.014 (0.004) (0.008) (0.008) (0.004) (0.008) (0.008) (0.004) (0.008) (0.009) 0.5 0.000 0.000 -0.001 0.000 0.006 -0.006 0.004 -0.015 0.011 (0.004) (0.007) (0.006) (0.004) (0.007) (0.006) (0.004) (0.007) (0.007) 1 0.000 0.000 0.001 -0.005 0.012 -0.006 0.003 -0.011 0.007 (0.003) (0.005) (0.004) (0.003) (0.006) (0.004) (0.003) (0.006) (0.005) 1.5 -0.001 0.000 0.001 -0.008 0.013 -0.005 0.002 -0.006 0.004 (0.002) (0.003) (0.003) (0.003) (0.004) (0.002) (0.002) (0.004) (0.003) γ: marginal classification accuracy rate, γ+: marginal false positive rate, γ- :marginal false negative rate Table B-2 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 1PL. Data Calibration Model: 1PL) Rudner Guo Lee Cut p κ p κ p κ -1.5 0.004 0.055 -0.005 0.113 -0.012 0.101 (0.005) (0.022) (0.005) (0.023) (0.005) (0.026) -1 0.005 0.024 0.002 0.057 -0.004 0.054 (0.004) (0.010) (0.004) (0.010) (0.004) (0.011) -0.5 0.004 0.008 0.006 0.023 0.003 0.026 (0.004) (0.010) (0.004) (0.01) (0.004) (0.011) 0 0.001 0.003 0.005 0.009 0.007 0.013 (0.004) (0.009) (0.004) (0.009) (0.005) (0.009) 0.5 0.000 0.002 0.000 0.008 0.006 0.007 (0.004) (0.009) (0.004) (0.009) (0.004) (0.009) 1 -0.001 0.006 -0.004 0.017 0.005 0.007 (0.004) (0.012) (0.004) (0.012) (0.004) (0.012) 1.5 -0.001 0.014 -0.006 0.038 0.003 0.009 (0.003) (0.017) (0.003) (0.018) (0.003) (0.017) p: marginal classification consistency rate, κ: Cohen’s kappa 62 Table B-3 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 2PL. Data Calibration Model: 2PL) Rudner Guo Lee Cut γ γ+ γ- γ γ+ γ- γ γ+ γ- -1.5 0.002 0.006 -0.008 -0.013 -0.007 0.019 -0.012 -0.009 0.021 (0.003) (0.002) (0.003) (0.004) (0.002) (0.004) (0.006) (0.004) (0.009) -1 0.002 0.006 -0.008 -0.004 -0.010 0.013 -0.004 -0.015 0.019 (0.003) (0.003) (0.003) (0.004) (0.002) (0.004) (0.005) (0.007) (0.009) -0.5 0.001 0.005 -0.006 0.002 -0.008 0.006 0.001 -0.018 0.017 (0.003) (0.003) (0.003) (0.004) (0.003) (0.004) (0.004) (0.007) (0.008) 0 0.000 0.003 -0.003 0.003 -0.002 -0.001 0.004 -0.017 0.013 (0.003) (0.003) (0.003) (0.003) (0.003) (0.003) (0.004) (0.007) (0.006) 0.5 -0.001 0.002 -0.001 -0.001 0.006 -0.005 0.003 -0.013 0.010 (0.004) (0.003) (0.003) (0.004) (0.003) (0.003) (0.004) (0.006) (0.005) 1 -0.002 0.001 0.001 -0.006 0.011 -0.005 0.001 -0.008 0.007 (0.003) (0.003) (0.002) (0.003) (0.003) (0.002) (0.003) (0.005) (0.003) 1.5 -0.002 0.000 0.001 -0.008 0.012 -0.004 0.000 -0.005 0.005 (0.002) (0.002) (0.002) (0.002) (0.002) (0.001) (0.002) (0.003) (0.002) γ: marginal classification accuracy rate, γ+: marginal false positive rate, γ- :marginal false negative rate Table B-4 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 2PL. Data Calibration Model: 2PL) Rudner Guo Lee Cut p κ p κ p κ -1.5 0.003 0.066 -0.004 0.137 -0.015 0.129 (0.004) (0.022) (0.004) (0.023) (0.006) (0.032) -1 0.003 0.023 0.003 0.065 -0.004 0.065 (0.004) (0.010) (0.004) (0.010) (0.004) (0.011) -0.5 0.001 0.004 0.006 0.024 0.003 0.028 (0.004) (0.010) (0.004) (0.010) (0.004) (0.011) 0 0.000 -0.001 0.004 0.007 0.005 0.011 (0.004) (0.009) (0.004) (0.009) (0.004) (0.009) 0.5 -0.002 0.000 -0.001 0.005 0.004 0.005 (0.004) (0.009) (0.004) (0.009) (0.004) (0.009) 1 -0.003 0.004 -0.004 0.014 0.002 0.005 (0.003) (0.011) (0.004) (0.011) (0.004) (0.010) 1.5 -0.003 0.016 -0.006 0.036 0.000 0.011 (0.003) (0.017) (0.003) (0.018) (0.003) (0.016) p: marginal classification consistency rate, κ: Cohen’s kappa 63 Table B-5 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 3PL. Data Calibration Model: 3PL) Rudner Guo Lee Cut γ γ+ γ- γ γ+ γ- γ γ+ γ- -1.5 0.011 0.011 -0.021 -0.038 -0.014 0.051 -0.008 -0.005 0.014 (0.005) (0.003) (0.005) (0.008) (0.002) (0.009) (0.006) (0.003) (0.008) -1 0.007 0.009 -0.016 -0.012 -0.025 0.037 0.003 -0.019 0.016 (0.004) (0.004) (0.004) (0.005) (0.003) (0.006) (0.005) (0.005) (0.008) -0.5 -0.003 0.012 -0.010 0.006 -0.025 0.019 0.009 -0.025 0.016 (0.005) (0.005) (0.004) (0.004) (0.004) (0.005) (0.005) (0.007) (0.007) 0 -0.010 0.014 -0.004 0.007 -0.012 0.005 0.007 -0.022 0.014 (0.006) (0.006) (0.004) (0.004) (0.004) (0.004) (0.005) (0.008) (0.007) 0.5 -0.009 0.009 0.000 0.001 0.003 -0.003 0.005 -0.016 0.011 (0.005) (0.005) (0.003) (0.004) (0.004) (0.003) (0.004) (0.007) (0.005) 1 -0.006 0.004 0.001 -0.005 0.010 -0.005 0.003 -0.011 0.008 (0.004) (0.004) (0.002) (0.003) (0.003) (0.002) (0.004) (0.005) (0.003) 1.5 -0.004 0.002 0.002 -0.008 0.012 -0.004 0.002 -0.007 0.005 (0.003) (0.003) (0.002) (0.003) (0.003) (0.002) (0.003) (0.004) (0.003) γ: marginal classification accuracy rate, γ+: marginal false positive rate, γ- :marginal false negative rate Table B-6 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 3PL. Data Calibration Model: 3PL) Rudner Guo Lee Cut p κ p κ p κ -1.5 0.016 0.100 -0.009 0.231 -0.012 0.172 (0.005) (0.026) (0.006) (0.024) (0.008) (0.027) -1 0.010 0.027 0.012 0.148 0.003 0.122 (0.006) (0.018) (0.006) (0.014) (0.005) (0.016) -0.5 -0.006 -0.025 0.019 0.068 0.013 0.061 (0.006) (0.018) (0.006) (0.014) (0.005) (0.016) 0 -0.018 -0.037 0.011 0.022 0.011 0.023 (0.008) (0.016) (0.005) (0.010) (0.005) (0.011) 0.5 -0.015 -0.027 0.001 0.009 0.007 0.009 (0.007) (0.014) (0.004) (0.009) (0.005) (0.010) 1 -0.010 -0.015 -0.004 0.018 0.004 0.006 (0.005) (0.015) (0.004) (0.012) (0.004) (0.011) 1.5 -0.007 -0.007 -0.005 0.042 0.002 0.010 (0.004) (0.023) (0.003) (0.019) (0.003) (0.017) p: marginal classification consistency rate, κ: Cohen’s kappa 64 Table B-7 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 2PL. Data Calibration Model: 1PL) Rudner Guo Lee Cut γ γ+ γ- γ γ+ γ- γ γ+ γ- -1.5 -0.005 0.010 -0.004 -0.020 -0.005 0.025 -0.018 -0.006 0.024 (0.003) (0.005) (0.006) (0.006) (0.003) (0.009) (0.007) (0.006) (0.012) -1 -0.003 0.009 -0.005 -0.009 -0.007 0.016 -0.008 -0.012 0.020 (0.003) (0.006) (0.006) (0.005) (0.004) (0.008) (0.006) (0.007) (0.011) -0.5 -0.001 0.006 -0.005 0.000 -0.006 0.006 0.000 -0.016 0.016 (0.004) (0.006) (0.006) (0.004) (0.005) (0.007) (0.005) (0.008) (0.010) 0 -0.001 0.004 -0.003 0.001 0.000 -0.001 0.003 -0.016 0.013 (0.003) (0.005) (0.005) (0.003) (0.005) (0.006) (0.004) (0.010) (0.010) 0.5 -0.004 0.003 0.001 -0.004 0.008 -0.004 0.001 -0.012 0.011 (0.004) (0.004) (0.004) (0.004) (0.005) (0.004) (0.004) (0.007) (0.007) 1 -0.007 0.003 0.004 -0.011 0.014 -0.004 -0.003 -0.007 0.010 (0.003) (0.004) (0.003) (0.003) (0.004) (0.003) (0.003) (0.006) (0.005) 1.5 -0.008 0.004 0.004 -0.015 0.017 -0.002 -0.005 -0.002 0.008 (0.002) (0.003) (0.002) (0.003) (0.003) (0.002) (0.003) (0.004) (0.003) γ: marginal classification accuracy rate, γ+: marginal false positive rate, γ- :marginal false negative rate Table B-8 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 2PL. Data Calibration Model: 1PL) Rudner Guo Lee Cut p κ p κ p κ -1.5 -0.007 0.078 -0.013 0.141 -0.022 0.145 (0.004) (0.022) (0.004) (0.024) (0.006) (0.031) -1 -0.005 0.032 -0.004 0.067 -0.009 0.076 (0.004) (0.010) (0.004) (0.01) (0.004) (0.011) -0.5 -0.002 0.006 0.002 0.023 0.002 0.033 (0.004) (0.010) (0.004) (0.01) (0.004) (0.011) 0 -0.002 -0.004 0.001 0.003 0.006 0.011 (0.004) (0.009) (0.004) (0.009) (0.005) (0.009) 0.5 -0.006 -0.005 -0.005 0.000 0.002 0.005 (0.004) (0.009) (0.004) (0.009) (0.004) (0.009) 1 -0.010 0.004 -0.011 0.012 -0.004 0.009 (0.003) (0.011) (0.003) (0.012) (0.004) (0.011) 1.5 -0.011 0.023 -0.015 0.041 -0.007 0.024 (0.003) (0.018) (0.003) (0.018) (0.003) (0.016) p: marginal classification consistency rate, κ: Cohen’s kappa 65 Table B-9 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 3PL. Data Calibration Model: 1PL) Rudner Guo Lee Cut γ γ+ γ- γ γ+ γ- γ γ+ γ- -1.5 0.058 -0.018 -0.040 0.042 -0.020 -0.022 0.048 -0.025 -0.023 (0.007) (0.003) (0.006) (0.008) (0.002) (0.007) (0.008) (0.004) (0.009) -1 0.033 -0.017 -0.016 0.018 -0.026 0.008 0.027 -0.033 0.006 (0.007) (0.006) (0.006) (0.007) (0.005) (0.007) (0.008) (0.008) (0.012) -0.5 0.004 0.000 -0.003 0.003 -0.012 0.010 0.010 -0.027 0.017 (0.005) (0.007) (0.007) (0.005) (0.006) (0.008) (0.005) (0.012) (0.013) 0 0.003 0.001 -0.003 0.005 -0.002 -0.003 0.012 -0.023 0.011 (0.004) (0.006) (0.006) (0.004) (0.006) (0.006) (0.005) (0.012) (0.011) 0.5 0.008 -0.004 -0.004 0.006 0.004 -0.009 0.016 -0.020 0.005 (0.004) (0.005) (0.004) (0.004) (0.005) (0.004) (0.005) (0.009) (0.007) 1 0.008 -0.005 -0.003 0.003 0.007 -0.009 0.013 -0.016 0.002 (0.003) (0.003) (0.003) (0.003) (0.004) (0.003) (0.004) (0.006) (0.005) 1.5 0.005 -0.004 -0.001 -0.002 0.008 -0.006 0.008 -0.010 0.002 (0.003) (0.002) (0.002) (0.003) (0.003) (0.002) (0.003) (0.004) (0.003) γ: marginal classification accuracy rate, γ+: marginal false positive rate, γ- :marginal false negative rate Table B-10 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 3PL. Data Calibration Model: 1PL) Rudner Guo Lee Cut p κ p κ p κ -1.5 0.078 -0.041 0.061 0.036 0.063 0.024 (0.010) (0.027) (0.010) (0.025) (0.011) (0.035) -1 0.045 -0.013 0.032 0.032 0.036 0.056 (0.006) (0.013) (0.006) (0.013) (0.005) (0.017) -0.5 0.005 0.003 0.006 0.021 0.014 0.055 (0.006) (0.013) (0.006) (0.013) (0.005) (0.017) 0 0.003 0.005 0.006 0.010 0.018 0.033 (0.005) (0.010) (0.005) (0.010) (0.006) (0.012) 0.5 0.011 -0.004 0.010 0.001 0.022 0.010 (0.005) (0.010) (0.005) (0.010) (0.006) (0.011) 1 0.011 -0.012 0.007 0.001 0.018 -0.007 (0.004) (0.013) (0.004) (0.012) (0.005) (0.012) 1.5 0.007 -0.012 0.002 0.018 0.012 -0.013 (0.003) (0.019) (0.003) (0.018) (0.004) (0.019) p: marginal classification consistency rate, κ: Cohen’s kappa 66 Table B-11 Average Differences (and SD) Between Sample and “True” CA Indices (Data Generating Model: 3PL. Data Calibration Model: 2PL) Rudner Guo Lee Cut γ γ+ γ- γ γ+ γ- γ γ+ γ- -1.5 0.029 -0.007 -0.022 0.005 -0.014 0.009 0.015 -0.016 0.000 (0.005) (0.003) (0.004) (0.005) (0.002) (0.004) (0.006) (0.004) (0.008) -1 0.012 -0.002 -0.010 0.001 -0.016 0.016 0.007 -0.021 0.015 (0.004) (0.003) (0.004) (0.004) (0.003) (0.004) (0.006) (0.007) (0.010) -0.5 0.006 0.001 -0.007 0.007 -0.012 0.005 0.011 -0.025 0.014 (0.004) (0.003) (0.004) (0.004) (0.003) (0.004) (0.005) (0.008) (0.008) 0 0.007 -0.002 -0.005 0.009 -0.004 -0.004 0.013 -0.024 0.011 (0.004) (0.003) (0.003) (0.004) (0.003) (0.003) (0.005) (0.009) (0.008) 0.5 0.004 -0.002 -0.001 0.002 0.005 -0.007 0.009 -0.018 0.009 (0.004) (0.003) (0.003) (0.004) (0.003) (0.003) (0.004) (0.007) (0.005) 1 -0.002 0.000 0.002 -0.007 0.013 -0.006 0.002 -0.011 0.008 (0.003) (0.003) (0.002) (0.003) (0.003) (0.002) (0.003) (0.005) (0.003) 1.5 -0.006 0.002 0.004 -0.015 0.018 -0.003 -0.004 -0.004 0.008 (0.003) (0.003) (0.002) (0.003) (0.003) (0.001) (0.003) (0.004) (0.002) γ: marginal classification accuracy rate, γ+: marginal false positive rate, γ- :marginal false negative rate Table B-12 Average Differences (and SD) Between Sample and “True” CC Indices (Data Generating Model: 3PL. Data Calibration: 2PL) Rudner Guo Lee Cut p κ p κ p κ -1.5 0.038 0.07 0.019 0.142 0.019 0.132 (0.007) (0.023) (0.006) (0.023) (0.008) (0.027) -1 0.016 0.063 0.010 0.102 0.009 0.124 (0.005) (0.013) (0.005) (0.013) (0.005) (0.017) -0.5 0.008 0.038 0.011 0.055 0.014 0.080 (0.005) (0.013) (0.005) (0.013) (0.005) (0.017) 0 0.008 0.015 0.012 0.021 0.019 0.034 (0.005) (0.010) (0.005) (0.010) (0.006) (0.011) 0.5 0.005 0.002 0.004 0.008 0.013 0.009 (0.004) (0.009) (0.004) (0.009) (0.005) (0.010) 1 -0.002 0.004 -0.006 0.015 0.003 0.003 (0.004) (0.012) (0.004) (0.013) (0.004) (0.011) 1.5 -0.009 0.024 -0.014 0.047 -0.006 0.016 (0.003) (0.018) (0.003) (0.020) (0.003) (0.018) p: marginal classification consistency rate, κ: Cohen’s kappa 67 Appendix C - R Code with Description library(dplyr) #data manipulation library(cacIRT) #Lee and Rudner estimates, transform theta to rawscore using TCC library(catR) #Estimate theta (ML) and its standard error from item parameters and responses library(irtoys) #Estimate IRT parameters given item responses through “ltm” engine #Function for generating IRT item parameters # #Arguments: # num_item: number of items # b_mean: mean of b parameter # b_sd: standard deviation of b parameter # a_mean: mean of a parameter # a_sd: standard deviation of a parameter # c_mean: mean of c parameter # c_sd: standard deviation of c parameter #Return: # A data frame with num_item number of rows and 4 columns (a, b, c, and d parameters) gen_parameters <- function(num_item, b_mean, b_sd, a_mean = 1, a_sd = 0, c_mean = 0, c_sd = 0){ a <- log(rlnorm(num_item, a_mean, a_sd)) #log normal distribution for a parameters b <- rnorm(num_item, b_mean, b_sd) c <- rnorm(num_item, c_mean, c_sd) c <- ifelse(c >= 0, c, 0) #Ensure no c values are negative. If negative, forces to 0. d <- rep(1, num_item) #d parameters are needed (all fixed at 1) for some functions to work item_parameters <- data.frame(a = a, b = b, c = c, d = d) return(item_parameters) } #Custom function for rounding to integers #Argument: # x: numerical value #Return: # a rounded integer rounding <- function(x){ if (x - floor(x) < .5){ return(floor(x)) } else { return(ceiling(x))}} 68 #Function for generating persons' thetas # #Arguments: # num_person: number of persons # theta_mean: mean of theta # theta_std: standard deviation of theta #Return: # A vector of length num_person gen_abilities <- function(num_person, theta_mean = 0, theta_std = 1){ thetas = rnorm(num_person, theta_mean, theta_std) return(thetas) } #Function for generating item responses # #Arguements: # thetas: vector of thetas # parameters: a matrix or data frame of item parameters, 1st, 2nd, and 3rd columns are a, b, and c # parameters #Return: # A number of persons by number of items data frame of item responses (1 and 0) gen_responses <- function(thetas, parameters){ num_person = length(thetas) num_item = nrow(parameters) #create an empty n by p matrix df <- matrix(0, nrow = num_person, ncol = num_item) for (i in seq(num_person)) { theta <- thetas[i] a <- parameters[,1] b <- parameters[,2] c <- parameters[,3] #calculate probabilities of correct response by each person for each item responses <- c + (1-c)*(exp(a*(theta-b))/(1+exp(a*(theta-b)))) #convert probabilities to correct (1) or incorrect (0) responses <- rbinom(length(responses), 1, responses) df[i,] <- responses } return(df) } 69 #Function for calculating “true” CC and CA indices # #Arguments: # theta_cut: single theta cut score # abiliites: vector of person thetas # parameters: matrix or data frame of item paraeters # response: matrix or data frame of item respones #Return: # a list of “true” CC and CA indices true_indices <- function(theta_cut, abilities, parameters, responses){ #convert person thetas to expected summed-score true_scores <- apply(cacIRT::irf(parameters, abilities, D = 1)$f, 1, sum) #convert theta cut to summed-score scale cut summed_cut <- sum(cacIRT::irf(parameters, theta_cut, D = 1)$f) #true classifications based on summed-score scale true_classifications_summed <- true_scores >= summed_cut #true classifications based on theta scale true_classifications_theta <- abilities >= theta_cut #rename item responses to responses_1 for better interpretation of code response_1 <- responses #generate item responses for the second administration of test #using the same thetas and item parameters from the first test response_2 <- gen_responses(abilities, parameters) #Calculate thetas from the generated responses est_theta_1 <- apply(response_1, 1, function(x){catR::thetaEst(parameters, x, method = 'ML', D = 1)}) est_theta_2 <- apply(response_2, 1, function(x){catR::thetaEst(parameters, x, method = 'ML', D = 1)}) #Calculate summed scores from generated responses summed_score_1 <- apply(response_1, 1, sum) summed_score_2 <- apply(response_2, 1, sum) #Observed classifications from generated responses based on theta scores class_1_theta <- est_theta_1 >= theta_cut class_2_theta <- est_theta_2 >= theta_cut #Observed classifications from generated responses based on summed-scores class_1_summed <- summed_score_1 >= summed_cut class_2_summed <- summed_score_2 >= summed_cut 70 #True indices (theta scale) #True CA true_CA_theta <- mean(true_classifications_theta == class_1_theta) #True false positive true_FP_theta <- mean(true_classifications_theta == FALSE & class_1_theta == TRUE) #True false negative true_FN_theta <- mean(true_classifications_theta == TRUE & class_1_theta == FALSE) #True CC true_CC_theta <- mean(class_1_theta == class_2_theta) #CC by chance pchance_theta <- mean(class_1_theta)*mean(class_2_theta) + (1-mean(class_1_theta))*(1-mean(class_2_theta)) #True kappa true_kappa_theta <- (true_CC_theta - pchance_theta) / (1 - pchance_theta) #True pass rate true_pass_rate_theta <- mean(class_1_theta) #Save true indices as named vector true_indices_theta <- c('T_CA' = true_CA_theta, 'T_FP' = true_FP_theta, 'T_FN' = true_FN_theta, 'T_CC' = true_CC_theta, 'T_K' = true_kappa_theta, 'T_P' = true_pass_rate_theta) #True indices (summed) #True CA true_CA_summed <- mean(true_classifications_summed == class_1_summed) #True false positive true_FP_summed <- mean(true_classifications_summed == FALSE & class_1_summed == TRUE) #True false negative true_FN_summed <- mean(true_classifications_summed == TRUE & class_1_summed == FALSE) #True CC true_CC_summed <- mean(class_1_summed == class_2_summed) #CC by chance pchance_summed <- mean(class_1_summed)*mean(class_2_summed) + (1-mean(class_1_summed))*(1-mean(class_2_summed)) #True kappa true_kappa_summed <- (true_CC_summed - pchance_summed) /( 1 - pchance_summed) #True pass rate true_pass_rate_summed <- mean(class_1_summed) #Save true indices as named vector true_indices_summed <- c('T_CA' = true_CA_summed, 'T_FP' = true_FP_summed, 'T_FN' = true_FN_summed, 71 'T_CC' = true_CC_summed, 'T_K' = true_kappa_summed, 'T_P' = true_pass_rate_summed) #store results as list results <- list() results'theta' <- true_indices_theta results'summed' <- true_indices_summed return(results) } #Function for estimating CC and CA indices using Lee method # (based on the Lee.P function from cacIRT library) # #Arguments: # responses: matrix or data frame of item responses # thetas: vector of person thetas # parameters: matrix or data frame of item parameters # theta_cut: theta cut score #Return: # a named vector of estimated CC and CA indices by Lee method lee_index <- function(responses, theta, parameters, theta_cut){ #convert theta cut to cut on summed-score scale raw_cut <- sum(cacIRT::irf(parameters, theta_cut, D = 1)$f) #conditional probabilities of having each summed score for each person probs <- cacIRT::recursive.raw(ip = parameters, theta = theta, D = 1) #true score for each person true_scores <- apply(cacIRT::irf(parameters, theta, D = 1)$f, 1, sum) #vector of fail and pass status of all persons fail_status <- true_scores < raw_cut pass_status <- true_scores >= raw_cut #conditional pass and fail probability of all persons fail_rate <- apply(probs[,1:(rounding(raw_cut)+1)], 1, sum) pass_rate <- apply(probs[,(rounding(raw_cut)+2):ncol(probs)], 1, sum) #estimated CA, FP and FN acc <- (sum(fail_rate[fail_status]) + sum(pass_rate[pass_status]))/length(theta) fp <- sum(pass_rate[fail_status])/length(theta) fn <- sum(fail_rate[pass_status])/length(theta) #estimated CC and kappa cons <- (sum(fail_rate^2) + sum(pass_rate^2)) / length(theta) pchance <- mean(pass_rate)^2 + mean(fail_rate)^2 kappa <- (cons - pchance) / (1 - pchance) 72 #observed summed score and pass rate summed_scores <- apply(responses, 1, sum) e_pass_rate <- mean(summed_scores >= rounding(raw_cut)) return(c('CA'= acc, 'FP' = fp, 'FN' =fn, 'CC'= cons, 'K' = kappa, 'E_P' = e_pass_rate)) } #Function for estimating CC and CA indices using Rudner method # (based on the Rud.P function from cacIRT library) # #Arguments: # thetas: vector of person thetas # sem: vector of standard error of theta estimates # theta_cut: theta cut score #Return: # a named vector of estimated CC and CA indices by Rud method rud_index <- function(theta, sem, theta_cut){ #vector of pass and fail status of all persons fail_status <- theta < theta_cut pass_status <- theta >= theta_cut #vector of pass and fail probabilities of all persons fail_rate <- pnorm(q = theta_cut, mean = theta, sd = sem) pass_rate <- 1 - fail_rate #estimaed CA, FP, and FN acc <- (sum(fail_rate[fail_status]) + sum(pass_rate[pass_status]))/length(theta) fp <- sum(pass_rate[fail_status])/length(theta) fn <- sum(fail_rate[pass_status])/length(theta) #estimated CC, kappa cons <- (sum(fail_rate^2) + sum(pass_rate^2)) / length(theta) pchance <- mean(pass_rate)^2 + mean(fail_rate)^2 kappa <- (cons - pchance) / (1 - pchance) #observed pass rate e_pass_rate <- mean(pass_status) return(c('CA'= acc, 'FP' = fp, 'FN' =fn, 'CC'= cons, 'K' = kappa, 'E_P' = e_pass_rate)) } 73 #Function for estimating CC and CA indices using Guo method # #Arguments: # responses: matrix of item responses # parameters: matrix or data frame of item parameters # thetas: vector of person thetas # theta_cut: single theta cut score #Return: # a named vector of estimated CC and CA indices by Guo method guo_index <- function(responses, parameters, thetas, theta_cut){ #vectors of pass and fail status for all persons fail_status <- thetas < theta_cut pass_status <- thetas >= theta_cut #extract a, b, and c item parameters as three separate vectors a <- parameters[,1] b <- parameters[,2] c <- parameters[,3] #Number of equally spaced theta points between cut scores num_points <- 100 #Exact locations of theta points in failing category failing_df <- matrix(0, ncol = num_points, nrow = nrow(responses)) width <- (theta_cut - -4)/num_points theta_points <- seq(-4 + width/2, theta_cut - width/2, length.out = num_points) #Calculate failing likelihoods for (i in seq(length(theta_points))){ theta <- theta_points[i] prop_correct <- c + (1 - c) * exp(a*(theta - b)) / (1 + exp(a*(theta - b))) likelihood_items <- t(apply(responses, MARGIN = 1, function(x){prop_correct^x * (1-prop_correct)^(1-x)})) failing_df[,i] <- apply(likelihood_items, MARGIN = 1, prod) } #Exact locations of theta points in passing category passing_df <- matrix(theta_cut, ncol = num_points, nrow = nrow(responses)) width <- (4 - theta_cut)/num_points theta_points <- seq(theta_cut + width/2, 4 - width/2, length.out = num_points) 74 #Calculate passing likelihoods for (i in seq(length(theta_points))){ theta <- theta_points[i] prop_correct <- c + (1 - c) * exp(a*(theta - b)) / (1 + exp(a*(theta - b))) likelihood_items <- t(apply(responses, MARGIN = 1, function(x){prop_correct^x * (1-prop_correct)^(1-x)})) passing_df[,i] <- apply(likelihood_items, MARGIN = 1, prod) } #sum of likelihoods failing <- apply(failing_df, MARGIN = 1, sum) passing <- apply(passing_df, MARGIN = 1, sum) #normalization of likelihodds fail_rate <- failing / (failing + passing) pass_rate <- passing / (failing + passing) #Estimated CA, FP, and FN acc <- (sum(fail_rate[fail_status]) + sum(pass_rate[pass_status]))/length(thetas) fp <- sum(pass_rate[fail_status])/length(thetas) fn <- sum(fail_rate[pass_status])/length(thetas) #Estimated CC and kappa cons <- (sum(fail_rate^2) + sum(pass_rate^2)) / length(thetas) pchance <- mean(pass_rate)^2 + mean(fail_rate)^2 kappa <- (cons - pchance) / (1 - pchance) #observed pass rate e_pass_rate <- mean(pass_status) return(c('CA'= acc, 'FP' = fp, 'FN' =fn, 'CC'= cons, 'K' = kappa, 'E_P' = e_pass_rate)) } 75 #Function for running simulations # #Arguments: # num_trials: number of trials/iterations # theta_cut: single theta cut score # num_items: number of items # num_persons: number of persons # b_mean: mean of b parameter # b_sd: standard deviation of b parameter # a_mean: mean of a parameter # a_sd: standard deviation of a parameter # c_mean: mean of c parameter # c_sd: standard deviation of c parameter # fitted.model: model used to fit item responses #Return: # a named vector of estimated CC and CA indices by Guo method run_simulation <- function(num_trials = 1, theta_cut = 0, num_items = 30, num_persons = 500, b_mean = .9, b_sd = .52, a_mean = 1, a_sd = 0, c_mean = 0, c_sd = 0, theta_mean = 0, theta_std = 1, fitted.model = '1PL'){ #Create template data frames for results df <- as.data.frame(matrix(0, nrow = num_trials, ncol = 17)) colnames(df) <- c('T_CA', 'E_CA', 'CA_Bias', 'T_FP', 'E_FP', 'FP_Bias', 'T_FN', 'E_FN', 'FN_Bias', 'T_CC', 'E_CC', 'CC_Bias', 'T_K', 'E_K', 'K_Bias', 'T_P', 'E_P') #Create list to save results from different methods results <- list() results'lee' <- df results'rudner' <- df results'guo' <- df #Start simulations for (trial in seq(num_trials)){ print(paste('Starting trial :', trial)) 76 #generate true thetas true_thetas <- gen_abilities(num_person = num_persons, theta_mean = theta_mean, theta_std = theta_std) #generate true item parameters true_parameters <- gen_parameters(num_item = num_items, b_mean = b_mean, b_sd = b_sd, a_mean = a_mean, a_sd = a_sd, c_mean = c_mean, c_sd = c_sd) #simulate item responses simulated_responses <- gen_responses(thetas = true_thetas, parameters = true_parameters) print('Done generating thetas, item parameters, and responses') #compute true indices true_indices <- true_indices(theta_cut = theta_cut, abilities = true_thetas, parameters = true_parameters, responses = simulated_responses) #save true indices results'lee'[trial,'T_CA'] <- true_indices'summed'['T_CA'] results'lee'[trial,'T_FP'] <- true_indices'summed'['T_FP'] results'lee'[trial,'T_FN'] <- true_indices'summed'['T_FN'] results'lee'[trial,'T_CC'] <- true_indices'summed'['T_CC'] results'lee'[trial,'T_K'] <- true_indices'summed'['T_K'] results'lee'[trial,'T_P'] <- true_indices'summed'['T_P'] results'rudner'[trial,'T_CA'] <- true_indices'theta'['T_CA'] results'rudner'[trial,'T_FP'] <- true_indices'theta'['T_FP'] results'rudner'[trial,'T_FN'] <- true_indices'theta'['T_FN'] results'rudner'[trial,'T_CC'] <- true_indices'theta'['T_CC'] results'rudner'[trial,'T_K'] <- true_indices'theta'['T_K'] results'rudner'[trial,'T_P'] <- true_indices'theta'['T_P'] results'guo'[trial,'T_CA'] <- true_indices'theta'['T_CA'] results'guo'[trial,'T_FP'] <- true_indices'theta'['T_FP'] results'guo'[trial,'T_FN'] <- true_indices'theta'['T_FN'] results'guo'[trial,'T_CC'] <- true_indices'theta'['T_CC'] results'guo'[trial,'T_K'] <- true_indices'theta'['T_K'] results'guo'[trial,'T_P'] <- true_indices'theta'['T_P'] print('Done computing true indices') 77 #estimate item parameters from item responses est_paras <- irtoys::est(simulated_responses, model = fitted.model, engine = 'ltm', rasch = TRUE) print('Done estimating item parameters') #estimate person's thetas using the item parameters and estimated item responses ab <- irtoys::ability(simulated_responses, est_paras, 'MLE') est_thetas <- ab[,1] #estimated thetas sem <- ab[,2] #standard error of estimates print('Done estimating thetas') #get Lee indices and save results lee_results <- lee_index(simulated_responses, est_thetas, est_paras$est, theta_cut) results'lee'[trial, 'E_CA'] <- lee_results['CA'] results'lee'[trial, 'E_FP'] <- lee_results['FP'] results'lee'[trial, 'E_FN'] <- lee_results['FN'] results'lee'[trial, 'E_CC'] <- lee_results['CC'] results'lee'[trial, 'E_K'] <- lee_results['K'] results'lee'[trial, 'E_P'] <- lee_results['E_P'] print('Done estimating Lee indices') #get Rudner indices and save results rudner_results <- rud_index(est_thetas, sem, theta_cut) results'rudner'[trial, 'E_CA'] <- rudner_results['CA'] results'rudner'[trial, 'E_FP'] <- rudner_results['FP'] results'rudner'[trial, 'E_FN'] <- rudner_results['FN'] results'rudner'[trial, 'E_CC'] <- rudner_results['CC'] results'rudner'[trial, 'E_K'] <- rudner_results['K'] results'rudner'[trial, 'E_P'] <- rudner_results['E_P'] print('Done estimating Rudner indices') #get Guo indices and save results guo_results <- guo_index(simulated_responses, est_paras$est, est_thetas, theta_cut) 78 results'guo'[trial, 'E_CA'] <- guo_results['CA'] results'guo'[trial, 'E_FP'] <- guo_results['FP'] results'guo'[trial, 'E_FN'] <- guo_results['FN'] results'guo'[trial, 'E_CC'] <- guo_results['CC'] results'guo'[trial, 'E_K'] <- guo_results['K'] results'guo'[trial, 'E_P'] <- guo_results['E_P'] print('Done estimating Guo indices') } #Calculate Bias for (li in names(results)){ resultsli[,'CA_Bias'] <- resultsli[,'E_CA'] - resultsli[,'T_CA'] resultsli[,'FP_Bias'] <- resultsli[,'E_FP'] - resultsli[,'T_FP'] resultsli[,'FN_Bias'] <- resultsli[,'E_FN'] - resultsli[,'T_FN'] resultsli[,'CC_Bias'] <- resultsli[,'E_CC'] - resultsli[,'T_CC'] resultsli[,'K_Bias'] <- resultsli[,'E_K'] - resultsli[,'T_K'] } print('Done calculating bias... Finished all trials.') ` return(results) } #To call the function to run simulations run_simulation(num_trials = 100, theta_cut = 1, num_items = 60, num_persons = 5000, b_mean = .52, b_sd = .92, a_mean = 1.2, a_sd = .25, c_mean = .17, c_sd = .08, theta_mean = 0, theta_std = 1, fitted.model = '1PL')
Thesis/Dissertation
eng
Measurement, Evaluation and Research Methodology
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Evaluating the correctness of IRT-based methods in computing classification consistency and accuracy indices in the presence of model misspecification
Text