Education, Faculty of
Educational and Counselling Psychology, and Special Education (ECPS), Department of
DSpace
UBCV
Friesen, Lucas
Master of Arts - MA
University of British Columbia
Differential item functioning (DIF) analysis is an invaluable tool for providing evidence that a test or measure is functioning as intended, and has been used extensively to show evidence that the items of tests are (not) favouring one group of test takers over another. Beyond evidence (for) against differential performance of items, DIF analysis provides extremely useful validation evidence in psychosocial measures because its application need not be limited to simple cases like, for example, detecting DIF based on a binary gender classification. However, the technical knowledge requirements of DIF analysis software can be prohibitively high in some research settings. The jamovi suite “Psychometric Post-Data Analysis” has been developed with a module for performing DIF analysis on dichotomously scored items using a generalized linear model framework (GLIM) that incorporates the GLIM family of methods for detecting DIF in a user- friendly interface. In addition, the idea of Type-M errors (Gelman & Carlin, 2014) was adapted and expanded to the DIF context, and then incorporated into the module. This tool is especially useful in the context of low-powered research settings. The goal of developing this module is to encourage the use of the GLIM framework for DIF analysis in the broader psychosocial measure validation praxis by lowering the technical barrier to assessing DIF and increasing the interpretability of DIF analysis results via post-data analytic techniques.
Psychometrics & Post-Data Analysis: A Software Implementation for Binary Logistic Regression in Jamovi by Lucas Friesen B.A., Simon Fraser University, 2014 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS in The Faculty of Graduate and Postdoctoral Studies (Measurement, Evaluation, and Research Methodology) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) August 2019 © Lucas Friesen, 2019 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, a thesis entitled: Psychometrics & Post-Data Analysis: A Software Implementation for Binary Logistic Regression in Jamovi submitted by Lucas Friesen in partial fulfillment of the requirements for the degree of Master of Arts in Measurement, Evaluation, and Research Methodology Examining Committee: Dr. Bruno D. Zumbo Supervisor Dr. Edward Kroc Supervisory Committee Member Dr. Richard Sawatzky Additional Examiner iii Abstract Differential item functioning (DIF) analysis is an invaluable tool for providing evidence that a test or measure is functioning as intended, and has been used extensively to show evidence that the items of tests are (not) favouring one group of test takers over another. Beyond evidence (for) against differential performance of items, DIF analysis provides extremely useful validation evidence in psychosocial measures because its application need not be limited to simple cases like, for example, detecting DIF based on a binary gender classification. However, the technical knowledge requirements of DIF analysis software can be prohibitively high in some research settings. The jamovi suite “Psychometric Post-Data Analysis” has been developed with a module for performing DIF analysis on dichotomously scored items using a generalized linear model framework (GLIM) that incorporates the GLIM family of methods for detecting DIF in a user-friendly interface. In addition, the module contains tools for assessing Type-M error rates (Gelman & Carlin, 2014) on DIF effects, a tool which is especially useful in the context of low-powered research settings. The goal of developing this module is to encourage the use of the the GLIM framework for DIF analysis in the broader psychosocial measure validation praxis by lowering the technical barrier to assessing DIF and increasing the interpretability of DIF analysis results via post-data analytic techniques. iv Lay Summary The developers of psychosocial measures and tests have long been concerned with differential item functioning (hereafter referred to as DIF). DIF is an undesirable characteristic of items on tests where the probability of endorsing an item a certain way is determined by construct-irrelevant factors. Which factors are deemed irrelevant is dependent on the test at hand. In a test of language proficiency, any test taker characteristic that is not a facet of their proficiency in the language construct being evaluated might be of concern. For instance, none of gender, ethnicity, height, or pizza-topping preference should be predictive of a test taker’s probability of endorsing an item. This manuscript presents a newly developed tool for detecting and assessing DIF in multiple-choice test items. v Preface The entirety of writing and research has been conducted by Lucas Friesen. Extensive discussion and consultation regarding statistical best practice, methodology, and avenues of research was provided by Dr. Zumbo. Extensive discussion and consultation regarding statistical best practice, methodology, avenues of research, and support with coding was provided by Dr. Kroc. vi Table of Contents Abstract ............................................................................................................................. iii Lay Summary ................................................................................................................... iv Preface .................................................................................................................................v Table of Contents ............................................................................................................. vi List of Figures ................................................................................................................. viii List of Abbreviations ....................................................................................................... ix Acknowledgements ............................................................................................................x Chapter 1: Introduction ................................................................................................... 1 1.1 Historical Background .................................................................................... 1 1.2 The Present State of Validity and Validation ................................................. 5 1.3 Validity as explanation ................................................................................... 6 1.4 Bridge .............................................................................................................. 9 Chapter 2: Post-Data Analysis Using Type-M Errors for Logistic Regression DIF .... 10 2.1 DIF Analysis Background and Software Implementation ............................ 13 2.1.1 Background ............................................................................................... 13 2.2 Design Analysis Background and Software Implementation ....................... 17 2.2.1 Background ............................................................................................... 17 2.2.2 Type-M Error in DIF Analysis .................................................................. 18 2.3 Jamovi Use .................................................................................................... 20 2.4 Calculations................................................................................................... 21 2.5 Results ........................................................................................................... 21 2.5.1 Procedure Notes ........................................................................................ 21 vii 2.5.2 DIF Analysis Table ................................................................................... 21 2.5.3 Design Analysis Table .............................................................................. 21 2.5.4 Item Response Curves ............................................................................... 22 2.6 Demonstration: DIF ...................................................................................... 22 2.6.1 The Stage ................................................................................................... 23 2.6.2 Analysis ..................................................................................................... 23 2.6.3 Interpretation ............................................................................................. 25 2.6.4 Results ....................................................................................................... 32 2.6.5 Discussion ................................................................................................. 35 2.6.6 Future directions ........................................................................................ 36 Chapter 3: Discussion and Future Directions ............................................................... 38 Bibliography .....................................................................................................................42 Appendix A Validity vs. Validation ..................................................................... 43 Appendix B Type-M and Type-S errors ............................................................... 46 Appendix C Documentation ................................................................................. 48 viii List of Figures Figure 1: Jamovi Interface ................................................................................................ 24 Figure 2: Procedure Notes ................................................................................................ 24 Figure 3: DIF Results Table .............................................................................................. 26 Figure 4: Design Analysis Table....................................................................................... 28 Figure 5: Item Response Curve 1...................................................................................... 29 Figure 6: Item Response Curve 2...................................................................................... 30 Figure 7: Item Response Curve 3...................................................................................... 31 Figure 8: Type-M Error Calculations ............................................................................... 33 ix List of Abbreviations DIF: Differential Item Functioning Δ R^2: Naeglekirke’s R^2 x Acknowledgements Special thanks are owed to Dr. Bruno Zumbo who has supported me throughout my years of education, both morally and financially, and Dr. Edward Kroc who has been an invaluable teacher. 1 Chapter 1: Introduction The following manuscript presents an introduction to an application of post-data analysis and explanatory pragmatic validity using a newly developed software for performing differential item functioning analysis. Chapter One begins with overview of historical trends in validity theory, followed by a summary of the current state of the field of validity theory. Next, an introduction to the view of validity as explanation is described, emphasizing the value of moving towards an explanatory view of validity and validation as scientific practice. A brief presentation of Gelman & Carlin’s Type-S and Type-M errors follows, laying the groundwork for a discussion of the value of post-data analysis in an explanatory view of validity. Finally, Chapter One concludes by re-emphasizing the purpose and goals of the manuscript: reorienting validation work in psychosocial measurement in order to develop a grounded scientific enterprise. Chapter Two presents a new software developed for performing differential item functioning analysis, followed by a demonstrative gedankenexperiment designed to illustrate the use of the software. Finally, a discussion is presented with an eye towards showing how this software functions as a tool for conducting explanatory validation research using post-data analysis. Chapter Three concludes the manuscript with a brief discussion of the broader applications of post-data analysis in validation and psychometrics in general. 1.1 Historical Background In the early 20th Century, the popularity of psychological and social (psychosocial) measurement erupted, from the Binet-Simon IQ test, to Army Alpha and Beta, to tests for employment suitability. Governments, corporations, and individuals had become enamoured 2 with the idea of measuring the unseen properties of human minds and societies. With popularity, however, came questions. The producers of psychosocial measurement instruments were asked to provide evidence that these new tools were performing as intended. In response, psychometricians performed studies which were intended to yield “validity evidence” that would assuage the concerns of stakeholders for a given test or measurement. If the evidence was deemed sufficient, the test was deemed to have validity (See Ruch 1924 p. 13 for one of the earliest accounts of validity). Validity theory was born. Over the next several decades, into the early 1950’s, a nearly absurd number of different types of test validities were proposed. In a comprehensive treatment of the history of validity theory, Newton & Shaw (2014) identified 150 distinct validities in published literature. Despite this proliferation, the conceptualization of validity as a metaphysical property of a test or measure that existed in different forms was largely, though not completely, put to rest with the publication of a seminal article by Samuel Messick (1989). Under Messick’s theory, validity is defined as “an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores” (Messick, 1989, p. 13). Messick’s 1989 article was a bookend to the work of validity theoreticians such as Cronbach and Meehl (Cronbach, 1951; Cronbach & Meehl, 1955), Loevinger (1957, 1979), and Anasatzi (1948, 1950), all of whom laid the collective groundwork for Messick’s unification of validity as a single construct validity. In this framework there were no longer different validities that reflected different aspects of quality in a test. Instead, all validity evidence was evidence for facets of a single unitary “construct validity”, and construct validity was related to test score interpretation, not to the test itself. In his paper, Messick emphasized strongly that validity was not a property of a test, but 3 rather of the inferences to be drawn on the basis of scores. This distinction was a key step in the evolution of validity theory because it incorporated the broader societal context1 of testing to the fore. In this framework, a score based on a given test or measure may be used to draw certain predetermined inferences with validity, but inferences which were not intended by the measure’s developers, or inferences based on scores from tests administered in a new context may not have the same validity. In contrast, if one holds validity to be a facet of a score, then the score is valid, or it is not; the context is not considered. As an example of the distinction between domains where a test may be used inferentially with and without strong validity, consider the use of a measure of depression in making hiring decisions. Given the stakes of their intended use as measures of depression, these scales are developed with extensive care, and thus inferences made on the basis of test scores related to whether or not, or the degree to which, an individual is experiencing clinical depression may be drawn with a fair degree of validity (Beck, Steer, & Carbin, 1988; Bruno D. Zumbo & Chan, 2014). However, that validity does not transfer to the domain of inference relevant to hiring decisions. This is because decidedly different evidence would be required to demonstrate validity for hiring decisions, and this evidence would not have been collected and assessed in the process of developing a measure of depression. Thus, there would be no evidence supporting the use of a measure of depression in making hiring decisions. To repeat, this logic holds in the inference- 1 Context here refers to both the varying features of test takers, as well as the varying features of the physical or societal context of the testing situation. 4 oriented unitary framework that Messick proposed, but is not immediately supported in frameworks where validity is attached to the test or test scores. This differential validity of inferences depending on the contextualized domain is frequently used as evidence for why validity must be a property of inferences, rather than a test itself, but there are competing views. The leading modern proponent of a test-centric conception of validity is Denny Borsboom (Borsboom, Mellenbergh, & van Heerden, 2004), who prefers a strongly realist conception of psychological measurement. In Borsboom’s framework, if a construct is not real, then it cannot be measured. If a construct is real, then validity is the degree to which responses on the test or measure can be causally accounted for, and validity is thus a property of the test or measure, rather than the inferences drawn on the basis of scores. Unfortunately, Borsboom’s realist framework falls short in that a clear ontology and epistemology of what is real or how it is knowably so is never given. The reader in this framework is thus left to define their own realism, adding to the confusion already present in discussions of validity theory. The fundamental disagreement between the conceptualizations of validity proposed by Messick and Borsboom is a reflection of the state of validity theory as a whole. Despite decades of debate, the questions of what is and is not validity, what evidence is relevant to validity, or even how validity ought to be operationally defined, remain unsettled. In all of these different facets of validity, conflict inevitably arises when a definition of validity is proposed because it never adequately captures the nuances that are required by the many diverse contexts in which the concept of validity is important. From testing for psychological disorders, to educational outcomes, to employment decisions, and social indicators, the stakeholders in psychosocial 5 measurement have so many different needs that some validity theorists have largely given up the search for a comprehensive consensus definition of validity. 1.2 The Present State of Validity and Validation Most treatments of validity theory or validation work begin with a statement regarding the widespread agreement about the degree of importance of validity. Unfortunately, importance is seemingly the singular agreed upon aspect of validity as a concept in psychosocial measurement. Many different conceptualizations of validity have arisen from this fractured state, none of which satisfy all of the diverse groups who claim validity as an important concept in their field. The current most agreed upon definition of validity is found in the Standards for Educational and Psychological Measurement (AERA, NCME, and APA, 2017) (Hereafter referred to as the Standards). The Standards have global reach, lending the following definition a great deal of credibility and impact (Zumbo, 2014): In the 2014 edition of the Standards, validity is defined as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests.” (AERA, NCME, and APA, 2017, p 11). Unfortunately, the Standards, like many other treatments of validity, fails to distinguish adequately between validity as a concept and validation as a process. This lacuna has plagued validity theory since its inception. Part of the reason for this failure to distinguish between validity and validation may be due to the absence of a concrete purpose driving validation research. In a survey of articles discussing the nature of validity theory spanning the past 20 years, including the first magnum opus treatment of the history of validity (Newton & Shaw, 2014), there is no mention of this purpose. Validation research is done more ‘because it is important’ than in an effort to answer any particular question. Indeed, the state of affairs in the validity theory literature is such that the 6 reason for conducting validation research appears largely forgotten. In the midst of ongoing debate over what is and is not validity, how it should and should not be conducted, its purpose has fallen to the wayside. What then, is that question, the reason for conducting validation research? The users of psychosocial tests and measures want assurance that the test is operating as intended. From the point of view of most non-psychometricians this will mean that “the test is measuring what it purports to measure”. For most psychometricians, “operating as intended” is more complicated than this simple definition, but can be understood as meaning that the measure possesses strong psychometric evidence for valid inferences of some particular sort, with a scope reaching from development and administration, to the societal feedback loop of continued use. For both parties, psychometricians and other users, the reason for conducting validation research is to provide confidence for stakeholders in their use of a measure; to show that inferences drawn on the basis of scores are adequately substantiated. The explanatory view of validity attempts to provide this assurance by reorienting psychosocial measure validation as a scientific practice, with an eye towards achieving the best possible explanation for variations in measure responses. 1.3 Validity as explanation In an explanatory theory of psychosocial validity, the goal of a program of validation research is to find the best possible explanation for variation in measure responses. That is, to determine whether or not a given explanation for variation in measure responses is the best explanation for that phenomenon. This addresses a major concern raised regarding Kane’s Inference/Use Argument approach (2003, 2014), which is that no guidance is offered on how to determine when an argument may be deemed “sufficient” for a given purpose. In an explanatory approach, the work of validation may be considered sufficient when a “best” explanation has 7 been chosen, and the evidence backing that explanation is sufficient to provide stakeholders with confidence in their use of the test in question. The process of accepting a given explanation as “best” will involve probing the explanation in order to discover whether or not it will pass tests which have a high probability of detecting error in the explanation, should that error exist. What then, does a conceptualization of validity such as Zumbo’s pragmatic theory of validity and validation yield in practice? The pragmatic explanatory approach to validity returns to the heart of the issue: stakeholders of psychosocial measures want confidence in inferences drawn on the basis of test scores, and the pragmatic validation researcher provides this by developing the most coherent possible explanation for variation in test scores. This is achieved by clearly distinguishing between validity as a concept and validation as a process. In particular, the validation researcher who is seeking to exploit Zumbo’s explanatory view of validity and validation, will (must) be proactive in theorizing about and probing accounts of test taker responding. This is in contrast with the standard argumentative approach which does not necessarily require a formal construct or rationalization of response processes. The process of developing and verifying an explanatory account of variation in test taker responses may involve developing new methods of data collection and experimentation, and undoubtedly will require a substantial degree of creativity. A benefit of adopting an explanatory view of validity whose scope goes beyond that of Borsboom’s realist causal explanations (Borsboom et al., 2004) is that there is no conflict with the current consensus notion of validity as defined in the Standards. The Standards define validity as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests”. As has been pointed out previously, while true, there is relatively little power behind this statement. It is agreed upon as the consensus because it is rather vague. 8 However, in developing any science (and in an explanatory account validity theory should indeed be a science), it is important to build upon what has been learned in the past (what has been called "vertical information gain" by Kroc and Zumbo). Explanatory validity does just that. Even more, validity as explanation does not limit itself to application in any particular domain (psycho-social research, industrial, theoretical…), and approaching the practice of validation research through an explanatory lens provides a scientific framework for validation that has been lacking in other frameworks of validity2. As a preview of the utility of the explanatory account of validity, consider the controversy of including testing consequences in validity. In certain circumstances, viewing validity as explanation ultimately requires that test takers and the decisions based upon measure results be considered in their broader psychosocial context. In a test validation context, the first stage of validation will undoubtedly focus on the psychometric qualities inherent to the test itself; that is, test format, item content, psychometric dimensionality, etc. Once reasonable confidence in the explanation has been attained at this level, it becomes apparent that a better explanation ought to also account for variation in test responses due to higher-level effects, such as person characteristics, and then higher-level still at the level of teachers, classrooms, and social contexts3. If the stakes of the use of a measure are high enough that confidence in the 2 Cronbach and Meehl’s 1955 nomological network might be an exception, however, the philosophy of science underpinning their work is rather out of fashion (Cronbach & Meehl, 1955). 3 This has been referred to as the ecological context of testing by Zumbo et al 2015. 9 measure’s use is diminished when the current best explanation does not account for the measurement context and consequential feedback loop, then this context must be addressed in order to achieve an adequate best explanation. This field of explanatory reach will continue to grow in an ongoing validation research effort until the best argument available sufficiently assuages the concerns of all relevant stakeholders. 1.4 Bridge To reiterate, the validation researcher attempting to develop an explanatory account of responses on a psychosocial measure is attempting to provide stakeholders with confidence in their use of the measure. This confidence comes in the form of a strong theoretical explanation as a foundation, and is then increased by performing research which yields evidence to either reinforce the claims made by theory, or discounts competing explanations. Like many discussions of validity theory, this high-level discussion does not immediately lend itself to application. For this reason, the next chapter will discuss a new software developed with an eye towards the explanatory pragmatic view of validation research that facilitates the adoption of this philosophy of validity through the use of highly accessible and widely distributed tools. This software is the first module in a suite of software being designed for jamovi (The jamovi project, 2019) which provides tools for conducting psychometric post-data analysis. Following an introduction to the software, a gedankenexperiment will walk through the actual process of conducting a gender DIF analysis using the Binary LogR module (Friesen, J. L., Kroc, E., Zumbo, B. D.) of the Psychometrics & Post-Data Analysis package. 10 Chapter 2: Post-Data Analysis Using Type-M Errors for Logistic Regression DIF This chapter presents a tool for assessing DIF on dichotomously scored items using a generalized linear model framework in a newly developed module for jamovi4 (The jamovi Project, 2019). The intended audience for this software is measurement specialists who are familiar with DIF methods and their use, with no assumptions made regarding technical proficiency in conducting DIF analysis. The module provides a user-friendly interface for conducting DIF analyses and is designed with the intention of promoting an explanatory focus in DIF analysis, as opposed to a purely descriptive one. This is achieved through both DIF visualizations and item-level post-data analysis. This term, post-data analysis, may be unfamiliar to some, so a moment will be spent to clarify this the idea. At its core, post-data analysis can be any statistical analysis that cannot be computed in the absence of data, where data is not raw data per se, but rather is a calculation based on the raw data, such as an effect size. As a whole, post-data analysis is a toolkit for assessing what happened in this case, rather than what should be expected to happen in the long run, as is the purpose of controlling for Type-I and Type-II error rates. Power analysis, sample size planning, methods consultation, etc, are tools for ensuring the usefulness, quality, and defensibility of research. Post-data analysis is part of the interpretation of the results of the current study. 4 Jamovi is a free GUI for performing statistical analyses using the R programming language. https://www.jamovi.org/ 11 The post-data analysis in this module moves beyond the standard ‘A’, ‘B’, or ‘C’ level DIF that is standard practice and is a sort of primitive post-data analysis. This module provides a quantification of Type-M error on the DIF effect size estimates (Gelman & Carlin, 2014). This is an extension of Gelman & Carlin’s novel conceptual work regarding post-data analysis, and appears to also be among the first implementations of post-data error calculations in psychometric work. This extension is achieved via a bootstrapping procedure (Davison & Hinkley, 1997) which derives the empirical distribution of the meta-statistic (change in Nagelkerke’s R² between baseline and full models) that is frequently used to quantify logistic regression DIF effect sizes. With the empirical distribution in hand, the standard error of the estimate of change in Nagelkerke’s R² becomes available, and the Type-M error rate becomes calculable. This novel procedure expands the range of applicability of Type-M errors dramatically beyond the case presented in Gelman & Carlin’s 2014 introduction, which applies only to standard t-tests of mean differences. As noted in Chapter One, the developers of psychosocial measures and tests have long been concerned with differential item functioning (hereafter referred to as DIF). DIF is an undesirable characteristic of items on tests5 where the probability of endorsing an item a certain way is determined by construct-irrelevant factors. Which factors are deemed irrelevant is dependent on the test at hand. DIF analysis gained prominence in the 1980’s as a result of lawsuits against testing organizations accused of discrimination on the basis of race. However, 5 Tests, measures, and surveys are referred to collectively as tests for the present discussion. 12 beyond the legal utility of DIF analysis, Zumbo’s third generation DIF points to DIF analysis as a key feature in psychosocial validation work. In a test of language proficiency, for example, any test taker characteristic that is not a facet of their proficiency in the language construct being evaluated might be of concern. For instance, none of gender, ethnicity, height, or pizza-topping preference should be predictive of a test taker’s probability of endorsing an item, and DIF analysis is a tool that is used to detect these construct-irrelevant effects (See Chapter One). The reason for DIF analysis being so important is that the validation researcher is attempting to instill confidence in a stakeholder’s use of a measure. In order to do so, they seek after a pragmatic explanation of variations in test taker responses. In the context of psychosocial testing and measurement, DIF analysis is a commonly used tool for showing that a test or measure is functioning as intended. When discussing differential item functioning, it is important to note that DIF is not synonymous with bias or fairness. Questions of fairness in tests and measures are directly related to the use of an individual’s score on that test or measure. For example, if men and women from a given nation receive different levels of training in speaking English, whether and how scores on a speaking proficiency test should be used in making immigration decisions is a question of fairness. On the other hand, DIF analysis seeks to find out if men and women from that nation who are equally proficient in spite of unfair circumstances have an equal probability of answering a given test item correctly. Thus, DIF analysis is a step removed from questions of external bias and fairness in test decisions, but is instead concerned with the internal functioning of the test and test items. However, should a test or test item be found to exhibit DIF in favour of one group over another, this result may be relevant to any decision-making process based on test scores, as well as a discussion of fairness regarding the use of that test. 13 2.1 DIF Analysis Background and Software Implementation 2.1.1 Background Logistic regression, the method used in this module, is among the most popular methods for detecting DIF for several reasons. Primarily, because most standardized tests and measures use questions which are dichotomized as being “correct” or “incorrect”, logistic regression is the natural tool in this vast majority of testing contexts: it allows one to easily model binary response data (e.g. Kutner et al. 2004 or McCulloch et al. 2008). Logistic regression can be used to detect both uniform and non-uniform DIF with two or more categorical groups or a continuous random variable. That is, it can be applied to questions of DIF between external variables such as binary-classified gender, an n-ary classification of ethnicity or pizza topping preference, or continuous variables such as height or age. This wide applicability makes logistic regression the most flexible method of detecting DIF to date (see Moses, Miao, & Dorans (2010) and Magis et al (2010) for additional discussion). However, no software has yet been developed which facilitates DIF analyses in a way which is widely accessible to researchers not already familiar with sophisticated programming languages, such as R6 (R Core Team, 2019). In addition to facilitating the process of conducting DIF analysis, the software presented below incorporates 6 The difR package by Magis, et al. is a notable contribution. DifR is a magnum opus R-package which computes many different DIF statistics. Its only drawback is the degree of technical skill required to use the package, and its lack of several options for the logistic regression method which we consider vital. 14 features of the post-data design analysis proposed by Gelman & Carlin (2014) which aids in the interpretation of DIF analysis results. The following explanation and demonstration use the default software settings provided by the Binary LogR module. These defaults are: • Assess both uniform and non-uniform DIF simultaneously • Test for model coefficient significance using the standard 2 degrees of freedom Wald test • Significance test alpha level set to 0.05 • Effect sizes are evaluated according to the Zumbo-Thomas scale The module performs DIF analysis on dichotomously scored items using binary grouping variables7 and either total-score matching or matching based on a supplied variable. The following explanation of the analysis will use an example where the grouping variable G is a binary classification 0/1, such as Male/Female, and the matching variable is a continuous random variable θ, such as scores on a measure of tendency towards verbal aggression. The matching variable is used to control for expected variation. For example, in a measure of language proficiency it is expected that a given item will be easier for individuals who are more highly trained in the language. Thus, if Males generally receive more training than Females, the effect 7 A common example encountered in the literature is the binary Male/Female classification. The analysis will accept more nuanced input, however, all groups will be collapsed into Reference/Non-Reference groups by default. 15 of this training would confound the examination of whether or not an item exhibits gender-DIF8 unless we account for this differential effect in our model via G * θ. Assessing DIF using logistic regression involves fitting two models and comparing the model fit of these models. The first is the baseline model which is fit using only the main effect of the matching variable: Logit of endorsement = β₀+ β1 * θ + ε, where beta_0 is a fixed intercept, beta_1 is the marginal effect of theta on the response variable, and epsilon is a random perturbation (assumed normally distributed). Note that the logit of endorsement is defined as: log(Pr(endorsement)/(1-Pr(endorsement))) A second model is then fit which contains the main effect of the matching variable, the main effect of the grouping variable, and an interaction between the matching and grouping variables: Logit of endorsement = β₀+ β1 * θ + β2 * G + β3 * (θ * G) + ε 8 Note again, DIF analysis is not addressing external sources of bias; DIF analysis concerns only the internal functioning of items. 16 Logically, if the model which includes the grouping variable fits the observed data more closely than the model which does not account for group differences, then the item would appear to be exhibiting DIF. The statistic most commonly used to quantify the difference in model fit between the baseline and full models is the change in model deviance, which is a measure of the poorness of model fit, ranging between 0 and 1, where a change in deviance of 0 indicates perfect fit. (Hancock, Mueller, & Stapleton, 2010, p 236). In standard industry practice, if the change in model deviance is not statistically significant (at a level specified by the researcher and using either the standard (asymptotic) chi-squared Wald test of joint significance or a likelihood ratio test), then the item is not flagged as exhibiting DIF. If the change in model deviance is significant, the item is flagged as exhibiting DIF. However, statistical significance alone is not a useful assessment of DIF. Items flagged as exhibiting DIF are typically categorized in terms of their effect sizes according to the Zumbo-Thomas scale (Zumbo & Thomas, 1997) which uses three levels to aid interpretation: A-level (Negligible effect), B-level (Moderate effect), and C-level (Large effect)9 (Zumbo, 1999). The following section elaborates on an adaptation of Gelman & Carlin’s techniques for measuring Type-M error (Gelman & Carlin, 2014) . This is a newly developed technique for improving the interpretability of effect size estimates in DIF analysis. 9 These classifications were developed by ETS to simplify interpretation and should be taken with a grain of salt. The effect size interpretation must be evaluated in a specific testing context. 17 2.2 Post-Data Analysis Background and Software Implementation 2.2.1 Background Post-data analysis is a relatively uncommon term, but it is not a new idea. Post-data analysis is the toolkit used for assessing what happened in this case, and even ideas such as the classification of effect sizes based on pre-determined thresholds can be considered a type of rudimentary post-data analysis. In the context of DIF, effect sizes ares commonly classified into ‘A’, ‘B’, or ‘C’ level DIF effects as an aide to interpretation. The present work extends this tradition by implementing a conceptual and practical extension of Gelman & Carlin’s (2014) work on Type-S and Type-M errors. Type-S and Type-M errors are post-data calculations that rely on the observed data and an a priori estimated or hypothesized effect size which is determined via information external to the study at hand, such as literature review and subject-matter expertise (Gelman & Carlin, 2014, p. 643). The Type-S error rate is the probability that the observed estimate of the effect size will have the wrong sign (i.e. - vs. +), if statistically significantly different from zero (Gelman & Carlin, 2014, p. 643). Type-M error, or the exaggeration ratio, is the expectation of the absolute value of the effect size divided by the hypothesized true effect size, if statistically significantly different from zero (Gelman & Carlin, 2014, p. 643). In other words, it is the factor by which the observed effect size might be expected to be exaggerated given the assumption of the hypothesized true effect size and statistical significance. To give the reader an intuition for assessing Type-S and Type-M errors, understanding their intimate relationship with power is helpful. In the simpler case of a classic t-test, when true power is 0 the Type-S error will be 0.5 and Type-M error will be infinite. This means that the testing procedure had an equal probability of observing an effect that was larger than the true 18 effect as it did of observing an effect which was smaller than the true effect. That is, it can tell us nothing about the direction of the true effect relative to our observed effect. Even worse, when the true power to detect an effect is 0, the Type-M error being infinite means that a statistically significant observed effect will be, on average, infinitely larger than the true effect. As the true power to detect an effect increases the Type-S and Type-M errors will decrease. When the true power is 1 the Type-S error will be 0 and the Type-M error will be 1. A Type-S error of 0 means that there is a 0% probability that the observed effect size is in the opposite direction of the true effect size, and a Type-M error of 1 means that the observed effect size (which is necessarily statistically significant by virtue of power = 1) is precisely equal to the true effect size (under replication). Taken together, this means that, as power increases, an observed effect is more and more likely to be an accurate estimate of the unknown true effect. Type-S and Type-M errors have provided a way to quantify the uncertainty associated with using an observed effect as an estimate of the unknown true effect that goes beyond the simple (and relatively uninformative) consequences of statistical power. The interpretation of Type-S and Type-M errors must always remain contextualized. There cannot exist a cutoff for what is “too large” of a Type-S or Type-M error, but some heuristics are available. For instance, a Type-M error of 100 indicates that statistically significant the observed effect size is likely exaggerated by an order of magnitude. Errors in this range can be acceptable when calculating the distances between the stars, but not when calculating differences between groups in a DIF analysis. 2.2.2 Type-M Error in DIF Analysis In the context of DIF analysis, the effect size most commonly encountered (and that referenced by the Zumbo-Thomas scale) is a derived (meta) statistic: The difference in 19 Nagelkerke’s R² (Δ R²) between the baseline and full models. Because this number is the difference in statistical fit of a logistic model with fewer terms (the baseline) and a logistic model with more terms (the full model), the difference between the two will always be positive. This means that it is not possible to compute a Type-S error rate for this statistic10. Regardless, in most contexts, the precise magnitude of the true DIF effect is of great interest, and it is here that the Type-M error rate proves its usefulness. Because the decisions made regarding the fate of an item on the basis of DIF analysis, such as whether or not it is to be used in a test, or whether a test as a whole is admissible for certain uses, depend almost entirely on the criteria used to decide whether an item is exhibiting “too much DIF”, understanding both the uncertainty around the estimated effect size and the nature of the unknown true effect size are vital. Calculating the Type-M error rate for Δ R² is completed via a bootstrapping process which generates an empirical distribution for the Δ R² for each item. The R code for this computation is available in Appendix B. The design analysis is then carried out according to the same principles as those laid out in Gelman & Carlin (2014) using three hypothesized true correlations. The hypothesized true correlations correspond to the level thresholds of the Zumbo-Thomas DIF evaluation scale11: 0 (Null/Negligible DIF), 0.13 (Moderate DIF), 0.26 (Large DIF). Those familiar with Type-M error calculations may immediately spot a potential problem with 10 See the final section on Future Directions for discussion of an implementation that would enable calculation of Type-S errors for DIF. 11 The software allows for the customization of this, as well as most other parameters, according the researcher’s purpose. This example represents the default settings. 20 the Null DIF hypothesized true effect. Type-M error is the ratio of the expectation of the observed effect and the hypothesized true effect; in the Null DIF hypothesis, this is 0. In order to bypass this problem, the Null effect is actually hypothesized as being 0 plus 2 times the standard error of the estimated effect size, which is derived during the process of bootstrapping the Δ R² empirical distribution. In effect, this hypothesis is that the unknown true effect is 0 +/- a certain degree of measurement error. 2.3 Jamovi Use Jamovi is a GUI wrapper for the R programming language. This means that the loading of data, selection of options, and collecting of output, is all handled graphically, rather than through the use of code. While the module provides many options for customizing the analysis, the basic process consists of loading the data, selecting which items are to be analyzed, selecting the grouping variable, and (optionally) selecting an external matching variable and anchor items12. A complete list of the customizable options is available in Appendix C. Once the analysis has begun, the output window of Jamovi will display four sections: (1) A descriptive overview of the analysis performed; (2) A DIF analysis table; (3) A design analysis table (optional); and (4). Item characteristic curves (optional) These four sections will be described in turn below. 12 Anchor items are those which are defined as a “gold standard”. That is, they are determined a priori to not exhibit DIF and are used to improve DIF estimation when using measure total score as a matching variable. 21 2.4 Calculations The details of the model fitting and design analysis calculations are described in detail above in sections 2.1 and 2.2. Further details are available in Appendix C, or in the online documentation hosted at: https://lucasjfriesen.github.io/jamoviPsychoPDA_docs/ 2.5 Results Once the data have been input and all options are selected, the module outputs results in up to four sections. Sections 1 (Procedure Notes) and 2 (DIF Analysis Table) are always provided, but Sections 3 (Design Analysis Table) and 4 (Item Response Curves) are optional, depending on the user’s choice. 2.5.1 Procedure Notes The descriptive overview provides a textual summary of the whole analysis, including what options were chosen, and documenting any warnings or errors that were encountered during the model fitting process. 2.5.2 DIF Analysis Table The DIF analysis table provides an overview of the analysis results. For each item the following information is displayed: DIF classification (Zumbo-Thomas criterion), p-value and chi-squared statistic of either the Wald or Likelihood ratio test, Δ R². The purpose of this table is to provide an easily viewable and shareable summary of results. 2.5.3 Design Analysis Table The design analysis table displays a list of all the items analyzed, the observed DIF effect, the bootstrapped standard error of the estimate, and Type-M errors rates calculated for each of the three thresholds in the Zumbo-Thomas scale. 22 2.5.4 Item Response Curves The item response curve can be plotted for any item being assessed for DIF. It provides a visual representation of the full model used in the DIF analysis. The different groups are colour coded, with 95% confidence intervals represented by gray shading overlaying each line. In these images the Y-axis is the probability of endorsement (0 to 1), and the X-axis is the range of the matching variable used in the logistic models. 2.6 Demonstration: DIF The following is a demonstration of a basic DIF analysis conducted using the PsychoPDA Binary LogR module. The data used is that supplied by Boeck and Wilson in their textbook13 (Boeck & Wilson, 2004). These data consist of responses from 316 test takers to 24 questions about verbal aggression. The measure is designed to have four factors, focusing on different situations where aggressive verbal behaviour might arise. Each factor is addressed by 6 questions. The dataset also contains a binary gender classification variable. The participant pool contained 243 females (coded as 1) and 73 males (coded as 0). In addition, scores on an external measure of anger as a personality trait are provided. The Trait Anger measure (STAXI; Spielberger, 1988, 1994) provides scores ranging continuously from 0 to 40. The responses recorded in the original data have been modified to magnify the gender DIF for illustrative purposes. The modified item is ‘S2DoScold’ and responses from Males were systematically changed to ‘0’ in order to achieve the desired effect for this illustration. 13 This data is freely available at http://bearcenter.berkeley.edu/page/materials-explanatory-item-response-models 23 2.6.1 The Stage Consider a scenario where a governing body wishes to employ this measure of aggressive tendencies in a scenario for which it was designed, but has concerns regarding the possibility of the test favouring females over males. This body could request that a gender DIF analysis be conducted by administering this new measure to some of the target population, in addition to the more costly measure that is currently in use. The more costly measure is used as a baseline because it has been subjected to rigorous scrutiny in the past. Scores on this measure provide the external matching variable14. 2.6.2 Analysis After administering both measures to the pilot test group and collecting the needed demographic information, the researcher loads the data into jamovi and selects “Binary LogR” from the dropdown menu of the PsychoPDA module. This will yield the shown in Figure 1. From here the variables may be selected via a drag-and-drop process. 14 The item total score of the new measure could also be used for matching, depending on the case at hand. 24 Figure 1: Jamovi Interface Once the necessary variables are selected, the analysis will automatically run, and interpretation must now begin. The first step in interpretation is to review the Procedure Notes section of the results window (See Figure 2). This allows the researcher to confirm that all of the correct analysis options were selected and to evaluate any warnings which might have surfaced during the model fitting process. An important step at this point is to confirm the coding of the grouping variable in order to properly evaluate the results of the logistic regression. Figure 2: Procedure Notes 25 2.6.3 Interpretation After confirming that the analysis was conducted properly, the DIF Analysis Table may be assessed. In Figure 3, we see that 6 of the 38 items have been flagged as exhibiting gender DIF. These 6 items all have p-values associated with the standard 2-degrees of freedom likelihood ratio test below the chosen threshold of 0.05. Of these 6 items, only one item (S2DoScold) has an effect size (Δ R²) passing the upper bound of A-level DIF (Δ R² = 0.29). The other 6 items have extremely small effect size estimates, a fact which must be evaluated alongside their statistical significance. 26 Figure 3: DIF Results Table The Design Analysis Table will be consulted next (Figure 4), in order to aid the interpretation of the effect size estimates. Recall that the software computes the Type-M error at three proposed values for the unknown true effect: the thresholds for A-, B-, and C-level DIF respectively15. 15 These values are found in the Procedure Notes table. 27 Here we see that in all cases there is a pattern in the Type-M error associated with the observed effect at each hypothesized true effect. As the hypothesized true effect increases, the estimated Type-M error decreases. This pattern is expected due to the relationship between Type-M error and the power to detect an effect of a given size. In addition, in the cases where the observed effect is smaller than the hypothesized true effect the Type-M error is below 1. That is to say that the post-data analysis has shown that, on average, a statistically significant observed effect will be smaller than the hypothesized true effect, rather than magnified. The Type-M error is correspondingly above 1 for all items at the ‘Null’ level and at all three levels for the ‘S2DoScold’ item, which is the one which was manipulated to increase the DIF effect. The most interesting results of this table are found in the rows of the ‘S2DoScold’ item. Several key observations can be made here. 1. First, under the hypothesis of a null true DIF effect, the post-data Type-M error calculation shows that a statistically significant effect will be magnified by approximately 16x. Under the hypothesis that the unknown true DIF effect is equal to the B-level threshold, the average statistically significant observed effect will be multiplied two times too large, and the Type-M error corresponding to a C-level DIF hypothesis for the unknown true effect shows minimal magnification. Second, for this item, the best estimate of the unknown true effect reduced by its Type-M error at the B-level hypothesis is at the B-level threshold of 0.29 / 2.27 = 0.13 And the corresponding best estimate of the unknown true effect if the C-level hypothesis is used is at the very upper end of B-level DIF. 0.29/1.13 = 0.25. 28 Figure 4: Design Analysis Table Taken together, what do these Type-M errors tell us? Given that the hypothesized effects are plausible, we should have sufficient power to instill confidence in our observed effect sizes. The interpretation of these results will be expanded on in the Results section of this demonstration. Finally, in order to communicate the results of the DIF analysis more effectively, the item response curves for three items have been produced: S2WantCurse, S3DoCurse, and S2DoScold. These show a visual representation of the probability of endorsement for the items (y-axis) chosen for both groups across the range of the matching variable (x-axis). The gray shading 29 represents the approximate 95% confidence bands of the logistic regression model coefficients, which is largely a function of the sample size of the group at each point along the scale represented by the x-axis. Each item is discussed below in turn. These plots are extremely useful when trying to understand what it means when an item is flagged as exhibiting DIF. The first item, S2WantCurse, was not flagged, and both groups are nearly matched all the way along the scale. This is an example of what one expects to see when there is no DIF effect present. Figure 5: Item Response Curve 1 30 The second item, S3DoCurse, was flagged as exhibiting A-level gender DIF. The second item response curve shows two notable characteristics: (1) the approximate confidence band on the lower range of the reference group (Females with Trait Anger scores below 40), is much wider than that of the contrast group (Figure 6). This reflects the fact that the large majority of respondents in the reference group did not endorse this item. Thus, the next step in evaluating this item might be to gain a larger sample of responses to this item in order to see if the DIF effect is an artifact of the potentially biased sample. Figure 6: Item Response Curve 2 31 The third item, S2DoScold, is the item which was manipulated to magnify the DIF effect. Responses from the contrast group (The pale grey line in these images) are clearly following a different pattern than that of the reference group in Figure 7. Figure 7: Item Response Curve 3 The Item response curve for this item quite clearly illustrate the effect of DIF on this item; Females are substantially more likely to endorse this item than Males, even controlling for scores on the Anger matching variable. 32 2.6.4 Results The interpretations of these results are similar in either the item/measure development or validation contexts, although the degrees of concern for DIF effects and their implications and costs to repair will differ circumstantially. Regardless of the context, if the estimated true effect of gender DIF on an item is substantial enough there will be consequences of some sort. Either the validity of test score interpretations will be impaired, if the gender DIF effect is not addressed, or time- and financial-resource costs will be incurred in addressing the gender DIF effect, either through removing or modifying the item. Here is where the Type-M error analysis can help us, as it captures a more accurate picture of the unknown true effect of interest, as quantified by our sample estimate, in order to both strengthen any validity argument and ensure unnecessary costs are not incurred. The results of this analysis show that of the 24 items on this scale, 6 exhibit evidence of gender DIF. Of those 6, only 1 item (‘S2DoScold’, Δ R² = 0.26) is above A-level. That is, only 1 item has a gender DIF effect that is beyond ‘negligible’, according to the Zumbo-Thomas scale16. In the design analysis table the Type-M error is estimated relative to three hypothesized true effects sizes, corresponding to the three DIF-level thresholds. This type of a priori hypothesis for the true effect is somewhat contrary to the recommendations of Gelman & Carlin (2014) who recommend basing the hypothesized true effect on a thorough review of extant literature on the effect in question. However, given the wide use of the Zumbo-Thomas scale, the possibility of 16 The qualitative characterization and discretization of effect sizes should always be evaluated relative to the local context of a measure and its consequences. 33 negligible, moderate, or large DIF effects are at the very least hypotheses of great practical interest. Of the six items exhibiting signs of gender DIF, the five which are A-level have relatively simple interpretations with respect to their Type-M error. The unknown true effect is smaller than the estimated effect size, meaning that if the B-level threshold is the minimum effect size that matters, these items are not of concern; the unknown true effect is very likely also ‘negligible’. The interpretation of the remaining item, ‘S2DoScold’, is somewhat more involved. The observed effect size is 0.29, which is in the C-level, or ‘large’, effect size range. Figure 8: Type-M Error Calculations There are several results to note in the above table. First, unsurprisingly, the estimated true effect is always smaller than the observed effect. This should be the case because we are dividing the observed effect by the Type-M error (Recall, it is also called the exaggeration ratio)17. Second, the estimated true effect under the null/negligible level is lower than the Null threshold18. That is, if we assume that the most likely scenario is that the true unknown gender 17 The estimated true effect is also smaller than the hypothesized true effect, although this is not a mathematical requirement, it simply happens to be the case here. 18 the Null threshold is really a fuzzy approximation of a 0 Null. This is computed as 0 + 2 * SE, where SE is the empirically derived bootstrap estimate of the standard error. The reason 34 DIF effect size on this item is no larger than twice the standard error of the estimate, i.e. a proxy for sampling error, the Type-M error indicates that the observed result is exaggerated by a factor of 16.69. However, given the large sample size used in this study, the likelihood of observing an effect which is 30x the standard error of the estimate is quite low. This a priori hypothesis seems quite weak. Third, the computations for the estimated true effects of both the B- and C-level hypotheses for the unknown true effect yield B-level results. That is, under both the a priori hypotheses that the unknown true effect is at the threshold for B-level or at the threshold for C-level DIF, the observed effect multiplied by the respective Type-M error is in the B-level effect size range. Again, given the large sample size used in this study, the observed effect arising by chance seems quite unlikely. This means that, so long as our primary concern is the DIF-level categorization, the choice of hypothesis here does not change the result. The best estimate of the unknown true effect of gender DIF on this item is in the B-level range. This conclusion, that the item ’S2DoScold’ is exhibiting evidence of a true gender DIF effect, is reinforced by the item response curves. There is a clearly visible difference in the probability of endorsement between Males and Females, with Males being less likely to endorse the item across the range of the Anger matching variable. In practice, this evidence should be taken together and assessed as a whole. The main points as they might be presented in practice are: 1. The statistical test of significance for DIF effects indicates that six of 24 items on this measure are exhibiting DIF effects. 2. Of these six, for this approximation is twofold. First, the Type-M error cannot be undefined when the hypothesized true effect is 0, and second, because using an arbitrarily small number results in arbitrarily inflated Type-M error rates. 35 five are in the A-level (negligible) range, and the Type-M error calculations reinforce the conclusion that the unknown true effect of DIF is indeed negligible. 3. The one item which exhibits a gender DIF effect larger than A-level shows evidence that the unknown true effect is in the B-level range, based on the Type-M error calculations. 4. Given the large sample size used in this study, it is highly unlikely that the results are due purely to sampling error. The final conclusion based on this evidence would (hopefully) be that this item should likely be reviewed by the relevant experts involved in its development or validation and either revised or rejected. 2.6.5 Discussion Differential item functioning was chosen as the first module to be developed and presented in the Psychometrics & Post-Data Analysis suite because its heart lies directly in line with the essence of the explanatory pragmatic view of validity. DIF analysis asks the question “What is the source of variation in test taker responses?. by positing competing theories (e.g. Test takers responses are explained by their gender, not/in addition to the theorized construct.) and evaluating those theories. If no explanation apart from that theorized at the outset can be found, the weight of evidence in favour of intended interpretations based on test scores (validity) is increased. By providing a user-friendly interface to performing DIF analysis, the technical barrier to this important analysis has been greatly reduced. The goal of this barrier reduction is to encourage researchers to incorporate DIF analysis in all stages of the development and use of psychosocial measures. The reason for this goal is that DIF analysis is fundamentally an explanation-oriented analysis which seeks to answer the question of why test takers respond on measures the way that they do. If there exists no explanation for test taker responses other than 36 random measurement and/or sampling error and variation in the measured trait or construct, then the best possible explanation for variation in test taker responses is that the measure is operating as intended (Zumbo, 2007, 2009). The addition of Design Analysis provides the researcher with an improved ability to practically assess the degree of DIF. Because statistically significant effects are often necessarily exaggerated (Gelman & Carlin, 2014), it is important to incorporate information on the degree of this inflation in a response to items flagged as exhibiting DIF. In a practical sense, consider the development of a measure, or the field testing of items for an item bank on an established measure: measures and items are resources which are not to be thrown away lightly, and the Type-M error rate can help the researcher to find the appropriate balance between psychometric integrity and economic responsibility. 2.6.6 Future directions There are several areas for expansion on this work. First, despite the change in Nagelkerke’s R² being a standard effect size measure in DIF analyses, its interpretability leaves much to be desired. By focusing instead on the logistic regression coefficients estimated for the model comparison stage of the analysis, the Type-S error could be calculated in addition to the Type-M error. The logistic regression coefficients, combined with Type-S and Type-M error rates would provide a more intuitive picture of the DIF effect direction and size. Related to this, the capacity for assumption and model fit checks in DIF analyses like the one demonstrated above is lacking. In the above example with 28 items, there were a total of 56 models fit, and it is not common practice in DIF analyses to report assumption checks or model fit indices. The jamovi software could be expanded to fill this void, increasing the robustness and interpretability of findings beyond the current professional standard. In particular, questions 37 about proper model specification, autocorrelation of errors, and omitted confounders should play a critical role in any sophisticated and thorough DIF analysis of the future. Finally, after incorporating both the use of logistic regression coefficients and improving model diagnostics, improving the model specification capacity of the jamovi software would enable researchers to pursue the ideal of 3rd generation DIF analysis (Zumbo, 2009) wherein the social and environmental context is given greater prominence. This is a key area of improvement because DIF analysis is often ultimately seeking a causal explanation of test taker responses. In this inferential realm, issues of exchangeability become paramount (e.g. Draper et al. 1993). Causal modeling is a distinctly different process than predictive or explanatory modeling (see e.g. Bollen 1989 Chapter 3, Pearl 2009), and its practical implementation relies heavily on those exact issues of model specification previously highlighted, among others. 38 Chapter 3: Discussion and Future Directions The previous chapter illustrated the utility of combining post-data analysis with an explanatory view of validity through a discussion of a software implementation of DIF analysis in jamovi. From out of that narrow scope, there is clearly a great deal of room for further work, including improving effect size measures, model fit procedures, and exploring more deeply the utility of post-data analysis in the context of Zumbo’s (2009) 3rd generation DIF. One of the stated goals of the software implementation of DIF analysis was to increase the adoption of a post-data analysis infused theory of explanatory validity by developing tools which are widely accessible and widely distributable. This has been accomplished in two ways. First, the module has already been accepted for publication into the jamovi library upon submission of this thesis. This means that it will be visible and accessible to anyone with access to a computer and internet, at no cost. Given that the jamovi software is receiving approximately 8,000 unique downloads per month in 2019, compared to 2,000 per month in 2018 (Personal communication), this means that the software will be widely visible. In addition, this remarkable growth clearly demonstrates the desire for open source tools that facilitate research methods in the jamovi interface. One area for growth in this module will be to incorporate more qualitative aspects of post-data analysis, such as providing heuristic interpretations of results in non-technical language, in order to “tell the story” of the analysis. This would facilitate the interpretation of results by the knowledge user, who may or may not have the technical expertise needed to accurately interpret the quantitative statistics without assistance. This could include providing clear prompts to the user to consider how differences in item-level response curves would manifest at the test-score level, or how the ‘reason’ for DIF might be further analyzed. It would 39 prompt the user to consider the common question from test takers of “Why did I get this score?”. The ability of the validation researcher to answer this question would be invaluable in providing stakeholders with confidence in the interpretation of test scores. Moving beyond DIF analysis software, let us explore for a moment a number of other areas for the application of post-data analysis in psychometrics. A recent comprehensive survey of validation work in psychosocial measurement by Bruno and Chan (2017, chapter 19) showed that validation work has by and large been limited to correlational research. This has been the case since the earliest conception of validity wherein the validity of a measure is its correlation with some criterion. This is the case despite the fact that more robust methods and evidence have been called for since the first edition of the Standards was published in 1954. There are many possible reasons that validation praxis and validation theory have diverged, but one probable explanation is simply that collecting evidence, especially for causal or externally validated inferences is hard. From Cronbach and Meehl’s (1954) nomological networks to Campbell and Fiske’s (1959) multi-trait multi-method matrix ,and from Messick’s unitary construct validity (1989) to Kane’s Inference Use Argument (2004, 2013), the techniques and implementations proposed for the improvement of validation research are complex, and theories of validity don’t usually propose how to conduct validation work. Correlations, however, are simple both to understand and to compute. By conceiving of validity as an explanation of variation in response patterns or measures, Zumbo’s pragmatic explanatory validity provides a framework wherein both the simple tools are still applicable, but the need for complex tools is harder to ignore. Validation in this framework is the the process of developing an explanation of variation in response patterns. Similar to Messick and Kane’s paradigms, there is no pre-defined end to the 40 act of validation. However, given the above definition of validity, the end of the validation process is knowable, and even attainable, if with difficulty. The process of validating a test or measure may be considered complete when the strength of the explanation for variation in response patterns in question is “strong enough”, where “strong enough” is measured by the confidence of stakeholders in their use of the test or measure. Post-data analysis is the toolbox toolbox containing the methods and techniques needed to provide this explanatory account. These methods, which link the needs of knowledge users to the robust theory of explanatory validity, are needed in order to move beyond ad hoc practices (Loevinger, 1957) posing as empiricism and into validity and validation as a productive area of scientific exploration. Developing tools used for validation research with an eye towards encouraging knowledge users to critically analyze hypotheses and results in ways that that is interpretable by both non-expert stakeholders and academics alike, seems a good place to start. Gelman and Carlin’s Type-S and Type-M errors are an example of an intuitive metric that encourages critical thought in practitioners and which is also widely applicable to work already being done in validation research, as well as to the work that experts agree should be done. Take, for example, correlations with gold standard measures or external criteria. Research in this area would benefit from an application of Gelman and Carlin’s Type-S and Type-M error calculations because it would require stake holders at all levels to more fully consider the plausibility of observed effect sizes within the larger context of their research problem, and to attempt to understand and develop an explanatory account of psychosocial measurement. Correlational methods showing high Type-S and Type-M errors would also (hopefully) encourage researchers to explore (and stakeholders to demand) additional forms of evidence, such as evidence related to response processes, relations to other variables, internal structure, and 41 test content, all of which are themselves amenable to post-data analysis in different ways when assessed statistically. By adding Type-S and Type-M error calculations into standard practice, the consideration of both observed and hypothesized effect sizes becomes necessary. This will require more explicit consideration of existing literature and extant theory in the formulation of hypotheses of effects leading to construct-irrelevant variation in measure responses, which is itself a necessary component in an explanatory account of variation in responses to measures. This emerging explanatory account will then feed into the consideration of what forms of evidence are needed to strengthen a validity argument. Thus, by grounding validation research, and psychometric research more generally, in a framework which encourages critical thinking from the beginning of test construction or research design to the presentation of validation research to stakeholders, post-data analysis acquires an explanatory bent which strengthens the argument used to rationalize test score interpretation and increases stakeholder confidence in the use of measures. 42 Bibliography American Psychological Association (Wash.). (1954). Technical recommendations for psychological tests and diagnostic techniques. Washington 6: American psychological Association, 1333 16th street N.W. Anastasi, A. (1948). The nature of psychological ’traits.’. Psychological Review, 55(3), 127–138. https://doi.org/10.1037/h0063619 Anastasi, A. (1950). The Concept of Validity in the Interpretation of Test Scores. Educational and Psychological Measurement, 10(1), 67–78. https://doi.org/10.1177/001316445001000105 Beck, A. T., Steer, R. A., & Carbin, M. G. (1988). Psychometric properties of the Beck Depression Inventory: Twenty-five years of evaluation. Clinical Psychology Review, 8(1), 77–100. https://doi.org/10.1016/0272-7358(88)90050-5 Boeck, P. de, & Wilson, M. (Eds.). (2004). Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach. New York: Springer-Verlag. Retrieved from //www.springer.com/gp/book/9780387402758 Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The Concept of Validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061 Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016 Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. 43 Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281. Davison, A.C. and Hinkley, D.V. (1997). Bootstrap Methods and Their Application, Chapter 5. Cambridge University Press. Draper, D., Hodges, J.S., Mallows, C.L., Pregibon, D. (1993). Exchangeability and Data Analysis. JRSS, Series B, 156(1), 9-37. Friesen, L. J., Kroc, E., Zumbo, B. D. (2019). PsychoPDA. [jamovi module]. Retrieved from: https://github.com/lucasjfriesen/jamoviPsychoPDA. Date Retrieved: July 24, 2019. Gelman, A., & Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 9(6), 641–651. https://doi.org/10.1177/1745691614551642 Hancock, G. R., Mueller, R. O., & Stapleton, L. M. (2010). The Reviewer’s Guide to Quantitative Methods in the Social Sciences. Routledge. Kutner, M.H., Nachtsheim, C.J., Neter, J. Applied Linear Regression Models, 4th Edition. McGraw-Hill Irwin, New York City, NY, USA, 2004. Loevinger, J. (1957). Objective Tests as Instruments of Psychological Theory. Psychological Reports, 3(3), 635–694. https://doi.org/10.2466/pr0.1957.3.3.635 Loevinger, J. (1979). Construct Validity of the Sentence Completion Test of Ego Development. Applied Psychological Measurement, 3(3), 281–311. https://doi.org/10.1177/014662167900300301 44 Magis, D., Béland, S., Tuerlinckx, F., & De Boeck, P. (2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862. https://doi.org/10.3758/BRM.42.3.847 McCulloch, C.E., Searle, S.R., Neuhaus, J.M. Generalized, Linear, and Mixed Models, 2nd Edition. John Wiley and Sons, Hoboken, NJ, USA, 2008. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York, NY: American Council on education and Macmillan. Moses, T., Miao, J., & Dorans, N. J. (2010). A Comparison of Strategies for Estimating Conditional DIF. Journal of Educational and Behavioral Statistics, 35(6), 726–743. https://doi.org/10.3102/1076998610379135 Newton, P. E., & Shaw, S. D. (2014). Validity in Educational & Psychological Assessment. 1 Oliver’s Yard, 55 City Road London EC1Y 1SP: SAGE Publications Ltd. https://doi.org/10.4135/9781446288856 Pearl, J. (2009). Causal inference in statistics: An overview. Statistics Surveys, 3, 96-146. Spielberger, C. D. (1988). Manual for the state-trait anger expression scale (STAXI). Odessa, FL: Psychological Assessment Resources. R Core Team. (2018). R: A Language and environment for statistical computing. [Computer software]. Retrieved from: https://cran.r-project.org/. Date Retrieved: July 24, 2019. Spielberger, C. D., & Sydeman, S. (1994). State-Trait Anxiety Inventory and State-Trait Anger Expression Inventory. In The Use of Psychological Testing for Treatment Planning and Outcome Assessment (pp. 292–321). The jamovi project. 2019. (Version 1.0) [Computer Software]. Retrieved from https://jamovi.org/. Date Retrieved: July 24, 2019. 45 Zumbo, B. D. (1999). A Handbook on the Theory and Methods of Differential Item Functioning (DIF), 57. Zumbo, B. D. (2007). Three Generations of DIF Analyses: Considering Where It Has Been, Where It Is Now, and Where It Is Going. Language Assessment Quarterly, 4(2), 223–233. https://doi.org/10.1080/15434300701375832 Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The Concept of Validity: Revisions, New Directions and Applications, Oct, 2008. IAP Information Age Publishing. Zumbo, B. D. (2014). What Role Does, and Should, the Test Standards Play Outside of the United States of America? Educational Measurement: Issues and Practice, 33(4), 31–33. https://doi.org/10.1111/emip.12052 Zumbo, B. D., & Chan, E. K. (Eds.). (2014). Validity and Validation in Social, Behavioral, and Health Sciences (Vol. 54). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-07794-9 Zumbo, B., D., & Thomas, R., D. (1997). A measure of effect size for a model-based approach for studying DIF. Working Paper of the Edgeworth Laboratory for Quantitative Behavioral Science, University of Northern British Columbia: Prince George, B.C. 46 Appendix A Type-M and Type-S errors A.1 Type-M Error Calculating the Type-M error rate for a derived statistic is done via a bootstrapping process which generates an empirical distribution for the Δ R^2 for each item. Once the empirical distribution has been derived, the following function is used to compute the Type-M error: The above function is derived from that provided by Gelman & Carlin (2014) in their appendix. The changes are made in order to accommodate the reality that we cannot assume a normal distribution in this case. 47 A.2 Type-S Error Type-S error is not calculated because it does not align conceptually with the tests being performed in this type of DIF analysis. The Δ R^2 effect size is a meta-statistic, as opposed to a normal measure of effect size like Cohen’s d, and one of the results of this is that Δ R^2 will only be negative in anomalous situations, meaning that there is no meaningful probability of achieving a statistically significant effect in the ‘wrong’ direction. 48 Appendix B Documentation B.1 DIF Analysis In any discussion of differential item functioning, it is important to note that DIF is not synonymous with bias or fairness. Questions of fairness in tests and measures are directly related to the use of an individual’s score on that test or measure. For example, if men and women from a given nation receive different levels of training in speaking English, whether and how scores on a speaking proficiency test should be used in making immigration decisions is a question of fairness. On the other hand, DIF analysis seeks to find out if men and women from that nation who are equally proficient in spite of unfair circumstances have an equal probability of answering a given test item correctly. Thus, DIF analysis is a step removed from external questions of bias and fairness in test decisions, but is instead concerned with the internal functioning of the test and test items. However, should a test or test item be found to exhibit DIF in favour of one group over another, this result may be relevant to any decision making process based on test scores, as well as a discussion of fairness regarding the use of that test. Logistic regression, the method used in this module, is among the most popular methods for detecting DIF for several reasons (see Moses et al. (2010) and Magis et al. (2010) for additional discussion). Logistic regression can be used to detect both uniform and non-uniform DIF with two or more groups, and can be used in either an IRT or non-IRT context. To date, however, no software has been developed which facilitates DIF analysis in a way which is widely accessible. B.2 Calculations The DIF module performs DIF analysis on dichotomously scored items using binary grouping variables and either total-score matching or matching based on a supplied variable. The 49 following explanation of the analysis will use an example where the grouping variable β₁ is a binary classification (0, 1) and the matching variable is a latent construct-level variable θ. This example is the case where we are simultaneously assessing for both uniform or non-uniform DIF. Using a GLIM model, items are fit first with a baseline model containing both the main effects and interaction effect: Logit of endorsement = β₀ + β₁ + θ + (β₁ * θ) A second model containing only the main effect of the matching variable is then fit: Logit of endorsement = β₀ + θ The reduction in model deviance is then calculated between the baseline and secondary models (ΔD). If ΔD is not statistically significant (at a level specifiable by the user and using either the Wald test of joint significance or a likelihood ratio test), then the item is not exhibiting DIF. If ΔD is significant, the item is flagged as exhibiting DIF and the effect size for ΔD may be assessed. B.3 Options 1. title: Items type: Variables Items must be a series (vector) of names referencing columns in the data containing exactly two unique values, such as [0, 1]. The data will be converted to integer type prior to analysis using jmvcore::toNumeric(). 2. title: Grouping Variable type: Variable Grouping Variable must be a single name referencing a column in the data containing at least two unique values, such as [0, 1]. The data will be converted to factor type prior to analysis using base:as.factor(). 50 The value of the first row in the column will be coded as Group A, and all rows containing non-Group A values will be coded as Group B. If more than two unique values are present in the column, all values different from the value of the first row will be collapsed into the same Group B. This restriction will be removed in a future version of the module, but is necessary for several aspects of the current version to perform. 3. title: Matching Variable type: Variable Matching Variable is an optional argument. This is the value which is used in the regression equations to account for differences in, for example, proficiency, levels of a latent construct, or some other psychosocial factor. If it is left empty, subjects will be matched on the Total Score, calculated internally by the module. This variable must be numeric. 4. title: Anchor Items type: Variables Anchor Items is an optional argument. If provided, Matching Variable must be left empty. Anchor Items must be a series (vector) of names referencing columns in the data which are considered to be ‘gold standard’. That is, containing no DIF. These items will be used in calculating the subject Total Score, but are not assessed for DIF. See Purification below for more details. 5. title: Group Type options: title: Discrete Groups title: Continuous Groups Group Type must take either “Discrete Groups” or “Continuous Groups” as its value. Discrete groups are a grouping variable such as binary gender classifications, or national identity. Continuous Groups are a grouping variable such as age. 6. title: Group contrasts 7. title: Evaluation Scale options: title: Zumbo-Thomas title: Jodoin-Gierl 51 Evaluation Scale refers to the classification scheme for DIF effect sizes on the Nagelkerke’s R^2 (Δ R^2) scale. 8. title: Type options: – title: Uniform DIF • title: Non-Uniform DIF • title: Uniform and Non-Uniform DIF Type refers to the type of DIF to be assessed for. The default, Uniform and Non-Uniform DIF will check simultaneously for significant main and interaction effects. 9. title: Flagging Criterion options: – name: “Wald” • name: “LRT” Flagging Criterion refers to what test should be used in testing for statistical significance of regression coefficients. 10. title: Alpha type: Number Alpha is the desired Type-I error rate for statistical significance tests.r 11. title: Item Purification 12. title: Number of Iterations type: Number Number of Iterations is the maximum number of purification iterations to be used if Item Purification is selected. 13. title: P-value Adjustment method options: – name: bonferroni • name: holm – name: hochberg 52 – name: hommel – name: BH – name: BY – name: none P-value Adjustment method is the method of correction for multiple comparisons desired: The adjustment methods include the Bonferroni correction (“bonferroni”) in which the p-values are multiplied by the number of comparisons. Less conservative corrections are also included by Holm (1979) (“holm”), Hochberg (1988) (“hochberg”), Hommel (1988) (“hommel”), Benjamini & Hochberg (1995) (“BH” or its alias “fdr”), and Benjamini & Yekutieli (2001) (“BY”), respectively. A pass-through option (“none”) is also included. The set of methods are contained in the p.adjust.methods vector for the benefit of methods that need to have the method as an option and pass it on to p.adjust. The first four methods are designed to give strong control of the family-wise error rate. There seems no reason to use the unmodified Bonferroni correction because it is dominated by Holm’s method, which is also valid under arbitrary assumptions. Hochberg’s and Hommel’s methods are valid when the hypothesis tests are independent or when they are non-negatively associated (Sarkar, 1998; Sarkar and Chang, 1997). Hommel’s method is more powerful than Hochberg’s, but the difference is usually small and the Hochberg p-values are faster to compute. The “BH” (aka “fdr”) and “BY” method of Benjamini, Hochberg, and Yekutieli control the false discovery rate, the expected proportion of false discoveries amongst the rejected hypotheses. The false discovery rate is a less stringent condition than the family-wise error rate, so these methods are more powerful than the others. 53 Note that you can set n larger than length(p) which means the unobserved p-values are assumed to be greater than all the observed p for “bonferroni” and “holm” methods and equal to 1 for the other methods. B.4 Design Analysis Type-S and Type-M errors are post-data calculations that rely on the observed data and an estimated “true” effect size which is determined via information external to the study at hand, such as literature review and subject-matter expertise (Gelman & Carlin (2014), p. 643). The Type-S error rate is the probability that the observed estimate of the effect size will have the wrong sign (I.e. - vs. +), if statistically significantly different from zero (Gelman & Carlin (2014), p. 643). Type-M error, or the exaggeration ratio, is the expectation of the absolute value of the effect size divided by the hypothesized true effect size, if statistically significantly different from zero (Gelman & Carlin (2014), p. 643). In other words, it is the factor by which the observed effect size might be expected to be exaggerated given the assumption of the provided true effect size. To give the reader a sense for assessing Type-S and Type-M errors, understanding their intimate relationship between power is helpful. In the cases where True Power is 0 and 1, Type-S error will be 0.5 and 0, and Type-M error will be infinite and 1, respectively. A Type-S error of 0.5 means that their is a 50% probability that the observed effect size is in the opposite direction of the True effect size, and a Type-M error of 1 means that the observed effect size is precisely what is expected (under replication) of a statistically significant result. In the context of DIF analysis, the effect size most commonly encountered (and that referenced by the Zumbo-Thomas scale) is a derived statistic - The change in Naeglekirke’s R^2 (Δ R^2) between the baseline and full models. Because this number will always be positive, it is 54 not possible to compute a Type-S error rate for this statistic. However, in most contexts, the precise magnitude of the true DIF effect is of great interest, and it is here that the Type-M error rate is of great interest. This is because decisions made regarding the fate of an item, such as whether or not it is to be used in a test, or whether a test as a whole is admissible for certain uses, depends almost entirely on the criteria used to decide whether an item is exhibiting “too much DIF”. The design analysis is carried out according to the same principles as those laid out in Gelman & Carlin (2014) using three hypothesized true correlations. The hypothesized true correlations correspond to the level thresholds of the zumbo-Thomas DIF evaluation scale: 0 (Null/Negligble DIF), 0.13 (Moderate DIF), 0.26 (Large DIF). B.5 Calculation Type-M Error Calculating the Type-M error rate for a derived statistic is done via a bootstrapping process which generates an empirical distribution for the Δ R^2 for each item. Once the empirical distribution has been derived, the following function is used to compute the Type-M error: 55 The above function is derived from that provided by Gelman & Carlin (2014) in their appendix. The changes are made in order to accommodate the reality that we cannot assume a normal distribution in this case. Type-S Error Type-S error is not calculated because it does not align conceptually with the tests being performed in this type of DIF analysis. The Δ R^2 effect size is a meta-statistic, as opposed to a normal measure of effect size like Cohen’s d, and one of the results of this is that Δ R^2 will only be negative in anomalous situations, meaning that there is no meaningful probability of achieving a statistically significant effect in the ‘wrong’ direction. B.6 Options 1. title: Design Analysis 56 Design Analysis is a True/False option, defaulted to False. This can be quite computationally intensive. If this option is selected, and then another option is changed, the module will incorporate the change as soon as possible, but several embedded loops mean that there may be a delay before you see the change take effect. 2. title: Constant Alpha Constant Alpha is a True/False option, defaulted to True. This option refers to whether or not you want to perform a post-data design analysis using the same alpha as in the DIF analysis, or using the minimum alpha required for significance for all items. 3. title: Bootstrap N type: Number Bootstrap N refers to the number of bootstrap replications to be used in creating the empirical distribution for Δ R^2. This is defaulted to 1000 as a balance between computation time and precision. Lower numbers will allow faster exploration, but will provide less stable estimates. 4. title: Observed Power Observed Power is a True/False option, defaulted to False. Toggling this option reveals the column showing the Empirical Observed Power. This is defaulted to False because its interpretation can be confusing, and will not always provide useful information. N.B. The Empirical Observed Power can equal 1 or 0. 5. title: Hypothesized True Effect Size type: String Hypothesized True Effect Size is used to perform design analysis using a non-default hypothesized true effect. If left blank, the analysis will be performed at each of the thresholds of the DIF Flag Scale selected above. 57 B.7 Item Response Curves Details 1. title: Item Response Curves based on Logistic Regression type: Variables Item Respnse Curves is a list (vector) of names referring to columns in the data selected for DIF analysis for which logistic regression Item Response Curves should be produced. B.8 Results Procedure Notes The descriptive overview provides a textual summary of the whole analysis, including what options were chosen, and documenting any warnings or errors that were encountered during the model fitting process. DIF Analysis Table The DIF analysis table provides an overview of the analysis results. For each item the following information is displayed: DIF classification, P-value, Chi^2 statistic, delta R^2. The purpose of this table is to provide an easily viewed and shared summary of results. Design Analysis Table The Design Analysis Table displays a list of all the items analyzed, their Type-M errors, and more. B.9 Item Response Curves The Item Response Curve can be plotted for any item being assessed for DIF. It provides a visual representation of the full model used in the DIF analysis. The different groups are colour coded, with 95% confidence intervals represented by gray shading overlaying each line. In these 58 images the Y-axis is the probability of endorsement (0 to 1), and the X-axis is the range of the matching variable used in the logistic models.
Thesis/Dissertation
eng
Measurement, Evaluation and Research Methodology
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Psychometrics & post-data analysis : a software implementation for binary logistic regression in jamovi
Text