Education, Faculty of
Educational and Counselling Psychology, and Special Education (ECPS), Department of
DSpace
UBCV
Coronel Villalobos, Mauricio
Doctor of Philosophy - PhD
University of British Columbia
In social sciences, the measurement of constructs requires decisions on how to quantify, record and analyse scores. The use of electronic surveys allows for additional measurement tools different from traditional paper surveys. However, the manner of how these choices affect the validity of scores has not been explored thoroughly. This dissertation uses the Center for Epidemiology Scale of Depression to explore the effects of scoring method and response formats. Study 1 compares four response formats commonly used in electronic surveys, and scores obtained using different electronic devices. Measurement invariance of the scale was examined using multi-group confirmatory factor analysis. Results showed that electronic response formats suggest response behaviours to participants. Furthermore, scores obtained using different response formats and electronic devices failed to show measurement invariance at a configural level. Study 2 compares changes in differential item functioning (DIF) using ordinal logistic regression based on the scoring method. Furthermore, DIF contrasts included binary sex, minority gender identity and sexual orientation. Results showed that changes in scoring methodology altered which items presented DIF. Moreover, contrasts with sexual and gender minority identity altered the presence of DIF. As such, treatment of scores and choice of focus for demographic characteristics impact the findings of psychometric properties of surveys. Overall, findings from both studies suggest that the structure of surveys can be fragile, and conclusions drawn from scores are sensitive to any changes to the measurement and analyses. Any such change, even if seemingly minor, may change the construct, and consequently affect the validity of scores. Researchers must be mindful of the specific needs of their target construct and use adequate forms of measurement and analyses to suit their studies, thus guiding more precise measurement. A fuller understanding of the factors that affect validity will result in more control to ensure accurate measurement and appropriate conclusions from obtained scores.
IMPACT OF SCORING AND RESPONSE FORMAT ON INFERENCES FROM ELECTRONIC SURVEYS FOR RATING SCALES by MAURICIO CORONEL VILLALOBOS M.A. The University of British Columbia, 2015 A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Measurement, Evaluation and Research Methodology) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) SEPTEMBER 2020 © Mauricio Coronel Villalobos, 2020 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the dissertation entitled: Impact of scoring and response format on inferences from electronic surveys for rating scales submitted by Mauricio Coronel Villalobos in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Measurement, Evaluation and Research Methodology Examining Committee: Dr. Bruno D. Zumbo, Professor, Educational and Counselling Psychology, and Special Education, UBC Supervisor Dr. Sheila Marshall, Professor, Department of Pediatrics, UBC Supervisory Committee Member Dr. Amery Wu, Professor, Educational and Counselling Psychology, and Special Education, UBC Supervisory Committee Member Dr. Kim T. Zebehazy, Professor, Educational and Counselling Psychology, and Special Education, UBC University Examiner Dr. Skye Barbic, Professor, Occupational Science & Occupational Therapy, UBC University Examiner Additional Supervisory Committee Members: Dr. Elizabeth Sanders, Professor, Educational Psychology, University of Washington External Examiner iii Abstract In social sciences, the measurement of constructs requires decisions on how to quantify, record and analyse scores. The use of electronic surveys allows for additional measurement tools different from traditional paper surveys. However, the manner of how these choices affect the validity of scores has not been explored thoroughly. This dissertation uses the Center for Epidemiology Scale of Depression to explore the effects of scoring method and response formats. Study 1 compares four response formats commonly used in electronic surveys, and scores obtained using different electronic devices. Measurement invariance of the scale was examined using multi-group confirmatory factor analysis. Results showed that electronic response formats suggest response behaviours to participants. Furthermore, scores obtained using different response formats and electronic devices failed to show measurement invariance at a configural level. Study 2 compares changes in differential item functioning (DIF) using ordinal logistic regression based on the scoring method. Furthermore, DIF contrasts included binary sex, minority gender identity and sexual orientation. Results showed that changes in scoring methodology altered which items presented DIF. Moreover, contrasts with sexual and gender minority identity altered the presence of DIF. As such, treatment of scores and choice of focus for demographic characteristics impact the findings of psychometric properties of surveys. Overall, findings from both studies suggest that the structure of surveys can be fragile, and conclusions drawn from scores are sensitive to any changes to the measurement and analyses. Any such change, even if seemingly minor, may change the construct, and consequently affect the validity of scores. Researchers must be mindful of the specific needs of their target construct and use adequate forms of measurement and analyses to suit their studies, thus guiding more precise measurement. A fuller understanding of the factors that affect validity will result in more control to ensure accurate measurement and appropriate conclusions from obtained scores. iv Lay Summary Researchers frequently use surveys to gather information; however, changes in how this data is collected and used affect the conclusions obtained. To ensure that these conclusions are as accurate or adequate as possible, researchers must understand the effect of these changes. This dissertation examines the effect of these changes on the adequateness of conclusions in two studies. Study 1 examines if different response formats of electronic surveys or the device used when responding to the survey changes how people respond to questions. Study 2 examines if changing how surveys are scored affects conclusions on how items work for different groups, and if these conclusions are the same if the groups compared are sexual orientation or gender minorities instead of the gender binary (male/female). This thesis shows how much these changes can impact the conclusions on research and emphasizes the importance of understanding these effects for accurate research. v Preface This dissertation is an original intellectual product of the author, M. Coronel Villalobos. Study in chapter 3 was an extension of the study by Gelin, M., & Zumbo, B. (2003); methodology in chapter 3 drafted in collaboration with Zumbo, B. No part of the dissertation is taken directly from previously published or collaborative articles. The fieldwork reported in chapter 2 was covered by UBC Ethics Certificate #H19-00619. The fieldwork reported in chapter 3 was covered by UBC Ethics Certificate # H16-02925. vi Table of Contents Abstract...................................................................................................................................... iii Lay Summary ............................................................................................................................. iv Preface ....................................................................................................................................... v Table of Contents ....................................................................................................................... vi List of Tables ............................................................................................................................ vii List of Figures .......................................................................................................................... viii Acknowledgements .................................................................................................................... ix Dedication ................................................................................................................................... x Chapter I - Introduction .............................................................................................................. 1 Validity of scores .................................................................................................................... 1 Response formats for rating scales ........................................................................................ 5 Online and electronic surveys ................................................................................................. 7 Item affordances ..................................................................................................................... 9 Effects of scoring ...................................................................................................................11 CES-D and CESD-R ..............................................................................................................12 Gender and sexual orientation as moderators to scores ........................................................15 Bringing concepts together ....................................................................................................17 Chapter II - Impact of Electronic Response Formats on Validity ................................................22 Introduction ...........................................................................................................................22 Methodology ..........................................................................................................................26 Results & discussion .............................................................................................................32 Conclusions ...........................................................................................................................38 Chapter III - Item Scoring and DIF for Non-binary Gender and Sexual Orientation ....................45 Introduction ...........................................................................................................................45 Methodology ..........................................................................................................................49 Results & discussion .............................................................................................................53 Conclusions ...........................................................................................................................63 Chapter IV - Conclusion: Consequences of scoring methods and response formats .................66 References ...............................................................................................................................76 vii List of Tables Table 2.1 Breakdown of demographics by response format.............................................. 29 Table 2.2 Totals of device used by response format.......................................................... 30 Table 2.3 Proportion of missing and multiple responses for response format by item....... 33 Table 2.4 Proportion of missing and multiple responses for device................................... 34 Table 2.5 Eigenvalues, ratios and ordinal alpha for response format and device.............. 35 Table 2.6 MG CFA results for response format.................................................................. 36 Table 2.7 Fit statistics of a one-factor model for each response format............................. 37 Table 2.8 MG CFA results for electronic devices............................................................... 37 Table 2.9 Fit statistics of a one-factor model for each device............................................ 38 Table 3.1 Demographics..................................................................................................... 30 Table 3.2 Mean total scores for the CES-D for each scoring method in each group......... 54 Table 3.3 Mean total scores for the CESD-R for each scoring method in each group....... 55 Table 3.4 CES-D ÎR2 values............................................................................................. 56 Table 3.5 CESD-R ÎR2 values........................................................................................... 57 Table 3.6 CES-D ÎR2 values contrasting DIF.................................................................... 63 viii List of Figures Figure 2.1 Radio buttons (Version A).................................................................................... 27 Figure 2.2 Checkboxes (Version B)...................................................................................... 28 Figure 2.3 Drop-Down menu (Version C).............................................................................. 28 Figure 2.4 Sliders (Version D)............................................................................................... 28 Figure 3.1 Scoring methods for the CES-D and CESD-R.................................................... 47 ix Acknowledgements I thank my family, to whom I owe everything. Their unconditional support and love has made this possible. I also thank Dr. B. Zumbo for his amazing insights and constant help, he is a brilliant mentor that keeps pushing me forward. I owe my gratitude to the faculty and fellow students of UBC, who aided me in this journey. In particular, I thank Dr. S. Marshall, Dr. A. Wu, Dr. E. Saewyc and Dr. A. Hubley, who have given me extraordinary learning opportunities. Finally, I thank my partner for the constant inspiration, support, help, patience, joy, laughs and love. Our mutual love of science has proven to be significant at p < .001. x Dedication To my partner, with whom all things are possible 1 Chapter I - Introduction Measurement in social sciences such as psychology, sociology, or economics, is to assign a numerical representation to a phenomenon, be it a physical property, a behaviour, or a mental construct (Campbell, 1952; Stevens, 1946). Not all measurement is equal; where physical properties and behaviours can often be observed, counted or establish proportion, mental constructs often rely on observable phenomena such as responses to questionnaires resulting in indirect measurement. In this setting, researchers develop rating scales, which are a collection of items that ask participants to provide an assessment of their attitudes, opinions and experiences to serve as indicators of a construct (Brill, 2008; Drasgow, Chernyshenko, & Stark, 2010; Hodge & Gillespie, 2003; Roiser, 1996). By assigning a numerical value to this assessment of a construct, we obtain scores. These scores represent a classification, quantity or magnitude of the measured construct, which practitioners and researchers use to make estimations, inferences and claims. Therefore, it is critical that this measurement is made in the most accurate manner possible, and that the inferences drawn from these scores represent the relevant population appropriately. It is the main interest of this thesis to explore some of the factors that may impact this accuracy and appropriateness; in other words, this thesis main focus is on validity evidence for scale scores. Validity of scores The inferences drawn from scores in their intended use must be supported by evidence and theory; this collection of supports is validity evidence (AERA, APA, & NCME, 2014). Validity is not a property of a measure or scale, but instead of the obtained scores (Zumbo & Chan, 2014), and is vital to ensure that scores are interpreted suitably in the context in which they are pertinent. 2 Validity evidence is not a static single source, and any changes that alter obtained scores will, in turn, affect validity. That is to say that it is not sufficient to obtain validity evidence for scores in a single manner; whenever the source of scores changes in a significant manner (e.g., different population, changes in item wording or response options) validity evidence must be examined again. Furthermore, examining validity evidence in different manners strengthens the support towards the appropriateness of the inferences obtained. There are various manners from which this validity evidence can be approached. This document focuses mainly on the guidelines provided by the Standards for Educational and Psychological Testing (AERA et al., 2014), which argue for validity based on the sources from which this evidence is drawn. However, this is complimentary with other validity perspectives. Cronbach (1988) viewed validity as the supporting arguments that tested the data and in need of continuous updating. Messick (1989) suggested a view of validity as a unified source centred on the construct. Kane (2006, 2013) argues for the examination of the utility and adequacy to the context. Zumbo (2007, 2009) proposes validity as an explanation for the variation of scores between samples. In short, these various views of validity aim to provide support for the adequate use and interpretation of scores. While there is value to the many approaches for validity evidence, the focus of this thesis will be approached from the lens of unified validity theory, as described by Messick, and The Standards. When classifying validity based on the sources of its evidence, The Standards (AERA et al., 2014) classifies it as validity based on content, on response process, on internal structure, in relation to other variables and on consequences of testing. The most common sources of validity evidence reported are based on internal structure and relation to other variables, while evidence based on response processes or consequences of testing is seldom examined (Chan, Munro et al., 2014; Chan, Zumbo, Chen et al., 2014; Chinni & Hubley, 2014; Cizek et al., 2008; Cox & Owen, 2014; Gunnell, Wilson et al., 2014; Hogan & Agnello, 2004). 3 Validity based on content examines how closely the intended construct relates to the scale measuring it, supported by the relevance and representativeness of the items to the construct. It can be examined by the themes, wording and scoring processes utilized on the scale and is threatened by construct-irrelevant variance and construct underrepresentation. Validity evidence based on content can be supported by presenting empirical evidence on the construction of the scale items, and by using subject matter experts or experiential experts to assess the appropriateness and representativeness of the scale and the response options (Grant & Davis, 1997; Haynes, Richard, & Kubany, 1995). Validity evidence based on response processes examines the interaction of participants to the scale; this validity evidence is supported when participants respond to the tests in an expected or intended manner, including understanding the task and responding to their best of their abilities. Validity evidence based on response processes does not evaluate subject ability, item appropriateness, or whether participants give the desired response; instead, this evidence source examines the appropriateness of the cognitive process that participants take during measurement. Think-aloud protocols and cognitive interviews can be used to assess response processes (Hubley, 2017; Launeanu, 2016; Maddox & Zumbo, 2017; Padilla & Benítez, 2014). Validity evidence based on internal structure refers to the relation between the test items to the latent variable intended to be measured based on reliability, invariance and dimensionality of the test (Furr & Bacharach, 2014; Rios & Wells, 2014). Many statistical analyses can be used to examine these psychometric properties of the test, and are commonly reported for scales as a standard practice. Reliability is the degree of consistency or repeatability of scores upon multiple testing and can be examined using alpha and test-retest coefficients. Measurement invariance examines whether tests obtain the same scores between individuals with the same level of the construct when these subjects belong to different groups; in other words, if the test is consistent between different populations. The dimensionality of a 4 scale is determined by how the items of a scale group together to represent one or more constructs, and their relation to the intended construct. Factor analyses, differential item functioning and item response theory, can be used to test scale invariance and dimensionality. Validity evidence based on relationships to other variables examines how scores of a test relate to scores of other tests or constructs. This validity evidence can be examined through convergent, discriminant, test-criterion and validity generalization. Convergent validity compares how closely scores compare to a theoretically similar measure; in contrast, discriminant validity is the lack of correlation with scales that evaluate a different construct. It is useful to examine convergent and discriminant validity evidence to assess the relationship of the scale scores with other measures with various degrees of relation to the construct (AERA et al., 2014; Furr & Bacharach, 2014). Similarly, test-criterion validity evidence refers to the ability of a test to measure or predict an expected outcome, either concurrently or predictively. Validity generalization refers to the degree to which scores and their inferences are relevant to various groups, that is, the scope of the population to which scores are valid. Validity based on consequences evaluates the outcomes that take effect when tests are used for their intended purpose. It does not relate to test misuse or abuse, or issues of ethics and policy (Cizek et al., 2010; Messick, 1995a; Messick, 1995b). Validity based on consequences relate to the value implications of a measure; these are observed on the influences, social and personal values that guide the development of a test. This source of validity also assesses the short- and long-term social consequences of the measure, including intended and unintended results. Validity evidence based on consequences is informed by the anticipation of benefits of testing and prevention of detriment caused by it, as well as the evidence that supports claims of these purported benefits of the use of scores and inferences drawn from them. Factors that threaten this source of validity are scores that favour a particular group or groups and results in a disproportionate outcome, such as being admitted into a school 5 or a job. This source of validity evidence can interact with other sources of validity evidence, such as with validity evidence based on content on having strong construct representation, or validity evidence based on internal structure supported by strong measurement invariance. The studies in the following chapters will examine different manners in which scores can be obtained, some ways in which these scores are affected, and how these changes may affect the inferences drawn from scores. As described earlier, these are some of the ways in which validity evidence is affected. Understanding these changes will allow for better control of the experimental factors that may threaten validity evidence, thus allowing more accurate measurement and validation. Response formats for rating scales The set of items that form a rating scale intend to quantify some facet of a person using self or peer-reports. This measurement reflects a personal evaluation of an individual or an experience, often measured by degrees of agreement with a stimulus item (e.g., I am satisfied with my life, rated from 1-strongly disagree to 5-strongly agree). When these values can be added for a total or average score, they are considered summated rating scales, which imply that responses are equal among items and are ordered responses with approximately equal intervals. The response options presented in rating scales can also imply order and magnitude. These properties of the response options determine their polarity. Response options that do not imply magnitude, or order relative to each other, are non-polar. Non-polar responses are often mutually-exclusive options (i.e., categorical) and can be used to analyze distributions and frequencies of choices (Downing et al., 1995; Hancock et al., 1993). Ordered responses that range from complete absence to some quantity, frequency or intensity are unipolar, such as âneverâ to âalways.â Conversely, bipolar responses range between two opposites, such as 6 âcompletely disagreeâ to âcompletely agree.â Both unipolar and bipolar responses can be used to measure a degree of agreement, skill, or quantity of a parameter in a distribution; unlike non-polar scales, they can imply attitudes and ranges within a construct (Ackerman et al., 2016; Downing et al., 1995; Wetzel et al., 2016). Response formats present different manners in which items are presented, and data is measured on rating scales. Different response formats will affect the response of participants, their cognitive process, and obtained scores (Ackerman et al., 2016; Keehner & Moon, 2018; Ritter & Sue, 2007; Wetzel & Greiff, 2018); as such, different response formats should be considered for what better suits the researcher needs and intended measure. Response formats may be classified by the labelling of their response options as verbal, numerical, and graphical. Verbal rating scales are those that present response options with verbal labels that represent a degree of agreement or a choice (Dictionary of Psychology, 2015), such as Likert scale, semantic differential, and phrase completion. For numerical rating scales, the response choices are given a numerical value to identify the amount of agreement or assessment of a construct and may label some or all their response intervals. Commonly, numerical rating scales are accompanied by instruction or verbal labels that describe that the numerical ratings represent. Graphic rating scales measure a construct, agreement or attitude using a symbolic representation such as a mark on a slider or facial expressions to represent the severity of pain. Some graphic rating scales may be presented without a verbal or numerical label as their response options are representative of the construct being measured; as such, they can be useful for young or illiterate participants. The choice of verbal, numerical or graphical rating scale can influence how a participant responds to traditional paper surveys. Verbal rating scales are frequently used in psychosocial measurement (Brill, 2008; Lee & Paek, 2014; Wakita et al., 2012), they identify the meaning of a response option and provide it with context. Item anchors with verbal formats produce higher 7 scale reliability, but participants will be more likely to endorse positive-leaning responses or central choices (Hjermstad et al., 2011: Krosnick & Berent, 1993: Weijters, Cabooter, & Schillewaert, 2010). Unlike verbal rating scales, numerical rating scales assign a value that participants often identify as more objective, and responses will often avoid extreme choices (Cabooter, Weijters, Geuens, & Vermeir, 2016; Schwarz, Knäuper, Hippler, Noelle-Neumann, & Clark, 1991). However, they rely on some indication of what the values mean, either through clarification on the instructions or the use of a verbal label to anchors. Furthermore, the numerical values assigned to response options will suggest some order, or magnitude, to the response options. Graphic rating scales have the advantage of their non-verbal format, which makes them more accessible and easier to relate to (ten Klooster et al., 2006; Reips & Funke, 2008); however, their meaning may be ambiguous to participants, and often require more space to be displayed correctly. Online and electronic surveys The rise of technology as an everyday commodity has made it possible for most of the population to have access to computers, phones and other similar electronic devices. This widespread availability of electronics has caused the use of electronic or online surveys to be on the rise, using devices such as computers and mobile devices to perform measurements on various sites. Electronic assessment or testing is when the measurement is performed using electronic devices; when this measurement is performed or can be accessed online, it is considered an online assessment. This method of assessment performs similarly to traditional paper surveys and can obtain similar responses and inferences from scores, they can allow the immediate upload of results, provide fast feedback to participants on their performance, allow live monitoring of the measurement, and reduces requirements of physical data storage or risk of loss of physical copies of the data (Giduthuri et al., 2014; Leisher, 2014; Ravert, Gomez-Scott, & Donnellan, 2015). 8 In self- and peer-rating, electronic assessment is useful to data collection, as it can expand the geographical range accessible to researchers, streamline the measurement process, reduce the need for an active supervisor or researcher, expedite the data capture and scoring, and reduce the input error and ambiguousness of responses (Evans & Mathur, 2005; Van Selm & Jankowski, 2006). Electronic assessment can be administered automatically and guide participants autonomously, which can provide a greater sense of anonymity and comfort, as well as reducing some effects of social desirability or having the presence of a researcher influence the responses (Fitton, Ahmedani, Harold & Shifflet, 2013; Lim, 2011; Van Selm & Jankowski, 2006). Moreover, the convenience of access on personal devices can allow greater representation of groups who might fear exposure or violence, as well as communities that may be physically distant or hard to reach (Wellman, 1997; Wright, 2005). The inclusion of larger populations and more representation of otherwise hard-to-find groups allows for more inclusive groups that better reflect the populations intended to be examined, and permit inferences that represent the intended constructs being measured more accurately. Response options for electronic rating scales have some common ground with Technology-Enhanced Items (TEI), which are computer-based items that measure responses with formats not available to multiple-choice or constructed-response items (Bryant, 2017); however, these are different classifications, and their terminology should not be used interchangeably. Although TEI includes some of the response formats included in this dissertation -such as drop-down menus- they exclude responses that are shared for electronic and paper testing -such as radio buttons or check boxes-, and utilize behavioural measurements (e.g., âhotspotâ items, AI interactions, microphone or webcam recordings). As such, it is important to draw this distinction between items created for electronic measurement and measurement done with the aid of technology. 9 Given the increasing ease of access and subsequent frequency of use of electronic testing, it is important to understand how this may affect measurement and scoring, and their impact to psychometric properties, statistical analyses and their implications for measurement validity. Validity evidence of testing is threatened when scales and scores are not adequately implemented, or the construct is over or under-represented (AERA et al., 2014). Similarly, the implementation of different response formats may affect the responses and behaviours of participants, affecting scoring and validity (Ackerman et al., 2016; Ritter & Sue, 2007; Wetzel & Greiff, 2018). However, there is little literature that examines the use of response formats and whether there are any effects on using different ones for recording scores. Without supporting literature that examines the use of electronic response formats, there is little guidance in making decisions for what formats can be used for accurate measurement (e.g., Ravert et al., 2015; Schleyer & Forrest, 2000; Sue & Ritter, 2007). Some of the few studies that examine electronic formats report scores have been reported to be unaffected by item orientation (vertically or horizontally), there are no significant differences between items that are presented increasing or decreasing in an ordered rating scale. There is no interaction effect between orientation and response order (Garbarski, Schaeffer, & Dykema, 2018). Item affordances While the orientation or order does not affect the response process of participants, different response formats or item presentation can affect the cognitive process of participants; these changes on perception are item affordances (Keehner & Moon, 2018). Affordances refer to when the environment provides a resource or support that leaves the user an opportunity to act in a certain way (Colman, 2015). In other words, people will intuit what actions are afforded to them based on what they can perceive, and what actions would be natural or simple responses based on these prompts. 10 Affordances provide cues based on visual design and interactivity that allow users to determine how to operate within the environment; this extends into graphical user interfaces, including those used for electronic measurement (Kaptelinin, 2013). Some of these affordances are conveyed by the choice of wording or the context in which the measurement takes place, but the response format itself may also have intrinsic affordances. How an item is designed will prompt the user to interact differently; for example, a true or false response format can shift the choice of an item as an explicit or an implicit decision, depending on whether the item forces the participant to select whether the statement is true or false (explicit choice) or it requires participants to mark only true statements (implicit choice); participants will omit responses during uncertainty, but will lean towards responding affirmatively (Cronbach, 1941; Pomplun & Omar, 1997). One way of understanding item affordances is by observing the response process of participants, including their missing response patterns. Understanding why some items may be more aversive or attractive can give insight on what affordances the items are providing to the participant, and provide response process validity evidence to the obtained scores. As some item affordances are provided by the manner in which an item is constructed, researchers must be aware of possible bias or background cues that may influence responses. Presenting more attractive wording, starting or default values, and options that are easier to select could be endorsed more frequently, resulting in an inaccurate measurement. One could think of this as participants being asked to choose one of five doors to walk through, and the researcher inadvertently leaving one of them open. As changes in item affordances affect the response process of participants, it affects the validity of obtained scores. Therefore, item construction should be a deliberate process that considers how its presentation will affect the assessment. However, these critical decisions do 11 not end once scores have been obtained; researchers also decide how to analyze and interpret these scores, including the choice on how to score these responses. Effects of scoring Obtained scores from the measurement are, in essence, quantification of a construct or behaviour; as these scores are often not naturally assigned, the manner in which responses are scored are determined by the values chosen by researchers. These choices in scoring can contextualize or specify how a specific measured construct is being examined; for example, a total score for a summated rating scale shows a gradient of intensity or frequency, and responses scored dichotomously can indicate high and low groups, or showing any vs no symptoms. There is scarce literature on the effects of collapsing scores and response formats of electronic surveys. These gaps in the literature hint at a lack of understanding of the repercussions of modifying these core elements in measurement, or the fundamental role of the inferences drawn from scores. Thus, it is vital to explore the impact of scoring methods and response formats on electronic measurement, and the inferences drawn from those scores. Some known effects on grouping or collapsing scores is that these changes affect the stated purpose of the instrument and interpretation of scores. Furthermore, collapsing response categories affect the scale psychometric properties, including variance and distribution. How scores are recorded and interpreted affect the internal structure of the measure, such as with differential item functioning (DIF), which in turn, alters the conclusions obtained from their analysis and can threaten their validity, as described by Gelin & Zumbo (2003). Gelin and Zumbo used the Center of Epidemiologic Studies Depression (CES-D) scale to test for DIF across three different scoring methods between male and female participants and identified that, depending on which scoring method used, different items would show gender DIF. This 12 finding demonstrated that DIF is a property of the item, the scoring method used, and the intended purpose of the test. As part of this thesis, this previous study by Gelin and Zumbo (2003) will be extended in study 2 by using the original CES-D and its revised version (CESD-R), as well as adding the contrast groups of gender minority/non-minority and sexual orientation minority/non-minority in addition to binary gender (male/female). CES-D and CESD-R The Center of Epidemiologic Studies Depression (CES-D) scale was initially developed for use in studies of the epidemiology of depressive symptomatology in the general population. As the original developer, Radloff, notes, the CES-D was designed to measure current level of depressive symptomatology by ascertaining the frequency of this symptomatology in the previous week (Eaton et al., 2004; Radloff, 1977). It was intended as general-population alternative to scales that were focused at clinical evaluation, and emphasizes the affective component. The CES-D contains 20 self-rated items rating common depressive symptoms (e.g., item 2 âI did not feel like eating; my appetite was poor."). Response options are scored from 0 to 3, where option 1 = rarely or none of the time (less than 1 day), option 2 = some or a little of the time (1-2 days), option 3 = occasionally or a moderate amount of the time (3-4 days), and option 4 = most of the time (5-7 days). The scale suggests that high scores relate to the high frequency of depressive symptomatology, and scores above 16 are considered highly likely to be clinically depressed (Radloff, 1997). Validity evidence for the original scale was explored from the lens of unified validity theory as described by The Standards (Radloff, 1977).The scale presents reliability evidence including alpha coefficients from .78 to .90, internal structure validity evidence for a single factor 13 with alpha between .85 and .92, test-retest r of .46, and criterion validity evidence compared to clinical diagnoses (r = .44 to .75), which support the use of the scale for its intended purpose (Lewinsohn, Seeley, Roberts, & Allen, 1997; Radloff, 1977; Radloff & Lock, 1986). The scale has been tested in diverse racial and ethnic groups, and ages ranging from teenagers to seniors over 70 years (Roth et al., 2008). The CES-D can be scored adding individual item scores (0-3) to obtain a total score up to 60; alternatively, the scale can be scored by presence of depressive symptoms or persistence of these symptoms over a period of time (Eaton et al., 2004; Gelin & Zumbo, 2003; Radloff & Lock, 1986). Similarly, the Center for Epidemiologic Studies Depression â Revised (CESD-R) scale is a self-report rating scale that revised the CES-D in 2004 (Eaton et al., 2004). Although some items are equal or similar, some items have been replaced. Furthermore, the response options for this scale now range from 0 to 4, where Option 1 = Not at all or less than 1 day last week, Option 2 = One or two days last week, Option 3 = Three to four days last week, Option 4 = Five to seven days last week, and Option 5 = Nearly every day for two weeks. With this additional response option, scores can now range from 0 to 40, but still maintain a recommended cut-off of 16 points to identify possible clinical depression; additionally, an alternative scoring method allows for grouping the last two categories into a single one to recreate the original scoring method and total (Eaton et al., 2004). Validity evidence for the use of this scale is similar to that of the CES-D (Eaton et al., 2004; Van Dam & Earleywine, 2011), presenting evidence of a single factor structure with high internal consistency (α = 0.93), and convergent and discriminant validity evidence with anxiety (r = 0.74, p < 0.01), schizotypy (r = 0.44, p < 0.01), positive affect (r = â0.26, p < 0.01), and negative affect (r = 0.58, p < 0.01). Both versions of the scale have similar intended uses. However, as there are wording and scoring differences between them âincluding a different maximum total score, psychometric properties for these scales cannot be treated equally, even when utilizing scoring methods that 14 modify total scale scores to the same range. Due to these changes, it is necessary to treat these scale versions separately when investigating their reliability and validity evidence. Similarly, inferences drawn from one version may not be directly extended to the other version. While the CES-D is a popular depression scale, there is evidence of its limitations in use; previous literature has reported some issues of validity evidence based on relations to other variables, as the scale has shown high correlation with measures of anxiety, demoralization, and distress (Orme, Reis, & Herz, 1986; Vernon & Roberts, 1981); similarly, its factor structure has been explored from one up to seven possible factors (Beals, Manson, Keane & Dick, 1991; Callahan & Wolinsky, 1994; Makambi, Williams, Taylor, Ronsenber & Adams-Campbell, 2009; Schroevers, Sanderman, van Sonderen & Ranchor, 2000; Thorson & Powell, 1993). Furthermore, research has indicated a gender effect in the scores of the CES-D, with females obtaining significantly larger scores (Arbona, Burridge, & Olvera, 2017; Callahan & Wolinsky, 1994; Verhoeven, Sawyer, & Spence, 2013). The main interest of this thesis is not to demonstrate further validity evidence for the use of the CES-D and the CESD-R; rather, these measures are used as illustrations and experimental instruments for the contrasts of the scoring method, DIF, and response format. However, it is important to note that the limitations of the scale may have some effects in the results of the experimental contrasts in the studies reported herein, as it is possible to expect analyses to show issues with the factor structure, and gender effects on scores. While acknowledging its limitations, the CES-D and CESD-R were chosen for these studies because they are popularly-used scales with documented validity evidence âas described earlier. Moreover, they have documented alternative methods of scoring, which were contrasted for changes in gender DIF (Gelin & Zumbo, 2003). It is noteworthy that the original scale and this thesis approach validity evidence from the lens of unified validity theory. Given 15 this, the present thesis intends to build on the study by Gelin and Zumbo (2003) by exploring in study 2, the differences in DIF by different scoring methods using the CES-D and CESD-R. Gender and sexual orientation as moderators to scores Whether reporting the demographic characteristics of our samples or comparing groups based on gender for purposes of DIF or interventions, researchers frequently utilize binary gender male-female (M/F) and omit other gender and sexual identities (Garvey, 2017; Palan, 2001; Saewyc, 2017; Treharne & Beres, 2016; Weston, 1991; Zumbo et al., 2015). However, these descriptors are insufficient for accurately portraying the sample or assessing differences accurately. Gender identity is an influencing factor in how participants perceive and respond to surveys, but the inclusion of degrees of masculinity and femininity are stronger predictors of survey response behaviour (Clarke, Ellis, Peel, & Riggs, 2010; Fischer & Arnold, 1994; Palan, 2001; Weston, 1991). Furthermore, the exclusion of gender identities generates inaccuracies responses that lead to measurement error or outliers (Treharne & Beres, 2016; Zumbo et al., 2015). Therefore, when making contrasts of gender identities, such as gender DIF, it is important to extend these contrasts to include gender identities beyond a traditional gender binary. Similarly, sexual orientation or identity can be closely related to gender identity (Weston, 1991). Studies that consider gender or orientation should be aware of gender minority âthose whose gender identity or sex-expression differs from binary masculinity or femininity, trans, transgender, gender-fluid, queer, or an analogous term, and sexual orientation minorities âgay, lesbian, bisexual, pansexual, queer, two-spirited, or an analogous term. Both gender and sexual minority groups frequently present a higher risk of depression, self-harm, suicidality and feelings of exclusion (Birnholz & Young, 2012; Garvey, 2017; King et al., 2008; Woodford, Kulick, 16 Garvey, Sinco, & Hong, 2018). Lack of inclusion in mental and physical health research means that the particular challenges and needs of these groups are systematically omitted, resulting in research that lacks relevance or direct applicability, and healthcare professionals that are uninformed, unable or unwilling to provide adequate care (Bauer et al., 2009; Saewyc, 2017). Measurement and report of sexual orientation can present oversimplified groups or non-representative descriptions; one challenge in measurement research is finding a balance for the thoroughness of information against the number of variables, as well as the need for simplicity and clarity (Chun & Katayama, 1996; Saewyc, 2017; Treharne & Beres, 2016; Weinrich, 2014). Examining participant demographics in a simplified binary identity systematically excludes populations that are often under-represented, reinforcing a lack of acknowledgement of minority populations. This under-representation can bias research and limit the generalizability of findings, which threatens the validity of inferences drawn from these scores. In other words, studies that include gender contrasts should be cognizant and inclusive of minority gender identities. To increase the generalizability of findings in research relevant to sexual orientation or gender identity, researchers must consider the relevance and adequacy of the items and measures utilized. In survey-based research, the inclusion of sexual orientation and gender diverse groups has increased in research years, particularly in the healthcare field (Garvey, 2017; Hart, Saperstein, Magliozzi, & Westbrook, 2019; Nagoshi & Brzuzy, 2010; Saewyc, 2017). However, this is still far from the norm, and there is no consensus on how to adequately measure gender or sexual identity (Clarke et al., 2010; Garvey, 2017; Gonsiorek, Sell, & Weinrich, 1995; National Institute of Mental Health, 2016; Saewyc et al., 2004; Treharne & Beres, 2016). In chapter 2 of this thesis, findings on differences in DIF between binary gender M/F will be extended to contrasts between gender minority to non-minority (i.e., cisgender binary), and 17 sexual minority to non-minority (i.e., heterosexual identity). While the choice of contrast groups simplifies sexual orientation and gender identity into binary groups, it is intended to illustrate that analyses for gender binary are insufficient, rather than providing a full description of differences for all gender and sexual identities. Bringing concepts together This thesis touches on various aspects of measurement. In part, it informs the decision-making process during test construction and score analysis by comparing differences between different response formats and scoring methods. It should be noted that throughout this thesis, the terms âtestâ, âmeasureâ and âscaleâ are used interchangeably. This usage of the interchangeable terminology, although somewhat contentious, is supported by the fact that unified validity theory has been recommended for survey instruments (Zumbo & Padilla, 2020). It also challenges some assumptions in applied measurement such as the treatment of electronic response formats as interchangeable, the use of binary response options for gender, and how rescoring may affect inferences drawn from scores. This thesis could have examined various statistical or psychometric features of the CES-D (or any other similar scale); however, the format and scoring methods are investigated through the lens of validity theory of a survey measure using the CES-D and CESD-R as illustrative tools. In other words, this thesis focuses on the validity of scores for electronic surveys. Although there are many factors that can affect the validity, this paper focuses on the relation of scores to the decision-making of researchers that take place during three stages: the design of a measure, recording of scores, and analysis/interpretation of those scores. During the design of a measure, researchers should design a number of items based on their intended construct in a specific population, and test them for content validity evidence (AERA et al., 2014; Boateng, Neilands, Frongillo, Melgar-Quiñonez, & Young, 2018; Carpenter, 2018). However, regarding the choice of response format for electronic surveys, there are 18 virtually no guidelines, whether they have an impact on scores, what is their intended use, and what is their relationship or effect to validity inferences. Subsequently, the recording of scores from a measure takes place when a survey has been piloted, and there is evidence of reliability and validity of its scores. During this stage, scores can be affected by measurement error introduced by interviewer behaviour, respondent behaviour, and errors introduced by the survey itself; in other words, measurement error affects validity evidence of response processes (AERA et al., 2014; Visser, Krossnick, & Lavrakas, 2000). A source of measurement error during the recording of scores can occur when a survey is unclear, difficult to respond, or participants are not recording their responses in a manner that reflects their intended response (Furr & Bacharach, 2014; Niemi, 1993; Visser et al., 2000). Understanding this source of error demonstrates why it is crucial to understand how response formats can affect response processes. Once scores have been recorded in a survey, researchers have multiple ways to analyse and interpret these scores. While the measured construct and the intended responses recorded (that is, a respondentâs input) do not change, how a researcher alters or analyses these scores may affect the construct. The practice of rescoring, for example, has implications on internal structure validity evidence, as it may alter the conclusions in evidence of DIF (Gelin & Zumbo, 2003). Furthermore, changes in how scores are analyzed and interpreted may affect the conclusions and use of said scores, with possible implications on validity evidence based on consequences of measurement (AERA et al., 2014; Messick, 1989). As such, researchers must take care to not alter the construct and interpretation by redefining how the scores reflect the intended construct. As in previous steps of research, it is the decision process that may affect the validity of scores. Across the research process, decisions and practices that can appear to have a minimal impact could, in fact, have substantial repercussions on the findings. Each of these steps is 19 closely related to each other and the validity of scores. Test items that have not been designed adequately will obtain scores that may deviate from the true score, which in turn can lead to distorted estimates; this measurement error can be further aggravated if these scores are, in turn, rescored or analyzed inadequately. (Boateng, Neilands, Frongillo, Melgar-Quiñonez, & Young, 2018; Carpenter, 2018). Thus, it is important that scores are obtained and analyzed as accurately as possible to support validity evidence. In other to test the possible effects that research decisions have on validity evidence, Study 1 uses Multi-Group Confirmatory Factor Analysis (MGCFA), and study 2 uses Differential Item Functioning (DIF). Although both of these analyses are rooted in confirmatory factor analyses and provide validity evidence based on internal structure, they examine different perspectives on measurement and have a different purpose. MGCFA is intended to examine whether the measurement is testing the same construct in different conditions, the generalizability of these constructs and whether the change in parameters results in measurement invariance (Brown, 2006; Gamst, Meyers & Guarino, 2008; Sörbom, 1974). That is, MGCFA tests whether multiple groups are equivalent to all other factors (such as demographics) randomly sampled and thus assumed to be equal. Therefore, in study 1, MGCFA examines whether the changes in the response formats alter the construct being measured. In other words, it focuses on changes in the scale. Conversely, DIF analyses are intended to compare whether items function in the same manner among different groups of people, or to provide validity evidence for use in new settings or populations (Brown, 2006; Furr & Bacharach, 2014; Zumbo, 2007a). DIF does not examine whether the test is measuring a different construct, but rather if it functions equally between contrast groups. In this case, it contrasts two specific characteristics (e.g., gender) of the groups in the random sample. In other words, the focus is on the changes between the scores of the participants. As described earlier, the CES-D has been found not to be strictly unidimensional in 20 different samples. However, in DIF analyses it is possible to examine multidimensional models, with the underlying assumption that scales can be, to some degree, multidimensional (Zumbo, 2007a) while the primary factor for the CES-D reflects depressive symptomatology. This is sometimes referred to as a position of âessential unidimensinalityâ wherein minor secondary dimensions may reflect sample and item-specific characteristics. In this case, the measurement of gender and sexual minorities and item affordances may be reflective of the secondary dimensions in DIF. This aids in explaining why a single-factor model may have issues of fit, but can still be essentially unidimensional (Nandakumar, 1991; Slocum-Gori & Zumbo, 2011; Stout, 1987). For example, as this research literature describes, a common way to quantify the essential unidimensionality is the ratio of the first to second factor eigenvalues of the correlation matrix of the item response data being greater than 4.0. An advantage of an essentially undimiensional scale is that it is sufficient for logistic regression methodologies, and yet allows comparisons of changes in fit based on this model for MGCFA. In sum, these two factor analysis-based methodologies provide validity evidence for the scale but allow for a more relevant contrast at the different stages of the test construction and validation studies: MGCFA for the design of a measure, and DIF for the analysis and interpretation of scores. In short, this dissertation centers around two empirical studies. The first study, reported in Chapter 2, investigates four different response formats in electronic surveys using multi-group confirmatory factor analysis to examine whether these changes affect the scoring and internal structure of the CES-D. These findings will contribute to the growing literature on the impacts of item scoring, response formats, and psychometric properties of measures, including DIF. The second study, reported in Chapter 3, is an extension of the research by Gelin and Zumbo (2003), who demonstrated that psychometric properties of a scale -specifically, differential item functioning (DIF)- can depend on how an item is scored; this was achieved by investigating how 21 changes in the scoring method for the Center for Epidemiology Scale of Depression (CES-D) altered the conclusions of binary gender DIF. To expand on these findings, study two included the analysis of DIF attributable to sex (M/F), gender identity and sexual orientation for the 7CESD, and included the CESD-R, which has different response options than the earlier version. It should be emphasized that this study is not about providing validity evidence of the CES-D or the CESD-R scores, rather, these scales are used as an illustration of a practical experimental instrument to allow comparisons across the experiemental conditions of interest. Thus, the main goal of two studies in this thesis is to provide evidence on the role of scoring and response formats in electronic surveys to the validity of inferences drawn from scores. 22 Chapter II - Impact of Electronic Response Formats n Validity Introduction Rating scales are items that ask participants to quantify their own or otherâs behaviours and attitudes, and whose scores can be interpreted as a group (Brill, 2008; Drasgow, Chernyshenko, & Stark, 2010; Hodge & Gillespie, 2003; Roiser, 1996). They can be grouped as verbal, numerical and graphic; verbal rating scales label their response options with one or more words to identify what the possible values are (e.g., Strongly Disagree, Completely like me, Almost Always), numerical rating scales identify their response values with a number. They often require at least one verbal anchor to contextualize the values presented. Graphic rating scales use symbols or image representations as response choices, such as a selection of different pictures or presenting a line where participants can mark their estimated value. Rating scales do not have a single universal response format, and rather, they have various forms in which they can be presented. Response formats are the different ways in which responses can be organized, presented and recorded, independently of the item (Ackerman et al., 2016; Sue & Ritter, 2007; Wetzel et al., 2016; Wetzel & Greiff, 2018); response formats can also change if the data is recorded as discrete or continuous values (Rausch & Zehetleitner, 2014), and affect which statistical analyses are appropriate. It is important that an adequate response format is selected in measurement, as different response formats can affect how participants will interpret an item, their cognitive process towards it, and their corresponding response (Keehner & Moon, 2018). Given the aforementioned effects that response formats have on scores and their subsequent inferences, it is imperative for supporting their validity to select an adequate response format for the intended measurement. Electronic measurement utilizes devices such as computers, tablets and mobile devices as a way of obtaining data as an alternative to traditional paper surveys. They can facilitate data 23 collection, reduce expenses, increase collection range and shorten data capture (Evans & Mathur, 2005; Van Selm & Jankowski, 2006). Online-assessment is an electronic measurement that is typically performed with the use of the internet, which can upload results immediately, provide feedback or reduce the requirements of the device storage. Data obtained in electronic testing obtains similar responses and inferences to their traditional paper counterparts (Giduthuri et al., 2014; Leisher, 2014; Ravert et al., 2015). Online surveys can automatically be administered, record and score responses, and guide participants; they can also provide feedback, send the recorded data to researchers or storage locations, access distant populations or groups who are hard to locate, are very distant or require additional anonymity (Wellman, 1997; Wright, 2005). When participants can respond to tests from home or in mobile devices, they can be more comfortable for the participant and reduce the influence of the presence of researchers during survey responding (Fitton, Ahmedani, Harold & Shifflet, 2013; Lim, 2011; Van Selm & Jankowski, 2006). It is possible to present analogous items on paper and electronically. However, there are multiple formats in which surveys can be adapted online, some of which are exclusively possible through electronic platforms due to their interactivity. Of all possible response formats, some commonly used are radio buttons, checkboxes, drop-down menus and sliders (Funke et al., 2011; Kero & Lee, 2015; Roster, Lucianetti, & Albaum, 2015). Radio buttons are a very commonly used form of response format, often considered the traditional or default option in electronic assessment, can be presented either horizontal or vertical without a significant difference, allow only a single response, and can be labelled in any number of the data points. They are visually similar to traditional paper rating scales and are perceived as easy to use and intuitive for users. They are better suited for categorical data, although there is evidence of their use for continuous data based on the chosen wording and operationalization of the responses (Funke et al., 2011; Kero & Lee, 2015; Roster et al., 2015). 24 Checkboxes are similar to radio buttons in that they can be presented in vertical or horizontal formats, and can be labelled in any number of their anchors. However, they allow for any number of responses to be selected, which is useful for measurements in which more than one condition may apply (Sue & Ritter, 2007). Although the research literature on the characteristics of this response format is limited, it can be inferred that this format is oriented towards categorical data with responses that may not be mutually exclusive. This response format could also be used for other data, although there is a risk of participants selecting more than one option. Drop-down menus are dialogue boxes that list the possible choices, often a numerical value or a verbal label (Healey, 2007; Liu & Conrad, 2016; Sue & Ritter, 2007). They can be useful to reduce the visual space of an item or accommodate lengthy response options and are overall comparable to radio buttons or visual analogues in regards to response bias. However, this format requires labelling of all levels of the responses, and present some technical difficulty with compatibility among different electronic devices, as well as participant errors, who might skip this item or select an unintended option. Thus, this format is not recommended unless there is a pressing need to reduce the visual space or minimize the effects of participant exhaustion due to perceived length. Sliders are a format of visual analogue scales that present a graphical gauge such as a bar or a line in which participants may indicate a position representing their rating (Funke, 2016; De Jong et al., 2015; Reips & Funke, 2008). They are intended to provide a continuous visual representation of a scale, intended for continuous data. They require to be provided context on the value ranges, often at both ends of the slider, although multiple points of the slider may be labelled for further context. Sliders may allow for a small numerical range, and these values can be displayed; this is discouraged, as participants will lean on middle-point or positive responses, and often select multiples of 2 or 5, reducing the effective range and meaningfulness of the 25 scoring. Furthermore, this format often presents a less reliable measure for categorical measurement compared to radio buttons, and evidence suggests that large ranges do not produce truly continuous data (Arnau, Thompson, & Cook, 2001; Kero & Lee, 2015; Liu & Conrad, 2016; Roster et al., 2015). The form in which response formats are presented will affect the response process of participants, as they encourage participants to interact with them in certain ways (Kaptelinin, 2013; Keehner & Moon, 2018). This implicit interactivity is referred to as an affordance (Colman, 2015), which gives context clues on possible actions and how participants can operate think or react to the stimuli. In measurement, item affordances can give implicit or explicit choices, suggest the desired outcome, or a default value, as well as how participants will use the response format (Keehner & Moon, 2018). The impact of item affordances suggests that a difference in design for an electronic measure will give a different user experience and cognitive process, which in return may give a different response score. The cognitive difference in participants based on the response format can be observed with formats with an implicit choice where a default response is already highlighted (such as a starting point in a slider) versus formats with an explicit choice, where participants have to select a response to be registered. Even if participants intend to respond to the same value (e.g., a low score in a depression scale), the manner in which they interacted with the scale will have gone through a different cognitive process (Keehner & Moon, 2018). Furthermore, if a participant is hesitant in a choice, or has low interest or effort in their responses, their responses may lean towards affirmative or medium statements (Cronbach, 1941; Pomplun & Omar, 1997). Similarly, participants are more likely to endorse the more visible options in surveys (Couper, Tourangeau, Conrad, & Crawford, 2004), suggesting that accessibility and readability are impactful factors in online or electronic measurement. 26 As different response formats have diverse intended uses and recommended purposes, it is important to examine how their implementation in analogous scales may affect how participants respond and whether there are differences in the measurement, interpretation and validity of ordinal scores. The goal of this study is to explore how response formats may affect response behaviour and the internal (factor) structure of a scale. A secondary goal is to examine how interaction with different electronic devices affects response behaviour and the internal structure of a scale. Methodology Measure This study used the CES-D as the measure for analysis to illustrate the differences between response formats. The CES-D is a four-point screening measure for depression symptomatology which asks participants to rate the frequency of their symptoms in the past week in four response options: Option 1 = rarely or none of the time (less than 1 day), Option 2 = some or a little of the time (1-2 days), Option 3 = occasionally or a moderate amount of the time (3-4 days), and Option 4 = most of the time (5-7 days). The CES-D was selected as the illustration measure because it is utilized widely in research with supporting evidence of reliability with alpha coefficients between .78 to .90, and internal structure validity evidence for a single factor between alpha coefficients of .85 and .92 (Eaton et al., 2004; Radloff, 1977; Radloff & Lock, 1986; Lewinsohn, Seeley, Roberts, & Allen, 1997; Van Dam & Earleywine, 2011). There is a revised version of the CES-D, the CESD-R, which has been examined for utility and validity evidence (Van Dam & Earleywine, 2011). The response options for the CESD-R change the range of responses from exclusively in the past week to include a fifth option âNearly every day for two weeks,â and evidence the effects of this change in metric was not available from a review of the literature. Furthermore, the change in response options disallows 27 for the score metric between measures to be compared. While both versions of this survey have been used in recent years (e.g., Jahn et al., 2018; Loe, Stillwell, & Gibbons, 2017; Owora & Carabin, 2018), this study only uses the original version, as it has more accumulated evidence of validity and use. The CES-D survey was presented in four different versions. All versions included the same wording for the responses, but the format of these response options was altered in each survey version. Version A utilized horizontal radio buttons, which only allow a single response. Version B utilized horizontal checkboxes, which allows participants to mark more than one response âalthough instructions stated to choose only one. Version C presented responses with a drop-down menu with no default value selected. Version D presented a slider with the four anchor labels on top at equal intervals defaulted to option 1. All survey versions presented responses from the least frequency to most frequent, and while no responses were mandatory, only completed surveys were used in the final analyses. Figures 3.1 through 3.4 show an excerpt from the versions used in this study. Figure 2.1. Radio buttons (Version A) 28 Figure 2.2. Checkboxes (Version B) Figure 2.3. Drop-Down menu (Version C) Figure 2.4. Sliders (Version D) Participants Data for this study was collected through Mechanical Turk, an online survey program. Participants have to be at least 18 years old of age with a registered Mechanical Turk account but are otherwise not restricted. To ensure survey responses were from real participants and not from automatized responses (âbotsâ or software-generated), only respondents with a 29 âMastersâ qualification from Mechanical Turk were be allowed, which is awarded to respondents who are identified as high-performing workers over time and pass continuous statistical monitoring. Participants were asked to provide their Mechanical Turk ID to corroborate against their survey identifier to disallow for automatic responders. Finally, individual entries were screened for incomplete surveys and irregular response patterns (e.g., incongruent or patterned responses, extremely fast completion), and were not be allowed to participate in more than one survey version. Incomplete surveys or those that were missing all demographics were removed. The final sample was a total of 3,034 mTurk workers (M = 759 per version) with an age range 18 to 81 (M = 33.69, Median=30); Table 2.1 shows a breakdown of the demographic characteristics of the participants by response format. The survey asked participants to identify their gender âeither male or female, as well as whether they identify as a gender minority (if their gender identity or sex-expression differs from conventional expectations of masculinity or femininity; trans, transgender, gender-fluid, or an analogous term), or as a sexual minority (self-identified as lesbian, gay, bisexual, two-spirited, or an analogous term). Table 2.1 Breakdown of demographics by response format Format N Age M (SD) Male Female Sexual Minority Gender Minority Both Minorities Checkbox 759 33.7 (11.1) 367 (48.4%) 389 (51.3%) 217 (28.6%) 152 (20.0%) 133 (17.8%) Dropdown Menu 739 33.8 (11.2) 355 (48.0%) 384 (52.0%) 202 (27.3%) 140 (18.9%) 127 (17.4%) Radio Button 793 33.6 (11.3) 369 (46.5%) 417 (52.6%) 234 (29.5%) 174 (21.9%) 163 (20.9%) Slider 743 33.6 (10.8) 344 (46.3%) 397 (53.4%) 202 (27.2%) 160 (21.5%) 143 (19.4%) Overall 3034 33.7 (11.1) 1435 (47.3%) 1587 (52.3%) 855 (28.2%) 626 (20.6%) 566 (18.9%) Similarly, participants were asked which device was used to respond to the survey. Table 2.2 shows a breakdown of the devices broken down by response format. Options were 1. Desk computer (PC/Mac), 2. Laptop or MacBook, 3. Tablet or iPad, 4. Phone or mobile device 30 (Android, iPhone, google phone, etc.), and 5. Other (please describe). Note that not every participant described what device they used to complete the survey. Table 2.2 Sample sizes of device used by response format Format Desktop Laptop Tablet Phone Overall (row) Checkboxes 282 (37.2%) 389 (51.3%) 28 (3.7%) 59 (7.8%) 758 (99.9%) Dropdown Menus 277 (37.5%) 373 (50.5%) 25 (3.4%) 64 (8.7%) 739 (100%) Radio Buttons 305 (38.5%) 396 (49.9%) 28 (3.5%) 60 (7.6%) 789 (99.5%) Sliders 284 (38.2%) 357 (48.0%) 22 (3.0%) 73 (9.8%) 736 (99.0%) Overall (column) 1148 (37.8%) 1515 (49.9%) 103 (3.4%) 256 (8.4%) 3022 (99.6%) Tablets were the least used device in this study, whereas laptops were the most common device. Participants did not indicate unlisted devices. There may be some overlap in what participants considered a laptop and a tablet due to how some devices are designed or advertised, which could reduce the representation of tablets in the survey. Given that the survey was distributed through mTurk, it is also possible that participants were more likely to use a desktop or laptop instead of phones or tablets, as the software is more oriented to survey workers. Surveys distributed via email or advertisement might see an increase in the use of mobile devices and tablets. Analysis This study intended to observe if the choice of response formats affects how the construct is measured and how participants respond to the survey. As a follow-up analysis, the study also examined if the use of a different device changed how participants behaved. First, participant behaviour is described by missing responses, multiple responses (for checkboxes) and implicit vs explicit choice (for sliders). It is important to describe these behaviours because they inform the data analyses as well as how different response formats and devices affect the behaviour of participants; in other words, what affordances do formats and devices present to 31 participants. Analyses were performed using R 3.4.4 with the packages Lavaan 0.6-3, semPlot 1.1.1 and semTools 0.5-1 as described by Hirschfeld and von Brachel (2014). A Multi-group Confirmatory Factor Analysis (MGCFA) with Diagonally Weighted Least Squares (DWLS) was used to examine the internal structure of the test and whether different response formats affect this structure. The CESD has shown to have a single-factor structure (Eaton et al., 2004; Zumbo, Gelin & Hubley, 2002), and it was expected that different response formats should not affect this if they have a negligible effect on participant responses. Responses were treated as categorical and missing responses were removed listwise. Although listwise deletion can potentially reduce power by reducing sample size, and introduce bias if missing is not at random (Graham, 2009; Kang, 2013; Little, Rubin, & Wiley, 2019); to minimize the impact of deletion data was examined to ensure responses did not show missing not at random patterns and intended for a sufficient sample size. While imputation methods such as Full-information Maximum Likelihood or Expectation Maximization can be used for categorical data, it does not provide a significant advantage over list-wise deletion in a sufficiently large sample size with a small proportion of random missing responses, as imputation may bias standard error or parameter estimates (Allison, 2001; Pepinsky, 2018). Similarly, Multiple Imputation was not used as a data imputation method as it introduces random variation and can be implemented in multiple ways, reducing replicability and thus the generalizability of the results (Allison, 2003; Dong & Peng, 2013; van Buuren, Boshuizen, & Knook, 1999). The use of an MGCFA also allows the examination of measurement invariance (MI) across all four response options. Similarly, DWLS is intended for categorical data and is less biased against non-normality or symmetrical distribution than alternative estimates such as weighted least squares mean or robust maximum likelihood (DiStefano & Morgan, 2014; Li, 2016; Mindrila, 2010). The study examined invariance from least to most strict as configural, weak and strong invariance, where configural invariance freely estimates factor loadings and 32 intercepts, weak invariance constrains the factor loadings, and strong invariance constrains both factor loadings and intercepts across groups. To confirm MI, this study utilized RMSEA, CFI, and âCFI as fit indices. âCFI for each level is estimated by subtracting the CFI of a model to the closest stricter model (e.g., weak and strong invariance models); thus, configural invariance does not have a âCFI index. Acceptable fit coefficients for supporting invariance at each level for RMSEA are suggested at <.08 as acceptable and <.06 as a good fit, for CFI and TLI >.9 as good fit, for âCFI â¤-0.01, and SRMR approximating 0, with <.08 for good fit (Shi & Maydeu-Olivares, 2020; Wu, Li, & Zumbo, 2007; de Wulf, 1999). The Ï2 index is overly sensitive for sample size and will reject most models (Wu, Li, & Zumbo, 2007); thus, it was only included as a reference, where a value of Ï2 <.05 would suggest non-invariance. Since the test is expected to be a single factor with a simple model structure, there is no concern for the effects of model complexity. These analyses allow for direct statistical contrasts between the different experimental conditions. Results & discussion Missing or multiple responses. Participants were able to mark multiple responses in the checkbox format; as these responses are invalid, they were coded as missing but marked differently than other missing data. Sliders had a default value of 1 - Rarely or none of the time (less than 1 day), and thus will not show any missing data even if participants skipped the item. Table 2.3 shows the proportion of missing responses for every possible response. Note that percentages are off of the total number of possible responses (n=60,680), as there are 20 items per survey. 33 Table 2.3 Proportion of missing and multiple responses for response format per entry Format n Missing (%) Multiple response (%) Total (%) Checkboxes 15180 89 (0.59%) 340 (2.23%) 429 (2.83%) Dropdown Menus 14780 21 (0.14%) NA 21 (0.14%) Radio Buttons 15860 66 (0.42%) NA 66 (0.42%) Sliders 14860 NA NA NA Overall 60680 176 (0.29%) 340 (0.56%) 516 (0.85%) For the checkboxes, of the total 759 participants, less than a third (26.09%) selected multiple responses for at least one item. On average, participants marked 0.45 items with multiple responses (SD = 1.03, range 0-8). Of the people who marked at least one item with multiple responses, on average, they responded to 1.72 items out of 20 like so (SD = 1.37). Out of 340 multiple responses, only 8 (2.35%) marked three multiple choices, and the rest marked only two choices. Of those who marked two choices, 243 were adjacent (Out of 332, 73.19%). Each item had, on average, 17 multiple responses (SD = 5.44, range 8-31), where item 8, âI feel hopeful about the future,â had the highest rate of multiple responses. Although sliders are unable to differentiate if a participant skipped an item or decided the default value corresponded to their perception, the survey tool indicated if participants interacted with the slider before selecting any value, which allows making a description between implicit and explicit choices in this format. Of the total of 743 slider participants, 185 (24.90%) left at least one of the items at their default value (no interaction). Out of 4006 responses of â1 - Rarely or none of the timeâ, 876 were by non-interaction (21.87%). Of the whole sample, participants on average had 1.18 sliders (out of 20) with no interaction (SD = 2.75), ranging from 0 to 17 responses as such. Participants who did not interact with at least one slider had a larger proportion, with an average of 4.73 (SD = 3.68) sliders with no interaction. Table 2.4 shows the proportion of missing responses by the device. Note that percentages are off of the total number of possible responses (n), as there are 20 items per survey. Missing responses represented less than 1% of the total responses, the majority of 34 participants responded to all items (no missing values): for desktops, 10.7% had at least one or more missing or multiple responses, for laptops 9.8% had one or more, for tablets 6.8%, and phones or mobile devices 9.4%. Table 2.4 Proportion of missing and multiple responses for device Format n Missing (%) Multiple response (%) Total (%) Desktop 22960 77 (0.34%) 135 (0.59%) 212 (0.92%) Laptop 30300 78 (0.26%) 172 (0.57%) 250 (0.83%) Tablet 2060 2 (0.10%) 5 (0.24%) 7 (0.34%) Phone 5120 17 (0.33%) 27 (0.53%) 44 (0.86%) Overall 60440 174 (0.29%) 339 (0.56%) 513 (0.85%) These response rates suggest that tablets present the least information loss of all devices; however, the sample size of tablets was much smaller than other groups and may not be representative of all users. Other devices showed a similar proportion of missing responses, which suggests that most devices do not affect the response rate or engagement in electronic surveys. Factor analysis A principal component analysis (PCA) with polychoric correlations for ordinal data was used to support the assumption of unidimensionality. The structure was considered to be essentially unidimensional if the ratio between the first to second eigenvalues is greater than 4.0 (Slocum-Gori & Zumbo, 2011). Essential unidimensionality was supported with Eigenvalues of 8.88 and a trailing factor of 1.66. Additionally, PCA for each response format separately obtained similar estimates. As seen in Table 2.5, the eigenvalues for the first factor ranged from 7.38 to 10.04, with a trailing factor of 1.16 to 1.91 resulting in ratios of 5.05 to 8.39. The ordinal alpha reliability ranged from .92 to .95. 35 Because of the ordinal item response data, the reliability estimate for the scale was an ordinal alpha (Zumbo, Gadermann, & Zeisser, 2007; Zumbo & Kroc, 2019). The ordinal alpha for the full sample was 0.94. Table 2.5 Eigenvalues, ratios and ordinal alpha for response format and device Group 1st Eigenvalue 2nd Eigenvalue Ratio Alpha Checkbox 9.63 1.79 5.38 .94 Dropdown menu 7.38 1.24 5.95 .92 Radio button 9.35 1.83 5.11 .94 Slider 9.13 1.58 5.78 .94 Desktop 8.69 1.72 5.05 .93 Laptop 8.82 1.69 5.22 .93 Phone 10.04 1.91 5.26 .95 Tablet 9.73 1.16 8.39 .95 The factor structure of the scale was analyzed using Multi-group confirmatory factor analysis (MG CFA). This method allows comparison for measurement invariance between groups, in this case, the four response formats. If all measurement formats are equal, a scale should result in strong (equal loadings and intercepts) or strict measurement invariance (equal loadings, intercepts and residuals) levels. Table 2.6 shows the contrasts for the response format using MGCFA. There are significant differences between models, which fail to support invariance beyond the configural level. Pairwise contrasts indicated statistical differences between all response formats. For most contrasts, these differences were at p < .001 except comparing radio buttons to sliders where p < .01. This finding suggests that the use of different response formats creates enough differences between measures that they cannot be considered equivalent. 36 Table 2.6 MG CFA results for response format df CFI ÎCFI X2 ÎX2 Îdf p Configural 680 .796 -- 5945.5 -- -- -- Weak 737 .787 .009 6248.6 303.1 57 <.001 Strong 794 .779 .008 6509.2 260.6 57 <.001 Strict 854 .770 .009 6794.2 285.0 60 <.001 As the measures cannot be compared equally, using all scores together to examine factor structure overall would be inappropriate; instead, the four measures should be examined separately. Table 2.7 shows the fit indices for a single factor model of each used response format. An overall model that does not differentiate among the response formats was also estimated. This model was reported for comparative purposes against the common estimation of fit treating all response formats as equal. Similarly, the Ï2 index is biased for sample size and was included as a reference. The fit statistics indicate a moderate fit to a single-factor structure. The CFI and TLI fit statistics support moderate fit, while RMSEA and SRMR suggest a poor fit. This finding of a moderate fit may be due to the precense of skewed item response distributions by the large sample size, which affects the estimations of model estimation and fit (Bien & Tibshirani, 2011; Lai & Green, 2016; Shi & Maydeu-Olivares, 2020). Furthermore, RMSEA and SRMS are dependent of the non-centrality parameters, and differences in variance and distribution for the sample will reduce fit. Moreover, fit may be reduced by the trailing second factor âas this scale was established to have essential unidimensionality. With these considerations, it is possible to continue with the cautious use of a single factor. 37 Table 2.7 Fit statistics of a one-factor model for each response format X2 df p CFI TLI RMSEA SRMR Checkboxes 2426.78 170 <.001 .95 .95 .16 .12 Dropdown menu 1200.45 170 <.001 .96 .95 .09 .08 Radio buttons 4167.92 170 <.001 .94 .93 .18 .13 Sliders 2624.44 170 <.001 .96 .95 .14 .10 Overall 9780.42 170 <.001 .95 .94 .14 .11 The contrasts between devices using MGCFA presented significant differences between models, suggesting invariance cannot be supported between these platforms. The results can be found in Table 2.8. Pairwise contrasts indicated significant differences at p < .001 for all models except the analyses between tablets and phones; this latter model supported strict invariance (ÎX2 = 75.78, p = 0.08). Non-invariance between devices suggests that measurement performed on different devices will impact how participants will respond to the measures, and thus cannot be considered equivalent regardless of presentation. Conversely, as tablets and phones showed evidence of measurement invariance at a strict level, it is possible to consider these devices to be equivalent. Table 2.8 MG CFA results for electronic devices df CFI ÎCFI X2 ÎX2 Îdf p Configural 680 .796 -- 5935.2 -- -- -- Weak 737 .794 .002 6046.7 111.4 57 <.001 Strong 794 .793 .001 6123.1 76.4 57 0.044 Strict 854 .791 .002 6228.9 105.8 60 <.001 Table 2.9 shows the fit indices for a single factor model of each device as well as an overall model that does not differentiate among the devices. This model was reported for comparative purposes, similar to the contrasts between response formats. Likewise, the Ï2 index is biased for sample size and was included as a reference. The model fit between different devices was similar to each other, and to the different response formats, CFI and TLI for every device model were within moderate fit parameters, whereas RMSEA and SRMR 38 suggested poor fit. As discussed above, the characteristics of this measure and sample are expected to hinder fit, so the analyses proceeded with a cautious interpretation of a single factor. Table 2.9 Fit statistics of a one-factor model for each device X2 df p CFI TLI RMSEA SRMR Desktop 4069.67 170 <.001 .94 .93 .15 .11 Laptop 5108.63 170 <.001 .94 .94 .15 .11 Tablet 518.70 170 <.001 .97 .96 .15 .13 Phone 622.07 170 <.001 .98 .97 .11 .09 Overall 9780.42 170 <.001 .95 .94 .14 .11 The expected single factor structure for the CESD holds true for the scale regardless of response format used, either when formats are examined individually, or all together. However, there are major differences in how participants engage with the different response formats, indicating a difference in the response process; this, in turn, may affect response process validity evidence. The estimation of model fit for response formats and used devices supported the scale as an essentially unidimensional model. However, the lack of support for measurement invariance across most experimental conditions suggests that the manner in which we present the measure to participants significantly changes the factor structure of the model. Conclusions Different response formats are intended to be used in different manners; this study shows how these differences can create a large impact by essentially modifying how the research question is asked and affected the obtained scores. Different response formats may produce different response rates, participant engagement, and response tendencies, which threatens the validity evidence of scores. Thus, researchers should engage actively in their 39 scale creation and distribution, considering which response format may obtain satisfactory results for their intended study. Furthermore, different response formats change how participants will respond to surveys based on the item affordances given by their presentation. Firstly, participants will be more likely to make an explicit choice when responding to measures when they are not provided with a default or expected value. Response formats like drop-down menus and sliders often provide a default value and a small increase in needed action, encouraging an implicit choice that minimizes mental and physical engagement (i.e., leaving the default choice as is). Dropdown menus might be attractive to reduce visual space in response formats; however, they do not immediately display all choices and can cause errors due to differences in visual space based on the display size of their device or location on a screen. A longer scale can also be tedious to scroll through when there are many response options, which can cause participant exhaustion and measurement error. Secondly, participants will attempt to mark more than one option where format allows it (such as checkboxes), even if the instructions direct the participant to choose only one. While this might allow participants to show if they are feeling hesitant or are not responding to the survey carefully, this means surveys might become unusable or questionable. A stricter response format that allows only one response will discourage recording errors and encourages participants to choose a single value when they are undecided. Moreover, it is up to the researcher to provide an easy and intuitive measure format that requires minimal instruction to be operated to ensure that participants follow these methods as accurately as possible. It should be noted that participants were part of an online worker community; that is, they frequently respond to online surveys as a paid activity. This frequent responding to online surveys presents with the benefit of users who are comfortable and familiar with various online formats, reducing the possible disparity of computer literacy or accessibility. It also allows to 40 focus on the habits of an online environment for those who frequently interact with the formats examined in this study. However, this also presents the possibility that some participants may not have been responding as attentively as possible. Although the data was cleaned for automatized responses or those generated with electronic software for automatic completion, this does not rule out possible inattentiveness. These findings would suggest that radio buttons are more reliable and easy to use as a response format for measuring categorical responses where only one response is expected. However, this does not mean other formats are without use; Checkboxes allow for multiple responses to be recorded, and sliders can be used for continuous data where specific scale points do not have to be labelled. Researchers must first consider what type of data they are expecting before selecting an appropriate response format that matches their intended use. The analyses of measurement invariance did not support invariance amongst different response formats or different devices beyond configural level (except for tablets and phones compared to each other). This finding suggests that these mediums are not interchangeable or equal at their core. In practice, this means that researchers who develop and implement surveys electronically must make decisions based on evidence for which response format is adequate for their measure to ensure the validity of scores. The differences between response formats also imply that an established measure cannot be adapted to different response formats, as this fundamentally changes the measure itself. If a measure is adapted to use a different response format, its scale scores must go through a validation process, as one would expect from a scale that had been edited, reworded, translated, or whose number of response options had changed. It should be noted that these results may have been affected by the reduced model fit of the scale, and stronger invariance might be found with a scale with better fit estimates. The differences between devices can be a concern for measurement. Although the magnitude of difference between formats might have been influenced by the differences in 41 sample sizes, evidence from this study suggests that the choice of a user device may affect the internal structure of the measure. Thus, researchers may endeavour to utilize the same device type across subjects to avoid confounding variables, if possible. However, when doing measurement remotely âdistributing online surveys, using surveying platforms such as mTurk or SurveyMonkey, or other online survey software-, it may not be possible to control this variable. If this is the case, researchers may have to acknowledge the possible differences in data and examine the validity of their scores. As the devices were not randomly assigned, one area of concern is that whether the differences between electronic devices are inherent to the devices themselves or if these are indicators of another characteristic âsuch as socioeconomic status, the location where the survey was responded, or computer literacy, and so these results need to be explored further on a controlled environment. The analysis of an overall fit model allowed to examine differences of fit if the response formats and devices were to be treated as equivalent. While the contrasts between models did not support invariance, and thus, equivalence between formats, the overall model showed similar estimates of fit to those of the individual models. This finding may provide some insight on why there is limited literature on contrasts between response formats and devices, where a philosophy of âgood enoughâ would overlook these contrasts. However, it begs reminding that appropriate measurement and validity rely on obtaining scores as accurately as possible. Strengths, Limitations, and Future Studies This study provides evidence on the implications and differences of response formats; this contributes to filling a gap in the literature that lacks evidence on the structural differences response formats for electronic measurement. Likewise, it contributes to similar evidence on the use of electronic devices. Furthermore, this study encourages critical and evidence-based decisions in creating and implementing electronic measures. 42 One more strength of this paper is the random assignment of response formats, which allowed an experimental study. Thus, the inferences drawn from the scores allow a causal interpretation to complement the descriptive portion of the missing response behaviour. Furthermore, the study gathered a robust sample size for every experimental condition, which ensures that there is enough statistical power to support the findings. The study was limited in the non-random nature of the electronic device. Participants were recruited online through a survey platform (mTurk) and were not assigned a device. As such, it is possible that differences between devices used were caused by an unmeasured variable, or were biased in their responses due to an unknown confounding factor. Thus, interpretations of the differences between devices are quasi-experimental at best and should be used cautiously. Another limitation of the study is that it may have been influenced by the choice of the scale. Although previous studies have examined the validity of its scores and the function of internal structure, it is possible that these findings may not extend to all measures equally; future studies may be needed to examine the generalizability of these findings. Similarly, fit estimates may be reduced due to the choice of a depression scale where item response data would be expected to have a skewed distribution, and the trailing second factor (essential unidimensionality) found by the factor analysis. Skew for items can cause spare data patterns of the covariance matrix, which, in turn, potentially impact the estimation of factor analysis, model parameters and fit (Bien & Tibshirani, 2011). Interpretation of the scores had to be mindful of this limitation, encouraging further studies to confirm these findings. One more limitation of the study is the possibility of random responses to the survey. The data was checked to remove cases where participants provided contradictory or incongruent responses and repeating patterns that suggest random responses. The survey also used an open-field item that asked for their user ID, which allowed them to identify fake surveys 43 based on irrelevant responses. Finally, as the survey contains reverse-scored items, it was possible to identify participants that responded equally to all items. However, it is possible that some false or inattentive respondents were not removed, which could affect the obtained scores. Similarly, reversed-coded items may cause a decrease in reliability and fit indices for the factor structure (Vigil-Colet, Navarro-Gonzalez, & Morales-Vives, 2020). A further limitation is the removal of missing cases to the analyses, as the removal of any amount of data may have a minor biasing effect on the results. However, the proportion of cases removed was very small, and the remainder of cases were sufficient for statistical power, and the responses were examined for the possibility of missing not at random to minimize the impact of these removed responses. Although the study explored possible differences between devices, it is limited in that this was a secondary goal, and thus the specificity of data collection was limited. Further studies may be necessary to focus on the differences between devices with more precision, including differences in models, display sizes, operating systems, and interactivity of their screens. Furthermore, future studies may need to randomly assign devices to participants in a controlled environment to determine whether the differences observed in this study are caused by the devices or another characteristic linked to the different choice of device that participants used to respond to surveys. These characteristics may include the level of computer literacy, access to electronic devices, the location where participants respond to the surveys, and the method used to interact with the survey (e.g., touch screen vs mouse). In this paper, the response format of sliders had a default value of â1â for its minimal value. While this is a common default setting for electronic survey software, it should be acknowledged that this might have some anchoring bias towards participants responses, and is an inherent limitation of this particular response format. If possible, future studies may consider implementing a slider that does not show an initial arrow that participants can drag until they 44 have interacted with the item first, if the available software permits such an option, and compare these two different versions of sliders to understand its effects on participants. Future studies might also examine the validity evidence of the scores based on response processes for different response formats and devices, which may describe the meaningfulness of participant choices and differences in their response behaviour. Other studies might replicate these experimental conditions on different measures, or generate a controlled environment to examine differences between devices, as this study did not randomly assign device. In conclusion, drawing a parallel between factor analysis and item response theory, the measurement of the same person or multiple people with an equal level of a latent variable, in this case depression, may obtain different scores depending on the format used, and thus different inferences might be drawn from these scores. This, in turn affects the measurement validity. In other words, measuring the same latent variable for different people at the same âlevelâ would result in the same expected item scores if the scale was invariant. As this is not the case, equal measurement across response formats is not guaranteed. Finally, researchers cannot forget that validity evidence of a scale only applies to the specific characteristics in which this evidence was found; this includes the demographics of the sample as much as the characteristics of the test, and any changes âincluding the use of different response formatsâ will require retesting to support validity evidence. Similarly, changes in the response format and device used for testing will affect item affordances. As seen in this study, different devices and response formats will suggest different manners of interaction with scales. However, it may also be possible that these choices also reflect the agency and choices by the researchers, providing implicit suggestions of how they intend participants to respond or how the construct is intended to be measured. 45 Chapter III - Item Scoring and DIF for Non-binary Gender and Sexual Orientation Introduction Scoring of item responses is one of the most basic processes in measurement, as part of quantifying an observed behaviour or a construct (Campbell, 1952; Stevens, 1946). Adequate item scoring is essential to assure the validity of inferences drawn from obtained scores (AERA et al., 2014), but it is not enough to obtain a valid score. It is important to understand how to interpret and use these scores for an intended purpose, and how any change to these scores will affect inferences and, in turn, validity (Chan, 2014; Messick, 1995a). Furthermore, re-scoring an item response scale may change the meaning of what is being measured and essentially alters the analyzed construct. A study by Gelin and Zumbo (2003) examined how different item scoring methods affected inferences based on the total scores. They demonstrated this effect with the original version of the Center for Epidemiologic Studies Depression (CES-D) scale while investigating gender differential item functioning (DIF) analyses, wherein participants identified as either male or female. The CES-D is a useful scale for this purpose because, as will be described below, the items can be (re-)scored in three different ways reflecting frequency, persistence, and presence of depressive symptoms; the three scoring methods reflect different use and interpretation of the scores. Gelin and Zumboâs findings were the first published findings that highlighted how different scoring methods result in different DIF results and affected the overall conclusions obtained from them. Gelin and Zumbo found gender DIF for two items in the frequency and presence scoring methods, but not in the persistence scoring method, whereas only the latter found DIF in two other items. As Gelin and Zumbo state, âthe scoring method has an effect on DIF; thus, DIF is a property of the item, scoring method, and purpose of the instrumentâ (p. 65). As DIF depends on these three characteristics, if any of these is altered, DIF is affected; collapsing categories of responses or scores will, in turn, affect the dependable 46 variable, distribution, variance and R2. Thus, collapsed scores are not treated as equal to the original scores. With a focus on extending Gelin and Zumboâs (2003) findings, the present paper has two purposes. The first purpose was to investigate if the conclusions from a DIF analysis change if one categorized survey respondents or participants using non-traditional conceptualizations of gender and sexual identity in DIF analyses rather than gender in the traditional binary (M/F). To our knowledge, this is the first DIF study to go beyond the traditional gender binary. The second purpose was to investigate if Gelin and Zumboâs findings generalize to the revised version of the CES-D, CESD-R, which has some items with similar question stems but different response formats, as well as new items. CES-D & CESD-R The Center for Epidemiologic Studies Depression (CES-D) and the revised (CESD-R) scale are self-report rating scales comprised of 20 items, developed as a diagnostic aid for depressive symptomatology (Eaton et al., 2004; Radloff, 1977). It measures the frequency of recent depressive symptoms and obtains a total score that can be used to identify individuals at risk for clinical depression. The CES-D and the CESD-R can be scored in different ways (see Figure 3.1). One method is the original scoring intended to identify the presence and severity of depressive symptomatology (Radloff & Lock, 1986). A presence method can be used when researchers are interested in identifying the presence or absence of depressive symptomatology. A persistence scoring method identifies individuals with a high or low likelihood of being depressed. Finally, the CESD-R presents an alternative scoring method to allow scores to be in the same range as the CES-D (Eaton et al., 2004). These different scoring methods may change what is being measured; the original score mainly measures frequency/intensity of symptoms, the presence scoring only quantifies a binary of whether 47 symptoms are present at all or not, and persistence scoring measures symptoms that are above a set threshold. Figure 3.1 Scoring methods for the CES-D and CESD-R CES-D CESD-R RESPONSE OPTIONS 0-Rarely or none of the time (less than 1 day) 0-Not at all or less than 1 day last week 1-Some or a little of the time (1-2 days) 1-One or two days last week 2-Occasionally or a moderate amount of time (3-4 days) 2-Three to four days last week 3-Five to seven days last week 3-All of the time (5-7 days) 4-Nearly every day for two weeks ORIGINAL scoring method Option score is equal to their response option value (0-3 and 0-4 respectively). Range: 0 - 60 Range: 0 - 80 PRESENCE scoring method The respondentâs report of having experienced the symptom at least some of the time. Option 0 > assigned 0 (indicating no depression) Option 0 > assigned 0 (indicating no depression) Option 1 > assigned 1 (indicating depression) Option 1 > assigned 1 (indicating depression) Option 2 Option 2 Option 3 Option 3 Option 4 Range: 0 - 20 Range: 0 - 20 PERSISTENCE scoring method The respondent reports experiencing the symptom for a moderate to large period of time. Option 0 > assigned 0 Option 0 > assigned 0 Option 1 Option 1 Option 2 > assigned 1 Option 2 > assigned 1 Option 3 Option 3 > assigned 2 Option 4 Range: 0 - 20 0 - 40 ALTERNATIVE scoring method Items are scored to reflect the original score range of the CES-D (0 - 60). Not Applicable Option 0 > assigned 0 Option 1 > assigned 1 Option 2 > assigned 2 Option 3 > assigned 3 Option 4 48 Gender, sexual identity and gender identity in survey research Whereas the study of gender as a binary male/female is attractive to simplify analyses, it is not a sufficient descriptor of a personâs identity (Garvey, 2017; Palan, 2001; Saewyc, 2017; Treharne & Beres, 2016). Although the inclusion of identities of gender and sexual orientation have somewhat increased in survey-based research (Garvey, 2017; Saewyc, 2017), measurement and psychometric research tend to only consider a gender binary (male/female) as a demographic item. However, the lack of consideration of these identities can create gaps in the measurement and interpretation of scores based on the characteristics of the population, and the fundamental differences between groups. Individuals that identify as a sexual minority (e.g., lesbian, gay or bisexual) or gender minority (e.g., trans, non-binary, genderfluid, two-spirited) have been documented to be at a higher risk of depression, self-harm and suicidality (King et al., 2008; Birnholz & Young, 2012). Because of these findings appearing systematically in the literature, it is important to consider the adequacy of the measuring instruments used to assess depression in a context of gender and sexual diversity beyond the male/female binary. For the CESD, previous research has examined the effects of gender differences on the original scale (Gelin & Zumbo, 2003), as well as the effects of sexual orientation among women (Brinholz & Young, 2012). This previous research has shown some differential item functioning for some items separately; however, there have been no contrasts for non-binary gender identities, sexual identities and the gender binary, or how these groups present similarities or differences for the scores on either the CES-D or the CESD-R. This gap in the literature can be extended beyond the context of the CESD-R, as it affects how a limited range of options for identity may not accurately reflect issues in measurement, such as systematic group differences. Investigating the roles gender and sexual identity, and comparing the different response formats of the CES-D and CESD-R, allows us to extend the Gelin and Zumbo line of 49 research to investigate if, and how, as they note, DIF is a property of the item, scoring method, purpose of the instrument, and the how one conceptualizes gender and sexual identity. This elaborated view captures the inherent complexity and multiple determinants of item responding, and the psychometric properties of measures (Zumbo, Liu, Wu, Shear, Astivia, & Ark, 2015); these contrasts between scoring methods, measures, and subgroups illustrate how these changes are not interchangeable or equivalent, and affect the psychometric properties in analyses. Methodology Participants Subjects in this study were recruited on the online survey program Mechanical Turk during the summer of 2017. Participants had to be at least 18 years old of age with a registered Mechanical Turk account. To ensure responses were real participants, only respondents with a âMastersâ qualification from Mechanical Turk were allowed (awarded to respondents who are identified as high-performing workers over time and pass continuous statistical monitoring). Participants were also asked to provide their Mechanical Turk ID, which was then corroborated against their survey identifier to disallow for automatic responders. Finally, individual entries were screened for irregular response patterns. The sample size per survey condition was intended to be between the sample size in the studies by Gelin and Zumbo (2003) and Gesicki (2015). The final sample consisted of 1,579 participants, of which 870 (55.1%) identified as male, 683 (43.3%) identified as female, and 26 (1.6%) participants did not respond to this question. Participantsâ age ranged from 18 to 74 years old (M = 33.46, SD = 10.74). Participants were also asked if they identified as a gender minority (if their gender identity or sex-expression differs from conventional expectations of masculinity or femininity; trans, 50 transgender, gender-fluid, or an analogous term); 203 (12.93%) participants identified as such and 27 (1.7%) chose not to respond. Participants were also asked if they identified as a sexual minority (self-identified as lesbian, gay, bisexual, two-spirited, or an analogous term); 288 (18.2%) identified and 27 (1.7%) chose not to respond. One hundred and sixty participants (10.6%) identified as both sexual and gender minority. Table 3.1 presents a breakdown of these demographics on the sample based on which version of the questionnaire they completed. Table 3.1 Demographics N Age M (SD) Male % Female % Gender minority % Sexual minority % Both minorities % CES-D 799 33.3 (10.4) 439 54.9% 346 43.3% 101 12.6% 145 18.1% 81 10.1% CESD-R 780 33.6 (11.0) 431 55.3% 337 43.2% 102 13.1% 143 18.3% 79 10.1% Total 1,579 33.5 (10.7) 870 55.1% 683 43.3% 203 12.9% 288 18.2% 160 10.1% Measure This study used the CES-D and CESD-R scales. Both scales ask respondents to rate the frequency/duration in which they have experienced symptoms of depression (e.g., Item 11. âMy sleep was restlessâ) during the previous week (or two, in the case of the CESD-R). The CES-D presents four response options: Option 1 = rarely or none of the time (less than 1 day), Option 2 = some or a little of the time (1-2 days), Option 3 = occasionally or a moderate amount of the time (3-4 days), and Option 4 = most of the time (5-7 days); the CESD-R presents five response options: Option 1 = Not at all or less than 1 day last week, Option 2 = One or two days last week, Option 3 = Three to four days last week, Option 4 = Five to seven days last week, and Option 5 = Nearly every day for two weeks. Analysis Test dimensionality. To confirm the initial assumptions of dimensionality, this was investigated for each scoring method. For the CES-D dimensionality was investigated for the 51 original, persistence, and presence scoring methods separately, and likewise, the CESD-R dimensionality was investigated for original, persistence, presence and alternative scoring methods. This study approached the question of dimensionality from an exploratory factor analysis point of view. As such, PCA-based methods with polychoric correlations for ordinal data were used as pointers to the number of factors to provide support for internal structure validity evidence of the scores (AERA et al., 2014). Support for the scale dimensionality is also necessary in order to support the analysis of reliability and differential item functioning (DIF) for a single total scale score (Slaney & Maraun, 2008). Parallel analysis, scree plots, and eigenvalues > 1 of the polychoric correlation matrix were used as scale dimensionality criteria; if these criteria pointed to more than one factor, we examined the ratio of the first to second eigenvalues to assess essential unidimensionality. If the ratio of the first to second eigenvalues is greater than 4.0, there is evidence for essential unidimensionality (Slocum-Gori & Zumbo, 2011). If the evidence does not support essential unidimensionality, then multidimensionality was considered, and parallel analysis was used to determine the number of factors. Parallel analyses were examined in R 3.4.4 using the package psych 1.8.12. Essential unidimensionality with a single dominant factor is the sufficient condition for the remaining analyses. The essential unidimensionality is also foundational validity evidence because internal structure using factor analysis not only determines if the dimensionality of the measure is in line with what is expected theoretically or empirically but also informs how a measure should be scored to appropriately capture the variability in the participantsâ responses to scale items (Flora & Flake, 2017; Hubley & Zumbo, 2013; Slocum-Gori & Zumbo, 2011; Slocum-Gori, Zumbo, Michalos, & Diener, 2009; Zumbo, Gelin, & Hubley, 2002; Zumbo, 2007b, 2009). Ordinal alpha. The reliability of the scale was examined using an ordinal reliability coefficient in R 3.4.4 using the package psych 1.8.12 as described by Gadermann, Guhn and 52 Zumbo (2012), as this is a more accurate estimate for ordinal item response data (Zumbo, Gadermann, & Zeisser, 2007; Zumbo & Kroc, 2019). The ordinal alpha for the full sample was 0.94. Ordinal Logistic Regression Differential Item Functioning. This study examined DIF between gender groups, gender minority groups, and sexual minority groups. In order to examine these groups, this study used an Ordinal Logistic Regression (OLR) Differential Item Functioning method (Swaminathan & Rogers, 1990, Zumbo, 1999) using a 2 degree of freedom test for each configuration of item across group across scoring method (e.g., Item 17 crying x Sexual non-minority x Persistence Scoring). OLR DIF was analyzed in SPSS 23.0. To obtain the estimates for DIF, three models were used. Model 1 uses the total score as a predictor. Model 2 uses the total score and group membership (e.g., male, female). Model 3 uses the total score, group, and the interaction between the two (ability*group). Contrasts between model 1 and 2 examine uniform DIF, and between 1 and 3 for non-uniform DIF. As there is a non-zero interaction parameter, this indicates non-uniform DIF (Swaminathan & Rogers, 1990); thus, the results focus on the latter. The effect size was quantified using Nagelkerke adjustment for pseudo R2. In order to identify an item as displaying DIF, two criteria were established, following recommendations from Gelin and Zumbo (2003) and Gesicki (2015): the chi-squared test of DIF and the corresponding measure of effect size between models 1 and 3. This chi-squared test of DIF compares the p values between model 1 and 3, and their âR2. For the two degrees of freedom chi-square test (which tests group and interaction effects together), p values must be equal or smaller than .01 in order to be considered significant. For a significant effect size differential, values of âR2 must be equal or larger than .00681 for moderate DIF, and equal or above .01514 for large DIF, using the criteria proposed by Gesicki (2015). These effect size criteria were the result of comparing the development of cutoffs in the literature for logistic regression (e.g., cut-offs suggested by Jodoin & Gierl, 2001) and the 53 conversion of the Mantel-Haenszel technique (Mantel & Haenszel, 1959; Holland & Thayer, 1988) cut-offs scaled as ÎR2. Results & discussion In order to facilitate the presentation of the results, these have been split into these categories: overall results, DIF grouped by gender binary (male or female), DIF grouped by sexual identity (minority or non-minority), DIF grouped by gender identity (minority or non-minority), comparisons in DIF between gender and sexual identities, and contrasts between the CES-D and the CESD-R for items in common. 1. Overall results a. CES-D For the CES-D, participants who reported being female, a gender minority, or a sexual non-minority scored higher than their counterparts. Statistically significant differences were found between sexual identity and gender identity across all scoring methods (see Table 3.2); no differences were found between groups of participants who identified as male or female (M/F). Table 3.2 shows the CES-D scores for the contrast groups and scoring methods. For each of the original and two scoring methods, the scale scores showed evidence for essential unidimensionality (a single factor with a minor trailing second factor) on a ratio of eigenvalues ranging from 5.67 to 8.12 times larger, as a residual second factor was within acceptable parameters but did not contribute significantly to the test interpretation (Slocum-Gori et al., 2009). Specifically, eigenvalue ratios between first and second factors were 6.37 for the original scoring, 8.12 for the presence scoring and 5.67 for the persistence scoring. The ordinal reliability estimate for the scale was .95. 54 Table 3.2 Mean total scores for the CES-D for each scoring method in each group Mean SD Mean SD t p α Gender Male Female Original 19.25 11.05 20.24 12.78 -1.17 .24 .94 Presence 12.22 5.54 12.22 5.80 -0.02 .98 .95 Persistence 7.03 6.49 8.02 7.98 -1.90 .06 .94 Sexual identity Minority Non-Minority Original* 26.05 10.88 18.25 11.59 7.70 <.01 .94 Presence* 15.34 4.47 11.51 5.66 7.62 <.01 .95 Persistence* 10.71 7.58 6.73 6.90 5.79 <.01 .94 Gender identity Minority Non-Minority Original* 27.34 10.19 18.54 11.65 7.94 <.01 .94 Presence* 15.83 3.79 11.68 5.69 7.10 <.01 .95 Persistence* 11.50 7.60 6.86 6.94 5.79 <.01 .94 *Significantly different at p < .01 b. CESD-R For the CESD-R, overall, males, sexual minorities, and gender minorities scored higher than their counterparts. Statistically significant differences were found between gender minority and sexual minority for all scoring methods (Table 3.3), but not for any M/F contrasts. Table 3.3 shows the CESD-R mean scores and statistical values for the contrast groups and scoring methods. For each of the original and three scoring methods, the scale scores showed strong evidence for a single factor on the PCA, using one factor with eigenvalues >1, and corresponding scree plot. The ordinal reliability estimate for the scale showed an alpha of .97, suggesting evidence of strong reliability. Eigenvalue ratios were 16.61 for the original score, 13.95 for the presence scoring, 21.46 for the persistence scoring, and 17.00 for the alternative. 55 Table 3.3 Mean total scores for the CESD-R for each scoring method in each group Mean SD Mean SD t p α Gender Male Female Original 21.85 17.31 21.64 17.37 0.17 .87 .97 Presence 10.57 6.06 9.74 5.56 1.95 .05 .97 Persistence 5.44 5.42 4.87 4.89 1.53 .13 .97 Alternative 20.89 15.94 20.40 15.41 0.44 .66 .97 Sexual identity Minority Non-Minority Original* 35.04 18.68 18.74 15.49 10.90 <.01 .97 Presence* 13.14 5.33 9.54 5.77 6.82 <.01 .97 Persistence* 8.56 5.28 4.43 4.86 8.56 <.01 .97 Alternative* 32.68 16.02 17.95 14.29 10.11 <.01 .97 Gender identity Minority Non-Minority Original* 36.25 18.73 19.54 16.00 8.54 <.01 .97 Presence* 13.43 5.43 9.71 5.77 6.39 <.01 .97 Persistence* 9.15 5.35 4.58 4.90 8.11 <.01 .97 Alternative* 33.89 16.23 18.65 14.61 8.94 <.01 .97 *Significantly different at p < .01 2. DIF in depressive symptomatology: Gender binary (male or female) Tables 3.4 and 3.5 show the DIF results between gender (male and female) for the CES-D and CESD-R. These tables identify the scoring method presented in Figure 3.1. Bolded items presented both a significant p-value (< .01) and moderate or large ÎR2 effect size. For gender DIF on the CES-D (Table 3.4), item 11 (sleep) presented DIF in all three scoring methods, items 13 (talked) and 17 (crying) presented DIF in the original and presence scoring, and item 7 (effort) presented DIF in original scoring only. The finding by Gelin and Zumbo for item 17 (crying) was replicated, but the persistence scoring of items 7 and 8 (effort and hopeful, respectively) did not. Regarding DIF between male and female respondents on the CESD-R (Table 3.5), the persistence scoring did not identify any items. DIF was identified in the original, presence and alternative scores for item 18 (weight). DIF was found for the original and alternative scores for items 7 (get going), 12 (slow), 15 (hurt), and 16 (tired). Item 6 (sad) presented DIF for the presence scoring method. 56 Table 3.4 CES-D ÎR2 values ITEM GENDER GENDER MINORITY SEXUAL MINORITY Original Presence Persistence Original Presence Persistence Original Presence Persistence 1 Bothered .0029 .0027 .0057 .0001 .0030 .0039 .0003 .0020 .0026 2 Appetite .0025 .0029 .0032 .0072 .0168** .0048 .0084 .0187** .0042 3 Blue .0001 .0025 .0022 .0021 .0007 .0084 .0003 .0000 .0032 4 Felt Good .0079 .0057 .0064 .0260** .0072 .0221** .0148* .0049 .0154** 5 Mindful .0019 .0036 .0035 .0006 .0000 .0020 .0037 .0050 .0035 6 Depressed .0001 .0012 .0008 .0000 .0018 .0047 .0004 .0005 .0059 7 Effort .0106* .0064 .0094 .0000 .0013 .0002 .0003 .0092 .0009 8 Hopeful .0003 .0005 .0012 .0233** .0079 .0304** .0253** .0091 .0279** 9 Failure .0030 .0026 .0057 .0012 .0035 .0021 .0004 .0045 .0008 10 Fearful .0022 .0057 .0024 .0040 .0038 .0058 .0011 .0013 .0019 11 Sleep .0150* .0169** .0165** .0033 .0007 .0074 .0006 .0002 .0023 12 Happy .0002 .0002 .0002 .0243** .0174** .0253** .0146* .0155** .0233** 13 Talked .0143* .0307** .0071 .0004 .0032 .0007 .0049 .0124 .0003 14 Lonely .0030 .0026 .0072 .0020 .0057 .0000 .0030 .0035 .0016 15 Unfriendly .0050 .0048 .0059 .0027 .0086 .0038 .0042 .0036 .0109 16 Enjoy Life .0001 .0027 .0003 .0195** .0076 .0213** .0286** .0145* .0334** 17 Crying .0112* .0201** .0013 .0086* .0138* .0129* .0091* .0096 .0171** 18 Sad .0040 .0043 .0059 .0014 .0037 .0009 .0027 .0031 .0009 19 Disliked .0017 .0005 .0004 .0010 .0023 .0013 .0023 .0051 .0030 20 Get Going .0059 .0090 .0059 .0012 .0017 .0007 .0009 .0038 .0001 *p < .01 and ÎR2 moderate effect **p < .01 and ÎR2 large effect 57 Table 3.5 CESD-R ÎR2 values ITEM GENDER GENDER MINORITY SEXUAL MINORITY Original Pres Persist Alt Original Pres Persist Alt Original Pres Persist Alt 1 Appetite .0021 .0030 .0044 .0012 .0016 .0041 .0020 .0024 .0004 .0051 .0034 .0002 2 Blue .0016 .0021 .0008 .0020 .0065 .0039 .0032 .0015 .0036 .0031 .0013 .0009 3 Mindful .0106 .0070 .0173 .0141 .0017 .0016 .0057 .0025 .0031 .0001 .0086 .0021 4 Depressed .0030 .0046 .0024 .0021 .0081* .0024 .0008 .0079* .0080* .0046 .0049 .0036 5 Restless .0036 .0018 .0069 .0035 .0051 .0007 .0014 .0035 .0065 .0006 .0010 .0036 6 Sad .0021 .0137* .0008 .0026 .0097* .0116* .0056 .0107* .0032 .0041 .0018 .0021 7 Get Going .0080* .0020 .0064 .0106* .0005 .0001 .0007 .0001 .0009 .0001 .0073 .0006 8 Happy .0051 .0044 .0029 .0053 .0005 .0007 .0021 .0011 .0032 .0016 .0025 .0009 9 Bad Person .0012 .0041 .0001 .0009 .0028 .0071 .0157** .0032 .0035 .0036 .0052 .0038 10 Interest .0015 .0009 .0001 .0011 .0041 .0035 .0053 .0027 .0038 .0017 .0048 .0023 11 Sleep .0050 .0091 .0013 .0044 .0027 .0145* .0026 .0017 .0013 .0050 .0014 .0002 12 Slow .0086* .0028 .0052 .0076* .0054 .0003 .0003 .0044 .0018 .0023 .0002 .0008 13 Fidget .0007 .0012 .0001 .0017 .0023 .0014 .0016 .0020 .0001 .0004 .0031 .0008 14 Dead .0076* .0020 .0065 .0065 .0050 .0194** .0160** .0052 .0103* .0217** .0144* .0096* 15 Hurt .0079* .0036 .0041 .0077* .0290** .0356** .0385** .0303** .0186** .0464** .0158** .0173** 16 Tired .0178** .0050 .0055 .0149* .0053 .0036 .0025 .0026 .0097* .0081 .0026 .0072* 17 Dislike .0009 .0010 .0006 .0000 .0031 .0118* .0117* .0010 .0023 .0079* .0074 .0008 18 Weight .0257** .0145* .0125 .0248** .0126* .0307** .0234** .0108* .0050 .0061 .0181** .0039 19 Insomnia .0030 .0018 .0005 .0020 .0049 .0023 .0033 .0035 .0077* .0001 .0076 .0074* 20 Focus .0010 .0044 .0000 .0010 .0024 .0069 .0109* .0028 .0029 .0047 .0110* .0015 *p < .01 and ÎR2 moderate effect **p < .01 and ÎR2 large effect 58 3. DIF in depressive symptomatology: Sexual identity (minority or non-minority) DIF between sexual minority and non-minorities on the CES-D was identified in all three scores in items 12 (happy) and 16 (enjoy life), in original and persistence scoring for items 4 (felt good), 8 (hopeful), and 17 (crying), and on the presence scoring only for item 2 (appetite), as seen in Table 3.4. DIF between sexual minority and non-minority was identified on the CESD-R in all four scoring methods for items 14 (dead) and 15 (hurt). DIF for the original and alternative scoring was identified for items 16 (tired) and 19 (insomnia). DIF was shown for the original scoring on item 4 (depressed), for the presence scoring on item 17 (dislike), and for the persistence scoring for items 18 (weight) and 20 (focus), as seen in Table 3.5. 4. DIF in depressive symptomatology: Gender identity (minority or non-minority) DIF between gender minorities and non-minorities on the CES-D (Table 3.4) was identified in all three scores for items 12 (happy) and 17 (crying), in the original and persistence scoring for items 4 (felt good), 8 (hopeful) and 16 (enjoy life), and on the presence scoring, only item 2 (appetite) showed DIF. For the CESD-R (Table 3.5), DIF between gender minority and non-minority groups was identified in all four scoring methods for items 15 (hurt) and 18 (weight). Item 6 (sad) presented DIF in the original, presence and alternative scoring. Items 14 (dead) and 17 (dislike) presented DIF in the presence and persistence scoring. Item 4 (depressed) presented DIF in the original and alternative scoring. Item 9 (bad person) showed DIF in the persistence scoring, item 11 (sleep) presented DIF in the presence scoring, and item 20 (focus) presented DIF on the persistence scoring. 59 5. Comparisons in DIF between gender and sexual identities a. CES-D When comparing the original scoring method across the contrast groups for the CES-D, only item 17 was found to display DIF in all 3, with an overall agreement of 13 out of 20 items (65%). When contrasting only the gender minority and the sexual minority groups, DIF is found equally in all items (100% agreement), so the agreement between gender and either minority group is 13 out of 20 (65%). When comparing the presence scoring method, no item showed DIF across all three groups, for an overall agreement of 14 out of 20 (70%). The gender group showed DIF in common with the gender minority group for item 17 (crying) for a total agreement of 16 out of 20 (80%), and with no item on the sexual minority group (in total, 14 out of 20, 70%). The gender minority and the sexual minority showed DIF in agreement for items 2 (appetite) and 12 (happy), with an overall agreement of 18 out of 20 (90%). For the persistence scoring method, no item showed DIF across all three groups, for a total agreement of 14 out of 20 (70%). The gender group did not show DIF in common with either the gender minority or sexual minority, with an overall agreement of 14 out of 20 items (70%) for both minority groups; sexual and gender minority groups showed complete agreement in DIF (100%). Item 17 (crying) presented DIF in the most number of data configurations (7 out of 9), while item 7 (effort) only showed DIF in the gender DIF original scoring method. Item 17 was the only item that presented DIF in all three grouping categories; otherwise, items that presented DIF between gender groups were not identified in the sexual and gender minority groups. Between the sexual and gender minority groups, most items presented similar behaviour, with an agreement in 58 out of 60 incidences (97%); whereas the gender group were in agreement 60 on 43 out of 60 (72%) with a gender minority group, and 41 out of 60 (68%) with the sexual minority group. b. CESD-R When comparing the original scoring method across the contrast groups of the CESD-R, item 15 (hurt) showed DIF in all 3, for a total agreement of 12 out of 20 items (60%). Additionally, the gender group found DIF in common with the gender minority group for item 18 (weight), for a total agreement between these two groups of 14 out of 20 items (70%). Similarly, the gender group found DIF in common with the sexual minority group for items 14 (dead) and 16 (tired), for a total agreement of 14 out of 20 (70%). The gender minority and sexual minority groups found DIF in agreement for item 4 (depressed), for a total agreement of 15 out of 20 (75%). For the presence scoring method, no item presented DIF in all three groups, for a total agreement of 14 out of 20 (70%). Between the gender and the gender minority groups, DIF was found in common for items 6 (sad), and 18 (weight), for an overall agreement of 16 out of 20 items (80%). Between gender and sexual minority groups, no items showed DIF equally, with a total agreement of 15 out of 20 (75%). Between gender minority and sexual minority groups, DIF was found in agreement for items 14 (dead), 15 (hurt) and 17 (dislike) for a total agreement of 17 out of 20 (85%). In the persistence scoring method across groups, no item showed DIF in all 3, for a total agreement of 14 out of 20 (70%). Since the gender group did not show DIF in any item, no agreement was shown between this group and the gender minority or sexual minority group, for a total agreement of 14 out of 20 (70%) and 16 out of 20 (80%) respectively. The gender minority and sexual minority groups identified DIF in common for items 14 (dead), 15 (hurt), 18 (weight), and 20 (focus), for a total agreement of 18 out of 20 items (90%). 61 The alternative scoring method identified DIF across all groups for item 15 (hurt), with an overall agreement of 12 out of 20 (60%) comparisons. Agreement in DIF between the gender and gender minority groups was also found for item 18 (weight), for a total agreement of 15 out of 20 (75%). Agreement between gender and sexual minority groups was found for item 16 (tired) for a total of 15 out of 20 (75%). The gender minority and sexual minority groups did not find DIF for additional items for a total agreement of 14 out of 20 (70%). Item 15 (hurt) presented DIF in the most number of configurations (10 out of 12), while items 9 (bad person) and 11 (sleep) only presented DIF in the gender minority persistence scoring method. Items 14 (dead), 15 (hurt), 16 (tired) and 18 (weight) presented DIF in all three grouping categories. Item 6 (sad) presented DIF in the gender and gender minority groups, but not in the sexual minority comparisons. Items 4 (depressed), 17 (dislike), and 20 (focus) only presented DIF in the sexual and gender minority groups. Gender and gender minority groups identified DIF in agreement in 38 out of 60 incidences (63%); gender and sexual minority groups agreed in 44 out of 60 incidences (73%); and sexual and gender minority groups in 46 out of 60 incidences (77%). 6. Comparing the common items between CES-D and CESD-R The mean scores among groups between the CES-D and the CESD-R show a change in statistically significant differences. The CES-D presented mean differences for gender in the original scoring method, and gender minority in the presence scoring; these differences were not present in the CESD-R. Similarly, only the CESD-R presented statistically significant differences for sexual minorities and non-minorities for the original scoring. Conversely, statistical differences were found in both the CES-D and the CESD-R for gender in the persistence scoring, and between sexual minority and non-minority for the presence scoring. 62 Because the CESD-R is a revised version of the CES-D, there are significant differences between versions, and not all items are comparable. Five items share the same wording between versions (items 3/ 5 Mindful, 6/4 Depressed, 11 Sleep/5 Restless, 18/6 Sad, and 20/7 Get Going for the CES-D and CESD-R, respectively). Two items were analogous, but the wording was different between versions (2/1 Appetite, and 3/2 Blue); two items were fundamentally different but had similar terminology (12/8 Happy and 19/17 Dislike); a total of 11 items in each version were entirely different. Because of these differences, findings on items cannot be compared in parallel to its entirety. For the items that were worded the same (items 3/ 5 Mindful, 6/4 Depressed, 11 Sleep/5 Restless, 18/6 Sad, and 20/7 Get Going), none of the items that presented DIF in one version did so in the other, even across groups or scoring method. Items were consistent in finding no DIF for either version in 33 out of 45 configurations (73.3%). For the two analogous items that had slightly changed wording (2/1 Appetite, and 3/2 Blue) item 3/2 (Blue) did not present DIF in any configuration, whereas item 2/1 (appetite) presented DIF on two configurations on the CES-D, for a total agreement of 16 out of 18 matches (88.9%). Overall, the CES-D presented DIF in 34 out of 180 configurations (18.9%). The CESD-R demonstrated DIF in 49 out of 240 configurations (20.4%) overall; when considering only the original, presence and persistence scoring, DIF was found in 34 out of 180 configurations (18.9%); and when considering only the presence, persistence and alternative scoring (meant to be analogous to the CES-D original scoring method), DIF was found in 36 out of 180 configurations (20%). These contrasts are meant to compare the density of DIF-presenting items across versions overall, and should not be interpreted as a contrast of repeated measures. 63 Conclusions Results from this paper replicate some of the findings by Gelin and Zumbo, as DIF between binary groups of respondents who identified as male or female were affected by the choice of the scoring method. These findings support Gelin and Zumboâs conclusion that DIF is a property of the item, the scoring method and the purpose of the instrument. DIF is a property of the item, as only some items presented DIF not attributable to a fundamental difference between groups. DIF is also a property of the scoring method, as different ways in which scoring was performed changed the results for DIF even between the same groups for the same item. Finally, DIF is a property of the purpose of the instrument, as the scoring method selected is dependent on this intention, the selection of items, groups assessed, and their interaction. As illustrated by Table 3.6, DIF may be found when comparing subgroups within the same sample, but not on others. Some of these differences in groups can be explained with the stressors that are part of being a minority group (Meyer, 2003; Mirowsky & Ross, 1989; Veale, Peter, Travers, & Saewyc, 2017). Table 3.6 CES-D ÎR2 values contrasting DIF ITEM GENDER GENDER MINORITY SEXUAL MINORITY Orig. Pres. Persist. Orig. Pres. Persist. Orig. Pres. Persist. 5 Mindful .0019 .0036 .0035 .0006 .0000 .0020 .0037 .0050 .0035 13 Talked .0143* .0307** .0071 .0004 .0032 .0007 .0049 .0124 .0003 17 Crying .0112* .0201** .0013 .0086* .0138* .0129* .0091* .0096 .0171** *p < .01 and ÎR2 moderate effect **p < .01 and ÎR2 large effect Moreover, these findings illustrate how the selection of scoring method alters the statistical inferences, as the construct changes âand thus, its psychometric characteristics may differ. Thus, Table 3.6 shows that while results may be similar between different scoring methods, they are not equivalent and should not be treated as such. 64 The current findings also expand on the previous study by Gelin and Zumbo. The use of a larger worldwide sample suggests that their conclusions are consistent and generalizable in a broad testing context. The study also extended the study on the revised version of the scale, the CESD-R, where we observed similar results where DIF between groups varied depending on the choice of scoring and grouping. Likewise, we compared items whose wordings did not change, or that had minimal differences and appeared on the surface as parallel or equivalent. Findings of DIF were affected in items that had varied wording and altered response options, suggesting that even small changes in wording or scale of measurement can result in changes in psychometric properties, urging researchers to consider their decisions during assessment carefully. Furthermore, this paper included groupings of gender and sexual identity beyond the traditional binary and discovered DIF differences between the groups. To our knowledge, this is the first study to investigate DIF, psychometric and measurement validity properties of a psychosocial measure from the lens of non-traditional identity. When the same population was grouped differently by binary gender, non-traditional gender identity and sexual orientation, the results of DIF were changed. Therefore, to extend Gelin and Zumboâs conclusion, DIF is not only a property of the items, scoring methods and purpose of the instrument, but also on the selection of grouping and the conceptualization of the group, specifically in this case the framing of gender and sexual identity. Research that considers the gender of the participants must expand beyond a binary male/female, as this is insufficient to represent the individual characteristics of the population adequately, limiting the scope of the findings. Researchers must be reminded about the generalizability of findings; inferences drawn from scores are limited in scope to the chosen characteristics of the individual characteristics of the sample, which are always present whether they are considered or not. It is up to the researchers to consider the characteristics of their sample in relation to the population they 65 represent. Neglecting these characteristics results in a limitation of the generalizability of findings, which threatens validity evidence. Furthermore, these choices for representation imply beliefs and values from the researchers, which are a concern of validity evidence based on consequences Findings in this study also have implications in the methodology and the interpretation of studies. Because of the changes in DIF, researchers must be mindful of how they choose to gather scores and interpret them. When lacking enough responses in a certain range of responses, one can decide to join score groups without much change in the interpretation. This approach is intrinsically erroneous, as these changes in scores change the operationalization of what is measured, and require to be interpreted adequately. Psychometricians need to be mindful that DIF is a property of the item, scoring method, purpose of the measure, groups and subgroups; scores that have been collapsed or rescored are not interchangeable, even if common practice treats them as equal. Thus, if the researcher suspects their sample may not be large or diverse enough, they should consider a priori whether they may change the measure or scores, and if they do, reflect these decisions on their inferences and conclusions. 66 Chapter IV - Conclusion: Consequences of scoring methods and response formats Measurement in social sciences requires accuracy and precision to ensure that valid inferences can be drawn from their scores (Chan, 2014; Messick, 1995a). This validation practice includes the response process âaffected by the wording and presentation of response formats, and the scoring methods. Thus, validation is a process that extends to scale creation, execution, and interpretation of scores. In other words, validation is a process that goes through the creation and deployment of a measure, its scores, and the extent of its consequences (AERA et al., 2014). This study examines this process within three main steps: design of a measure, recording of scores, and interpretation of scores. The validation process for each of these steps can be inferred to be highly related, as a flawed measure will obtain flawed scores and thus lead to misinformed interpretations. The process of validation is equally important in traditional paper surveys as it is in electronic or online surveys. However, whereas paper surveys have been examined in multiple settings and response formats, electronic surveys do not have as much evidence of the use and validity of scores obtained with different methods of measurement. Studies have shown that obtained scores between paper and electronic surveys produce similar outcomes (Giduthuri et al., 2014; Leisher, 2014; Ravert et al., 2015), but properties of scales that affect scores differ between them. For example, while the horizontal or vertical layout of rating scales can affect paper surveys (Paul-Dauphin, Guillemin, Virion, & Briançon, 1999; Scott & Huskisson, 1979), evidence suggests electronic formats are commonly unaffected by it (Garbarski, Schaeffer, & Dykema, 2018). However, linearly presenting the response options produces different results than those with response options in two ordered columns, or with the inclusion of numerical labels (Toepoel, Das, & van Soest, 2009). Moreover, while some electronic response formats may be similar to paper measures, some are unique to electronic versions, and as such, require 67 to be examined separately. Therefore, in the design of a measure, it is not enough to ensure content validity through the selection of accurate construct-relevant items; researchers must also select a response format that suits their needs and allows for an accurate collection of scores for their intended use. Rating scales are the set of items that quantify a construct based on agreement or endorsement (rating) of a response option. These rating scales can be classified as verbal, numerical and graphical-based on the presentation of their response options, although this terminology can be used inconsistently or ambiguously (Hjermstad et al., 2011). As suggested by their name, verbal rating scales use mainly verbal anchors and often label all their response items. These numerical rating scales utilize mainly an explicit numerical value to indicate response options, which may be accompanied by a verbal anchor or instruction to contextualize these values; graphical rating scales utilize images or symbols to indicate response options, and are seldom accompanied by a label or a numerical value. These response options in rating scales may imply order and magnitude, or suggest categorical responses. For traditional paper measurement, there is considerable evidence on the uses of these response formats and their implications on validity. For example, wording, phrasing and context affect score interpretation (e.g., Brunelli et al., 2010; Hjermstad et al., 2011; Mutebi et al., 2016), increasing the number of labelled anchors produces higher reliability but increases central or positive tendency responses (Krosnick & Berent, 1993; Weijters, Cabooter, & Schillewaert, 2010). Similarly, numerical labels tend to be perceived as more objective by respondents, and all-positive values are perceived as less extreme than a shift from negative to positives (Cabooter et al., 2016; Schwarz et al., 1991). Likewise, graphical rating scales have more discriminating power for thresholds of pain and can be easier for respondents to relate to the items (de Jong et al., 2015; ten Klooster et al., 2006; Rausch & Zehetleitner, 2014). 68 Whereas this evidence is widely available for traditional paper measurement, electronic response formats have little evidence of their use and their effects on score validity; some researchers may assume erroneously that this evidence of use is equivalent between paper and electronic measurement. However, different response formats affect the cognitive process of participants, and therefore it affects their response process and the obtained scores (Ackerman et al., 2016; Ritter & Sue, 2007; Wetzel & Greiff, 2018). These changes in recorded behaviour are partly caused by item affordances (Keehner & Moon, 2018), that is, the form in which the design of an item prompts or encourages the user to interact with it. Likewise, computer users often are familiar with response formats and have an expectation of the limitations and intended use of response formats. For example, a radio button format suggests a single choice, checkboxes suggests the possibility of multiple choices, and dropdown menus imply that the user can display a list of choices when interacting with the item. These visual and interactivity cues provide participants with the opportunities or affordances to act in specific ways (Colman, 2015) and minimize the mental effort needed to intuit how to operate with the presented items. One of the implications of item affordances is the explicit or implicit nature of choices. When a response format suggests a default value where participants have to modify their choice, it suggests implicit choice, whereas unmarked choices where participants have to engage actively suggest an explicit choice being made. Participants will also omit responses during uncertainty or learn towards positive responses (Cronbach, 1941; Pomplun & Omar, 1997). As shown in study 1, a small number of participants will respond to multiple options even when the instruction indicates otherwise when the format allows for it. Evidence collected in study 1 suggests that radio buttons are the most intuitive format for participants to use; they have low rates of missing responses, ensure that participants only select one choice, and resemble a traditional response format. Furthermore, they are used widely in formal and informal measurement settings, being a default option in many electronic 69 survey software packages. They appear to be well-suited for rating scales, which are widely used in psychosocial measurement (Brill, 2008; Lee & Paek, 2014; Wakita, Ueshima, & Noguchi, 2012). However, they cannot be used as a universal response format, as they lack the capability of selecting multiple responses, and are less suited for continuous scoring than a slider might be. In study 1, when presented with checkboxes, many participants attempted to mark multiple responses. It is unknown if this is an error caused by participants confusing which response box corresponded to their current item if they intended to mark more than one response or some other issue. As such, unless the researcher intends to record multiple options, this response format is not suggested. With this in mind, if the researcher does intend for multiple selections, this format is visually very similar to paper formats and another frequently seen format, so it may be intuitive for participants to use. It should be noted that participants in study 1 are likely familiar with different response formats due to the recruitment process having been done on an online survey platform so that less-familiar users may have more non-responses, that is, missing information. Another factor to consider for radio buttons and checkboxes in regards to missing or invalid responses is readability. When multiple items are presented on the same page, it may be harder for participants to identify the line of responses in the item grid. To reduce these user input errors, researchers can modify their surveys to highlight response rows and columns, use larger fonts, and limit the number of items presented per screen. It is advantageous to use intuitive response formats to facilitate test-taking to users, regardless of computer literacy levels. Similarly, the choice of the device seems to have an impact on the internal structure of a measure, where mobile devices âincluding tabletsâ showed differences to laptops and PC. Along with the evidence on response formats seen in study 1, this indicates how participants approach and interact with a survey affects how they will respond to it. This approach and 70 interaction may also be related to visual cues, interactivity, and accessibility. Because of these differences, researchers must attempt to ensure that their measures are presented as homogeneously as possible across devices to compliment compatibility and accessibility. Once data collection begins, researchers using electronic formats âparticularly online surveysâ are limited in their interaction with the recording of scores. Thus, it is important to ensure the measures are adequately constructed and to test their functionality ahead of time. While some survey software allows the modification of measures after it goes live, any changes may render the previous measure unusable. Therefore, researchers need to monitor the data collection as closely as possible. Whereas data collection is a process where participants are an active participant, the interpretation of scores, and by extension, their effects on validity, are affected mainly by researcher decisions. The choice of scoring method is one such decision that can facilitate the interpretation of the measurement, such as to compensate low endorsement for middle-point options, dichotomizing responses for certain analyses, or grouping scores based on specific cut-offs. At face value, it could seem that merging similar scoring groups might change this interpretation, as recorded responses are not shifted in quantity, just grouped. However, this change in precision has been shown to change the internal factor structure of a measure (Gelin & Zumbo, 2003). It bears reminding that these scoring choices have to be disclosed in the methodology and should follow a rationale supported by evidence. Scoring methods can serve different purposes in a study. In study 2, a depression scale was chosen to illustrate these changes, where one could intend to score by the presence or absence of any symptoms, to classify them by low and high persistence scoring, or to obtain a total score as per the original scoring method for this summated rating scale-. These three scoring methods reflect a theoretical definition of what the construct being measured is, even though test participants are presented with the same items and response options. 71 As observed in the study by Gelin and Zumbo (2003) and, by extension, the findings in this thesis in study 2, the changes in score affect the internal structure of a scale. Different scoring methods alter the interpretations based on obtained scores, highlighted in the changes for measurement invariance examined. Although it could be expected that scoring methods that more used more general classifications such as dichotomous classifications (high vs low scores) would hide the smaller differences between groups and thus one would expect more invariant items in these methods, findings in study 2 did not support this. On the other hand, changes in scores affected the outcomes of measurement invariance based on what subgroups were being compared, where sexual and gender minorities showed more changes based on the scoring method. As such, these differences in scoring methods result in non-trivial changes to the inferences from scale scores, and researchers should be clear on the function of these decisions. In other words, the use of a binary gender option (male/female) is not sufficient to describe the population adequately. In short, changing the response options and scoring methods will change the internal structure of the measure. Put another way, the researcher's decisions on the scale creation and score interpretation will guide how the intended construct is captured. Consequently, it is up to the researcher to inform these decisions and direct the measurement through the adequate choice of wording, scale and response format, data collection, and score interpretation. During these three phases in the process of measurement (design of a measure, recording of scores, and interpretation of scores), researchers should be aware of how the many small decisions on creation and implementation of measures can affect obtained scores. Even when using scales with supporting literature of validity evidence, small deviations from the original response format and scoring may result in differences that can affect their internal structure. Furthermore, as different response formats affect the measurement, and by 72 extension, the obtained scores, an inadequate response format can result in an inaccurate measurement. An interesting methodological point is observed in study 1, wherein the devices or the response formats had were not invariant, however, in either case the model fit for the overall data showed similar fit as the separate models. This may seem, at first, contradictory; however, these findings highlight that in cases of less than perfect model fit, it is possible for a model that is not invariant to still show adequate fit for the aggregate data. This is akin to finding that one-factor between groups ANOVA is not statistically significant, however, independent t-tests (or post-hoc tests) show differences between the groups. This also highlights that the intentionality of the specificity for choices of formats, grouping and scoring is important for measurement validity when considering differences between groups. Commonly, researchers are expected to report in their methodology what scales were used, and any changes that were made; the evidence found in previous chapters suggests that the choices for response format and scoring method should be considered in these reports. When researchers are not aware of the impact that differences of response formats and scoring methods, studies using electronic measurement could suffer from under-documentation, researchers may not feel compelled to discuss their decision making for recording and scoring if these appear trivial. Changes in scoring going under-reported would result in a lack of clarity on how the construct was defined and measured, and results may not be replicable. Without solid reasoning for these choices, researchers should avoid grouping scores or changing the pre-existing response formats on electronic measurement. To aid in the choice of response formats and scoring methods, researchers may find it useful to refer to their research interests and goals; a clear construct defined a priori should lead the choice of measure and any scoring differences. Once the intended measurement of the construct is defined, choosing a response 73 format that can be selected based on how information is intended to be recorded. In short, electronic measurement should be tailored to the research needs. To reduce the need for grouping scores or recoding categories, researchers should use item response formats that have only as many data points as needed, no more or less. Too few item response points and scores may be imprecise or too broad, and too many item response points will result in unnecessary clutter and additional effort for participants. Identifying the internal structure of the data and the expected number of participants can also aid in this decision, as complex models may need more straightforward data or a larger number of participants. It might be preferable to err on the side of detail, as responses can always be grouped and recoded, but few data points cannot be separated for further precision. This decision about the number of response options is particularly important if the data will be open-sourced or for multiple uses. The studies in the previous chapters have described some of the effects that changes in response formats have in the data scoring; however, there are still gaps in the literature. Future studies may be necessary to understand the effects on validity based on other modalities of electronic response formats; this includes comparing response formats with similar intended purposes (e.g., checkboxes vs text boxes that allow multiple choice), and changes in the visual presentation of formats (e.g., using stars vs using faces in rating scales). Furthermore, it might be necessary to utilize measures with a significantly larger number of items to observe if the response formats have a different effect on respondent fatigue. Likewise, these observed changes based on response formats should be replicated using different scales, and extended to other electronic response formats. Additionally, examining response processes for different response formats may give further insight into the behaviour of participants, such as think-aloud protocols or cognitive interviews. 74 Study 1 introduced an examination of whether devices had any implication on responses and obtained scores. The obtained results suggest that the choice of device used to record responses does have an effect on responses. As such, future studies may be necessary to obtain a further descriptive view around these effects and examine various experimental conditions based on various electronic devices, including device size, interaction method (e.g., touch vs mouse), operating system, and even location of test-taking (e.g., at home, on the street, at a public location). One way to interpret the findings seen in the previous chapters is that questionnaires and survey designs are somewhat fragile, as minor changes or revisions in what might be seemingly a minor feature may change the construct, and consequently affect the validity of scores. This finding was illustrated in this thesis with changes of scoring method (total score, presence and persistence), the response format (checkbox, dropdown menu, radio button, and slider), population characteristics selected or ignored, and their grouping for analyses (binary gender, sexual minority and gender minority), use of a device (desktop, laptop, phone and tablet), item wording, and the number of response options (CES-D vs CESD-R). Just how constructs are defined and specified by the researcher, it is also shaped by the choices made in how to observe, measure and analyze it. There is still a limited amount of research literature on the psychometric properties of electronic measurement. For the field of measurement validity, this limitation presents the need to examine this phenomenon thoroughly, to communicate the findings to the community, and for the community to integrate these findings into their practice. The more we understand how changes in response formats and scoring methods affect measurement validity, the more control we can have to ensure accurate measurement and appropriate conclusions from the obtained scores. 75 It may be that a perfect measurement of constructs in psychosocial sciences may be impossible, as it attempts to give a simple numerical representation to a complex system that is inextricable to its environs. However, a more precise measurement for a solidly defined construct can approximate scores that reflect these constructs more accurately. Improvements in measurement are critical in the betterment of research; therefore, it is essential to have the best possible understanding of how we record and analyze scores. 76 References Ackerman, R., Donnellan, M., Roberts, B., & Fraley, R. (2016). The effect of response format on the psychometric properties of the narcissistic personality inventory: Consequences for item meaning and factor structure. Assessment, 23(2), 203-220. doi:10.1177/1073191114568113 Allison, P. D. (2003). Missing data techniques for structural equation modeling. Journal of Abnormal Psychology, 112(4), 545-557. doi:10.1037/0021-843X.112.4.545 Allison, P. D. (2001). Missing data (Vol. 136). Thousand Oaks, CA: Sage. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. [AERA, APA, NCME] (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Arbona, C., Burridge, A., & Olvera, N. (2017). The center for epidemiological studies depression scale (CES-D): Measurement equivalence across gender groups in hispanic college students. Journal of Affective Disorders, 219, 112-118. doi:10.1016/j.jad.2017.05.024 Arnau, R. C., Thompson, R. L., & Cook, C. (2001). Do different response formats change the latent structure of responses? An empirical investigation using taxometric analysis. Educational and Psychological Measurement, 61(1), 23-44. doi:10.1177/00131640121971040 Bauer, G., Hammond, R., Travers, R., Kaay, M., Hohenadel, K. M., & Boyce, M. (2009). âI don't think this is theoretical; this is our livesâ: How erasure impacts health care for transgender people. Journal of the Association of Nurses in AIDS Care, 20, 348-361. Beals, J., Manson, S. M., Keane, E. M., & Dick, R. W. (1991). Factorial structure of the Center for Epidemiologic StudiesâDepression Scale among American Indian college students. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 3(4), 623â627. https://doi.org/10.1037/1040-3590.3.4.623 Bien, J., & Tibshirani, R. J. (2011). Sparse estimation of a covariance matrix. Biometrika, 98(4), 807-820. doi:10.1093/biomet/asr054 Birnholz, J. L., & Young, M. A. (2012). Differential item functioning for lesbians, bisexual, and heterosexual women in the center for epidemiological studies depression scale. Assessment, 19(4), 502-505. doi:10.1177/1073191112440032 Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149. doi:10.3389/fpubh.2018.00149 Brill, J. (2008). Likert scale. In P. J. Lavrakas (Ed.), Encyclopedia of survey research methods (pp. 428-429). Thousand Oaks, CA: SAGE Publications Ltd. doi: 10.4135/9781412963947.n273 Brown, T. A. (2006). Confirmatory factor analysis for applied research (2nd ed). New York: Guilford Press. Brunelli, C., Zecca, E., Martini, C., Campa, T., Fagnoni, E., Bagnasco, M., Lanata, L., & Caraceni, A. (2010). Comparison of numerical and verbal rating scales to measure pain exacerbations in patients with chronic cancer pain. Health and Quality of Life Outcomes, 8(1), 42-42. doi:10.1186/1477-7525-8-42 Bryant, W. (2017). Developing a strategy for using technology-enhanced items in large-scale standardized tests. Practical Assessment, Research and Evaluation, 22(1). van Buuren, S., Boshuizen, H. C., & Knook, D. L. (1999). Multiple imputation of missing blood pressure covariates in survival analysis. Statistics in Medicine, 18(6), 681. 77 Cabooter, E., Weijters, B., Geuens, M., & Vermeir, I. (2016). Scale format effects on response option interpretation and use. Journal of Business Research, 69(7), 2574-2584. doi:10.1016/j.jbusres.2015.10.138 Callahan, C. M., & Wolinsky, F. D. (1994). The effect of gender and race on the measurement properties of the CES-D in older adults. Medical Care, 32(4), 341-356. doi:10.1097/00005650-199404000-00003 Campbell, N. R. (1952). Measurement. In What is science? (pp. 109-134). New York: Dover Publications. Carpenter, S. (2018). Ten steps in scale development and reporting: A guide for researchers. Communication Methods and Measures, 12(1), 25-44. doi:10.1080/19312458.2017.1396583 Chan, E. K. H, Zumbo, B. D., Chen, M.Y., Zhang, W., Darmawanti, I., & Mulyana, O. P. (2014). Reporting of measurement validity in articles published in Quality of Life Research. In E. K. H. Chan & B. D. Zumbo (Eds.), Validity and Validation in Social, Behavioral, and Health Sciences (pp. 217-228). New York: Springer. Chan, E. K. H. (2014). Standards and guidelines for validation practices: development and evaluation of measurement instruments. In B. D. Zumbo and E.K.H. Chan (Eds.), Validity and Validation in Social, Behavioral, and Health Sciences (pp. 35-66). New York: Springer. Chan, E. K. H., Munro, D. W., Huang, A. H. S., Zumbo, B. D., Vojdanijahromi, R., & Ark, N. (2014). Validation practices in counseling: Major journals, mattering instruments, and the Kuder Occupational Interest Survey (KOIS). In E. K. H. Chan & B. D. Zumbo (Eds.), Validity and Validation in Social, Behavioral, and Health Sciences (pp. 67-87). New York: Springer. Chinni, M. L., & Hubley, A. M. (2014). A research synthesis of validation practices used to evaluate the satisfaction with life scale (SWLS). In E. K. H. Chan & B. D. Zumbo (Eds.), Validity and Validation in Social, Behavioral, and Health Sciences (pp. 35-66). New York: Springer. Chung, Y. B., & Katayama, M. (1996). Assessment of sexual orientation in lesbian/gay/bisexual studies. Journal of Homosexuality, 30(4), 49. Cizek, G. J., Bowen, D., & Church, K. (2010). Sources of validity evidence for educational and psychological tests: A follow-up study. Educational and Psychological Measurement, 70(5), 732-743. doi: 10.1177/0013164410379323 Cizek, G. J., Rosenberg, S. L., & Koons, H. H. (2008). Sources of validity evidence for educational and psychological tests. Educational and Psychological Measurement, 68(3), 397-412. doi: 10.1177/0013164407310130 Clarke, V., Ellis, S. J., Peel, E., & Riggs, D. W. (2010) Lesbian, gay, bisexual, trans and queer psychology: An introduction. Leiden: Cambridge University Press. Colman, A. (2015). Affordance. In A Dictionary of Psychology. Oxford University Press. Retrieved 10 Jun. 2019, from https://www-oxfordreference-com.ezproxy.library.ubc.ca/view/10.1093/acref/9780199657681.001.0001/acref-9780199657681-e-205. Couper, M. P., Tourangeau, R., Conrad, F. G., & Crawford, S. D. (2004). What they see is what we get: Response options for web surveys. Social Science Computer Review, 22(1), 111-127. doi:10.1177/0894439303256555 Cox, D. W., & Owen, J. J. (2014). Validity evidence for a perceived social support measure in a population health context. In E. K. H. Chan & B. D. Zumbo (Eds.), Validity and Validation in Social, Behavioral, and Health Sciences (pp. 229-241). New York: Springer. Cronbach, L. J. (1941). An experimental comparison of the multiple true-false and multiple multiple-choice tests. Journal of Educational Psychology, 32(7), 533-543. 78 Cronbach, L. J. (1988). Five perspectives on validation argument. In H. Wainer & H. Braun (Eds.), Test validity (pp. 3-17). Hillsdale: Lawrence Erlbaum Associates. van Dam, N. T., & Earleywine, M. (2011). Validation of the center for epidemiologic studies depression ScaleâRevised (CESD-R): Pragmatic depression assessment in the general population. Psychiatry Research, 186(1), 128-132. doi:10.1016/j.psychres.2010.08.018 DiStefano, C., & Morgan, G. B. (2014). A comparison of diagonal weighted least squares robust estimation techniques for ordinal data. Structural Equation Modeling: A Multidisciplinary Journal, 21(3), 425-438. doi:10.1080/10705511.2014.915373 Dong, Y., & Peng, C. J. (2013). Principled missing data methods for researchers. Springerplus, 2(1), 1-17. doi:10.1186/2193-1801-2-222 Downing, S. M., Baranowski, R. A., Grosso, L. J., & Norcini, J. J. (1995). Item type and cognitive ability measured: The validity evidence for multiple true-false items in medical specialty certification. Applied Measurement in Education, 8(2), 187-207. doi:10.1207/s15324818ame0802_5 Drasgow, F., Chernyshenko, O. S., & Stark, S. (2010). 75 years after likert: Thurstone was right. Industrial and Organizational Psychology, 3(4), 465-476. doi:10.1111/j.1754-9434.2010.01273.x Eaton, W.W., Muntaner, C., Smith, C., Tien, A., & Ybarra, M. (2004). Center for Epidemiologic Studies Depression Scale: Review and revision (CESD and CESD-R). In M. E. Maruish (Ed.), The Use of Psychological Testing for Treatment Planning and Outcomes Assessment (3rd ed., pp. 363-377). Mahwah, NJ: Lawrence Erlbaum. Evans, J. R., & Mathur, A. (2005). The value of online surveys. Internet Research, 15(2), 195-219. Fischer, E., & Arnold, S. J. (1994). Sex, gender identity, gender role attitudes, and consumer behavior. Psychology & Marketing, 11(2), 163-182. doi:10.1002/mar.4220110206 Fitton, V. A., Ahmedani, B. K., Harold, R. D., & Shifflet, E. D. (2013). The role of technology on young adolescent development: Implications for policy, research and practice. Child and Adolescent Social Work Journal, 30(5), 399-413. doi:10.1007/s10560-013-0296-2 Flora, D. B., & Flake, J. K. (2017). The purpose and practice of exploratory and confirmatory factor analysis in psychological research: Decisions for scale development and validation. Canadian Journal of Behavioural Science, 49, 78-88. Funke, F. (2016). A web experiment showing negative effects of slider scales compared to visual analogue scales and radio button scales. Social Science Computer Review, 34(2), 244. doi:10.1177/0894439315575477 Funke, F., Reips, U., & Thomas, R. K. (2011). Sliders for the smart: Type of rating scale on the web interacts with educational level. Social Science Computer Review, 29(2), 221-231. doi:10.1177/0894439310376896 Furr, R., & Bacharach, V. R. (2014). Psychometrics: An introduction (2nd ed.). Thousand Oaks, CA, US: Sage Publications, Inc. Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal reliability for likert-type and ordinal item response data: A conceptual, empirical, and practical guide. Practical Assessment, Research & Evaluation, 17(3), 1-13. Gamst, G., Meyers, L. S., & Guarino, A. J. (2008). Analysis of variance designs: A conceptual and computational approach with SPSS and SAS. Cambridge University Press. https://doi.org/10.1017/CBO9780511801648 Garbarski, D., Schaeffer, N. C., & Dykema, J. (2018). The effects of features of survey measurement on self-rated health: Response option order and scale orientation. Applied Research in Quality of Life, doi:10.1007/s11482-018-9628-x Garvey, J. (2017). Considerations for queer as a sexual identity classification in education survey research. Journal of College Student Development, 58(7), 1113-1118. doi:10.1353/csd.2017.0088 79 Gelin, M. N., & Zumbo, B. D. (2003). Differential item functioning results may change depending on how an item is scored: An illustration with the center for epidemiologic studies depression scale. Educational and Psychological Measurement, 63(1), 65-74. doi:10.1177/0013164402239317 Gesicki, A. (2015). Decision rules based on hypothesis tests and effect sizes for logistic regression differential item functioning (Masterâs thesis). University of British Columbia. Retrieved from https://open.library.ubc.ca/cIRcle/collections/24/items/1.0165814 Giduthuri, J. G., Maire, N., Joseph, S., Kudale, A., Schaetti, C., Sundaram, N., Schindler, C., & Weiss, M. G. (2014). Developing and validating a tablet version of an illness explanatory model interview for a public health survey in Pune, India. PloS One, 9(9), e107374. doi:10.1371/journal.pone.0107374 Gonsiorek, J. C., Sell, R. L., & Weinrich, J. D. (1995). Definition and measurement of sexual orientation. Suicide and LifeâThreatening Behavior, 25(SUP-95), 40-51. doi:10.1111/j.1943-278X.1995.tb00489.x Grant, J. S., & Davis, L. L. (1997). Selection and use of content experts for instrument development. Research in Nursing & Health, 20(3), 269-274. doi: 10.1002/ (SICI) 1098-240X (199706)20:3<269::AID-NUR9>3.0.CO;2-G Gunnell, K. E., Wilson, P. M., Zumbo, B. D., Crocker, P. R. E., Mack, D. E., & Schellenberg, B. J. I. (2014). Validity theory and validity evidence for scores derived from the Behavioural Regulation in Exercise questionnaire. In E. K. H. Chan & B. D. Zumbo (Eds.), Validity and Validation in Social, Behavioral, and Health Sciences (pp. 175-191). New York: Springer. Hancock, G. R., Thiede, K. W., Sax, G., & Michael, W. B. (1993). Reliability of comparably written two-option multiple-choice and true-false test items. Educational and Psychological Measurement, 53(3), 651. Hart, C. G., Saperstein, A., Magliozzi, D., & Westbrook, L. (2019). Gender and health: Beyond binary categorical measurement. Journal of Health and Social Behavior, 60(1), 101-118. doi:10.1177/0022146519825749 Haynes, S. N., Richard, D. S., & Kubany, E. S. (1995). Content validity in Psychological Assessment: A functional approach to concepts and methods. Psychological Assessment, 7(3), 238-247. doi:10.1037/1040-3590.7.3.238 Healey, B. (2007). Drop downs and scroll mice: The effect of response option format and input mechanism employed on data quality in web surveys. Social Science Computer Review, 25(1), 111-128. 10.1177/0894439306293888 Hirschfeld, G., & Brachel, R. v. (2014). Multiple-group confirmatory factor analysis in R--A tutorial in measurement invariance with continuous and ordinal indicators. Practical Assessment, 19(7), 1-12. Hjermstad, M. J., Fayers, P. M., Haugen, D. F., Caraceni, A., Hanks, G. W., Loge, J. H., Fainsinger, R., Aass, N., Stein, K., & European Palliative Care Research Collaborative (EPCRC). (2011). Studies comparing numerical rating scales, verbal rating scales, and visual analogue scales for assessment of pain intensity in adults: A systematic literature review. Journal of Pain and Symptom Management, 41(6), 1073-1093. doi:10.1016/j.jpainsymman.2010.08.016 Hodge, D.R., & Gillespie, D. (2003). Phrase completions: An alternative to Likert scales. Social Work Research, 27(1), 45-55. Hogan, T. P., & Agnello, J. (2004). An empirical study of reporting practices concerning measurement validity. Educational and Psychological Measurement, 64(5), 802-812. doi:10.1177/0013164404264120 Holland, P. W., & Thayer, D. T. (1988). Differential item functioning and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Erlbaum. 80 Hubley, A. M. (2017). Expanding views on response processes evidence for validity. Measurement: Interdisciplinary Research and Perspectives, 15(3-4), 140-142. doi:10.1080/15366367.2017.1404366 Hubley, A. M., & Zumbo, B. D. (2013). Psychometric characteristics of assessment procedures: An overview. In K. F. Geisinger (Ed.). APA handbook of testing and assessment in psychology (Vol. 1, pp. 3-19). Washington, D.C.: American Psychological Association Press. Jahn, R., Baumgartner, J., van den Nest, M., Friedrich, F., Alexandrowicz, R., & Wancata, J. (2018). Kriteriumsvalidität der deutschsprachigen version der CES-D in der allgemeinbevölkerung. Psychiatrische Praxis, doi:10.1055/a-0584-9803 [Epub ahead of print] Jodoin, M. G., & Gierl, M. J. (2001). Evaluating Type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14(4), 329-349. de Jong, A. E. E., Bremer, M., Hofland, H. W. C., Schuurmans, M. J., Middelkoop, E., & van Loey, N. E. E. (2015). The visual analogue thermometer and the graphic numeric rating scale: A comparison of self-report instruments for pain measurement in adults with burns. Burns, 41(2), 333-340. Kane, M. T. (2006). Educational measurement. In R. L. Brennan (Ed.), Validation (4th ed., pp. 17-64). Westport: American Council on Education/Praeger. Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1-73. Kang, H. (2013). The prevention and handling of the missing data. Korean Journal of Anesthesiology, 64(5), 402-406. doi:10.4097/kjae.2013.64.5.402 Kaptelinin, V. (2013). Affordances. In The Encyclopedia of Human-Computer Interaction (2nd ed.). Interaction Design Foundation. Retrived from https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/affordances Keehner, M. & Moon, J. A. (2018). What Cognitive Science Can Tell Us about Technology-Enhanced Item Design: Foundational Principles and Empirical Findings. Paper presented at the National Conference on Student Assessment, San Diego, CA. Kero, P., & Lee, D. (2015). Slider scales and web-based surveys: A cautionary note. Journal of Research Practice, 11(1) King, M., Semlyen, J., Tai, S., Killaspy, H., Osborn, D., Popelyuk, D., & Nazareth, I. (2008). A systematic review of mental disorder, suicide, and deliberate self harm in lesbian, gay and bisexual people. Bmc Psychiatry, 8(1), 70-70. doi:10.1186/1471-244X-8-70 ten Klooster, P. M., Vlaar, A. P. J., Taal, E., Gheith, R. E., Rasker, J. J., El-Garf, A. K., & van de Laar, Mart A. F. J. (2006). The validity and reliability of the graphic rating scale and verbal rating scale for measuring pain across cultures: A study in egyptian and dutch women with rheumatoid arthritis. Clinical Journal of Pain, 22(9), 827-830. 10.1097/01.ajp.0000210939.96557.c6 Krosnick, J. A., & Berent, M. K. (1993). Comparisons of party identification and policy preferences: The impact of survey question format. American Journal of Political Science, 37(3), 941-964. Lai, K., & Green, S. B. (2016). The problem with having two watches: Assessment of fit when RMSEA and CFI disagree. Multivariate Behavioral Research, 51(2-3), 220-239. doi:10.1080/00273171.2015.1134306 Launeanu, M. S. (2016). Response processes as a source of validity evidence in self-report measuresâ¯: theoretical and methodological implications (Doctoral dissertation). University of British Columbia. Retrieved from https://open.library.ubc.ca/collections/ubctheses/24/items/1.0300124 81 Lee, J., & Paek, I. (2014). In search of the optimal number of response categories in a rating scale. Journal of Psychoeducational Assessment, 32(7), 663-673. doi:10.1177/0734282914522200 Leisher, C. (2014). A comparison of tablet-based and paper-based survey data collection in conservation projects. Social Sciences, 3(2), 264-271. doi:10.3390/socsci3020264 Lewinsohn, P. M., Seeley, J. R., Allen, N. B., & Roberts, R. E. (1997). Center for epidemiologic studies depression scale (CES-D) as a screening instrument for depression among community-residing older adults. Psychology and Aging, 12(2), 277-287. doi:10.1037/0882-7974.12.2.277 Li, C. (2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48(3), 936-949. doi:10.3758/s13428-015-0619-7 Lim, K. Y. (2011). What does the tablet PC mean to you? A phenomenological research. Innovations in Education and Teaching International, 48(3), 323-333. Liu, M., & Conrad, F. G. (2016). An experiment testing six formats of 101-point rating scales. Computers in Human Behavior, 55, 364-371. doi:10.1016/j.chb.2015.09.036 Loe, B. S., Stillwell, D., & Gibbons, C. (2017). Computerized adaptive testing provides reliable and efficient depression measurement using the CES-D scale. Journal of Medical Internet Research, 19(9), e302. doi:10.2196/jmir.7453 Little, R. J. A., Rubin, D. B., Wiley Online Library, & Wiley Frontlist All Obook 2019. (2019). Statistical analysis with missing data (Third ed.). Hoboken, NJ: Wiley. Maddox B., Zumbo B.D. (2017) Observing Testing Situations: Validation as Jazz. In: Zumbo B., Hubley A. (eds) Understanding and Investigating Response Processes in Validation Research. Social Indicators Research Series, vol 69. Springer, Cham Makambi, K. H., Williams, C. D., Taylor, T. R., Rosenberg, L., & Adams-Campbell, L. L. (2009). An assessment of the CES-D scale factor structure in black women: The black women's health study. Psychiatry Research, 168(2), 163-170. doi:10.1016/j.psychres.2008.04.022 Mantel, N., & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22, 719-748. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). New York: American Council on Education and Macmillan. Messick, S. (1995a). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741. Messick, S. (1995b). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14(4), 5-8. doi:10.1111/j.1745-3992.1995.tb00881.x Meyer, I.H. (2003). Prejudice, social stress, and mental health in lesbian, gay and bisexual populations: Conceptual issues and research evidence. Psychological Bulletin, 129, 674-697. doi:10.1037/0033-2909.129.5.674 Mindrila, D. (2010). Maximum Likelihood (ML) and Diagonally Weighted Least Squares (DWLS) Estimation Procedures: A Comparison of Estimation Bias with Ordinal and Multivariate Non-Normal Data. International Journal of Digital Society, 1(1). 60-66. doi:10.20533/ijds.2040.2570.2010.0010 Mirowsky, J., & Ross, C E. (1989). Social causes of psychological distress. Hawthorne, NY: Aldine de Gruyter. Mutebi, A., Slack, M., Warholak, T. L., Hudgens, S., & Coons, S. J. (2016). Interpretation of verbal descriptors for response options commonly used in verbal rating scales in patient-reported outcome instruments. Quality of Life Research, 25(12), 3181-3189. doi:10.1007/s11136-016-1333-3 82 Nagoshi, J. L., & Brzuzy, S. (2010). Transgender theory: Embodying research and practice. Affilia, 25(4), 431-443. doi:10.1177/0886109910384068 Nandakumar, R. (1991). Traditional dimensionality versus essential dimensionality. Journal of Educational Measurement, 28(2), 99-117. doi:10.1111/j.1745-3984.1991.tb00347.x National Institute of Mental Health. (2016). Evaluations of Sexual Orientation and Gender Identity Survey Measures: What Have We Learned? Retrieved from https://dpcpsi.nih.gov/sites/default/files/Evaluations_of_SOGI_Questions_20160923_508.pdf Niemi, I. (1993). Systematic Error in Behavioural Measurement: Comparing Results from Interview and Time Budget Studies. Social Indicators Research, 30(2-3), 229-244. Orme, J. G., Reis, J., & Herz, E. J. (1986). Factorial and discriminant validity of the center for epidemiological studies depression (CESâD) scale. Journal of Clinical Psychology, 42(1), 28-33. doi:10.1002/1097-4679(198601)42:1<28::AID-JCLP2270420104>3.0.CO;2-T Owora, A. H., & Carabin, H. (2018). Impact of misclassification error in the estimation of maternal major depression disorder prevalence in home visitation programs. Psychiatry Research, 261, 80-87. doi:10.1016/j.psychres.2017.12.047 Padilla, J., & Benítez, I. (2014). Validity evidence based on response processes. Psicothema, 26(1), 136-144. Palan, K. M. (2001). Gender identity in consumer behavior research: A literature review and research agenda. Academy of Marketing Science Review, 2001, 1. Paul-Dauphin, A., Guillemin, F., Virion, J. M., & Briançon, S. (1999). Bias and precision in visual analogue scales: A randomized controlled trial. American Journal of Epidemiology, 150(10), 1117-1127. doi:10.1093/oxfordjournals.aje.a009937 Pepinsky, T. B. (2018). A note on listwise deletion versus Multiple Imputation. Political Analysis, 26(4), 480-488. doi:10.1017/pan.2018.18 Pomplun, M., & Omar, M. H. (1997). Multiple-mark items: An alternative objective item format? Educational and Psychological Measurement, 57(6), 949-962. Radloff, L. S. (1977). The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1(3), 385-401. doi:10.1177/014662167700100306 Radloff, L. S., & Locke, B. Z. (1986). The community mental health assessment survey and the CES-D Scale. In A. E. Slaby (Series Ed.), Community surveys of psychiatric disorders (pp. 177-189). New Brunswick, NJ: Rutgers University Press. Ramsay, J. O. (2001). TestGraf: A program for the graphical analysis of multiple choice test and questionnaire data. Montreal, Canada: McGill University. Rausch, M., & Zehetleitner, M. (2014). A comparison between a visual analogue scale and a four point scale as measures of conscious experience of motion. Consciousness and Cognition, 28, 126-140. doi:10.1016/j.concog.2014.06.012 Ravert, R., Gomez-Scott, J., & Donnellan, M. (2015). Equivalency of paper versus tablet computer survey data. Educational Researcher, 44(5), 308-310. doi:10.3102/0013189X15592845 Reips, U., & Funke, F. (2008). Interval-level measurement with visual analogue scales in internet-based research: VAS generator. Behavior Research Methods, 40(3), 699-704. doi:10.3758/BRM.40.3.699 Rios, J., & Wells, C. (2014). Validity evidence based on internal structure. Psicothema, 26, 108-116. Ritter, L. A., & Sue, V. M. (2007). Systematic planning for using an online survey. New Directions for Evaluation, 15(22). doi:10.1002/ev.231 Roiser, M. (1996). Consensus, attitudes and guttman scales. Papers on Social Representations, 5, 11-19. 83 Roster, C. A., Lucianetti, L., & Albaum, G. (2015). Exploring slider vs. categorical response formats in web-based surveys. Journal of Research Practice, 11(1) Roth, D.L., Ackerman, M. L., Okonkwo, O. C., & Burgio, L. D. (2008). The four-factor model of depressive symptoms in dementia caregivers: A structural equation model of ethnic differences. Psychology and Aging, 23, 567-576. Saewyc, E. M. (2017). Respecting variations in embodiment as well as gender: Beyond the presumed âbinaryâ of sex. Nursing Inquiry, 24(1), e12184-n/a. doi:10.1111/nin.12184 Saewyc, E. M., Bauer, G. R., Skay, C. L., Bearinger, L. H., Resnick, M. D., Reis, E., & Murphy, A. (2004). Measuring sexual orientation in adolescent health surveys: Evaluation of eight school-based surveys. Journal of Adolescent Health, 35(4), 345.e1-345.e15. doi:10.1016/j.jadohealth.2004.06.002 Schwarz, N., Knäuper, B., Hippler, H., Noelle-Neumann, E., & Clark, L. (1991). Rating scales: Numeric values may change the meaning of scale labels. The Public Opinion Quarterly, 55(4), 570-582. 10.1086/269282 Schleyer, T. K., & Forrest, J. L. (2000). Methods for the design and administration of web-based surveys. Journal of the American Medical Informatics Association: JAMIA, 7(4), 416-425. doi:10.1136/jamia.2000.0070416 Schroevers M.J., Sanderman R., van Sonderen E., Ranchor A.V.: The evaluation of the Center for Epidemiologic Studies Depression (CES-D) scale: depressed and positive affect in cancer patients and health reference subjects. Quality of Life Research 2000;9(9): pp. 1015-1029. Van Selm, M., & Jankowski, N. W. (2006). Conducting online surveys. Quality & Quantity, 40(3), 435-456. doi:10.1007/s11135-005-8081-8 Scott, J., & Huskisson, E. C. (1979). Vertical or horizontal visual analogue scales. Annals of the Rheumatic Diseases, 38(6), 560-560. doi:10.1136/ard.38.6.560 Shi, D., & Maydeu-Olivares, A. (2020). The effect of estimation methods on SEM fit indices. Educational and Psychological Measurement, 80(3), 421-445. doi:10.1177/0013164419885164 Slaney, K. L., & Maraun, M. D. (2008). A proposed framework for conducting data-based test analysis. Psychological Methods, 13, 376-390. Slocum-Gori, S. L., & Zumbo, B. D. (2011). Assessing the unidimensionality of psychological scales: Using multiple criteria from factor analysis. Social Indicators Research, 102, 443-461. doi:10.1007/s11205-010-9682- Slocum-Gori, S. L., Zumbo, B. D., Michalos, A. C., & Diener, E. (2009). A note on the dimensionality of quality of life scales: An illustration with the satisfaction with life scale (SWLS). Social Indicators Research, 92(3), 489-496. doi:10.1007/s11205-008-9303-y Sörbom, D. (1974). A general method for studying differences in factor means and factor structure between groups. British Journal of Mathematical and Statistical Psychology, 27, 229-239. doi:10.1111/bmsp.1974.27.issue-2 Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677-680. Stout, W. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52(4), 589-617. doi:10.1007/BF02294821 Sue, V. M. & Ritter, L. A. (2007). Designing and developing the survey instrument. In Conducting online surveys (pp. 59-87). SAGE Publications Ltd doi: 10.4135/9781412983754.n5 Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361-370. Thorson, J. A., & Powell, F. C. (1993). The CES-D: Four or five factors? Bulletin of the Psychonomic Society, 31(6), 577-578. doi:10.3758/BF03337359 Toepoel, V., Das, M., & van Soest, A. (2009). Design of web questionnaires: The effect of layout in rating scales. Journal of Official Statistics, 25, 509-28. 84 Treharne, G. J., & Beres, M. A. (2016). Writing survey questions to operationalise sex, gender identity, and sexual orientation in new zealand: Perspectives from psychological and sociological research with the LGBTQ community. New Zealand Sociology, 31(1), 173-180. Veale, J. F., Peter, T., Travers, R., & Saewyc, E. M. (2017). Enacted stigma, mental health, and protective factors among transgender youth in Canada. Transgender Health, 2(1), 207-216. doi:10.1089/trgh.2017.0031 Verhoeven, M., Sawyer, M. G., & Spence, S. H. (2013). The factorial invariance of the CES-D during adolescence: Are symptom profiles for depression stable across gender and time? Journal of Adolescence, 36(1), 181-190. doi:10.1016/j.adolescence.2012.10.007 Vernon, S. W., & Roberts, R. E. (1981). Measuring nonspecific psychological distress and other dimensions of psychopathology. Archives of General Psychiatry, 38, 1239-1247 Vigil-Colet, A., Navarro-Gonzalez, D., & Morales-Vives, F. (2020). To reverse or to not reverse likert-type items: That is the question/Invertir o no invertir items tipo likert: Esa es la cuestion. Psicothema, 32(1), 108. doi:10.7334/psicothema2019.286 Visser, P. S., Krossnick, J. A., & Lavrakas, P. J. (2000). Survey Research. In Reis, H. T. & Judd, C. M. (Eds.) Handbook of research methods in social and personality psychology (pp. 223-252). United States: Cambridge University Press. Wakita, T., Ueshima, N., & Noguchi, H. (2012). Psychological distance between categories in the likert scale: Comparing different numbers of options. Educational and Psychological Measurement, 72(4), 533-546. Weijters, B., Cabooter, E., & Schillewaert, N. (2010). The effect of rating scale format on response styles: The number of response categories and response category labels. International Journal of Research in Marketing, 27(3), 236-247. 10.1016/j.ijresmar.2010.02.004 Weinrich, J. D. (2014). Multidimensional measurement of sexual orientation: Ideal. Journal of Bisexuality, 14(3-4), 544-556. doi:10.1080/15299716.2014.960958 Wellman, B. (1997). An electronic group is virtually a social network. In S.Kiesler (Ed.), Culture of the Internet (pp. 179-205). Mahwah, NJ : Lawrence Erlbaum. Weston, Kath. 1991. Families We Choose: Lesbians, Gays, Kinship. New York: Columbia University Press. Wetzel, E., & Greiff, S. (2018). The world beyond rating scales: Why we should think more carefully about the response format in questionnaires. European Journal of Psychological Assessment, 34(1), 1-5. doi:10.1027/1015-5759/a000469 Wetzel, E., Roberts, B., Fraley, R., & Brown, A. (2016). Equivalence of narcissistic personality inventory constructs and correlates across scoring approaches and response formats. Journal of Research in Personality, 61, 87-98. doi:10.1016/j.jrp.2015.12.002 Woodford, M. R., Kulick, A., Garvey, J. C., Sinco, B. R., & Hong, J. S. (2018). LGBTQ policies and resources on campus and the experiences and psychological well-being of sexual minority college students: Advancing research on structural inclusion. Psychology of Sexual Orientation and Gender Diversity, 5(4), 445-456. https://doi.org/10.1037/sgd0000289 Wright, K. B. (2005). Researching InternetâBased populations: Advantages and disadvantages of online survey research, online questionnaire authoring software packages, and web survey services. Journal of ComputerâMediated Communication, 10(3), 00-00. doi:10.1111/j.1083-6101.2005.tb00259.x Wu, A. D., Li, Z., & Zumbo, B. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment, 12(3), 1-26. 85 de Wulf, K. (1999). The role of the seller in enhancing buyer-seller relationships: Empirical studies in a retail context. Ghent University. Faculty of Economics and Business Administration, Ghent, Belgium. Zumbo, B. D. (2007a). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223-233. doi:10.1080/15434300701375832 Zumbo, B. D. (2007b). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics, Vol. 26: Psychometrics (pp. 45-79). Amsterdam,The Netherlands: Elsevier Science B.V. Zumbo, B. D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions and applications (pp. 65-82). Charlotte, NC: IAP â Information Age Publishing, Inc.. Zumbo, B. D. & Chan, E. K. H. (2014). Setting the Stage for Validity and Validation in Social, Behavioral, and Health Sciences: Trends in Validation Practices. In E. K. H. Chan & B. D. Zumbo (Eds.), Validity and Validation in Social, Behavioral, and Health Sciences (pp. 3-8). New York: Springer. Zumbo, B. D., Gadermann, A. M., & Zeisser, C.. (2007). Ordinal Versions of Coefficients Alpha and Theta For Likert Rating Scales. Journal of Modern Applied Statistical Methods, 6, 21-29. Zumbo, B. D., Gelin, M. N., & Hubley, A. M. (2002). The construction and use of psychological tests and measures. Encyclopedia of life support systems (EOLSS): Psychology theme (pp. 1-28). Oxford, UK: Eolss Publishers. http://www.eolss.net[entry,6.27.3.4] Zumbo, B.D., & Kroc, E. (2019). A measurement is a choice and Stevens' scales of measurement do not help make it: A Response to Chalmers. Educational and Psychological Measurement, 79, 1184-1197. Zumbo, B. D., Liu, Y., Wu, A. D., Shear, B. R., Olvera Astivia, O. L., & Ark, T. K. (2015). A methodology for Zumbo's third generation DIF analyses and the ecology of item responding. Language Assessment Quarterly, 12(1), 136-151. doi:10.1080/15434303.2014.972559 Zumbo, B.D. and Padilla, J. L. (2020). The Interplay Between Survey Research and Psychometrics, with a Focus on Validity Theory. In P. Beatty, D. Collins, L. Kaye, J.L. Padilla, G. Willis and A. Wilmot (Eds.), Advances in Questionnaire Design, Development, Evaluation and Testing (pp. 593-612). Hoboken, NJ: Wiley. doi:10.1002/9781119263685.ch24
Thesis/Dissertation
eng
Measurement, Evaluation and Research Methodology
Vancouver : University of British Columbia Library
Attribution-ShareAlike 4.0 International
Graduate
Impact of scoring and response format on inferences from electronic surveys for rating scales
Text