Education, Faculty of
Educational and Counselling Psychology, and Special Education (ECPS), Department of
DSpace
UBCV
Lynch, Sarah Elizabeth
Doctor of Philosophy - PhD
University of British Columbia
It has been suggested that the use of complex syntax in psychological test items may hinder some respondents’ comprehension of items (Graesser et al., 2000; Tourangeau et al., 2000). However, the prevalence and patterns of syntactic complexity features in psychological tests are not well understood, nor is their relationship with item responses. This two-part study addresses this underexplored area. Study 1 uses the Register Functional approach, which involves a corpus-based analysis, to identify the prevalence of one type of syntactic complexity, dependent clauses, in non-commercial psychological tests. It reveals nonfinite complement clauses—commonly used to express cognitive states, desires, stances, or probabilities (Biber et al., 2021)—as the most prevalent type. Study 2 then investigates the relationship between dependent clause types and item responses and subsequently examines any moderating effect of respondent’s vocabulary familiarity for associations found. Of the five dependent clause types analyzed, only nonfinite complement clauses showed a significant association with item responses, and this association was moderated by respondents’ vocabulary familiarity scores. Overall, these studies highlight the value of an interdisciplinary approach to measurement issues. The research emphasizes the importance of considering linguistic complexity in psychological test development, offering important insights into patterns of syntactic complexity in non-commercial psychological tests and the relationship between clausal complexity and response processes. By establishing foundational findings, this research sets the stage for future studies to explore the interplay between linguistic complexity and response processes, with the potential to inform test design and enhance validity and fairness for diverse populations.
UNPACKING THE GRAMMAR OF TESTS AND MEASURES: THE HIDDEN COMPLEXITY IN PSYCHOLOGICAL TEST ITEMS AND ITS ASSOCIATION WITH ITEM RESPONSES by SARAH ELIZABETH LYNCH B.A., St. Thomas University, 2004 B.Ed., The University of New Brunswick, 2006 M.A., Carleton University, 2012 A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Measurement, Evaluation, and Research Methodology) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) April 2025 © Sarah Elizabeth Lynch, 2025 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the dissertation entitled: Unpacking the Grammar of Tests and Measures: The Hidden Complexity in Psychological Test Items and Its Association with Item Responses submitted by Sarah Elizabeth Lynch in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Measurement, Evaluation, and Research Methodology Examining Committee: Dr. Bruno Zumbo, Professor and Distinguished University Scholar, Department of Educational and Counselling Psychology, and Special Education, UBC Supervisor Dr. Anita Hubley, Professor, Department of Educational and Counselling Psychology, and Special Education, UBC Supervisory Committee Member Dr. Anne Gadermann, Associate Professor, School of Population and Public Health, UBC Supervisory Committee Member Dr. Nancy Perry, Professor and Associate Dean, Research, Department of Educational and Counselling Psychology, and Special Education, UBC University Examiner Dr. Elizabeth Saewyc, Professor and Distinguished University Scholar, Director, School of Nursing, UBC University Examiner iii Abstract It has been suggested that the use of complex syntax in psychological test items may hinder some respondents’ comprehension of items (Graesser et al., 2000; Tourangeau et al., 2000). However, the prevalence and patterns of syntactic complexity features in psychological tests are not well understood, nor is their relationship with item responses. This two-part study addresses this underexplored area. Study 1 uses the Register Functional approach, which involves a corpus-based analysis, to identify the prevalence of one type of syntactic complexity, dependent clauses, in non-commercial psychological tests. It reveals nonfinite complement clauses—commonly used to express cognitive states, desires, stances, or probabilities (Biber et al., 2021)—as the most prevalent type. Study 2 then investigates the relationship between dependent clause types and item responses and subsequently examines any moderating effect of respondent’s vocabulary familiarity for associations found. Of the five dependent clause types analyzed, only nonfinite complement clauses showed a significant association with item responses, and this association was moderated by respondents’ vocabulary familiarity scores. Overall, these studies highlight the value of an interdisciplinary approach to measurement issues. The research emphasizes the importance of considering linguistic complexity in psychological test development, offering important insights into patterns of syntactic complexity in non-commercial psychological tests and the relationship between clausal complexity and response processes. By establishing foundational findings, this research sets the stage for future studies to explore the interplay between linguistic complexity and response processes, with the potential to inform test design and enhance validity and fairness for diverse populations. iv Lay Summary Psychological tests are tools used to measure people’s feelings, thoughts and behaviours. These tests are made up of individual items (questions or statements) designed to gather information about certain attributes, such as depression, motivation, or attitudes. Because test scores are used to make decisions about individuals or in research, it is important that these scores accurately reflect the attribute of interest and are not influenced by other factors. One factor that could affect test scores is complex grammar, which may make some items harder to understand, particularly for people with lower reading or language skills. In this two-part study, I explore one specific aspect of complex grammar, dependent clauses. I examine a) which types of dependent clauses are most common in psychological tests, b) whether these clause types relate to how people respond to items, and c) whether a person’s vocabulary familiarity affects this relationship. v Preface This dissertation is an original intellectual contribution of the author, Sarah Elizabeth Lynch. Dr. Bruno Zumbo provided invaluable expert guidance and feedback throughout the process, particularly on the development of the statistical models and analyses. The data used in this dissertation were sourced from two key repositories: the APA PsycTests database and the Open-Source Psychometrics Project (2019). Permission for research purposes was granted for all psychological tests from the APA PsycTests database used to create the corpus. The Depression Anxiety Stress Scales (DASS) data from the Open-Source Psychometrics Project are publicly available through their website. As these datasets were either publicly accessible or accompanied by the necessary permissions, no institutional ethics approval (BREB) was required for these studies. The texts in the corpus analyzed for this study were annotated by Dr. Randi Reppen and Dr. Douglas Biber at the Northern Arizona University Corpus Research Lab using their proprietary tools, the Biber Tagger (Biber, 1988; Biber et al., 1999) and Biber Complexity program (Biber & Gray, 2010). Subsequently, to address accuracy issues in the annotations, additional tagging was performed by a UBC research assistant, Daniel Jimenez, using the Stanford CoreNLP (Manning et al., 2014). The use of Generative AI in this thesis was limited to the use of ChatGPT (OpenAI) for checking sentences and paragraphs for correctness and clarity, and for resolving issues with R code. vi Table of Contents Abstract ......................................................................................................................................... iii Lay Summary ............................................................................................................................... iv Preface ............................................................................................................................................ v Table of Contents ......................................................................................................................... vi List of Tables ................................................................................................................................. ix List of Figures ................................................................................................................................ x Acknowledgements ...................................................................................................................... xi Chapter 1. Study Motivation and Purpose ................................................................................. 1 Chapter 2. Background ................................................................................................................ 6 The Register Functional Approach to Grammatical Complexity ........................................... 6 Corpus-Based Analysis ......................................................................................................... 12 Validity, Validation and Response Processes ........................................................................ 14 Mixed-Effects Modeling ....................................................................................................... 16 Chapter 3. Unpacking the grammar of tests and measures: The hidden complexity in psychological test items and its association with item responses ............................................ 18 The Register-Functional Approach ....................................................................................... 22 Literature Review.................................................................................................................. 25 Study 1...................................................................................................................................... 27 Research Purpose and Questions .......................................................................................... 27 Methods .................................................................................................................................... 28 vii Corpus Creation .................................................................................................................... 28 Corpus Analysis .................................................................................................................... 30 Results ...................................................................................................................................... 32 Corpus Description ............................................................................................................... 32 Patterns of Clausal Complexity ............................................................................................ 33 Discussion................................................................................................................................. 34 Study 2...................................................................................................................................... 35 Methods .................................................................................................................................... 36 Study Sample ........................................................................................................................ 36 Measures ............................................................................................................................... 38 Data Analyses........................................................................................................................ 41 Results ...................................................................................................................................... 49 Structure of the Random Effects ........................................................................................... 49 Proportional Odds Assumption ............................................................................................. 50 Dependent Clauses ................................................................................................................ 51 Discussion................................................................................................................................. 54 Conclusion ............................................................................................................................... 57 Chapter 4. Conclusion ................................................................................................................ 61 Review .................................................................................................................................. 61 Novel Contributions .............................................................................................................. 62 Limitations and Future Directions ........................................................................................ 64 References .................................................................................................................................... 66 viii Appendices ................................................................................................................................... 79 Appendix A: Calculation of Precision and Recall ................................................................ 79 Appendix B: Estimation of Corpus Sample Size .................................................................. 80 Appendix C: Tests Included in the Corpus ........................................................................... 81 Appendix D: DASS Codebook ............................................................................................. 91 Appendix E: DASS CFA Results and Descriptive Statistics ................................................ 95 Appendix F: DASS Coding of Dependent Clause Types ................................................... 100 Appendix G: Sample R Code .............................................................................................. 103 ix List of Tables Table 1 Grammatical Complexity Features in English ................................................................... 7 Table 2 Overview of Dependent Clauses in English..................................................................... 24 Table 3 Presence of Dependent Clauses across Instruments in Corpus ........................................ 33 Table 4 Demographic Characteristics of the Study and Full Samples .......................................... 38 Table 5 Model Summaries for Baseline Models ........................................................................... 50 Table 6 Model Summaries for Each Clause Type ......................................................................... 53 Table 7 Precision and Recall for Dependent Clauses ................................................................... 79 Table 8 Estimation of Sample Size Using Pilot Corpus ............................................................... 80 Table 9 Tests Included in the Corpus ............................................................................................ 81 Table 10 CFA Factor Loadings for DASS-42 ............................................................................... 95 Table 11 Distribution of Item Responses ...................................................................................... 98 Table 12 Presence/Absence Coding of DASS Items .................................................................. 100 x List of Figures Figure 1 Multilevel Data Structure for Persons and Items ........................................................... 16 Figure 2 Dependent Clauses by Syntactic Function ..................................................................... 34 Figure 3 Estimated Marginal Effect of Vocabulary ...................................................................... 54 xi Acknowledgements I am truly grateful to everyone who has supported me throughout my PhD journey. First and foremost, I would like to thank my supervisor and mentor, Dr. Bruno Zumbo, for his unwavering support, encouragement, and enthusiasm throughout my studies at UBC. I count myself extremely fortunate to have learned from you, worked with you, and shared many laughs along the way. I would also like to thank my dissertation committee members, Dr. Anita Hubley and Dr. Anne Gadermann, for their valuable feedback and guidance. Your constructive feedback strengthened my research, challenged my thinking, and helped me reflect deeply on this project. I am also grateful for the support and encouragement of my colleagues and classmates at UBC, whose friendship and collaboration helped me through challenging moments. I would also like to extend my gratitude to the UBC Paragon Research Initiative for their generous financial support during my studies, and to Dr. Randi Reppen, Dr. Douglas Biber, and Daniel Jimenez for their help with annotating the corpus used in this study. On a personal note, I would like to thank my family and friends for their love, encouragement, and understanding. This journey has taken up a large part of my life and I look forward to having more time to share with you soon! 1 Chapter 1. Study Motivation and Purpose Psychological tests are used to measure a variety of mental attributes and behaviours, including “attitudes, emotional functioning, intelligence and cognitive abilities..., aptitudes, values, interests, and personality characteristics” (VandenBos, 2015). The terms test, measure, and instrument are used synonymously throughout this dissertation to refer broadly to “a set of items, tasks or stimuli that are designed to elicit responses that describe or provide a sample of a respondents’ thoughts, feelings, or behaviors” (Hubley, 2021, p. 414). Scores from tests are used to gather information about people and make a variety of decisions. Ensuring that these inferences are appropriate is a matter of validity. This research integrates two complementary perspectives on validity: Messick’s (1989; 1995) unified model, which has been endorsed by the Standards for Educational and Psychological Testing (American Educational Research Association [AERA] et al., 2014), and Zumbo’s (2007; 2009; 2023) explanation-focused view. Based on Messick’s framework, the Standards define validity as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests” (AERA et al., 2014, p. 11). Zumbo defines validity as “the explanation for the test score variation” (2009, p. 69). From both perspectives, validation involves gathering evidence to support the inferences made from test scores. One major threat to validity arises when test scores reflect variance from factors unrelated to the attribute being measured, leading to inaccurate inferences. For instance, when respondents have low levels of literacy or proficiency in the target language, the linguistic features (i.e., vocabulary and grammar) of the items may introduce hidden invalidity. Therefore, test developers and test users must carefully consider the diversity of respondents’ language proficiency and literacy levels to ensure fair and valid interpretations of test scores. 2 This consideration is particularly relevant in countries like Canada and the United States, which are rich in linguistic diversity and have varying literacy levels. Recent census data reveal that around 26% of Canadians and just over 20% of Americans used a language other than English at home (Statistics Canada, 2022; United States Census Bureau, 2021). It is reasonable to assume that within this linguistic diversity lies variability in English proficiency. Canadian and American adults also vary in terms of their literacy levels. Results from the OECD’s Survey of Adult Skills indicate that roughly 20% of Canadian adults and 30% of American adults scored at or below the lowest literacy level (OECD, 2024). This variation in language and literacy in Canada and the US has important implications for researchers and other professionals in the psychological and social sciences, where psychological tests are a prevalent means of gathering information about individuals (Urbina, 2014). The inferences made about some individuals based on test scores may not be appropriate if the items are capturing something besides the attribute of interest, such as reading comprehension, grammar knowledge, and/or vocabulary knowledge. In other words, linguistic complexity in a test’s items may interfere with the measurement of the attribute of interest for respondents with low levels of literacy or language proficiency. In an effort to reduce sources of invalidity in tests, numerous guidelines on item writing have been published in the educational testing, psychological testing, and survey development literature. As an applied linguist and language test developer, the linguistic-related guidelines have always been of particular interest. Among these is the recommendation to avoid complex syntax (or grammar) in test items, with some experts specifically cautioning against the use of dependent clauses (e.g., Abedi, 2015; Dillman et al., 2014; Reynolds et al., 2021; Tourangeau et al., 2000). However, there does not appear to be much empirical research to support this recommendation. 3 My background in applied linguistics, second language teaching, and item writing provides me with a unique vantage point from which to evaluate these recommendations. From an applied linguist’s and item writer’s perspective, I recognize that dependent clauses are often necessary to fully convey experiences and ideas, and that not all dependent clauses are created equal—different types of dependent clauses serve different communicative functions. Therefore, this particular item writing guideline, though well-intentioned, seems unrealistic. This guideline also seems overly broad, as some types of dependent clauses in English are learned early in a person’s language acquisition, a pattern I have observed as an English language teacher. Investigating linguistic sources of invalidity is an area of research naturally suited to applied linguists, given our deep understanding of language structures, functions, and our motivations to address language-related problems. As Zumbo (2023) highlights, multidisciplinary approaches to assessment and validity research can expand our understanding of measurement by drawing on diverse expertise. Motivated by my interest in linguistic sources of invalidity, this research allows me to combine my two areas of expertise: applied linguistics and educational and psychological measurement. The goals of this research project are twofold. Accordingly, it is divided into two interrelated studies. The first study aims to identify the prevalent syntactic complexity features of psychological test items, specifically focusing on dependent clause types. This study uses the Register-Functional framework (Biber et al., 2022), which involves a corpus-based analysis, to identify the prevalent dependent clause types in a small, specialized corpus of non-commercial psychological tests. It uncovers the dependent clauses most frequently used in non-commercial psychological tests and examines the functions that they serve in the context of psychological testing. This research also sheds light on how item writers and test developers adhere to syntax- 4 related guidelines, and the practicality of these guidelines. The dependent clauses investigated in this study are then examined in the subsequent response process validation study. The second study investigates the relationship between item responses and the presence of five dependent clause types using data from the Depression, Anxiety Stress Scales (DASS; Lovibond & Lovibond, 1995). It then explores the moderating role of vocabulary familiarity on any associations identified. Generalized linear mixed effects models are used for the analysis, as they specifically deal with ordinal outcomes and random effects (Bono et al., 2021), making them suitable for hierarchical or multilevel data of the nature in this investigation. The findings offer preliminary insights into the relationship between syntactic complexity and response processes, with the potential to inform item writing guidelines and enhance validity and fairness for diverse populations. This dissertation is organized into four chapters. Chapter 1 (the current chapter) introduces the context and rationale for my two studies. Chapter 2 provides background on the theoretical frameworks adopted in these studies: Biber et al.’s (2022) Register Functional Approach, Messick’s (1989; 1995) unified model of validity and Zumbo’s (2007; 2009; 2023) explanation focused view of validity as well as response processes as validity evidence (Hubley & Zumbo, 2017). It also details the methods used in my two studies: corpus-based analysis and mixed-effects modeling. Chapter 3 presents the multi-study manuscript, of which the first study examines the grammatical complexity features in a corpus of non-commercial psychological tests, while the second study examines the relationship between these features and the item responses in an extant DASS dataset. Chapter 4 summarizes the key conclusions from these research studies, highlights the novel contributions to educational and psychological measurement, and outlines directions for future research. Because this dissertation follows a 5 manuscript style format, some of the content in the chapters is intentionally repetitive to align with the standalone nature of the manuscript. 6 Chapter 2. Background This chapter describes the theoretical frameworks that underpin this research project: Biber et al.’s (2022) Register Functional (RF) approach, Messick’s unified model of validity (1989; 1995) and Zumbo’s explanation-focused view of validity (2007; 2009; 2023). It also outlines the methods used in the two studies: corpus-based analysis and mixed-effects modeling. The Register Functional Approach to Grammatical Complexity It has been suggested in the survey design and psychometrics literature that one reason a respondent may misunderstand a test item is due to complex syntax (or grammar) (Graesser et al., 2000; Tourangeau et al., 2000). In alignment with the terminology used in the RF approach, the term syntax will hereafter be replaced with grammar. Complex grammar can place a burden on a respondent’s ability to process a sentence (Norman et al., 1992; Tourangeau et al., 2000), and in turn can affect their response and score. Some item writing guidelines recommend avoiding complex grammar, specifically by using simple sentence structures and avoiding dependent clauses (Dillman et al., 2014; Reynolds et al., 2021; Tourangeau et al., 2000). However, this description of grammatical complexity is quite narrow, and it assumes that the various types of dependent clauses are equally complex. This research project uses Biber et al.’s (2022) contemporary, empirically informed framework of grammatical complexity, the Register Functional (RF) approach, which broadens the conceptualization of grammatical complexity to include the various forms (finite dependent clause, nonfinite dependent clause, dependent phrase) and functions (adverbial, complement, noun modifier [relative], or other phrasal modifier) of dependent clauses and phrases. As Biber et al. (2022) highlight, not all dependent clauses and phrases are equally complex; their structural forms and syntactic functions vary, and this variation should be recognized. Table 1 provides a 7 detailed overview of the various grammatical complexity features (underlined) in terms of their forms and functions. Table 1 Grammatical Complexity Features in English Form and function Example Finite adverbial clause “If I could live my life over, I would change almost nothing.” (SWLS; Diener et al., 1985, p. 72) Finite complement clause “I feel that I have a number of good qualities.” (RSE; Rosenberg, 1965, p. 17) Finite relative clause “I was bothered by things that usually don't bother me.” (CES-D; Radloff, 1977, p. 387) Nonfinite adverbial clause “I've been taking action to try to make the situation better.” (Brief-COPE; Carver, 1997, p. 96) Nonfinite complement clause “I've been trying to find comfort in my religion or spiritual beliefs.” (Brief-COPE; Carver, 1997, p. 96) Nonfinite relative clause “It is easy for me to feel the pain (and joy) experienced by my loved ones.” (CLS; Sprecher & Fehr, 2005, p. 649) Adverb phrase as adverbial “All in all, I am inclined to feel that I am a failure.” (RSE; Rosenberg, 1965, p. 17) Prepositional phrase as adverbial “In most ways my life is close to my ideal.” (SWLS; Diener et al., 1985, p. 72) Attributive adjective as noun premodifier positive attitude, emotional support, social groups Noun as noun premodifier human condition, group membership Prepositional phrase as noun postmodifier “I spend a lot of time concerned about the well-being of those people close to me.” (CLS; Sprecher & Fehr, 2005, p. 649) Adverb as adjective premodifier relatively rare, so upset Prepositional phrase as adjective postmodifier “On the whole, I am satisfied with myself.” (RSE; Rosenberg, 1965, p. 17) To accommodate readers with differing levels of familiarity, clarification of key linguistic terms is provided. 8 Dependent Clauses One way that the grammatical complexity of a sentence can be increased is by adding dependent clauses. A dependent clause (also referred to as a subordinate or embedded clause) is a clause that is an incomplete idea on its own and is connected to a main clause. For example, the following item from the Satisfaction with Life Scale (SWLS; Diener et al., 1985) begins with a dependent clause: “If I could live my life over, I would change almost nothing” (p. 72). Dependent clauses vary in structural form, that is they can be finite or nonfinite. A finite dependent clause contains a verb phrase that indicates tense or modality (e.g., had, could go), such as “could live” in the aforementioned example from the SWLS. A nonfinite dependent clause contains a verb that does not indicate tense or modality, typically in the form of a to-clause, ing-clause, or ed- clause. For example, the following item from the Brief-COPE (Carver, 1997) contains a nonfinite to-clause: “I've been trying to find comfort in my religion or spiritual beliefs” (p. 96). Dependent clauses also vary in terms of syntactic functions within each structural form (i.e., adverbial, complement, and noun modifier [relative] clause). Dependent Phrases Another way in which grammar can be made more complex is by adding dependent phrases. While not features of interest in this research, dependent phrases are part of the RF framework and will be mentioned in the dissertation; thus, they warrant description. A phrase is a word or group of words that functions as a unit in a clause. The five main types of phrases in English are noun phrases, verb phrases, adjective phrases, adverb phrases, and prepositional phrases. In its simplest form, a phrase consists of one word (i.e., the head). For example, the following item from the Flourishing Scale (Diener et al., 2010), “People respect me” (p. 155) contains two simple noun phrases consisting of one head word each: “people” and “me”. 9 However, oftentimes phrases and clauses are modified with additional phrases in order to clarify or expand their meaning. These additional dependent phrases may be located before the head word (i.e., pre-modifiers), or after the head word (i.e., post-modifiers). A good example is the following item from the SWLS “The conditions of my life are excellent” (Diener et al., 1985, p. 72); it contains a head noun “conditions” that is post-modified with the prepositional phrase “of my life” in order to specify the meaning of “conditions”. The RF approach is grounded in the theory of functional grammar and the concept of register. Functional grammar emphasizes how language serves as a resource for making meaning, highlighting that writers or speakers have choices regarding the linguistic features they use to convey meaning (Halliday & Matthiessen, 2014). A register is “a language variety associated with a particular situation of use” (Biber & Conrad, 2019, p. 31); each field or discipline has distinct registers, which are characterized by linguistic features suited to the context of communication. In any given situation, the range of available choices that a person has to communicate their intended meaning is limited, and these choices are influenced by contextual characteristics of the register such as communicative purpose, mode of communication, audience, and the relationship between participants. For example, the following item from the Brief COPE (Carver, 1997) uses an adverbial clause to indicate the reason for a behavior: “I've been using alcohol or other drugs to make myself feel better” (p. 96). Alternatively, the same idea could have been conveyed using a prepositional phrase: I've been using alcohol or other drugs for mood improvement. The item writer had options to communicate their intended meaning. The original item uses a dependent clause that communicates a personal and explicit reason for the behavior, making it 10 more appropriate for informal situations where the goal is to express personal feelings or emotions. In contrast, the alternative version uses a dependent phrase that is more concise and abstract, offering a more formal and impersonal tone better suited to an academic or professional audience. These examples show that, by choosing between a clause and a phrase, the item writer can tailor the level of formality, detail, and emotional engagement according to the audience and purpose of the communication. Applying the RF approach to psychological tests (which are designed to measure feelings, thoughts, behaviors) suggests that certain linguistic features are prevalent because they effectively serve the needs of this context. The features most frequently used in this register are those that item writers choose because they are well suited for achieving the intended communicative goals. The prevalent linguistic features of a register are typically identified via corpus linguistics methods that use computers to search and analyze a collection of natural texts, called a corpus (Biber & Conrad, 2019). Problems with Traditional Conceptualizations of Grammatical Complexity Biber et al. (2022) argue that traditional conceptualizations of grammatical complexity have typically equated it to structural elaboration and emphasized the addition of dependent clauses; that is, the longer the structure and the more dependent clauses, the greater the grammatical complexity. However, their empirical corpus-based research conducted over the past 25 years provides contradictory evidence. In their large-scale corpus-based studies on spoken and written registers in English, Biber (1988) and Biber et al. (1999; 2021) found that spoken conversations tend to be quite structurally elaborated, often using numerous dependent clauses, while academic research writing tends to use more densely packed phrases than elaborated clauses. This finding can be more clearly illustrated with examples. Example 1 below presents an 11 excerpt from a spoken conversation. Spoken conversations take place in real time, leaving the speakers with little time to plan their utterance, and potentially needing to edit or clarify what they have said. This often results in elaborate sentence structures with multiple dependent clauses, as depicted in Example 1. The dependent clauses have been separated by a vertical bar | to illustrate the clausal elaboration. Example 1 “The trouble is | if you’re the only one in the house | he follows you | and you’re looking for him | so you can’t find him. | I thought | I wonder | where the hell he’s gone | I mean | he was immediately behind me.” (Biber et al., 2021, p. 1063) Example 2 below shows an excerpt from an academic research article. In contrast to spoken texts, written texts are typically planned, and writers have time to edit and revise their ideas. In academic writing, this often results in fewer clauses and denser noun phrases (Biber et al., 2022). Example 2 contains only one finite clause and extensive modification at the phrase level. The dependent phrases have been underlined to illustrate the complex phrasal elaboration. Example 2 “Despite the abnormal morphogenesis observed in such grafts, the range of differentiated tissues formed in such an ‘experimental teratoma’ can be used to provide an estimate of the developmental potential of the transferred tissue.” (Biber et al., 2021, p. 1063) Due to these findings, Biber et al. (2022) argue that since spoken conversation is a less complex register than academic writing, a reconceptualization of grammatical complexity is required. Based on the evidence from corpus-bases studies, they have operationally defined grammatical complexity as complexity that “varies with respect to three major underlying parameters: 1) the different structural types of complexity (e.g., finite versus nonfinite dependent clauses), and 2) 12 the different syntactic functions served by complexity features (e.g., adverbial versus noun-modifying) 3) the different functional patterns of register variation associated with each complexity feature.” (Biber et al., 2022, p. 4). This conceptualization of grammatical complexity challenges the notion that the most complex structures are those that are longest and include dependent clauses. Moreover, it is based on the full range of grammatical structures, including types of phrases, types of nonfinite and finite clauses, and emphasizes the need to examine each structural type on its own (Biber et al., 2022). The RF approach diverges from earlier frameworks in two important ways (Biber et al. 2022). Firstly, unlike the traditional theory-driven conceptualizations, the RF conceptualization of grammatical complexity was arrived at inductively based on the findings of empirical corpus-based research. Secondly, the RF approach acknowledges that grammatical structures can be added in various ways, each representing a distinct type of grammatical complexity. Hence, this approach is unique in that it recognizes the “full range of structural devices and syntactic functions that result in grammatically complex phrases and clauses” (Biber et al, 2022, p. 23). It also recognizes that the syntactic function of certain grammatical structures makes a clause more or less complex (rather than the traditional notion of clause length). Corpus-Based Analysis A corpus-based analysis is used in Study 1 to analyze the patterns of grammatical complexity features, specifically the types of dependent clauses, in the register of non-commercial psychological tests. This analytical approach, rooted in corpus linguistics, investigates “the actual patterns of language use, investigating the highly systematic patterns that structure our everyday use of grammatical features in speech and writing” (Biber, 2006, p. 204). The overarching goal of corpus-based analysis is to produce generalizable descriptions of 13 language used within a specific domain, which requires the corpus to be as representative of the target domain as possible (Egbert et al., 2022). Corpus-based analysis can be used to examine a number of aspects of language use. It can reveal patterns of use of various linguistic features such as words, multi-word units, or grammatical structures; it can also provide insights into broader textual characteristics, including registers and genres (O’Keefe & McCarthy, 2022). Widely used in fields such as applied linguistics and language teaching, corpus-based analysis provides the means for understanding language in context. The central methods of corpus linguistics are divided into (i) identifying or building the corpus and (ii) analysis of the corpus data. A key consideration in the first stage is whether an appropriate corpus already exists; if not, one must be created. When creating a corpus, researchers must carefully account for domain and distribution factors to ensure representativeness, balancing theoretical goals with practical constraints (Biber, 1993; Egbert et al., 2022). This process begins with defining and operationalizing the domain, followed by sampling texts that adequately capture the domain’s variability (Egbert et al., 2022). The practical steps of corpus creation include identifying and collecting texts, preparing the data by converting the texts into plain text (.txt) format, and analyzing the texts using specialized tools. Popular tools for corpus analysis include open-source programs like AntConc, Sketch Engine, and Python libraries (e.g., NLTK, spaCy), as well as proprietary tools like the Biber Tagger or Complexity Tool. These tools support a range of analyses, such as extracting keywords, identifying multi-word units and collocations, and investigating grammatical structures. In the analysis stage, researchers calculate normalized rates of occurrence for the linguistic features across texts to account for differences in text length. Normalization allows for 14 meaningful comparisons and helps identify patterns of language use across the corpus. This approach ensures that the findings are both systematic and reflective of the linguistic characteristics of the target register. The corpus-based analysis method aligns well with the objectives of Study 1, as it enables the identification of dependent clauses that serve the communicative purposes within the register of non-commercial psychological tests. By applying this approach, Study 1 uncovers the specific types of dependent clauses most commonly used in this register. Additionally, this analysis provides valuable insights into the practicality and applicability of clause-specific item-writing guidelines, offering a clearer understanding of their relevance to the language used in psychological tests. Validity, Validation and Response Processes This research project adopts two complementary approaches to validity: Messick’s unified model of validity (1989; 1995), endorsed by the Standards for Educational and Psychological Testing (AERA et al., 2014), and Zumbo’s explanation-focused view of validity (2007, 2009, 2023). These frameworks offer distinct but related perspectives for examining the relationship between the linguistic features of test items and item responses. From Messick’s unified model, validity is all about the construct and inferences made based on test scores. The Standards (AERA et al., 2014) defines validity as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests” (p. 11), and validation as the process of gathering evidence to support the test score interpretations. One source of validity evidence proposed by the Standards is response processes, which they refer to as the “cognitive processes engaged in by test takers” (p. 15). Hubley and Zumbo (2017) expand this definition to include not only cognition, but also the motivations, actions and emotions 15 involved, describing response processes as “the mechanisms that underlie what people do, think, or feel when interacting with, and responding to, the item or task and are responsible for generating observed test score variation” (p. 2). Zumbo’s (2009) explanation-focused view frames validity as “the explanation for the test score variation” (p. 69) and validation as “the process of developing and testing the explanation” (p. 69). The central idea is that understanding variation in item responses and test scores can help bridge the inferential leap involved when making inferences about measured attributes based on test scores (Zumbo, 2009). Study 2 investigates validity evidence based on response processes by modeling item responses and exploring whether grammatical complexity features are associated with item responding. These findings have implications for test score interpretations. Hubley (2021) underscores the significance of response process as validity evidence, noting that it addresses the core validity question: “to what degree does evidence and theory support the intended interpretation of test scores given the proposed test use?” (p. 421). Thus, examining whether the grammatical complexity features of test items introduce hidden invalidity is important for supporting test score interpretations. From the perspective of Messick’s unified model, Study 2 examines whether the grammatical complexity features in test items introduce construct-irrelevant variance, that is variance stemming from factors other than the construct of interest (Messick, 1989, 1995; AERA et al., 2014). From Zumbo’s explanation-focused lens, this study models item responses to determine whether grammatical complexity features act as confounds that undermine validity. Both frameworks agree that the linguistic features of items should not impact item responses, 16 and together they offer complementary insights into how these features may compromise validity. By integrating Messick’s and Zumbo’s frameworks, this study provides evidence of how grammatical complexity features may introduce hidden invalidity into test responses. This integration enhances our understanding of their role as potential confounds and advances validity research by addressing their implications for test score interpretations. Mixed-Effects Modeling Mixed-effects modeling is a statistical approach designed to analyze data where observations are not independent (Garson, 2014). This issue often arises when variables are clustered—for example, when children are nested within schools or items are nested within respondents. In such cases, the assumption of independence is violated, as illustrated in Figure 1, where items (1–3) are clustered within respondents. Figure 1 Multilevel Data Structure for Persons and Items 17 Traditional methods like ANOVA or OLS regression assume independence among observations, and violating this assumption can result in inflated Type I error rates (Peugh, 2010). Mixed-effects models overcome this limitation by accounting for clustering in the data. Moreover, these models enable generalizations to larger populations when predictor variables are random samples from those populations, as the inclusion of random effects captures variability across clusters (Garson, 2014). In Study 2, mixed-effects modeling is used to explore the association between dependent clauses and item responses. This approach is well-suited for data that includes both fixed and random effects. Fixed effects represent variables that do not vary systematically across observations. For example, in Study 2, the type(s) of dependent clauses present in test items were treated as fixed effects. Random effects capture variability associated with clusters in the data, such as individual differences among respondents or variations among test items. Study 2 uses mixed-effects modeling with crossed random effects for respondents and items, a common approach in psycholinguistic research. In crossed designs, each participant responds to each item, and each item is responded to by each participant (Quené & van den Bergh, 2008). Including random effects for both participants and items ensures that the findings can be generalized beyond the specific sample of respondents and test items (Baayen et al., 2008; Judd et al., 2012; Quené & van den Bergh, 2008). Given the ordinal nature of the dependent variable in Study 2 (the 4-point response scale from the DASS), generalized linear mixed-effects models (GLMMs) were employed. GLMMs are versatile, as they support various link functions and allow for the modeling of non-continuous dependent variables, such as binary or ordinal data (Garson, 2014). This flexibility made GLMMs an appropriate choice for the analysis. 18 Chapter 3. Unpacking the grammar of tests and measures: The hidden complexity in psychological test items and its association with item responses Even a cursory glance through the contents of the APA PsycTests database of psychological tests created for social and behavioral science research (American Psychological Association, 2015) shows that these tests and measures come in a variety of forms and lengths. They can be administered individually or in groups, through surveys or questionnaires, and are utilized to measure a broad range of psychological attributes. The terms test, measure, and instrument are used synonymously throughout this paper to refer broadly to “a set of items, tasks or stimuli that are designed to elicit responses that describe or provide a sample of a respondents’ thoughts, feelings, or behaviors” (Hubley, 2021, p. 414). Although they possess diverse attributes, all psychological tests and measures share the essential feature of being composed of one or more items, tasks, or questions that require an individual's response. Put simply, these items serve as the fundamental building block of tests and measures, with the responses to these items being scored to draw conclusions about the respondent or as dependent variables or covariates in subsequent research. Because item-level flaws can undermine later phases of test development and validation, it is essential that each item clearly represents the intended attribute and is understood by respondents in the way intended by the test developer. Since language is the foundation of these test items, a deeper understanding of the linguistic features (i.e., grammar and vocabulary) and their relationship with response processes is essential, yet remains underexplored. It is widely recognized that items in tests that are administered on paper or electronically inherently measure reading to some degree, as a certain level of reading ability is required to understand and respond to the items. However, if the linguistic features (vocabulary and 19 grammar) of a test interferes with respondents’ understanding of the test items, the inferences made based on those test scores may not be valid. This research integrates two complementary perspectives on validity: Messick’s (1989; 1995) unified model, endorsed by the Standards for Educational and Psychological Testing (American Educational Research Association [AERA] et al., 2014), and Zumbo’s (2007; 2009; 2023) explanation-focused view. Based on Messick’s framework, the Standards define validity as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests” (AERA et al., 2014, p. 11). Zumbo (2023) defines validity as “the explanation for the test score variation” (p. 1). From both perspectives, validation involves gathering evidence to support the inferences made from test scores. For respondents with low levels of reading ability or limited proficiency in the target language, the language of the test items may introduce a source of invalidity, that is, the test scores could reflect factors unrelated to the attribute being measured. Consequently, inferences based on these test scores may not be appropriate if the items are capturing something unintended, such as literacy or language ability. This ‘hidden invalidity’ may significantly undermine measurement validity, especially for diverse populations with considerable sociolinguistic variation, adding complexity to the item response process (Zumbo, 2023). For example, in Canada and the United States, linguistic and literacy diversity is notable, with around 26% of Canadians and over 20% of Americans using a language other than English at home (Statistics Canada, 2022; United States Census Bureau, 2021), while roughly 20% of Canadian and 30% of American adults score at or below the lowest literacy level on the Survey of Adult Skills (OECD, 2024). These variations underscore the importance of addressing potential linguistic sources of item invalidity in psychological and social science research, where 20 psychological tests are a prevalent means of gathering information about individuals or groups (Urbina, 2014). In the survey development literature, one linguistic feature believed to have the potential to interfere with a respondents’ understanding of a test item is complex syntax (Graesser et al., 2000; Tourangeau et al., 2000). Syntax refers to “the structure of sentences through the arrangement of words into phrases, phrases into clauses, and clauses into sentences” (Aarts, 2014, Syntax). Syntax can be made more complex by adding optional elements at the clause or phrase level. Syntactic complexity in test items may impact some respondents’ response processes, “the mechanisms that underlie what people do, think, or feel when interacting with, and responding to, the item or task and are responsible for generating observed test score variation” (Hubley & Zumbo, 2017, p. 2). As Hubley (2021) underscores, response processes are a source of validity evidence that get “to the heart of the validity question: to what degree does evidence and theory support the intended interpretation of test scores given the proposed test use?” (p. 421). The validity of any score interpretation hinges on the assumption that a respondent interacted with the items as intended by the test developer, and that the items are tapping into the attribute of interest. However, since a certain level of literacy or language proficiency is required to understand and respond to items in instruments administered in writing, the syntactic complexity of the items may impact the response processes of some respondents. To minimize sources of invalidity in psychological tests, numerous item-writing guidelines have been published, primarily in the survey development literature and less frequently in the psychological testing literature. While some item-writing experts advocate for the use of simple sentences, which are those containing only a main clause and excluding 21 dependent clauses (e.g., Dillman et al., 2014; Reynolds et al., 2021; Tourangeau et al., 2000), others suggest minimizing the number of clauses overall (e.g., Peterson, 2013). A dependent clause, sometimes referred to as a subordinate or embedded clause, is an incomplete idea on its own and relies on a main clause to convey meaning. For example, the following item from the Satisfaction with Life Scale (Diener et al., 1985) demonstrates a complex sentence: “If I could live my life over, I would change almost nothing” (p. 72). This sentence contains a dependent clause (italicized) preceding the main clause. Another related recommendation in many item writing guides is to write items at a middle school reading level (e.g., Johnson & Morgan, 2016; Peterson, 2013), which is typically evaluated using a readability formula such as the Flesh-Kincaid Grade Level index or the Gunning Fog index. However, readability formulas (which rely on factors such as word frequency, word length, and sentence length) are not necessarily accurate predictors of language difficulty and often favour survey items considered more linguistically complex (Lenzner, 2014). Linguistic sources of invalidity present an area of research particularly well-suited to applied linguists. These language experts, driven by a desire to solve "real-world language-based problems" (Grabe, 2010, p. 35), are uniquely positioned to analyze and address issues related to the language used in psychological tests. Their expertise can uncover specific linguistic features that may contribute to invalidity, such as ambiguous phrasing or complex grammar. From an applied linguistics perspective, there are several concerns with the aforementioned item writing guidelines. One issue is that they conflate the various syntactic structures and functions of dependent clauses. Dependent clauses come in different forms and serve different syntactic functions in a sentence, and, as Biber et al. (2022) highlight, not all types of dependent clauses should be assumed to be equally complex. Additionally, the readability guidelines imply that 22 longer sentences are more syntactically complex, a notion contradicted by corpus research (Biber et al., 1999, 2021) and survey research (Lenzner, 2014), which show that shorter sentences can also exhibit significant complexity, particularly when noun phrases contain a lot of embedding, meaning they include other phrases or clauses nested within them. Furthermore, research in writing studies has associated different grammatical complexity features with different stages of literacy development (Larsson et al., 2023; Staples et al., 2023), implying that dependent clause structures acquired earlier in one’s literacy development may be more easily understood. Finally, these syntactic guidelines may be impractical, as constructing test items exclusively with simple sentences is unrealistic; complex sentences are often necessary to convey logical links between ideas and experiences (Eggins, 2004). In order to develop comprehensible test items, a more nuanced understanding of the relationship between dependent clauses and item responding is essential. The Register-Functional Approach A promising lens through which to gain a richer understanding of the prevalent syntactic complexity features of psychological test items is the Register Functional (RF) framework (Biber et al., 2022). In alignment with the terminology used in the RF approach, the term syntactic complexity will hereafter be replaced with grammatical complexity. The RF approach is grounded in the theory of functional grammar and the concept of register. Functional grammar emphasizes how language serves as a resource for making meaning, highlighting that writers or speakers have choices regarding the linguistic features they use to convey meaning (Halliday & Matthiessen, 2014). In any given situation, the range of available choices is limited, and these choices are influenced by contextual factors such as communicative purpose, mode of communication, audience, and the relationship between participants. For 23 example, the following item from the Brief COPE (Carver, 1997) uses an adverbial clause to indicate the reason for a behavior: “I've been using alcohol or other drugs to make myself feel better.” (p. 96) Alternatively, the same idea could have been conveyed using a prepositional phrase: I've been using alcohol or other drugs for mood improvement. The item writer had options to communicate their intended meaning. The original item uses a dependent clause that communicates a personal and explicit reason for the behavior, making it more appropriate for informal situations where the goal is to express personal feelings or emotions. In contrast, the alternative version uses a dependent phrase that is more concise and abstract, offering a more formal and impersonal tone better suited to an academic or professional audience. These examples show that, by choosing between a clause and a phrase, the item writer can tailor the level of formality, abstraction, and emotional engagement according to the audience and purpose of the communication. A register can be defined as “a language variety associated with a particular situation of use” (Biber & Conrad, 2019, p. 31); each field or discipline has distinct registers, which are characterized by linguistic features suited to the context. Applying the RF approach to psychological tests (which are designed to measuring feelings, thoughts, and behaviors) suggests that certain linguistic features are prevalent because they effectively serve the needs of this context. The features most frequently used in this register are those that item writers choose because they are well suited to the intended communicative goals. The RF approach expands the understanding of grammatical complexity by differentiating between various forms (finite dependent clauses, nonfinite dependent clauses, 24 dependent phrases) and functions (adverbials, complements, noun-modifying relatives) of dependent clauses. As Biber et al. (2022) highlight, not all dependent clauses are equally complex; their structural forms and syntactic functions vary, and this variation should be recognized. Table 2 provides a detailed overview of the functions and forms of dependent clauses. Table 2 Overview of Dependent Clauses in English Function Description Form Example Adverbial Adds circumstantial information (e.g., time, place, manner, reason, or condition) to the main idea in the sentence (Biber et al., 2021) Finite “When I am working, I forget everything else around me.” (UWES; Schaufeli & Bakker, 2004, p. 48) Nonfinite “I've been taking action to try to make the situation better.” (Brief-COPE; Carver, 1997, p. 96) Complement Completes the meaning of a verb, adjective or noun in the main part of the sentence (Biber et al., 2021) Finite “I feel that I have a number of good qualities.” (RSE; Rosenberg, 1965, p. 17) Nonfinite “I've been trying to find comfort in my religion or spiritual beliefs.” (Brief-COPE; Carver, 1997, p. 96) Relative Defines or specifies a noun (finite), or provides extra, but not essential, details about a noun (nonfinite) (Biber et al., 2021; Humphrey et al., 2012) Finite “I was bothered by things that usually don't bother me.” (CES-D; Radloff, 1977, p. 387) Nonfinite “It is easy for me to feel the pain (and joy) experienced by my loved ones.” (CLS; Sprecher & Fehr, 2005, p. 649) Biber et al. (2022) argue that traditional views of grammatical complexity, which typically equate it with elaborated structures and embedded dependent clauses, are outdated. Their empirical, corpus-based studies show that spoken conversations tend to be structurally elaborated with many dependent clauses, whereas academic writing (a more complex register) tends to contain densely embedded phrases (Biber et al., 1999, 2021). Due to these findings, they have proposed a reconceptualization of grammatical complexity that emphasizes the need to take 25 into account both grammatical form and syntactic function. This new framework disputes the idea that the most complex structures are longer sentences containing dependent clauses and emphasizes the need to consider the full range of grammatical structures individually. Thus, the RF approach offers a fresh perspective for examining the clausal complexity in psychological test items and assessing the relevance of item writing guidelines. Literature Review A review of the literature suggests that few, if any, studies have investigated the grammatical complexity of psychological test items. Graesser et al. (2000) built a corpus of survey items by the U.S. Bureau of Census in order to develop a software tool (QUAID) that helps improve the “wording, syntax, and semantics” (p. 254) of survey items. This was based on prior research (Graesser et al., 1996) that identified common problems with survey questions, including several linguistic features believed to impede respondents’ comprehension, one of which was complex syntax. However, their research aims were not to describe census survey items as a register, thus the prevalent linguistic features of the survey items are not described. Beyond this study, there do not appear to be any published corpus-based register analyses of psychological test items; thus, their prevalent linguistic features are not well understood. Other studies have examined the relationship between the grammatical complexity of survey items and response processes. Yan and Tourangeau (2008) investigated several factors believed to be related to online survey item response times, one being item complexity, which was operationalized as the number of words per clause, and the number of clauses per item. They observed an association between response times and item complexity, with items containing more words per clause and more clauses requiring longer response times. Lenzner et al. (2010) investigated whether several linguistic features of online survey items administered in German, 26 one being complex syntax, increased respondents’ cognitive burden, as indicated by response times. These researchers operationalized complex syntax as consisting of “ambiguous structures,” “propositionally dense sentences,” or “left-embedded sentences” (p. 1007). They found that each of the aforementioned complexity features significantly increased response times, and, thus, the cognitive burden on respondents. Lenzner (2012) later examined how the clarity of survey questions affects the quality of responses. He administered two versions of a survey in German: one with questions that were difficult to understand due to certain problematic linguistic features (including complex syntax), and another with clearer questions that avoided those issues. Response quality was evaluated by analyzing the frequency of non-substantive and neutral responses. He found that respondents who took the less comprehensible survey version provided significantly more non-substantive and neutral responses than those who took the more comprehensible version. This prior research suggests that the grammatical complexity of psychological test items may impact response processes. However, prior studies have defined or operationalized grammatical complexity differently than the RF approach, and the prevalence of the various grammatical complexity features in the registers was unknown. Given the varying definitions and operationalizations of grammatical complexity in previous studies, and the lack of detailed knowledge about the prevalence of these features in psychological tests, further research is needed. The research presented here consists of two interconnected studies that examine grammatical complexity in psychological test items. The first study identifies prevalent dependent clause types in non-commercial psychological tests and evaluates their functions and adherence to syntax-related item-writing guidelines. The second study investigates how five 27 types of dependent clauses relate to test item responses, using data from a widely used psychological measure (the DASS) included in the first study while also considering the moderating role of vocabulary familiarity. Such research could have important implications for the validity of inferences drawn from psychological test scores. Study 1 Research Purpose and Questions The RF approach allows for the exploration of grammatical complexity in psychological test items through corpus-based register analysis. This approach involves describing the key linguistic features and situational context of a register, as well as explaining the functional relationship between them (Biber & Conrad, 2019). By employing the methods of computational corpus linguistics, which use computational algorithms and tools to analyze large databases of natural texts, register analysis makes language use patterns explicit (Biber & Conrad, 2019). A corpus-based register analysis can reveal how selected linguistic features are functionally adapted to the context of psychological testing. In the context of this study, it can help reveal the dependent clause types most frequently used to convey meaning, organize information, and fulfil an item writer’s intent. Moreover, understanding these features can shed light on how item writers and test developers adhere to syntax-related guidelines, as well as evaluate the practicality of these guidelines given the communicative purpose and context of the register. Additionally, the insights into grammatical complexity will inform the subsequent response process validation study. The goal of this first study is to explore psychological test items as a register by answering the following research questions: 28 1. Which dependent clause types are most prevalent in the register of non-commercial, non-cognitive psychological tests, designed for adults and administered in writing? 2. What functions do the prevalent dependent clause types serve in the context of this register? Methods The central methods of corpus linguistics are divided into (i) identifying a suitable corpus or building one if none exists, and (ii) analysing the corpus data. Corpus Creation A small, specialized corpus of psychological tests was built to address the research questions because, to our knowledge, no such corpus existed. In the RF approach, complete texts are typically analyzed as this permits the exploration of variation across texts (Biber et al., 2022). In this study, each text is defined as the complete set of test items from a given psychological test. Analyzing individual test items separately would not be appropriate as the items collectively function as a cohesive text with the specific purpose of measuring one or more attributes. Moreover, corpus analysis involves calculating normalized rates of occurrence in order to account for differences in text length and ensure accurate comparison of features across texts (Biber et al., 2012). To describe the predominant linguistic features within the register, the domain of non-commercial psychological tests was analyzed to ensure the corpus included a representative range of tests from the target population. This required describing the domain, operationalizing it, and sampling texts (Egbert et al., 2022). The domain was defined as non-commercial psychological tests, which are tests typically available for free in academic publications such as 29 journals or dissertations, with author permission often granted for their use (Urbina, 2014). The domain boundaries were further narrowed to non-cognitive tests (those measuring thoughts, feelings and behaviours), administered in writing (on paper or electronically), and designed for adult (18+) populations. Next, the operational domain was specified, which involves identifying a suitable set of texts available for sampling. The APA PsycTests database was selected as it is a reputable source containing over 75,000 primarily non-commercial psychological tests, and is regularly updated by APA staff with entries from dissertations, journal articles, books, and websites (American Psychological Association, 2015). Thus, this collection of psychological tests was determined to be the best approximation of the broader domain that was available for our research purposes. The database was filtered to identify the operational domain of interest: non-cognitive tests, which were filtered by instrument type (index/indicators, inventories/questionnaires, rating scales, screeners, surveys, and tests), available for research purposes, written in English, designed for adult (18+ years old) populations, administered by paper or electronically and published between 1972 and 2022. To avoid duplication of items, the search was limited to original measures, which excluded short forms and translations, and also excluded tests labeled as "Model", as these contained items from existing instruments. This process yielded a sampling frame of approximately 7,100 tests identified in a November 2023 search of the PsycTests database. From this sampling frame, a random sample of 200 tests was drawn. A simple random sample was chosen over a stratified random sample due to the APA PsycTests’ labelling of instrument types. Neither the APA Thesaurus of Psychological Index Terms (Tuleya & American Psychological Association, 2005) nor the APA Dictionary of Psychology (VandenBos, 2015) 30 clearly define or distinguish between the labels of instrument types used in the database, which limits the possibility of meaningful categorization. The files were then prepared for corpus analysis. The randomly selected tests were downloaded as PDF files, and their items were transferred to .docx files for screening. Tests not administered in English, knowledge tests, or those with single-word items were excluded, and additional tests were randomly sampled to replace them. Details for each test were recorded in an MS Excel spreadsheet, including instrument type, test name, grammatical unit of items (e.g., declarative sentence, interrogative sentence, dependent clause, phrase), number of items, and word count. Any instructions or response options were removed, minor typos were corrected, and blanks within items were filled with appropriate words to facilitate software annotation. The files were then converted to plain text (.txt) format for compatibility with corpus analysis tools. Corpus Analysis The analysis of the corpus data was an iterative process that required considerable linguistic and computational linguistic expertise. First, the formatted texts were sent to the Northern Arizona University Corpus Research Lab where they were annotated (or tagged) using their proprietary software tools. The Biber Tagger program (Biber, 1988; Biber et al., 1999) was used to annotate the tests for part of speech (POS), then the Biber Complexity program (Biber & Gray, 2010) was used to annotate the grammatical complexity features based on the POS tagging. The normalized rates of occurrence per 100 words were calculated as shown in Equation (1) (Gray, 2015, p. 52). These normalized rates standardize the frequency of grammatical complexity features across tests of varying lengths, facilitating the identification of patterns across different texts. 31 Normalized rate of occurrence =𝑟𝑎𝑤 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 𝑜𝑓 𝐺𝐶 𝑓𝑒𝑎𝑡𝑢𝑟𝑒 𝑖𝑛 𝑇𝑒𝑠𝑡 𝐴𝑡𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑤𝑜𝑟𝑑𝑠 𝑖𝑛 𝑇𝑒𝑠𝑡 𝐴 × 100 (1) The accuracy of the annotations was then evaluated since the reliability of annotation tools should not be assumed (Biber et al., 2022). A subsample of five annotated tests from the corpus was assessed based on two accuracy measures: precision, “the degree to which an identification is correct”, and recall, “the extent to which a tag captures all instances of a feature” (Biber et al., 2022, p. 35). Details of these calculations are provided in Appendix A. Grammatical complexity annotations from the Biber Complexity program were verified by comparing them to those from the Berkeley Neural Parser (Berkeley NLP Group, 2019), an online tool that identifies and annotates dependent clauses and phrases. Discrepancies were resolved using the author’s linguistic expertise and a grammar reference text, Grammar of spoken and written English (Biber et al., 2021). Features with precision or recall below 0.90 were flagged for accuracy remediation based on thresholds from prior research (Gray, 2015; Staples et al., 2016). Most grammatical complexity features did not meet this threshold, indicating the need for thorough review and correction of annotations. As a result, a research assistant was enlisted to annotate the 200 corpus files using the open-source Stanford CoreNLP tool (Manning et al., 2014), effectively providing a second round of coding. The Stanford CoreNLP tool was selected for its functional annotations, which closely align with those of the Biber tools, although it lacks the same level of granularity and form-specific tagging. Discrepancies between the Biber and Stanford annotations were again resolved using the author’s expertise and the same grammar reference. This iterative process ensured a high degree of accuracy in the grammatical complexity annotations. Due to the need to review and correct annotations across all tests, it was no longer feasible to analyze all 200 randomly sampled tests as originally intended. A smaller sample was 32 needed to refine the annotations. Therefore, the minimum required sample size was estimated using Equation (2) below, 𝑛 =𝑠2(0.5 ∗ 𝐶𝐼 𝑟𝑎𝑛𝑔𝑒𝑡)2 , (2) where n denotes the required sample size, s denotes the standard deviation, CI the confidence interval, and t represents the t-statistic (Egbert et al., 2022, p. 130). Following Egbert et al.’s (2022) recommendation to use a sample size greater than 30, 32 tests were randomly selected from the annotated sample. Any errors in the grammatical complexity feature annotations were examined and corrected for these tests, and the revised data were used to determine the necessary sample size for each feature. The calculations of the corpus sample size are provided in Appendix B. Although the recommended minimum sample size was approximately 30 tests, in the final step of this stage of the corpus analysis, we opted to create a corpus of 100 tests to enhance representativeness while still being practical. A list of tests included in the corpus is provided in Appendix C. Results Corpus Description The specialized corpus of non-commercial psychological tests created for this study consisted of 100 instruments and included a total of 2,160 items and 24,556 words. Table 3 details the number of tests in the corpus that include each clause type. 33 Table 3 Presence of Dependent Clauses across Instruments in Corpus Dependent Clause Type Instruments Per Clause Type (n) Finite adverbial clauses 71 Finite complement clauses 76 Finite relative clauses 67 Nonfinite adverbial clauses 52 Nonfinite complement clauses 86 Nonfinite relative clauses 28 Note: Most instruments contain multiple dependent clause types, so the total number of instruments per clause type exceeds the number of instruments in the corpus. Patterns of Clausal Complexity Figure 2 displays the frequency of dependent clauses based on their syntactic functions. As illustrated in Figure 2, the most frequently used dependent clause in the corpus is the nonfinite complement clause, appearing on average 2.7 times per 100 words (SD = 2.3). This is followed by finite complement clauses, which occur at a rate of 1.5 times per 100 words (SD = 1.6), and finite adverbial clauses, which occur at a rate of 1.3 times per 100 words (SD = 1.4). Notably, the standard deviations for all clause types other than nonfinite complement clauses slightly exceed their respective means, suggesting a considerable degree of variability in how frequently these features are used across tests. Therefore, while the usage of dependent clause types varies significantly across tests, the nonfinite complement clause shows the most consistent pattern of use in this register. 34 Figure 2 Dependent Clauses by Syntactic Function Discussion The aim of this study was twofold: to identify the prevalent dependent clause types in a corpus of non-commercial psychological tests and to understand the functions these features serve within this specific register. The findings reveal nonfinite complement clauses to be the most commonly used dependent clause type in the corpus. These clauses are typically used to express cognitive and perceptual states, efforts, desires, stance, probabilities, or other actions (Biber et al., 2021). Their prevalence in the corpus of psychological test items aligns with the purpose of this register: to elicit responses about an individual's behavior, feelings or thoughts. The following example items from the corpus demonstrate nonfinite complement clauses that respectively convey behaviour (effort), thought (cognition), and feeling (stance): 0.01.02.03.04.0FiniteAdverbialClausesNonfiniteAdverbialClausesFiniteComplementClausesNonfiniteComplementClausesFinite RelativeClausesNonfiniteRelativeClausesFrequency per 100 words 35 “You tried to show that your partner was wrong” (Conflict Regulation Questionnaire; Darnon & Butera, 2007, p. 148) “I believe in making a responsible and well-considered decision” (Utilitarian Meaning and Piecemeal Judgment Scale; Allen & Ng, 1999, p. 25) “I couldn’t seem to experience any positive feeling at all” (DASS; Lovibond & Lovibond, 1995, p. 339). This finding aligns with previous corpus studies which have also found nonfinite complement clauses to be more common in written registers compared to other dependent clause types (Biber et al., 2021; Gray, 2015). Interestingly, finite complement clauses, which also serve to report a person’s attitudes, emotions, and thoughts, are less prevalent in the corpus, for instance: “I feel that I am a positive influence to my coworkers” (Teng et al., 2007, p. 50). This suggests that item writers may avoid such dependent clauses, opting instead for a more direct statement of the emotion, feeling, attitude or thought, such as “I am a positive influence to my coworkers”. This shift toward direct expression may reflect a preference for clarity and simplicity in communication within the corpus. Study 2 As previously mentioned, the existing literature suggests that certain clausal complexity features may impact item responses for some test takers; however, the relationship between these features and item responses is not well understood. Thus, the goal of this second study is to answer the following research questions: 1. Is there an association between item responses and the presence of the various types of dependent clauses? 36 2. If an association is found, does a respondent’s vocabulary familiarity moderate it? Methods Study Sample For this study, we used publicly available extant data from the Open-Source Psychometrics Project (2019). The data were filtered to include only Canadian and US adults (age 18+) who submitted a single survey from their network and provided complete and valid responses (described in the Vocabulary Familiarity subsection below). From the 4808 eligible respondents in the full sample who met our selection criteria, a study sample of 500 was randomly selected for analysis. The sample size of 500 was selected because the full sample of 4808 respondents exceeds the computational limits of the ordinal R package (Christensen, 2023) that was used for data analysis, which encounters memory constraints during maximum likelihood estimations for large datasets. Even in cases where computation is possible, processing time for each model can take hours, making analysis impractical. The minimum sufficient sample size was determined by balancing the maximum number of participants the ordinal package can handle for complex models and the necessary level-2 sample size for a two-level ordinal multilevel model. For the most complex cases, at least 100 level-2 units are recommended to ensure reliable estimation and convergence, with more complex models potentially requiring 100-150 level-2 units (Hox et al., 2012; Maas & Hox, 2005; Snijders & Bosker, 2012). This sample size of 500 was sufficient to meet these requirements while maintaining statistical rigor. Full descriptive statistics for the study and full samples are provided in Table 4. The study sample had a mean age of 31.0 (SD = 12.7), 63% were female, 48% were university educated, 37 and 89% indicated that English was their native language. The vast majority of respondents were from the US (92%) and identified as White (80%). Effect sizes were computed for the difference between the study and full samples for each demographic variable reported in Table 4 and interpreted descriptively, focusing on demographic variables exceeding Cohen’s (1988, 1992) small-effect threshold (e.g., h = 0.2 for proportions). Direct hypothesis testing was avoided because the study sample is a subset of the full sample, which would inflate the test statistic and produce an artificially low p-value (Zimmerman et al., 1993). The full sample’s mean age is 30.5 (SD = 13.0). Using the Statistics Kingdom (2017) website, Cohen’s d was calculated as 0.039, indicating a small effect size. Since a paired t-test was not feasible, an independent samples t-test was used, acknowledging that this approach inflates d (Zimmerman et al., 1993). While the true d is slightly smaller, it still supports age similarity between the study and full samples. In addition, Cohen’s h for proportions was computed using the same website to determine whether differences exceeded the small-effect threshold (h = 0.2). For example, 63% of the study sample and 64% of the full sample are female. The Cohen’s h for this proportion difference (0.63 and 0.64) is 0.021, which falls below the small-effect threshold. According to Zimmerman et al. (1993), this observed h represents an upper bound, as the actual value is smaller when accounting for the subset relationship. Overall, all demographic variables for the study sample fell below a small effect size compared to the full sample, providing strong evidence of their equivalence. 38 Table 4 Demographic Characteristics of the Study and Full Samples Study Sample Full Sample M (SD) n % M (SD) n % Effect Size Age 31.0 (12.7) 500 30.5 (13.0) 4808 0.039 Gender Female 316 63 3094 64 0.021 Male 169 34 1559 32 0.043 Other 15 3 155 3 0.000 Country of Residence Canada 41 8 587 12 0.134 USA 459 92 4221 88 0.134 Native Language English 446 89 4344 90 0.033 Other 54 11 464 10 0.033 Race White 400 80 3664 76 0.097 Asian 41 8 400 8 0.000 Other 32 6 469 10 0.149 Black 14 3 187 4 0.055 Native American 13 3 67 1 0.148 Arab 0 0 21 0 0.000 Education Level Less than high school 14 3 159 3 0.000 High school 248 50 2367 49 0.020 University degree 168 34 1608 33 0.021 Graduate degree 70 14 674 14 0.000 Measures The DASS was selected as the focus of this study because it is widely used, is included in the APA PsycTests database, and, therefore, is included in Study 1, resulting in the calculated distribution of dependent clause types across its items. Moreover, a dataset using the DASS was available from the Open-Source Psychometrics (2019) database of psychological tests and included variables suitable for constructing an index of vocabulary familiarity. 39 Depression Anxiety Stress Scales (DASS) The data was collected using the DASS (Lovibond & Lovibond, 1995), a psychological test available in the public domain. The DASS codebook for the Open-Source Psychometrics (2019) dataset is provided in Appendix D. The DASS consists of three self-report sub-scales – depression, anxiety and stress – each containing 14 items, designed to measure their respective states. Respondents rate the extent to which each statement applied to them over the past week on a 4-point scale ranging from 0 (did not apply to me at all) to 3 (applied to me very much, or most of the time). Responses were recoded from 1- 4 to facilitate the data analysis. Composite scores for the DASS are calculated by summing the item responses. Although originally conceptualized as having three moderately correlated factors (Lovibond & Lovibond, 1995), recent evidence from a systematic review and meta-analytic factor analysis of the DASS by Yeung et al. (2020) suggests that bifactor models (with one general factor and three specific factors reflecting the sub-scales, all four of which are uncorrelated) consistently provide the best fit for the DASS data. Therefore, the DASS was analyzed using a bifactor model. A confirmatory factor analysis (CFA) was conducted in R (R Core Team, 2024) with the lavaan package (Rosseel, 2012) fitting one general factor and three specific factors reflecting the sub-scales, all four of which were uncorrelated, to a polychoric correlation matrix of the ordinal item responses. The analysis used robust diagonally weighted least squares estimation, and demonstrated acceptable model fit (CFI = .95, TLI = .95, RMSEA = .05, SRMR = .05). These results are consistent with findings from other studies examining the DASS’s internal structure (Ciobanu et al., 2018; Gomez, 2013; Marshall et al., 2018; Yeung et al., 2020). The factor loadings, which ranged from 0.36 to 0.82, are provided in Appendix E. The fit of the bifactor model, with its 40 general factor, and the fact that all but one of the factor loadings were greater than 0.5, is sufficient evidence to support using an overall score. A reliability analysis was conducted for the overall scale, with Cronbach’s alpha internal consistency reliabilities estimated using the psych R package (Revelle, 2024). Consistent with prior DASS studies based on normative adult samples, the overall scale showed a reliability estimate of .96, aligning with previous findings ranging from .97 to .99 (Crawford & Henry, 2003; Marshall et al., 2018). Therefore, for the purposes of this study, given the psychometric dimensionality and reliability evidence, the DASS was treated as a unidimensional scale with a single overall negative emotional state score. Descriptive statistics on the distribution of DASS item responses for the study sample and the distribution of dependent clause types across DASS items (see Study 1) are also provided in Appendix E. Vocabulary Familiarity The data set also included a vocabulary measure designed to assess respondents' familiarity with English words (see Appendix D). Respondents were asked to select all the words whose definitions they were sure they knew. The test consisted of 16 words: 13 actual words and 3 fake words. These fake words were included to identify inattentive responses. A total vocabulary score of the 13 real words was calculated for each respondent. The words on the measure represented a range of frequencies, with some being high frequency (e.g., boat, decide, funny) and others being low frequency (e.g., abysmal, epistemology). Word frequency was evaluated using an online Vocabulary Profiler tool (Smith, 2024) that analyzes word occurrences in two large-scale English corpora, the BNC and COCA, which are comprehensive databases of English language usage across various contexts. This test also served as a validity check. Respondents who claimed to know two or more fake words (n = 328) were removed from the 41 sample as this suggested inattentiveness. Furthermore, those who failed to recognize the three high-frequency words (n = 429), which were among the 1000 most frequently used English words, were removed from the sample, as such responses indicated insufficient English proficiency to meaningfully engage with the DASS instructions and items, or inattentive responding. Three remaining outliers on the vocabulary score variable were identified using the interquartile range (IQR) approach, where values exceeding Q1 – 1.5 x IQR or Q3 + 1.5 x IQR were flagged and removed, resulting in a final full sample of 4808 respondents. Since vocabulary knowledge, which encompasses both breadth and depth, is a strong predictor of reading ability and language proficiency (Jeon & Yamashita, 2014; Qian & Lin, 2019; Sarbazi et al., 2021), this vocabulary recognition score can be considered the most reliable available proxy for literacy or language proficiency in interpreting the current study’s results. Data Analyses We used mixed-effects modeling with crossed random effects for subjects and items for the analysis. This approach is common in psycholinguistic research, where clusters are often described as crossed—each participant responds to each item, and each item is responded to by each participant (Quené & van den Bergh, 2008). By including random effects for both participants and items, these models allow for the generalization of findings beyond the specific sample of respondents and test items (Baayen et al., 2008; Judd et al., 2012; Quené & van den Bergh, 2008). Given the study’s crossed design and the ordinal nature of the dependent variable, generalized linear mixed effects models were used, specifically cumulative link mixed models (CLMMs). As Tutz (2022) notes, numerous models and software options are available today, but a standard nomenclature has yet to be established. The term cumulative link models was introduced by Agresti (2002) and is used by Christensen (2023) in the ordinal R package, which 42 was used for our analysis. However, this class of models appears under various names in the literature, including ordinal regression, ordered logit models, and probit models (Greene & Hensher, 2010), depending on the link function. The broader term generalized linear models is also commonly applied. CLMMs are designed to handle ordinal outcomes and random effects, making them well-suited for hierarchical or multilevel data of the nature in our investigation. The description of CLMM is provided within the context of our research setting, which involved a two-level hierarchical data structure. At Level 1, the data comprised 42 items for each participant, with each row of the data file corresponding to an item and its associated level-1 covariate and explanatory variables. At level 2, the data consisted of 500 respondents, with 42 rows nested within each respondent. This structure resulted in a dataset with 21,000 rows (42 × 500), organized in the long format. The outcome (dependent) variable for the statistical model used to investigate the two research questions is the responses to the 42 items of the DASS, each measured on a 4-point response scale. The DASS reflects the extent to which respondents experienced negative emotional states—specifically depression, anxiety, and stress—over the past week. Each item’s four response levels form an ordinal variable with ordered categories. These ordinal regression models are often described using a “threshold concept” (Bock, 1975). This approach assumes that a continuous latent variable Y𝑖𝑗∗ underlies the observed ordinal response 𝑌𝑖𝑗, where 𝑖 denotes items, 𝑖 = 1,2. … , 42, and 𝑗 denotes persons, 𝑗 = 1,2, … 500. Typically, the continuous latent variable is assumed to follow either a normal distribution, with a mean of zero and variance of one, or a logistic distribution, with a mean of zero and variance of 𝜋23⁄ , leading to probit or logit link functions, respectively. Under the threshold concept, the observed ordinal outcome 𝑌𝑖𝑗 = 𝑐 (for the DASS, 𝑐 = 1,2,3,4) is determined by the latent 43 variable falling within specific thresholds: 𝑡𝑐−1|𝑐 ∶= 𝑡𝑐−1 ≤ 𝑌𝑖𝑗 ≤ 𝑡𝑐, with 𝑡0 = −∞ and 𝑡𝑐 = ∞. These thresholds are the properties of items, meaning that each of the 42 items has three thresholds that need to be estimated during model fitting. Importantly, the thresholds need not be identical across items, and for any given item, the psychological distance between thresholds is unknown and need not be equal. In this study, the CLMM formulation of the generalized linear mixed effects model with probit link was fitted with Laplace approximation. This approach aligns with the ordinal extension of the item response theory model (Tutz, 1990) and reflects a multilevel formulation in which item parameters are fixed, person abilities are random, and Bock’s (1975) threshold concept serves as a plausible interpretation. The threshold concept links observed response categories to an underlying latent response tendency for that category, a common and meaningful framework in psychometric models. In the present study, the class of cumulative models assumes that the observed ordinal variable is the responses to the DASS items. The 𝑌𝑖𝑗, originates from the categorization of a latent (not observable) continuous variable, Y𝑖𝑗∗ . More specifically, if we considered item i=1, which elicits the response to the statement, I found myself getting upset by quite trivial things, then the latent continuum, Y𝑖𝑗∗ , reflects the extent to which respondent j experienced this negative emotional state in the past week. To model this categorization item response process, the cumulative model assumes that there is one less threshold than response options, which partition the latent continuous variable into observable, ordered categories of the dependent variable. In this example of the DASS, there are three thresholds (𝑡1|2, 𝑡2|3, and 𝑡3|4). That is, the 1|2 threshold, 𝑡1|2, is the first threshold separating response options 1 and 2 for item 1 on the latent psychological continuum of the extent to which respondent j experienced the negative emotional 44 state in the past week. The same interpretation applies to the two remaining thresholds (𝑡2|3, and 𝑡3|4), separating the 2 to 3 and 3 to 4 responses: the 2|3 and 3|4 cut points. Following best practices, we assessed the need for a mixed-effects model, built the level-1 model, built the level-2 models, reported effect sizes, and used likelihood ratio tests to compare competing models (Heck & Thomas, 2020; Peugh, 2010). Given that item responses naturally vary across respondents, the need for mixed-effects modeling was evident but nonetheless evaluated. To do so, the intraclass correlation (ICC) and the design effect statistics were calculated from the baseline model. The baseline model includes no predictors and partitions variance into variance attributable to level-2 units (e.g., persons) and level-1 units (e.g., items). An ICC value greater than 0 indicates that a portion of the variability is explained by the level-2 grouping factor, suggesting the need for multilevel modeling (Peugh, 2010). A design effect > 2.0 further confirms the need for a multilevel model (Muthén & Satorra, 1995). In order to compute this ICC, the baseline model for random person effects is specified as follows: Level 1: Yij∗ = 𝑏0𝑗 + 𝑒𝑖𝑗 (3) Level 2: 𝑏0𝑗 = 𝛾00 + 𝑢𝑜𝑗 (4) Combined: Yij∗ = 𝛾00 + 𝑢𝑜𝑗 + 𝑒𝑖𝑗. (5) In Equation (3), the probit transformed item response for item i for person j (Yij∗) is expressed as a function of the mean response for person j (b0j) plus a residual error term (eij) that captures the variability not captured by the person-level random effects. The probit transformation implies that eij follows a standard normal distribution (mean 0, variance 1). In Equation (4), the person-specific mean response (b0j) is modeled as a function of the overall grand mean (γ00) and a random deviation from the grand mean for person j (u0j), where u0j is normally distributed with a 45 mean of zero and variance denoted var(u0j) that is estimated in the process of fitting Equation (5). Combining these two levels, we arrive at the baseline model shown in Equation (5), where the overall response is expressed as the sum of the grand mean, the person-specific random effect, and the residual error term—the main purpose of Equation (5) is to compute the ICC with the random person effect. Three points are noteworthy in the description of the statistical models. First, although the statistical models include three thresholds (𝑡1|2, 𝑡2|3, and 𝑡3|4), as part of the probit-transformed dependent variable (the item response 𝑌𝑖𝑗∗ for person j on item i) they are expressed with 𝑌𝑖𝑗∗ as the outcome to emphasize its role. Second, a key constraint in the CLMM variant of the generalized linear mixed-effects model specification in Equation (5) is that the thresholds (𝑡1|2, 𝑡2|3, and 𝑡3|4) and the intercept γ00 are not jointly identifiable, implying that all four parameters cannot be estimated when fitting the model. This issue is usually addressed by either setting one of the thresholds or the intercept to a pre-specified value (usually zero) to avoid estimating it. The default approach in the analysis software (the CLMM function in the ordinal package) estimates all the thresholds and sets the intercept to zero so that it is not estimated. One of the thresholds (usually the first one) does double duty as both a threshold and, if needed, for interpretational purposes, stands in for the intercept. Therefore, the statistical model in Equation (5) has four parameters (model df=4) that were estimated: the three thresholds (𝑡1|2, 𝑡2|3, and 𝑡3|4), and the random intercept capturing between-person variability, also called the random effect of persons (u0j). Following conventions in psycholinguistics regarding crossed random effects (Baayen et al., 2008; Quené & van den Bergh, 2008), a random intercept was included for both persons and items to account for the variability across both, as shown in Equation (6) below, 46 Y𝑖𝑗∗ = 𝑢0𝑗 + 𝑣𝑖0 + 𝑒𝑖𝑗 . (6) Equation (6) is an extension of Equation (5) with the constraint that 𝛾00=0, which includes the unique contribution of each test item 𝑣𝑖0 and its corresponding random effects variance (𝑣𝑖0), and a residual error term (eij) that captures the variance not explained by the person-level or item-level random effects. Again, the distribution of 𝑒𝑖𝑗 in Equation (6) was set as standard normal, implying the probit link function for the response variable, 𝑌𝑖𝑗∗ . Finally, together (𝑢0𝑗 + 𝑣𝑖0 + 𝑒𝑖𝑗) are the random structure of Equation (6), where an implication of the unique contribution of the random effect of test items is that 𝑢0𝑗 and 𝑣𝑖0 are independent and hence uncorrelated, and because it is the level-1 residual, the 𝑒𝑖𝑗 random effect is uncorrelated with the other random terms in the model. This model accounts for the crossed structure of persons and items, where each person responds to all items, and each item is responded to by all persons, ensuring that variability is properly attributed to both sources. The statistical model in Equation (6) represents how both respondents (people) and items independently contribute to differences in the ordinal response and has five parameters (model df=5) that were estimated: the three thresholds (𝑡1|2, 𝑡2|3, and 𝑡3|4); the random intercept capturing between-person variability, also called the random effect of persons (u0j); and the random intercept capturing the between-item variability, also called the random effect of items (v𝑖0). To address the first research question, a main effects model was initially fit. This model treated the dependent clause type as a level-1 predictor in order to examine its association with item responses. If a significant main effect of clause type was found, an interaction effects model was subsequently fit. This second model included the vocabulary total score as a level-2 predictor in order to investigate whether vocabulary familiarity moderates the relationship 47 between item responses and the dependent clause type. Notably, the random effect of persons can be interpreted as a proxy for each test respondent’s total test score. For the main effects and interaction effects model, the random effects of persons, therefore, can be thought of as a test score covariate adjusting (controlling) for the level of the attribute being measured by the DASS. A two-level main effects model was built to examine the relationship between the predictor of interest, dependent clause type, and item responses. The level-1 predictor variable was group-mean centered as group-mean centering (subtracting the cluster-specific mean) highlights within-cluster (level-1) effects and prevents confounding with between-cluster (level-2) variation. All dependent clause types were group-mean centered, following recommendations for examining interactions between variables (Peugh, 2010). A model was fit with a random intercept and slope for each dependent clause type (DC) to account for variation across persons, as shown in Equation (7) below, Level 1: Yij∗ = 𝑏0𝑗 + 𝑏1𝑗𝐷𝐶𝑖𝑗 + 𝑒𝑖𝑗 (item) Level 2: 𝑏0𝑗 = 𝛾00 + 𝑣𝑖0 + 𝑢0𝑗 (person) 𝑏1𝑗 = 𝛾10 + 𝑢1𝑗 (person) Combined: Yij∗ = 𝛾10𝐷𝐶𝑖𝑗 + (𝑢0𝑗 + 𝑢1𝑗𝐷𝐶𝑖𝑗) + 𝑣𝑖0 + 𝑒𝑖𝑗. (7) In Equation (7), the fixed effect of the group-mean centered dependent clause type (DC), 𝛾10, is the average slope reflecting the relationship between 𝐷𝐶𝑖𝑗 and the item response for that person across all 500 persons in the study. More precisely, 𝛾10, is the average slope reflecting the relationship between presence/absence of the DC and the score (z-value) on the latent psychological continuum of the extent to which that respondent experienced the negative emotional state in the past week for that person across all 500 persons in the study. 48 This fixed effect is the key parameter of interest for the first research question. Extending the random effects structure of Equation (5) and Equation (6), the model in Equation (7) has two independent groupings of random effects: (a) two random effects that allow for person-specific deviations in both intercepts (𝑢𝑜𝑗) and slopes (𝑢1𝑗) of the relationship between the items’ responses and the DC of interest in the model, and (b) the random intercept capturing the between-item variability (random effect of items) denoted 𝑣𝑖0 shown in Equation (6). Regarding the number of parameters estimated for the model in Equation (7), there were three thresholds (𝑡1|2, 𝑡2|3, and 𝑡3|4), one fixed effect 𝛾10, three variances for the random effects of person intercept, person slope, and item intercept, plus one random effect covariance (correlation) between the person intercept and slope for a total of eight parameters; that is, model df=8. If a significant relationship was identified in the main effects model (7), an interaction effects model was fit by adding the level-2 predictor variable, vocabulary score, to examine its potential moderating effect on the relationship between item responses and the dependent clause type. This level-2 predictor variable was grand-mean centered, meaning the overall mean across all clusters was subtracted from each individual score (Peugh, 2010). This process ensures that the predictor reflects differences at the between-cluster (i.e. between person) level, while also supporting the interpretation of model coefficients by making the intercept correspond to the average predictor value. Because we were interested in the interaction of the level-2 predictor and the level-1 predictor, the level-2 predictor was added to both the level-2 intercept and level-2 slope equations, as shown in Equation (8) below, Level 1: (item) Level 2: (person) (person) 49 Combined: . (8) The statistical model represented in Equation (8) retains the same fixed and random effects structure as Equation (7), with the addition of two fixed effects: the main effect of vocabulary familiarity ( ) and its cross-level interaction with the item-level dependent clause (DC) of interest ( ). Therefore, 10 parameters were estimated (model df=10) for the model in Equation (8). Based on research question two, the key parameter of interest is , which quantifies the degree to which the respondent’s vocabulary familiarity moderates the association between the item response (i.e., the extent of the negative emotional state experienced in the past week) and the presence of the dependent clause type. In addition, this model provides a standard error of , supports statistical testing, and informs the moderation pattern for simple effects testing of statistically significant interactions. Results This section presents the results of the main effects models used to analyze the association between item responses and each type of dependent clause individually. A significant association is further investigated using an interaction effects model. Model assumptions are addressed, and a post-hoc analysis is conducted for the interaction effects model to provide additional insights. Structure of the Random Effects Table 5 displays the baseline model summaries. Equation (5) includes a random effect of persons, and the ICC indicated that 42% of the item response variation occurred across persons. Moreover, the design effect of 18 exceeded the suggested threshold of 2.0 (Muthén & Satorra, 1995), both confirming the need for mixed-effects modeling. Following conventions in 50 psycholinguistic research, an independent random effect of items was added. Equation (6) shows the random effect of persons and items, with the ICC of this model showing that 54% of the item response variation occurred across both persons and items. Equation (6) fit the data best, as indicated by the likelihood ratio test, and the lower AIC value. Table 5 Model Summaries for Baseline Models Equation (5) (Random Persons) Equation (6) (Random Persons & Items) Random Effects Residual var(eij) 1.00 1.00 Intercept Persons var(u0j) 0.74 0.92 Intercept Items var(vi0) 0.24 ICC 0.42 0.54 Evaluation Design effect 18.40 23.00 Marginal R2 / Conditional R2 0.00 / 0.43 0.00 / 0.54 AIC 49777 46561 log-Likelihood -24885 -23275 Likelihood ratio test comparing Equations (3) and (4) χ2(df =1) = 3218, p < .001 Proportional Odds Assumption In a cumulative link mixed model (CLMM), the proportional odds assumption means that a predictor has the same effect on cumulative outcome probabilities across all thresholds. In other words, each predictor is described by just one coefficient. This assumption was tested by comparing a proportional odds model to a non-proportional model (with a main effects model that includes the DC and Vocabulary score) using a likelihood ratio test (LRT) (Harrell, 2022). Effect sizes quantifying the differences in threshold for non-proportional models were assessed to evaluate practical significance. The proportional odds assumption is supported for all the 51 dependent clause types. In each case, the LRT was statistically non-significant, or if statistically significant, the effect sizes were small, supporting the proportional odds assumption. Dependent Clauses Table 6 presents the results of the main effects models fit for each clause type. As shown, there is no significant direct association between item responses and the presence of finite adverbial, finite relative, nonfinite relative and finite complement clauses, as indicated by their non-significant p-values. This finding suggests that these clause types are not associated with item responses. For finite relative clauses, the correlation between the random intercept and slope for person was 1.00, indicating an overparameterized random structure. As a result, a more restricted model was fit without a correlation to reduce redundancy (as presented in Table 6), but the overall conclusion remained unchanged. However, for nonfinite complement clauses, a significant direct association is observed (b = 0.469, p < .001). Specifically, nonfinite complement clauses are associated with a .469 standard deviation increase in the latent response variable, a near-medium effect by Cohen’s standards (Cohen’s d effects: small ≥ .20, medium ≥ .50, large ≥ .80; Cohen, 1988, 1992). Respondents are more likely to select higher response options when nonfinite complement clauses are present, suggesting that nonfinite complement clauses are associated with a greater tendency for respondents to report more negative emotional states over the past week. Furthermore, the results of fitting the model in Equation (8) are: model evaluation goodness-of-fit supported adequate model fit with LRT (df =7) = 10,958, p < .001, conditional R2 = 0.542, AIC = 46499 and log-Likelihood = -23240. There was a statistically significant direct effect of the presence of nonfinite complement clauses (b = 0.456, p < .001), significant vocabulary score covariate (b = -0.105, p <.001), and a statistically significant interaction 52 between nonfinite complement clauses and vocabulary scores (b = 0.036, p < .001), marginal R2 of 0.040. This interaction model suggests that the direct effect of the presence of nonfinite complement clauses continues to result in respondents tending to report more negative emotional states; however, in this case, by 0.456 standard deviations of the latent tendency scale, which continues to be a near-medium effect by Cohen’s d standards. Furthermore, for every one-unit increase in vocabulary score, the presence of nonfinite complement clauses resulted in an additional .036 standard deviations (a less than small effect by Cohen’s standards for d) increase in the underlying item response tendency, meaning that although the vocabulary score was negatively related to the latent item response tendency, higher vocabulary scores may magnify the positive effect of nonfinite complement clauses on item responses. Notably, the difference in marginal R2 of the interaction model in Equation (8) compared to the direct effects model in Equation (7) is 0.015 (i.e., 0.040 – 0.025), which is further evidence for a less than small effect by Cohen’s standards (Cohen’s R2 effects: small ≥ .02, medium ≥ .13, large ≥ .26; Cohen, 1988, 1992) for the change in accounted-for variance with the additional fixed effects terms in Equation (8) for the covariate and interaction. 53 Table 6 Model Summaries for Each Clause Type Finite Adverbial Finite Relative Nonfinite Relative Finite complement Nonfinite complement Fixed Effects Direct association 0.035 -0.142 -0.187 0.139 0.469*** Marginal R2 0.000 0.002 0.001 0.002 0.025 LR Chisq 0.014, df=1 p = .907 0.299, df=1 p = .585 0.271, df=1 p = .603 0.720, df=1 p = .396 10.212, df=1 p = .001 Random Effects Intercept Items var(vi0) 0.243 0.240 0.242 0.237 0.192 Intercept Persons var(u0j) 0.922 0.917 0.925 0.918 0.928 Slope (τ11) 0.085 0.000 0.182 0.012 0.047 Correlation (ρ01) 0.086 NA -0.496 0.488 -0.311 Model Evaluation Conditional R2 0.539 NA 0.541 0.537 0.542 AIC 46561 46565 46529 46559 46522 log-Likelihood -23272 -23275 -23257 -23271 -23253 Goodness-of-fit LRT compared to intercept-only model χ2(df =5) = 10893, p < .001 χ2(df =5) = 10887, p < .001 χ2(df =5) = 10925, p < .001 χ2(df =5) = 10895, p < .001 χ2(df =5) = 10932, p < .001 Note: *** p < .001 To further explore the interaction, estimated marginal means (EMMs) were computed using the emmeans package in R (Lenth, 2024) at five vocabulary levels based on summary statistics (minimum = 4.00, 1st quartile and median = 9.00, mean = 9.68, 3rd quartile = 11.00, maximum = 13.00). The results shown in Figure 3 reveal that the average item response was greater in the presence of nonfinite complement clauses across all vocabulary levels except the lowest. The lack of significance at the lowest level may be attributable to the small number of respondents at this level (n=2), or a lack of sensitivity to this clause type for respondents with very low vocabulary familiarity. In addition, there was a statistically significant effect of a lower 54 item response with a higher vocabulary level, regardless of the presence of nonfinite complement clauses. Figure 3 Estimated Marginal Effect of Vocabulary Discussion This study investigated associations between psychological test item responses and different types of dependent clauses, as well as whether vocabulary familiarity moderated any associations found. Of the five dependent clause types analyzed, only nonfinite complement clauses showed a significant positive association with item responses. This is after adjusting for 55 the respondents’ level of negative emotional state. The positive association between nonfinite complement clauses and item responses was slightly amplified by respondents’ vocabulary scores. Specifically, respondents with higher vocabulary familiarity were more likely to select higher response options, reflecting greater endorsement of negative emotional states over the past week. Notably, nonfinite complement clauses were identified as the most prevalent type in our corpus-based register analysis of non-commercial psychological tests in Study 1. Nonfinite complement clauses convey different semantic meanings, one of which is to express one’s stance toward a particular proposition. Among the 18 DASS items containing nonfinite complement clauses, five elicited respondents’ stance on the difficulty of an activity (e.g., I found it hard to wind down, I found it difficult to work up the initiative to do things), while four elicited respondents’ stance on the likelihood of an action (e.g., I couldn’t seem to experience any positive feeling at all, I tended to overreact to situations). It is possible that respondents with greater vocabulary familiarity, a potential proxy for reading or language ability (Guo et al., 2011; Qian & Lin, 2019; Sarbazi et al., 2021), engaged with these items differently than those with lower vocabulary scores, and that differences in reading or language ability may have led to varying interpretations of these items. Another semantic meaning conveyed by nonfinite dependent clauses found in four of the DASS items was to report simple descriptions of experiences (e.g., I found myself getting agitated, I found myself getting upset by quite trivial things). However, the connection between these meanings and the observed findings remains unclear. The remaining dependent clause types (finite adverbial, finite relative, nonfinite relative and finite complement) were not associated with respondents’ item responding. A few possible reasons exist for the non-significant finding. Firstly, very few items contained finite adverbial, 56 finite relative or nonfinite relative clauses. It is possible that the low presence of these clauses made associations difficult to detect. Secondly, regarding finite complement and adverbial clauses, one possible reason for the lack of association between these clauses and item responses could be due to the fact that they are hypothesized to be among the earliest grammatical complexity features acquired (Biber et al., 2011; Staples et al., 2023); thus, they are likely quite familiar to respondents and perhaps the easiest dependent clause types to read and process. These findings have implications for validity and item writing guidelines. The observed lack of significant association between item responses and four of the dependent clause types suggests that these clause types may have minimal impact on item responding. However, more empirical evidence is needed to substantiate these preliminary findings. From a validity perspective, the results of this preliminary study imply that certain syntactic structures, such as specific dependent clause types, may not be a significant source of invalidity in test scores. This is particularly relevant when considering the response processes engaged by respondents, as the cognitive and linguistic demands imposed by these clause types could be less impactful than hypothesized. Furthermore, the findings suggest that some dependent clause types may pose little threat to validity or fairness, even for respondents with lower literacy or language proficiency levels. However, it is necessary to confirm whether these results generalize across broader populations and test contexts. By integrating these insights, this study highlights the importance of re-examining language-related item-writing guidelines based on empirical research. While broadly framed discouragement of certain linguistic features, such as dependent clauses, may be intended to enhance validity and fairness, the current findings suggest that a more nuanced understanding of their effects on item responses is needed. At present, the existing body of evidence remains 57 limited, underscoring the need for further research. Once sufficient evidence is available, a revised, empirically informed set of guidelines could support item writers in writing test items that balance clarity and fairness while preserving the validity of test score interpretations. Finally, it is worth noting that, during the process of model building, main effects models for each clause type were fit that included vocabulary familiarity as a covariate. In each case, vocabulary familiarity had a significant direct association with item responses. While these models were not the primary focus of this study, this finding suggests that respondents' vocabulary familiarity, and the familiarity or frequency of vocabulary used in test items, may have a greater influence on item responses than the specific types of dependent clauses included in the items. However, further research is needed before drawing any definitive conclusions. Conclusion This manuscript represents an interdisciplinary effort to understand the grammatical complexity features of non-commercial psychological test items, and to explore their relationship with item responding. By combining insights from these two studies, we lay the groundwork for future psychometric research that expands on the explanatory variables used to understand item responses. These studies also demonstrate how linguistic expertise and approaches from applied linguistics can be used to better understand the linguistic features of psychological test items, opening up opportunities to explore other linguistic features believed to impact item comprehension, such as low frequency words, densely modified noun phrases, or vague or imprecise terms. Consistent with the findings of Study 1, nonfinite complement clauses were the most prevalent clause type in the DASS, present in 18 of the 42 items, while finite complement clauses were the second most common, present in 13 items. This alignment reinforces the 58 patterns identified in Study 1’s corpus-based analysis of non-commercial psychological tests, highlighting the central role of these clause types in the linguistic structure of psychological test items. Nonfinite complement clauses were the only dependent clause type to be associated with item responses, and this association was moderated by respondents’ vocabulary score. Zumbo et al. (2023) highlight that, despite varying definitions, response processes involve assumptions about how respondents engage with test items and the alignment between expected and actual response processes. Traditionally, researchers assess this alignment using methods such as cognitive interviewing; however, digital test administration (e.g., via computers, tablets, or smartphones) provides additional opportunities to gather evidence through techniques like eye tracking, process recording, and response time analysis (Hubley, 2021). These methods can enhance the evaluation of validity by identifying potential hidden invalidities across subgroups. Building on this perspective, the present research contributes by investigating prevalent grammatical complexity features in psychological test items through a corpus-based register analysis. It further examines the relationship between these complexity features and item responses, revealing how the presence of certain types of dependent clauses may or may not disrupt the fit between expected and actual item response processes. These studies offer valuable insights into the grammatical complexity of non-commercial psychological test items and their relationship with item responses. However, several limitations highlight directions for future research. First, unlike traditional register analyses, Study 1 did not compare the linguistic features of psychological test items to other registers, or situational contexts within the same register. This decision was intentional, as the primary goal was to identify the prevalent dependent clause types in the register of non-commercial psychological tests, establishing a foundation for subsequent research on response processes. Future studies 59 could extend these findings by exploring variations across different test modes (e.g., written vs. aural), target audiences (e.g., expert vs. general), or subfields of psychology. Additionally, the relatively small sample size of 100 tests in Study 1 (due to the need to manually remediate grammatical feature annotations) limits the representativeness of the broader target domain of non-commercial psychological tests. Future studies could address this limitation by analyzing a larger and more diverse corpus, enabling a more comprehensive understanding of grammatical complexity in this register. One major limitation of Study 2 was the vocabulary recognition measure provided in the dataset. The codebook for the dataset did not include information about the measure’s origin or any validity evidence, limiting its robustness as an indicator of respondents’ vocabulary familiarity. Despite these shortcomings, using this existing dataset was the most practical alternative, given the time and financial constraints associated with collecting our own data online. Moreover, identifying a brief, reliable measure of reading ability that would not increase respondents’ cognitive burden, risk survey abandonment, or encourage cheating proved unfeasible. Future studies could address this limitation by incorporating more robust measures of vocabulary or language ability. This would enable a deeper understanding how language ability moderates the association between item responses and the linguistic features of test items. Another limitation was the composition of the sample. Our aim was to select a sample of respondents who were fairly representative of the type of individuals who opt-in to surveys or to participate in research. However, the random sampling strategy to create the study sample used in the generalized linear mixed modelling resulted in a smaller group of respondents at the lowest levels of vocabulary and education. To address this, future studies should consider 60 oversampling from this subgroup to better capture how individuals with lower education levels and vocabulary familiarity interact with grammatically complex test items. Overall, this research emphasizes the importance of considering linguistic complexity in psychological test development, offering important preliminary insights into how different dependent clause types are associated with item responding. By establishing foundational findings, it sets the stage for future investigations into the interplay between linguistic complexity and response processes, with the potential to improve test design and enhance fairness and accessibility for diverse populations. 61 Chapter 4. Conclusion Review This research project had two overarching purposes related to understanding the linguistic features of psychological test items and their relationship with item responses. As a result, it was divided into two interrelated studies. Study 1 examined a corpus of non-commercial psychological tests to identify prevalent dependent clause types and their functional roles within this register. Nonfinite complement clauses emerged as the most frequently used type, serving functions such as expressing cognitive and perceptual states, efforts, desires, stance, probabilities, and other actions (Biber et al., 2021). Their prevalence aligns with the purpose of psychological tests: eliciting responses about an individual's behavior, thoughts, and feelings. For example: “You tried to show that your partner was wrong” (Darnon & Butera, 2007, p. 148) “I believe in making a responsible and well-considered decision” (Allen & Ng, 1999, p. 25) “I couldn’t seem to experience any positive feeling at all” (Lovibond & Lovibond, 1995, p. 339). Study 2 investigated how dependent clause types in test items were associated with item responses and whether vocabulary familiarity moderated any associations identified. Among the five analyzed clause types, only nonfinite complement clauses showed a significant relationship with item responses. Specifically, items containing these clauses were linked to a greater tendency to report negative emotional states over the past week. Moreover, respondents with higher vocabulary scores experienced a slightly magnified effect, suggesting that vocabulary ability may amplify the influence of these clauses on responses. 62 Consistent with the findings of Study 1, nonfinite complement clauses were the most prevalent type in the DASS used in Study 2, appearing in 18 of the 42 items. Finite complement clauses, the second most common type, were found in 13 items. This alignment reinforces the patterns observed in Study 1, underscoring the central role of nonfinite complement clauses in the linguistic structure of psychological test items. Notably, these clauses were the only type associated with item responses, and this relationship was moderated by vocabulary familiarity, further emphasizing their unique relationship with response processes. Novel Contributions This research offers several novel contributions to educational and psychological testing. First and foremost, this study highlights the value of taking an interdisciplinary lens to issues of measurement. Although many experts in the psychometric literature have raised concerns about the language in test items, few have examined this from the viewpoint of an applied linguist. It takes someone with expertise in linguistics and/or language learning to understand which specific linguistic features of test items may be problematic, and to identify them in the items. It also takes expertise to review the annotations provided from NLP tools to check for accuracy. These tools are able to annotate many grammatical forms, but there are often nuances in the functions that clauses or phrases serve in a sentence, and this requires human judgment. Study 1 highlights the value of register-based corpus analyses in enhancing our understanding of the predominant linguistic features in psychological test items. This is an approach that does not appear to have been explored in previously published studies. Using methods from corpus linguistics and natural language processing tools, the linguistic analysis in Study 1 provides a generalizable description of dependent clause use in the register of non-commercial psychological tests. This approach can serve as a blueprint for the analysis of other 63 linguistic features identified in the psychometric literature and item writing guidelines as potential sources of invalidity, such as unfamiliar vocabulary or noun phrase complexity. Study 2 contributes to the growing body of response process validity research by investigating the relationship between specific dependent clause types and item responses, as well as the moderating role of respondents’ vocabulary familiarity. Unlike prior studies that have focused on sentence length or different aspects of complex syntax (Lenzner, 2012; Lenzner et al., 2010; Yan & Tourangeau, 2008), this study uniquely examines how the specific forms and functions of dependent clauses influence response patterns. To my knowledge, Study 2 is the first to explore whether dependent clauses explain variation in item responses and whether vocabulary familiarity moderates these effects. Furthermore, Study 2 broadens the methodological toolkit for response processes research. While cognitive interviews, response times and eye tracking have been commonly used in this line of validity research (Hubley, 2021), this study demonstrates the utility of mixed-effects modeling for analyzing item responses. This approach permits the exploration of a wide range of variables that could be associated with respondents’ item responding. Additionally, by focusing on specific linguistic features associated with grammatical complexity, dependent clauses, this research expands the explanatory variables explored in psychometric analyses of response processes. While studies on positive and negative wording have been of interest (e.g., Koutsogiorgi & Michaelides, 2022, 2023), relatively few researchers, aside from Lenzner (2012) and Lenzner et al. (2010, 2011), have examined the role of grammatical complexity features in response processes. Thus, further examination of the linguistic features of test items is a promising area of research. 64 In summary, this interdisciplinary research lays the groundwork for future investigations into the role of linguistic features in response processes. Its findings aim to improve test fairness and validity across linguistically diverse populations. Limitations and Future Directions This research provides valuable insights into the grammatical complexity of non-commercial psychological test items and their relationship with item responses. However, several limitations exist that can provide directions for future research. First, Study 1 did not compare the linguistic features of psychological test items with those of other registers or situational contexts within the same register. This was intentional, as the primary goal was to identify prevalent dependent clause types in the register of non-commercial psychological tests. Future research could expand upon this by examining how grammatical complexity varies across different test modes (e.g., written vs. aural), audiences (e.g., expert vs. general), or subfields of psychology. Additionally, the relatively small sample size of 100 tests (due to the need to manually remediate grammatical feature annotations) limits the representativeness of the findings. A larger and more diverse corpus would provide a more comprehensive picture of grammatical complexity in psychological test items. For Study 2, the scope was constrained by the linguistic features of the DASS items, which did not encompass the full spectrum of dependent clause types. Additionally, the linguistic complexity of the response options, which feature complex verb phrases modified by prepositional phrases, could also have impacted responses, particularly for individuals with lower vocabulary familiarity. Future research could employ a psychological test with simpler response options, and greater grammatical complexity in the items, including aspects not covered in this study, such as noun phrase density. Moreover, while this study focused on the moderating 65 effect of vocabulary familiarity on the relationship between item responses and dependent clauses, the vocabulary familiarity measure provided in the dataset was not ideal. The codebook for the dataset did not include information about the measure’s origin or any validity evidence, limiting its robustness as an indicator of respondents’ vocabulary familiarity. Despite these shortcomings, using this existing dataset was the most practical alternative, given the time and financial constraints associated with collecting our own data online. Moreover, identifying a brief, reliable measure of reading ability that would not increase respondents’ cognitive burden, risk survey abandonment, or encourage cheating proved unfeasible. Future studies could address this limitation by incorporating more robust measures of vocabulary or language ability. This would enable a deeper understanding how language ability moderates the association between item responses and the linguistic features of test items. Another limitation was the composition of the sample. Our aim was to select a sample of respondents who were fairly representative of the type of individuals who opt-in to surveys or to participate in research. However, the random sampling strategy used to create the study sample for the generalized linear mixed modeling resulted in a smaller group of respondents at the lowest levels of vocabulary and education. To address this, future studies should consider oversampling from this subgroup to better capture how individuals with lower education levels and vocabulary familiarity interact with grammatically complex test items. Overall, this research highlights the importance of considering linguistic complexity in psychological test development. By addressing these limitations, future studies can deepen our understanding of the interplay between linguistic features of test items and item responding, ultimately advancing test fairness and validity for diverse populations. 66 References Aarts, B. (2014). Syntax. In The Oxford Dictionary of English Grammar (2nd ed.). Oxford University Press. https://doi.org/10.1093/acref/9780192800879.001.0001 Abedi, J. (2015). Language issues in item development. In S. Lane, M. R. Raymond, & T. M. Haladyna (Eds.), Handbook of Test Development (pp. 355–373). https://doi.org/10.4324/9780203874776 Agresti, A. (2002). Categorical data analysis. Wiley. https://doi.org/10.1002/0471249688 Allen, M. W., & Ng, S. H. (1999). The direct and indirect influences of human values on product ownership. Journal of Economic Psychology, 20, 5–39. American Educational Research Association [AERA], American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association. American Psychological Association. (2015). FAQ: Finding information about psychological tests. https://www.apa.org/science/programs/testing/find-tests Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, 59(4), 390–412. https://doi.org/10.1016/j.jml.2007.12.005 Berkeley NLP Group. (2019). Berkeley Neural Parser. Retrieved June 29, 2024, from https://parser.kitaev.io/ Biber, D. (1988). Variation across speech and writing. Cambridge University Press. https://doi.org/10.1017/cbo9780511621024 67 Biber, D. (2006). Parsing and grammar description, corpus-based. In K. Brown (Ed.), Encyclopedia of language and linguistics (2nd ed., pp. 197–205). Elsevier. Biber, D. (1993). Representativeness in corpus design. Literary and Linguistic Computing, 8(4), 243–257. https://doi.org/10.1093/LLC/8.4.243 Biber, D., & Conrad, S. (2019). Register, genre, and style (2nd ed.). Cambridge University Press. https://doi.org/10.1017/9781108686136 Biber, D., Conrad, S., & Reppen, R. (2012). Corpus linguistics: Investigating language structure and use. Cambridge University Press. https://doi.org/10.1017/CBO9780511804489 Biber, D., & Gray, B. (2010). Challenging stereotypes about academic writing: Complexity, elaboration, explicitness. Journal of English for Academic Purposes, 9, 2–20. https://doi.org/10.1016/j.jeap.2010.01.001 Biber, D., Gray, B., & Poonpon, K. (2011). Should we use characteristics of conversation to measure grammatical complexity in L2 writing development? Source: TESOL Quarterly, 45(1), 5–35. https://doi.org/10.5054/tq.201 Biber, D., Gray, B., Staples, S., & Egbert, J. (2022). The Register-Functional approach to grammatical complexity: Theoretical foundation, descriptive research findings, application. Routledge. Biber, D., Johansson, S., Leech, G., Conrad, S., & Finegan, E. (1999). Longman grammar of spoken and written English. Longman. Biber, D., Johansson, S., Leech, G. N., Conrad, S., & Finegan, E. (2021). Grammar of spoken and written English. John Benjamins Publishing Company. https://doi.org/10.1075/Z.232 68 Bock, R. D. (1975). Multivariate statistical methods in behavioral research. McGraw-Hill. Bono, R., Alarcón, R., & Blanca, M. J. (2021). Report quality of generalized linear mixed models in psychology: A systematic review. Frontiers in Psychology, 12, 666182. https://doi.org/10.3389/fpsyg.2021.666182 Carver, C. S. (1997). You want to measure coping but your protocol’s too long: Consider the Brief COPE. International Journal of Behavioral Medicine, 4(1), 92–100. https://doi.org/10.1207/s15327558ijbm0401_6 Christensen, R. H. B. (2023, December 4). ordinal: Regression models for ordinal data. R Package Ordinal Version 2023.12-4.1; Comprehensive R Archive Network (CRAN). https://doi.org/10.32614/cran.package.ordinal Ciobanu, T., Brodard, F., Antonietti, J. P., Genoud, P. A., & Brandner, C. (2018). Screening negative affectivity in young adults: Validation and psychometric evaluation of the French version of the Depression Anxiety Stress Scales. Canadian Journal of Behavioural Science, 50(4), 238–247. https://doi.org/10.1037/cbs0000110 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates. https://doi.org/10.4324/9780203771587 Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. Crawford, J. R., & Henry, J. D. (2003). The Depression Anxiety Stress Scales (DASS): Normative data and latent structure in a large non-clinical sample. British Journal of Clinical Psychology, 42, 111–131. https://doi.org/10.1348/014466503321903544 69 Darnon, C., & Butera, F. (2007). Learning or succeeding? Conflict regulation with mastery or performance goals. Swiss Journal of Psychology, 66(3), 145–152. https://doi.org/10.1024/1421-0185.66.3.145 Diener, E., Emmons, R. A., Larsen, R. J., & Griffin, S. (1985). The Satisfaction with Life Scale. Journal of Personality Assessment, 49(1), 71–75. https://doi.org/10.1207/s15327752jpa4901_13 Diener, E., Wirtz, D., Tov, W., Kim-Prieto, C., Choi, D., Oishi, S., & Biswas-Diener, R. (2010). New well-being measures: Short scales to assess flourishing and positive and negative feelings. Social Indicators Research, 97(2), 143-156. https://doi.org/10.1007/s11205-009-9493-y Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail and mixed-mode surveys: The tailored design method, 4th edition. John Wiley & Sons. Egbert, J., Biber, D., & Gray, B. (2022). Designing and evaluating language corpora. Cambridge University Press. Eggins, S. (2004). An introduction to systemic functional linguistics (2nd ed.). Continuum. Garson, G. (2014). Hierarchical linear modeling: Guide and applications. SAGE Publications, Inc. https://doi.org/10.4135/9781483384450 Gomez, R. (2013). Depression Anxiety Stress Scales: Factor structure and differential item functioning across women and men. Personality and Individual Differences, 54(6), 687–691. https://doi.org/10.1016/j.paid.2012.11.025 70 Grabe, W. (2010). Applied linguistics: A twenty-first-century discipline. In R. B. Kaplan (Ed.), The Oxford Handbook of Applied Linguistics, (2nd ed., pp. 34-44). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780195384253.013.0002 Graesser, A. C., Bommareddy, S., Swamer, S., & Golding, J. M. (1996). Integrating questionnaire design with a cognitive computational model of human question answering. In N. Schwarz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 143-174). Jossey-Bass Publishers. Graesser, A. C., Wiemer-Hastings, K., Kreuz, R., & Wiemer-Hastings, P. (2000). QUAID: A questionnaire evaluation aid for survey methodologists. Behavior Research Methods, Instruments, & Computers, 32(2), 254–262. Gray, B. (2015). Linguistic variation in research articles: When discipline tells only part of the story. John Benjamins Publishing Co. https://doi.org/10.1075/scl.71 Greene, W. H., & Hensher, D. A. (2010). Modeling ordered choices: A primer. Cambridge University Press. https://doi.org/10.1017/cbo9780511845062 Guo, Y., Roehrig, A. D., & Williams, R. S. (2011). The relation of morphological awareness and syntactic awareness to adults’ reading comprehension. Journal of Literacy Research, 43(2), 159–183. https://doi.org/10.1177/1086296X11403086 Halliday, M. A. K., & Matthiessen, C. M. I. M. (2014). Halliday’s introduction to functional grammar (4th ed.). Routledge. 71 Harrell, F. (2022, March 9). Assessing the proportional odds assumption and its impact. Statistical Thinking. https://www.fharrell.com/post/impactpo/#further-reading Heck, R. H., & Thomas, S. L. (2020). An introduction to multilevel modeling techniques (4th ed.). Routledge. https://doi.org/10.4324/9781410604767 Hox, J. J. C. M., Schoot, R. van de, & Matthijsse, S. (2012). How few countries will do? Comparative survey analysis from a Bayesian perspective. Survey Research Methods, 6(2), 87–93. https://doi.org/10.18148/srm/2012.v6i2.5033 Hubley, A. M. (2021). Response processes validity evidence: Understanding the meaning of scores from psychological measures. In P. Graf & D. J. A. Dozois (Eds.), Handbook on the state of the art in applied psychology (pp. 413–434). John Wiley & Sons. Hubley, A. M., & Zumbo, B. D. (2017). Response processes in the context of validity: Setting the stage. In B. D. Zumbo & A M. Hubley (Eds.), Understanding and investigating response processes in validation research (pp. 1–12). Springer. https://doi.org/10.1007/978-3-319-56129-5 Humphrey, S., Droga, L., & Feez, S. (2012). Grammar and meaning. PETAA. Jeon, E. H., & Yamashita, J. (2014). L2 reading comprehension and its correlates: A meta-analysis. Language Learning, 64(1), 160–212. https://doi.org/10.1111/lang.12034 Johnson, R. L., & Morgan, G. B. (2016). Survey scales: A guide to development, analysis, and reporting. Guildford Publications. Judd, C. M., Westfall, J., & Kenny, D. A. (2012). Treating stimuli as a random factor in social psychology: A new and comprehensive solution to a pervasive but largely ignored 72 problem. Journal of Personality and Social Psychology, 103(1), 54–69. https://doi.org/10.1037/a0028347 Koutsogiorgi, C. C., & Michaelides, M. P. (2022). Response tendencies due to item wording using eye-tracking methodology accounting for individual differences and item characteristics. Behavior Research Methods, 54(5), 2252–2270. https://doi.org/10.3758/s13428-021-01719-x Koutsogiorgi, C. C., & Michaelides, M. P. (2023). Response tendencies to positively and negatively worded items of the Rosenberg Self-Esteem Scale with eye-tracking methodology. European Journal of Psychological Assessment, 39(4), 307–315. https://doi.org/10.1027/1015-5759/a000772 Larsson, T., Berber Sardinha, T., Gray, B., & Biber, D. (2023). Exploring early L2 writing development through the lens of grammatical complexity. Applied Corpus Linguistics, 3(3), 100077. https://doi.org/10.1016/j.acorp.2023.100077 Lenth, R. V. (2024). emmeans: Estimated marginal means, aka least-squares means (Version 1.10.1) [R package]. https://CRAN.R-project.org/package=emmeans Lenzner, T. (2012). Effects of survey question comprehensibility on response quality. Field Methods, 24(4), 409–428. https://doi.org/10.1177/1525822X12448166 Lenzner, T. (2014). Are readability formulas valid tools for assessing survey question difficulty? Sociological Methods and Research, 43(4), 677–698. https://doi.org/10.1177/0049124113513436 73 Lenzner, T., Kaczmirek, L., & Galesic, M. (2011). Seeing through the eyes of the respondent: An eye-tracking study on survey question comprehension. International Journal of Public Opinion Research, 23(3), 361–373. https://doi.org/10.1093/ijpor/edq053 Lenzner, T., Kaczmirek, L., & Lenzner, A. (2010). Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology, 24, 1003–1020. https://doi.org/10.1002/acp.1602 Lovibond, P. F., & Lovibond, S. H. (1995). The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behaviour Research and Therapy, 33(3), 335–343. https://doi.org/10.1007/BF02511245 Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1(3), 86–92. https://doi.org/10.1027/1614-1881.1.3.86 Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., & McClosky, D. (2014). The Stanford CoreNLP natural language processing toolkit. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, https://aclanthology.org/P14-5010. Marshall, A. J., Evanovich, E. K., David, S. J., & Mumma, G. H. (2018). Separating common from unique variance within emotional distress: An examination of reliability and relations to worry. Behavioural and Cognitive Psychotherapy, 46(5), 633–638. https://doi.org/10.1017/S1352465817000777 Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational measurement (3rd ed., pp. 13-103). Macmillan. 74 Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performance as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741 Muthén, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316. Norman, S., Kemper, S., & Kynette, D. (1992). Adults’ reading comprehension: Effects of syntactic complexity and working memory. Journals of Gerontology, 47(4). https://doi.org/10.1093/geronj/47.4.P258 OECD. (2024). Do adults have the skills they need to thrive in a changing world?: Survey of Adult Skills 2023 (OECD Skills Studies). OECD. https://doi.org/10.1787/b263dc5d-en O'Keeffe, A.& McCarthy, M. (2022). The Routledge handbook of corpus linguistics (2nd ed.). Routledge. Open-Source Psychometrics Project. (2019, November 5). Raw data from online personality tests. https://openpsychometrics.org/_rawdata/ Peterson, R. (2013). Constructing effective questionnaires. Sage Publications. https://doi.org/10.4135/9781483349022 Peugh, J. L. (2010). A practical guide to multilevel modeling. Journal of School Psychology, 48(1), 85–112. https://doi.org/10.1016/j.jsp.2009.09.002 Qian, D. D., & Lin, L. H. F. (2019). The relationship between vocabulary knowledge and language proficiency. In S. Webb (Ed.), The Routledge Handbook of Vocabulary Studies (pp. 66–80). Taylor and Francis. https://doi.org/10.4324/9780429291586-5 75 Quené, H., & van den Bergh, H. (2008). Examples of mixed-effects modeling with crossed random effects and with binomial data. Journal of Memory and Language, 59(4), 413–425. https://doi.org/10.1016/j.jml.2008.02.002 R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org/ Radloff, L. S. (1977). The CES-D Scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1(3), 385–401. https://doi.org/10.1177/014662167700100306 Revelle, W. (2024). psych: Procedures for psychological, psychometric, and personality research (2.4.12). Reynolds, C. R., Altmann, R. A., & Allen, D. N. (2021). Mastering modern psychological testing: Theory and methods. Springer International Publishing. https://doi.org/10.1007/978-3-030-59455-8 Rosenberg, M. (1965). Society and the adolescent self-image. Princeton University Press. Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software (v. 0 6-7; Vol. 48, pp. 1–36). American Statistical Association. https://doi.org/10.18637/JSS.V048.I02 Sarbazi, M., Khany, R., & Shoja, L. (2021). The predictive power of vocabulary, syntax and metacognitive strategies for L2 reading comprehension. Southern African Linguistics and Applied Language Studies, 39(3), 244–258. https://doi.org/10.2989/16073614.2021.1939076 76 Schaufeli, W., & Bakker, A. (2004). Utrecht work engagement scale: Preliminary manual. Utrecht University. Smith, S. (2024, March 20). Vocabulary profiler. https://www.eapfoundation.com/vocab/profiler/ Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). Sage. Sprecher, S., & Fehr, B. (2005). Compassionate love for close others and humanity. Journal of Social and Personal Relationships, 22(5), 629–651. https://doi.org/10.1177/0265407505056439 Staples, S., Egbert, J., Biber, D., & Gray, B. (2016). Academic writing development at the university level: Phrasal and clausal complexity across level of study, discipline, and genre. Written Communication, 33(2), 149–183. https://doi.org/10.1177/0741088316631527 Staples, S., Gray, B., Biber, D., & Egbert, J. (2023). Writing trajectories of grammatical complexity at the university: Comparing L1 and L2 English writers in BAWE. Applied Linguistics, 44(1), 46–71. https://doi.org/10.1093/applin/amac047 Statistics Canada. (2022). While English and French are still the main languages spoken in Canada, the country’s linguistic diversity continues to grow. The Daily. https://www150.statcan.gc.ca/n1/daily-quotidien/220817/dq220817a-eng.htm Statistics Kingdom. (2017). Effect size calculator. https://www.statskingdom.com/effect-size-calculator.html 77 Teng, C. I., Lotus Shyu, Y. I., & Chang, H. Y. (2007). Moderating effects of professional commitment on hospital nurses in Taiwan. Journal of Professional Nursing, 23(1), 47–54. https://doi.org/10.1016/J.PROFNURS.2006.10.002 Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press. https://doi.org/10.1017/CBO9780511819322 Tuleya, L. G., & American Psychological Association. (2005). Thesaurus of psychological index terms (10th ed.). American Psychological Association. Tutz, G. (1990). Sequential item response models with an ordered response. British Journal of Mathematical and Statistical Psychology, 43(1), 39–55. https://doi.org/10.1111/j.2044-8317.1990.tb00925.x Tutz, G. (2022). Ordinal regression: A review and a taxonomy of models. WIREs Computational Statistics, 14(2), e1545. https://doi.org/10.1002/wics.1545 United States Census Bureau. (2021). Language spoken at home. American Community Survey. https://data.census.gov/table?q=language&t=Language+Spoken+at+Home&tid=ACSST1Y2021.S1601 Urbina, S. (2014). Essentials of psychological testing (2nd ed.). John Wiley and Sons. VandenBos, G. R. (2015). APA dictionary of psychology (2nd ed). American Psychological Association. Yan, T., & Tourangeau, R. (2008). Fast times and easy questions: The effects of age, experience and question complexity on web survey response times. Applied Cognitive Psychology, 22(1), 51–68. https://doi.org/10.1002/acp.1331 78 Yeung, A. Y., Yuliawati, L., & Cheung, S. H. (2020). A systematic review and meta-analytic factor analysis of the depression anxiety stress scales. Clinical Psychology: Science and Practice, 27(4). https://doi.org/10.1111/cpsp.12362 Zimmerman, D. W., Williams, R. H., & Zumbo, B. D. (1993). Reliability of measurement and power of significance tests based on differences. Applied Psychological Measurement, 17(l), 1–9. Zumbo, B.D. (2007). Validity: Foundational issues and statistical methodology. In C.R. Rao & S. Sinharay (Eds.), Handbook of statistics (Vol. 26, pp. 45–79). Elsevier. Zumbo, B.D. (2009). Validity as contextualized and pragmatic explanation, and its implications for validation practice. In R.W. Lissitz (Ed.) The concept of validity: Revisions, new directions, and applications (pp. 65–82). IAP Information Age Publishing. Zumbo, B. D. (2023). A dialectic on validity: Explanation-focused and the many ways of being human. International Journal of Assessment Tools in Education, 2023, 1–96. https://doi.org/10.21449/ijate.1406304 Zumbo, B. D., Maddox, B., & Care, N. M. (2023). Process and product in computer-based assessments: Clearing the ground for a holistic validity framework. European Journal of Psychological Assessment: Official Organ of the European Association of Psychological Assessment, 39(4), 252-262. https://doi.org/10.1027/1015-5759/a000748 79 Appendices Appendix A: Calculation of Precision and Recall The following formulas from Biber et al. (2022, p. 35) were used to calculate precision (9) and recall (10). Precision = true positivestrue positives+false positives (9) Recall = true positivestrue positives+false negatives (10) Table 7 presents the calculations of precision and recall for each grammatical complexity feature using a subsample of 5 tests. Table 7 Precision and Recall for Dependent Clauses Clause Function and Form Complexity Tag True Positives False Positives False Negatives Precision Recall Finite adverbial finite advl cls 16 5 11 .76 .59 Nonfinite adverbial ing advl cls 0 3 18 .00 .00 to advl cls 0 1 6 .00 .00 Finite complement adj that comp 0 0 0 - - N_thatcomp 0 0 1 - .00 V that comp 1 2 1 .33 .50 verb + wh 1 0 4 1.00 .20 prep + wh 3 0 1 1.00 .75 Nonfinite complement adj to comp 3 0 4 1.00 .43 N to comp 1 0 3 1.00 .25 V to comp 5 5 8 .50 .38 prep + ing 0 0 4 - .00 v + ing 13 0 10 1.00 .57 Finite relative finite rel 16 1 8 .94 .67 Nonfinite relative ING rel 1 1 3 .50 .25 ED-rel 0 0 0 - - to-rel 0 0 0 - - 80 Appendix B: Estimation of Corpus Sample Size Table 8 presents the parameters used to estimate a generalizable sample size for the corpus and the calculated sample size (n) using Equation (2) below. 𝑛 =𝑠2(0.5 ∗ 𝐶𝐼 𝑟𝑎𝑛𝑔𝑒𝑡)2 , (2) Table 8 Estimation of Sample Size Using Pilot Corpus Clause Function and Form s2 CI Range t Required n finite adverbial 1.96 1.01 1.96 30 nonfinite adverbial 2.02 1.02 1.96 30 finite complement 4.84 1.59 1.96 29 nonfinite complement 6.81 1.88 1.96 30 finite relative 1.08 .75 1.96 30 nonfinite relative 0.20 .32 1.96 30 Note: n = 32 81 Appendix C: Tests Included in the Corpus Table 9 below provides the test name and source for each of the tests in Study 1’s corpus. Table 9 Tests Included in the Corpus Test Name Source Acceptance of Sugar Relationship in Older Men and Women Scale Láng, A., Birkás, B., Zsidó, A. N., Ipolyi, D., & Meskó, N. (2021). It takes two to tango: Development, validation, and personality correlates of the acceptance of sugar relationships in older men and women scale (ASR-OMWS). Frontiers in Psychology, 12, 592138-592138. https://doi.org/10.3389/fpsyg.2021.592138 Adaptive Functions of Music Listening Scale Groarke, Jenny M., & Hogan, Michael J. (2018). Development and psychometric evaluation of the Adaptive Functions of Music Listening Scale. Frontiers in Psychology, 9. https://dx.doi.org/10.3389/fpsyg.2018.00516 Adolescent Relationship Abuse Measure McCauley, H. L., Dick, R. N., Tancredi, D. J., Goldstein, S., Blackburn, S., Silverman, J. G., Monasterio, E., James, L., & Miller, E. (2014). Differences by sexual minority status in relationship abuse and sexual and reproductive health among adolescent females. Journal of Adolescent Health, 55(5), 652-658. https://doi.org/10.1016/j.jadohealth.2014.04.020 Adolescent Sexting Scale Molla Esparza, C., Nájera, P., López-González, E., & Losilla, J. (2020). Development and validation of the adolescent sexting scale (A-SextS) with a Spanish sample. International Journal of Environmental Research and Public Health, 17(21), 8042. https://doi.org/10.3390/ijerph17218042 Air Traffic Controller Work Environment Questionnaire Chang, Yu-Hern, & Yeh, Chung-Hsing. (2010). Human performance interfaces in air traffic control. Applied Ergonomics, 41(1), 123-129. https://dx.doi.org/10.1016/j.apergo.2009.06.002 Atheism Perceptions Measure Burris, C. T. (2022). Poker-faced and godless: Expressive suppression and atheism. Psychology of Religion and Spirituality, 14(3), 351-361. https://doi.org/10.1037/rel0000361 Attitude Toward Mental Vigor Scale Beute, Femke, Kaiser, Florian G., Haans, Antal, & de Kort, Yvonne. (2017). Striving for mental vigor through restorative activities: Application of the Campbell Paradigm to construct the Attitude toward mental vigor scale. Mental Health and Prevention, 8, 20-26. https://dx.doi.org/10.1016/j.mhp.2017.09.001 Baby Care Questionnaire Winstanley, Alice, & Gattis, Merideth. (2013). The Baby Care Questionnaire: A measure of parenting principles and practices during infancy. Infant Behavior & Development, 36(4), 762-775. https://dx.doi.org/10.1016/j.infbeh.2013.08.004 Barriers to Help Seeking Scale Mansfield, A. K., Addis, M. E., & Courtenay, W. (2005). Measurement of men's help seeking: Development and evaluation of the barriers to help seeking scale. Psychology of Men & Masculinity, 6(2), 95-108. https://doi.org/10.1037/1524-9220.6.2.95 82 Behavioral Limitations Measure Lehmiller, J. J. (2009). Secret romantic relationships: Consequences for personal and relational well-being. Personality and Social Psychology Bulletin, 35(11), 1452-1466. https://dx.doi.org/10.1177/0146167209342594 Beliefs About Sexual Functioning Scale Pascoal, P. M., Alvarez, M., Pereira, C. R., & Nobre, P. (2017). Development and initial validation of the beliefs about sexual functioning scale: A gender invariant measure. Journal of Sexual Medicine, 14(4), 613-623. https://doi.org/10.1016/j.jsxm.2017.01.021 Beliefs and Behaviour Questionnaire George, J., Mackinnon, A., Kong, D. C. M., & Stewart, K. (2006). Development and validation of the beliefs and behaviour questionnaire (BBQ). Patient Education and Counseling, 64(1), 50-60. https://doi.org/10.1016/j.pec.2005.11.010 Body Image Measurement Scale Thomas, Erin Vinoski, Warren-Findlow, Jan, Reeve, Charlie L., Webb, Jennifer B., Laditka, Sarah B., & Quinlan, Margaret M. (2021). Universal design for measurement: Centering the experiences of individuals with disabilities within health measurement research. Evaluation & the Health Professions, 44(3), 203-209. https://dx.doi.org/10.1177/0163278719900530 Carer's Irritability Questionnaire Craig, K. J., Hietanen, H., Markova, I. S., & Berrios, G. E. (2008). The irritability questionnaire: A new scale for the measurement of irritability. Psychiatry Research, 159(3), 367-375. https://doi.org/10.1016/j.psychres.2007.03.002 Community Norms of Child Neglect Scale Goodvin, R., Johnson, D. R., Hardy, S. A., Graef, M. I., & Chambers, J. M. (2007). Development and confirmatory factor analysis of the community norms of child neglect scale. Child Maltreatment, 12(1), 68-85. https://doi.org/10.1177/1077559506296667 Compassionate Love Scale Sprecher, S., & Fehr, B. (2005). Compassionate love for close others and humanity. Journal of Social and Personal Relationships, 22(5), 629-651. https://doi.org/10.1177/0265407505056439 Complementary Medicine Disclosure Index McIntyre, Erica, Foley, Hope, Diezel, Helene, Harnett, Joanna, Adams, Jon, Sibritt, David, & Steel, Amie. (2020). Development and preliminarily validation of the Complementary Medicine Disclosure Index. Patient Education and Counseling, 103(6), 1237-1244. https://dx.doi.org/10.1016/j.pec.2020.01.008 Conflict Regulation Questionnaire Darnon, Céline, & Butera, Fabrizio. (2007). Learning or succeeding? Conflict regulation with mastery or performance goals. Swiss Journal of Psychology / Schweizerische Zeitschrift für Psychologie / Revue Suisse de Psychologie, 66(3), 145-152. https://dx.doi.org/10.1024/1421-0185.66.3.145 COVID-19 Experiences Measure Cohen, A. K., Hoyt, L. T., & Dull, B. (2020). A descriptive study of COVID-19–Related experiences and perspectives of a national sample of college students in spring 2020. Journal of Adolescent Health, 67(3), 369-375. https://doi.org/10.1016/j.jadohealth.2020.06.009 COVID-19 Impact Scale Srinivasan, T., & Sulur Nachimuthu, G. (2022). COVID-19 impact on employee flourishing: Parental stress as mediator. Psychological Trauma, 14(2), 281-290. https://doi.org/10.1037/tra0001037 83 Creative Mindsets Scale Karwowski, M. (2014). Creative mindsets: Measurement, correlates, consequences. Psychology of Aesthetics, Creativity, and the Arts, 8(1), 62-70. https://doi.org/10.1037/a0034898 Danger Appraisal Questionnaire Poulton, R. G., & Andrews, G. (1996). Change in danger cognitions in agoraphobia and social phobia during treatment. Behaviour Research and Therapy, 34(5), 413-421. https://doi.org/10.1016/0005-7967(96)00009-5 Depression Anxiety Stress Scales Lovibond, P. F., & Lovibond, S. H. (1995). The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behaviour Research and Therapy, 33(3), 335–343. https://doi.org/10.1007/BF02511245 Differentiation Inventory Pace, R. C. (1990). Personalized and depersonalized conflict in small group discussions: An examination of differentiation. Small Group Research, 21(1), 79-96. https://doi.org/10.1177/1046496490211006 Discrimination Experiences During COVID-19 Outbreak Scale Lui, P. P., Parikh, K., Katedia, S., & Jouriles, E. N. (2022). Anti-Asian discrimination and antiracist bystander behaviors amid the COVID-19 outbreak. Asian American Journal of Psychology, 13(3), 295-304. https://doi.org/10.1037/aap0000258 Distress Intolerance Scale McHugh, R. K., & Otto, M. W. (2012). Refining the measurement of distress intolerance. Behavior Therapy, 43(3), 641-651. https://doi.org/10.1016/j.beth.2011.12.001 Dramaturgical Perspective Endorsement Measure Sullivan, D., Landau, M. J., Young, I. F., & Stewart, S. A. (2014). The dramaturgical perspective in relation to self and culture. Journal of Personality and Social Psychology, 107(5), 767-790. https://doi.org/10.1037/a0037904 Emirati Marital Satisfaction Scale Al-Darmaki, F. R., Hassane, S. H., Ahammed, S., Abdullah, A. S., Yaaqeib, S. I., & Dodeen, H. (2016). Marital satisfaction in the United Arab Emirates: Development and validation of a culturally relevant scale. Journal of Family Issues, 37(12), 1703-1729. https://doi.org/10.1177/0192513X14547418 Ethnic and Racial Minority Student Program Satisfaction Questionnaire Tram, J. M., Dhaliwal, R. K., Kiyokawa, J. M., & Caceres-Licos, C. E. (2022). Impact of staff mentoring on ethnic and racial minority student program satisfaction. Scholarship of Teaching and Learning in Psychology, 8(4), 330-341. https://doi.org/10.1037/stl0000316 Fan Intentions to Buy Club Shares Questionnaire de Ruyter, K., & Wetzels, M. (2000). With a little help from my fans – extending models of pro-social behaviour to explain supporters’ intentions to buy soccer club shares. Journal of Economic Psychology, 21(4), 387-409. https://doi.org/10.1016/S0167-4870(00)00010-6 Focus of Attention Questionnaire Woody, S. R., Chambless, D. L., & Glass, C. R. (1997). Self-focused attention in the treatment of social phobia. Behaviour Research and Therapy, 35(2), 117-129. https://doi.org/10.1016/S0005-7967(96)00084-8 Gains from Dream Interpretation Heaton, K. J., Hill, C. E., Petersen, D. A., Rochlen, A. B., & Zack, J. S. (1998). A comparison of therapist-facilitated and self-guided dream interpretation sessions. Journal of Counseling Psychology, 45(1), 115-122. https://doi.org/10.1037/0022-0167.45.1.115 84 Healthcare Triggering Questionnaire Schnur, J. B., Chaplin, W. F., Khurshid, K., Mogavero, J. N., Goldsmith, R. E., Lee, Y., Litman, L., & Montgomery, G. H. (2017). Development of the healthcare triggering questionnaire in adult sexual abuse survivors. Psychological Trauma, 9(6), 714-722. https://doi.org/10.1037/tra0000273 Healthy Selfishness and Pathological Altruism Scales Kaufman, S. B., & Jauk, E. (2020). Healthy selfishness and pathological altruism: Measuring two paradoxical forms of selfishness. Frontiers in Psychology, 11, 1006-1006. https://doi.org/10.3389/fpsyg.2020.01006 Holistic Cognition Scale Lux, A. A., Grover, S. L., & Teo, S. T. T. (2021). Development and validation of the holistic cognition scale. Frontiers in Psychology, 12, 551623-551623. https://doi.org/10.3389/fpsyg.2021.551623 Importance of Organizational Political Affiliation Questionnaire Roth, P. L., Arnold, J. D., Walker, H. J., Zhang, L., & Van Iddekinge, C. H. (2022). Organizational political affiliation and job seekers: If I don't identify with your party, am I still attracted? Journal of Applied Psychology, 107(5), 724-745. https://doi.org/10.1037/apl0000932 index of Nighttime Offline Distress Scott, H., Biello, S. M., & Woods, H. C. (2021). Nodding off but can't disconnect: Development and validation of the iNOD index of nighttime offline distress. Sleep Medicine, 81, 430-438. https://doi.org/10.1016/j.sleep.2021.02.045 Information Seeking Behavior Scale Lagoe, C., & Atkin, D. (2015). Health anxiety in the digital age: An exploration of psychological determinants of online health information seeking. Computers in Human Behavior, 52, 484-491. https://doi.org/10.1016/j.chb.2015.06.003 Integrity Climate Questionnaire Zahari, A. I., Said, J., & Arshad, R. (2019). Integrity climate questionnaire. Cogent Psychology, 6(1). https://doi.org/10.1080/23311908.2019.1626541 Intention to Use Technology-Enabled Learning Scale Watson, J. H., & Rockinson-Szapkiw, A. (2021). Predicting preservice teachers' intention to use technology-enabled learning. Computers and Education, 168, 104207. https://doi.org/10.1016/j.compedu.2021.104207 Intimate Partner Violence Knowledge, Skills, and Attitude Survey Manuel, B., Roelens, K., Tiago, A., Keygnaert, I., & Valcke, M. (2019). Gaps in medical students' competencies to deal with intimate partner violence in key Mozambican medical schools. Frontiers in Public Health, 7, 204. https://doi.org/10.3389/fpubh.2019.00204 Intrusions at Work Measure Lin, B. C., Kain, J. M., & Fritz, C. (2013). Don't interrupt me! An examination of the relationship between intrusions at work and employee strain. International Journal of Stress Management, 20(2), 77-94. https://doi.org/10.1037/a0031637 Latino Mental Health Needs Measure Kanel, K. (2002). Mental health needs of Spanish-speaking Latinos in southern California. Hispanic Journal of Behavioral Sciences, 24(1), 74-91. https://doi.org/10.1177/0739986302024001005 Leave No Trace Attitudes Measure Mueller, J. T., Taff, B. D., Wimpey, J., & Graefe, A. (2018). Small-scale race events in natural areas: Participants’ attitudes, beliefs, and global perceptions of leave no trace ethics. Journal of Outdoor Recreation and Tourism, 23, 8-15. https://doi.org/10.1016/j.jort.2018.03.001 85 Leisure Satisfaction Scale Coyle, C. P., Lesnik-Emas, S., & Kinney, W. B. (1994). Predicting life satisfaction among adults with spinal cord injuries. Rehabilitation Psychology, 39(2), 95-112. https://doi.org/10.1037/h0080309 Likelihood Judgments of Future Events Measure Zelenski, J. M., & Larsen, R. J. (2002). Predicting the future: How affect-related personality traits influence likelihood judgments of future events. Personality & Social Psychology Bulletin, 28(7), 1000-1010. https://doi.org/10.1177/014616720202800712 Manipulation of Reimbursement Rules Measure VanGeest, J. B., Wynia, M. K., Cummins, D. S., & Wilson, I. B. (2002). Measuring deception: Test-retest reliability of physicians’ self-reported manipulation of reimbursement rules for patients. Medical Care Research and Review, 59(2), 184-196. https://doi.org/10.1177/1077558702059002004 Marital Conflict Resolution Style Measure Rands, M., Levinger, G., & Mellinger, G. D. (1981). Patterns of conflict resolution and marital satisfaction. Journal of Family Issues, 2(3), 297-321. https://doi.org/10.1177/0192513X8100200303 Maternal Disintegrative Responses Scale Chasson, M., & Taubman – Ben‐Ari, O. (2023). The maternal disintegrative responses scale (MDRS): Development and initial validation. Journal of Clinical Psychology, 79(2), 415-430. https://doi.org/10.1002/jclp.23414 Medical Consultation Experience Questionnaire Sanford, K., Rivers, A. S., Braun, T. L., Schultz, K. P., & Buchanan, E. P. (2018). Medical consultation experience questionnaire: Assessing perceived alliance and experienced confusion during medical consultations. Psychological Assessment, 30(11), 1499-1511. https://doi.org/10.1037/pas0000594 Metacomprehension Scale Moore, DeWayne, Zabrucky, Karen, & Commander, Nannette Evans. (1997). Validation of the Metacomprehension Scale. Contemporary Educational Psychology, 22(4), 457-471. doi: https://dx.doi.org/10.1006/ceps.1997.0946 Morality Questionnaire Hone, L. S. E., McCauley, T. G., Pedersen, E. J., Carter, E. C., & McCullough, M. E. (2021). The sex premium in religiously motivated moral judgment. Journal of Personality and Social Psychology, 120(6), 1621-1633. https://doi.org/10.1037/pspp0000296 Multiculturalism and Colorblindness Measure Morrison, K. R., Plaut, V. C., & Ybarra, O. (2010). Predicting whether multiculturalism positively or negatively influences white Americans’ intergroup attitudes: The role of ethnic identification. Personality & Social Psychology Bulletin, 36(12), 1648-1661. https://doi.org/10.1177/0146167210386118 Music Experience Questionnaire Werner, P. D., Swope, A. J., & Heide, F. J. (2006). The music experience questionnaire: Development and correlates. The Journal of Psychology, 140(4), 329-345. https://doi.org/10.3200/JRLP.140.4.329-345 National Image, Psychological Ownership, and Civilized Tourism Behavior Qu, Z., Cao, X., Ge, H., & Liu, Y. (2021). How does national image affect tourists’ civilized tourism behavior? The mediating role of psychological ownership. Journal of Hospitality and Tourism Management, 47, 468-475. https://doi.org/10.1016/j.jhtm.2021.04.019 86 Negative Emotions Scale Dunn, J., Ruedy, N. E., & Schweitzer, M. E. (2012). It hurts both ways: How social comparisons harm affective and cognitive trust. Organizational Behavior and Human Decision Processes, 117(1), 2-14. https://doi.org/10.1016/j.obhdp.2011.08.001 Nonsuicidal Self-Injury Transtheoretical Model of Change Scales Kruzan, K. P., Whitlock, J., & Hasking, P. (2020). Development and initial validation of scales to assess decisional balance (NSSI-DB), processes of change (NSSI-POC), and self-efficacy (NSSI-SE) in a population of young adults engaging in nonsuicidal self-injury. Psychological Assessment, 32(7), 635-648. https://doi.org/10.1037/pas0000821 Parent Self-Disclosure to Child Measure Kerr, M., Stattin, H., & Trost, K. (1999). To know you is to trust you: Parents' trust is rooted in child disclosure of information. Journal of Adolescence, 22(6), 737-752. https://doi.org/10.1006/jado.1999.0266 Parental Perceptions of Technology Scale Sanders, W., Parent, J., Forehand, R., Sullivan, A. D. W., & Jones, D. J. (2016). Parental perceptions of technology and technology-focused parenting: Associations with youth screen time. Journal of Applied Developmental Psychology, 44, 28-38. https://doi.org/10.1016/j.appdev.2016.02.005 Patient-Centred COPD Questionnaire Pommer, A. M., Prins, L., van Ranst, D., Meijer, J., Van't Hul, A., Janssen, J., Pouwer, F., & Pop, V. J. M. (2013). Development and validity of the patient-centred COPD questionnaire (PCQ). Journal of Psychosomatic Research, 75(6), 563-571. https://doi.org/10.1016/j.jpsychores.2013.10.001 Pediatrician-Reported Need for Behavioral Health and Openness to Technology-Based Solutions Survey McGoron, L., & Ondersma, S. J. (2015). Pediatrician and parent-reported interest in technology-based positive parenting promotion. Clinical Practice in Pediatric Psychology, 3(4), 282-287. https://doi.org/10.1037/cpp0000110 Perceived Rejection Scale Berenson, K. R., Downey, G., Rafaeli, E., Coifman, K. G., & Leventhal Paquin, N. (2011). The rejection-rage contingency in borderline personality disorder. Journal of Abnormal Psychology, 120(3), 681-690. https://doi.org/10.1037/a0023335 Perceptions of Teacher, Student, and Family Factors Influencing Student Performance Measure Patterson, M. M., Kravchenko, N., Chen-Bouck, L., & Kelley, J. A. (2016). General and domain-specific beliefs about intelligence, ability, and effort among preservice and practicing teachers. Teaching and Teacher Education, 59, 180-190. https://doi.org/10.1016/j.tate.2016.06.004 Perceptions of the Postgraduate Taught Experiences Survey Webber, M., Lynch, S., & Oluku, J. (2013). Enhancing student engagement in student experience surveys: A mixed methods study. Educational Research, 55(1), 71-86. https://doi.org/10.1080/00131881.2013.767026 Pharmacists' Alignment Questionnaire Harrison, J., Scahill, S., & Sheridan, J. (2012). New Zealand pharmacists’ alignment with their professional body’s vision for the future. Research in Social and Administrative Pharmacy, 8(1), 17-35. https://doi.org/10.1016/j.sapharm.2010.12.001 87 Positive and Negative Bystander Intervention Measure Miller, E., Tancredi, D. J., McCauley, H. L., Decker, M. R., Virata, M. C. D., Anderson, H. A., Stetkevich, N., Brown, E. W., Moideen, F., & Silverman, J. G. (2012). “Coaching boys into men”: A cluster-randomized controlled trial of a dating violence prevention program. Journal of Adolescent Health, 51(5), 431-438. https://doi.org/10.1016/j.jadohealth.2012.01.018 Post Entrance Exam Questionnaire Arnulf, I., Grosliere, L., Le Corvec, T., Golmard, J., Lascols, O., & Duguet, A. (2014). Will students pass a competitive exam that they failed in their dreams? Consciousness and Cognition, 29, 36-47. https://doi.org/10.1016/j.concog.2014.06.010 Pre-Service Teacher Use and Perceptions of Edmodo Questionnaire Krutka, D. G., Bergman, D. J., Flores, R., Mason, K., & Jack, A. R. (2014). Microblogging about teaching: Nurturing participatory cultures through collaborative online reflection with pre-service teachers. Teaching and Teacher Education, 40, 83-93. https://doi.org/10.1016/j.tate.2014.02.002 Preparedness for the Future Questionnaire Heyland, D. K., Pope, J. P., Jiang, X., & Day, A. G. (2021). Determining the psychometric properties of a novel questionnaire to measure "preparedness for the future" (prep FQ). Health and Quality of Life Outcomes, 19(1), 122-122. https://doi.org/10.1186/s12955-021-01759-z Prescriptive and Descriptive Meritocracy Scales Zimmerman, J. L., & Reyna, C. (2013). The meaning and role of ideology in system justification and resistance for high- and low-status people. Journal of Personality and Social Psychology, 105(1), 1-23. https://doi.org/10.1037/a0032967 Prescriptive Beliefs Scale for Democracy Zimmerman, J. L., & Reyna, C. (2013). The meaning and role of ideology in system justification and resistance for high- and low-status people. Journal of Personality and Social Psychology, 105(1), 1-23. https://doi.org/10.1037/a0032967 Professional Nursing Commitment Questionnaire Teng, C., Lotus Shyu, Y., & Chang, H. (2007). Moderating effects of professional commitment on hospital nurses in Taiwan. Journal of Professional Nursing, 23(1), 47-54. https://doi.org/10.1016/j.profnurs.2006.10.002 Propensity for Online Community Contribution Scale Fullwood, C., Chadwick, D., Keep, M., Attrill-Smith, A., Asbury, T., & Kirwan, G. (2019). Lurking towards empowerment: Explaining propensity to engage with online health support groups and its association with positive outcomes. Computers in Human Behavior, 90, 131-140. https://doi.org/10.1016/j.chb.2018.08.037 Protection Motivation Theory Measure Harbeck, E. L., Glendon, A. I., & Hine, T. J. (2018). Young driver perceived risk and risky driving: A theoretical approach to the “fatal five”. Transportation Research. Part F, Traffic Psychology and Behaviour, 58, 392-404. https://doi.org/10.1016/j.trf.2018.06.018 Protective Drinking Practices Scale Martin, J. L., Colvin, K. F., Madson, M. B., Zamboanga, B. L., & Pazienza, R. (2020). Optimal assessment of protective behavioral strategies among college drinkers: An item response theory analysis. Psychological Assessment, 32(4), 394-406. https://doi.org/10.1037/pas0000799 88 Psychologist Perceptions of Continuing Education Survey Neimeyer, G. J., Taylor, J. M., & Wear, D. M. (2009). Continuing education in psychology: Outcomes, evaluations, and mandates. Professional Psychology, Research and Practice, 40(6), 617-624. https://doi.org/10.1037/a0016655 Questionnaire for Assessing the Childbirth Experience Carquillat, P., Vendittelli, F., Perneger, T., & Guittier, M. (2017). Development of a questionnaire for assessing the childbirth experience (QACE). BMC Pregnancy and Childbirth, 17(1), 279-279. https://doi.org/10.1186/s12884-017-1462-x Reasons of Atheists and Agnostics for Nonbelief in God’s Existence Scale Bradley, D. F., Exline, J. J., Uzdavines, A., Stauner, N., & Grubbs, J. B. (2018). The reasons of atheists and agnostics for nonbelief in god's existence scale: Development and initial validation. Psychology of Religion and Spirituality, 10(3), 263-275. https://doi.org/10.1037/rel0000199 Relationship Marketing Orientation Scale Sin, L. Y. M., Tse, A. C. B., Yau, O. H. M., Chow, R. P. M., Lee, J. S. Y., & Lau, L. B. Y. (2005). Relationship marketing orientation: Scale development and cross-cultural validation. Journal of Business Research, 58(2), 185-194. https://doi.org/10.1016/S0148-2963(02)00493-9 Religious Head: Heart Beliefs Scale Park, C. L., & Carney, L. M. (2022). Religious head versus heart beliefs: Measurement development and validation. Psychology of Religion and Spirituality, 14(4), 572-584. https://doi.org/10.1037/rel0000352 Responsibility and Interpersonal Behaviors and Attitudes Questionnaire Moritz, S., Wahl, K., Ertle, A., Jelinek, L., Hauschildt, M., Klinge, R., & Hand, I. (2009). Neither saints nor wolves in disguise: Ambivalent interpersonal attitudes and behaviors in obsessive-compulsive disorder. Behavior Modification, 33(2), 274-292. https://doi.org/10.1177/0145445508327444 Restaurant Authenticity Scale Lehman, D. W., O’Connor, K., & Carroll, G. R. (2019). Acting on authenticity: Individual interpretations and behavioral responses. Review of General Psychology, 23(1), 19-31. https://doi.org/10.1177/1089268019829470 S-Scale Velten, J., Scholten, S., Brailovskaia, J., & Margraf, J. (2021). Psychometric properties of the S-scale: Assessing a psychological mindset that mediates the relationship between socioeconomic status and depression. PloS One, 16(10), e0258333-e0258333. https://doi.org/10.1371/journal.pone.0258333 Satisfaction with Simulation Experience Scale Levett-Jones, T., McCoy, M., Lapkin, S., Noble, D., Hoffman, K., Dempsey, J., Arthur, C., & Roche, J. (2011). The development and psychometric testing of the satisfaction with simulation experience scale. Nurse Education Today, 31(7), 705-710. https://doi.org/10.1016/j.nedt.2011.01.004 School Support Measure McGoron, L., Wargo Aikins, J., Trentacosta, C. J., Gómez, J. M., & Beeghly, M. (2022). School support, chaos, routines, and parents' mental health during COVID-19 remote schooling. School Psychology, 37(2), 173-182. https://doi.org/10.1037/spq0000467 Scientific Research Competency Scale Duru, P., & Örsal, Ö. (2021). Development of the scientific research competency scale for nurses. Journal of Research in Nursing, 26(7), 684-700. https://doi.org/10.1177/17449871211020061 Self-Disclosure Index Miller, L. C., Berg, J. H., & Archer, R. L. (1983). Openers: Individuals who elicit intimate self-disclosure. Journal of Personality and Social Psychology, 44(6), 1234-1244. https://doi.org/10.1037/0022-3514.44.6.1234 89 Self-efficacy Measure--Experiment Version Hecht, T. D., & Allen, N. J. (2005). Exploring links between polychronicity and well-being from the perspective of person–job fit: Does it matter if you prefer to do only one thing at a time? Organizational Behavior and Human Decision Processes, 98(2), 155-178. https://doi.org/10.1016/j.obhdp.2005.07.004 Self-Talk Questionnaire Jansen, A., Merckelbach, H., Oosterlaan, J., Tuiten, A., & van den Hout, M. (1988). Cognitions and self-talk during food intake of restrained and unrestrained eaters. Behaviour Research and Therapy, 26(5), 393-398. https://doi.org/10.1016/0005-7967(88)90072-1 Sexual Disgust Inventory Crosby, C. L., Durkee, P. K., Meston, C. M., & Buss, D. M. (2020). Six dimensions of sexual disgust. Personality and Individual Differences, 156, 109714. https://doi.org/10.1016/j.paid.2019.109714 Sport-Related Worry and Rumination Scale Balk, Y. A., Tamminen, K. A., & Eccles, D. W. (2021). Too tired to switch off? how post-training physical fatigue impairs mental recovery through increased worry. Sport, Exercise, and Performance Psychology, 10(4), 489-503. https://doi.org/10.1037/spy0000258 Student Rating of Environmental Stressors Scale Suldo, Shannon M., Dedrick, Robert F., Shaunessy-Dedrick, Elizabeth, Roth, Rachel A., & Ferron, John. (2015). Development and initial validation of the student rating of environmental stressors scale: Stressors faced by students in accelerated high school curricula. Journal of Psychoeducational Assessment, 33(4), 339-356. doi: https://dx.doi.org/10.1177/0734282914552164 Surgeons’ Interactions with and Attitudes Toward E-Patients Measure Masters, K., Loda, T., Johannink, J., Al-Abri, R., & Herrmann-Werner, A. (2020). Surgeons' interactions with and attitudes toward E-patients: Questionnaire study in Germany and Oman. Journal of Medical Internet Research, 22(3), e14646-e14646. https://doi.org/10.2196/14646 Task-Contingent Conscientiousness Scale Huang, Jason L., & Bramble, Reed J. (2016). Trait, state, and task-contingent conscientiousness: Influence on learning and transfer. Personality and Individual Differences, 92, 180-185. doi: https://dx.doi.org/10.1016/j.paid.2015.12.043 Teachers' Preferences for Learning Activities Scale Louws, M. L., Meirink, J. A., van Veen, K., & van Driel, J. H. (2017). Teachers' self-directed learning and teaching experience: What, how, and why teachers want to learn. Teaching and Teacher Education, 66, 171-183. https://doi.org/10.1016/j.tate.2017.04.004 Trans and Nonbinary Coping Measure Lindley, L., & Budge, S. L. (2024). Development and validation of the trans and nonbinary coping measure (TNCM): A measure of trans and nonbinary specific ways of coping with gender-related stress. Psychology of Sexual Orientation and Gender Diversity, 11(3), 425-441. https://doi.org/10.1037/sgd0000618 UCLA Multidimensional Condom Attitudes Scale Helweg-Larsen, M., & Collins, B. E. (1994). The UCLA multidimensional condom attitudes scale: Documenting the complex determinants of condom use in college students. Health Psychology, 13(3), 224-237. https://doi.org/10.1037/0278-6133.13.3.224 Utilitarian Meaning and Piecemeal Judgment Scale Allen, M. W., & Ng, S. H. (1999). The direct and indirect influences of human values on product ownership. Journal of Economic Psychology, 20(1), 5-39. https://doi.org/10.1016/S0167-4870(98)00041-5 90 Widowhood Resilience Scale West, C. L., Dreeben, S. J., & Busing, K. (2021). The development of the widowhood resilience scale. Omega: Journal of Death and Dying, 83(4), 958-975. https://doi.org/10.1177/0030222819873489 Workplace Safety Management Practices and Employees’ Organizational Citizenship Behavior During COVID-19 Vu, T., Vo-Thanh, T., Nguyen, N. P., Nguyen, D. V., & Chi, H. (2022). The COVID-19 pandemic: Workplace safety management practices, job insecurity, and employees’ organizational citizenship behavior. Safety Science, 145, 105527-105527. https://doi.org/10.1016/j.ssci.2021.105527 91 Appendix D: DASS Codebook This data was collected with an on-line version of the Depression Anxiety Stress Scales (DASS), see http://www2.psy.unsw.edu.au/dass/. This data was collected 2017 – 2019. The survey was open to anyone, and people were motivated to take it to get personalized results. At the end of the test, they also were given the option to complete a short research survey. This dataset comes from those who agreed to complete the research survey and answered yes to the question "Have you given accurate answers and may they be used for research?" at the end. DASS The following items were included in the survey: D1 I couldn't seem to experience any positive feeling at all. D2 I just couldn’t seem to get going. D3 I felt that I had nothing to look forward to. D4 I felt sad and depressed. D5 I felt that I had lost interest in just about everything. D6 I felt I wasn’t worth much as a person. D7 I felt that life wasn’t worthwhile. D8 I couldn’t seem to get any enjoyment out of the things I did. D9 I felt down-hearted and blue. D10 I was unable to become enthusiastic about anything. D11 I felt I was pretty worthless. D12 I could see nothing in the future to be hopeful about. D13 I felt that life was meaningless. D14 I found it difficult to work up the initiative to do things. A1 I was aware of dryness of my mouth. A2 I experienced breathing difficulty (eg, excessively rapid breathing, breathlessness in the absence of physical exertion). A3 I had a feeling of shakiness (eg, legs going to give way). A4 I found myself in situations that made me so anxious I was most relieved when they ended. A5 I had a feeling of faintness. 92 A6 I perspired noticeably (eg, hands sweaty) in the absence of high temperatures or physical exertion. A7 I felt scared without any good reason. A8 I had difficulty in swallowing. A9 I was aware of the action of my heart in the absence of physical exertion (eg, sense of heart rate increase, heart missing a beat). A10 I felt I was close to panic. A11 I feared that I would be “thrown” by some trivial but unfamiliar task. A12 I felt terrified. A13 I was worried about situations in which I might panic and make a fool of myself. A14 I experienced trembling (eg, in the hands). S1 I found myself getting upset by quite trivial things. S2 I tended to over-react to situations. S3 I found it difficult to relax. S4 I found myself getting upset rather easily. S5 I felt that I was using a lot of nervous energy. S6 I found myself getting impatient when I was delayed in any way (eg, elevators, traffic lights, being kept waiting). S7 I felt that I was rather touchy. S8 I found it hard to wind down. S9 I found that I was very irritable. S10 I found it hard to calm down after something upset me. S11 I found it difficult to tolerate interruptions to what I was doing. S12 I was in a state of nervous tension. S13 I was intolerant of anything that kept me from getting on with what I was doing. S14 I found myself getting agitated. Each item was presented one at a time in a random order for each new participant along with a 4-point rating scale asking the user to indicate how often that had been true of them in the past week where: 1 = Did not apply to me at all 2 = Applied to me to some degree, or some of the time 93 3 = Applied to me to a considerable degree, or a good part of the time 4 = Applied to me very much, or most of the time Vocabulary Recognition Measure The following items were presented as a checklist and subjects were instructed "In the grid below, check all the words whose definitions you are sure you know". A value of 1 is checked, 0 means unchecked. The words at V6, V9, and V12 are fake words and can be used as a validity check. V1 boat V2 incoherent V3 pallid V4 robot V5 audible V6 cuivocal V7 paucity V8 epistemology V9 florted V10 decide V11 pastiche V12 verdid V13 abysmal V14 lucid V15 betray V16 funny 94 Demographic Questions The following demographic questions were then asked: Education "How much education have you completed?" 1=Less than high school 2=High school 3=University degree 4=Graduate degree Gender "What is your gender?" 1=Male 2=Female 3=Other Eng_nat "Is English your native language?", 1=Yes 2=No Age "How many years old are you?" Race "What is your race?" 10=Asian 20=Arab 30=Black 40=Indigenous Australian 50=Native American 60=White 70=Other The following values were derived from technical information: Country ISO country code of where the user connected from Unique_network 1=only one survey from user's specific network in dataset, 2=multiple surveys submitted from the network of this user 95 Appendix E: DASS CFA Results and Descriptive Statistics The factor loadings for the bifactor model (one general factor and three specific factors) as well as the covariances are shown in Table 10. Table 10 CFA Factor Loadings for DASS-42 General Estimate D1 0.608 D2 0.684 D3 0.574 D4 0.559 D5 0.543 D6 0.561 D7 0.589 D8 0.555 D9 0.625 D10 0.582 D11 0.659 D12 0.642 D13 0.584 D14 0.553 A1 0.554 A2 0.566 A3 0.361 A4 0.629 A5 0.530 A6 0.616 A7 0.597 A8 0.703 A9 0.727 A10 0.697 A11 0.793 A12 0.752 A13 0.766 A14 0.543 S1 0.692 S2 0.760 96 S3 0.821 S4 0.755 S5 0.799 S6 0.756 S7 0.739 S8 0.697 S9 0.708 S10 0.690 S11 0.635 S12 0.613 S13 0.575 S14 0.598 Group1 D1 0.564 D2 0.563 D3 0.637 D4 0.665 D5 0.758 D6 0.746 D7 0.678 D8 0.711 D9 0.616 D10 0.582 D11 0.521 D12 0.552 D13 0.453 D14 0.478 Group2 A1 0.416 A2 0.352 A3 0.393 A4 0.543 A5 0.445 A6 0.633 A7 0.553 A8 0.263 A9 0.281 A10 0.101 A11 0.319 A12 0.235 97 A13 0.298 A14 0.417 Group3 S1 -0.108 S2 0.150 S3 -0.130 S4 -0.110 S5 -0.191 S6 0.424 S7 0.372 S8 0.510 S9 0.297 S10 0.546 S11 0.224 S12 0.417 S13 0.403 S14 0.425 Covariances General ~~ Group1 0.000 Group2 0.000 Group3 0.000 Group1 ~~ Group2 0.000 Group3 0.000 Group2 ~~ Group3 0.000 98 Table 11 presents the distribution of item response for the 42 DASS items. The counts and percentages for each response option are provided as is the mean, median and range for each item. Table 11 Distribution of Item Responses Item Response 1 Response 2 Response 3 Response 4 Mean Median Range n % n % n % n % D1 73 14.6 139 27.8 135 27 153 30.6 2.7 3 1-4 D2 65 13 134 26.8 134 26.8 167 33.4 2.8 3 1-4 D3 162 32.4 146 29.2 94 18.8 98 19.6 2.3 2 1-4 D4 139 27.8 136 27.2 96 19.2 129 25.8 2.4 2 1-4 D5 181 36.2 108 21.6 84 16.8 127 25.4 2.3 2 1-4 D6 168 33.6 119 23.8 96 19.2 117 23.4 2.3 2 1-4 D7 130 26 144 28.8 90 18 136 27.2 2.5 2 1-4 D8 141 28.2 120 24 101 20.2 138 27.6 2.5 2 1-4 D9 127 25.4 145 29 102 20.4 126 25.2 2.5 2 1-4 D10 107 21.4 155 31 128 25.6 110 22 2.5 2 1-4 D11 137 27.4 186 37.2 108 21.6 69 13.8 2.2 2 1-4 D12 97 19.4 170 34 119 23.8 114 22.8 2.5 2 1-4 D13 57 11.4 150 30 114 22.8 179 35.8 2.8 3 1-4 D14 44 8.8 125 25 124 24.8 207 41.4 3.0 3 1-4 A1 180 36 163 32.6 93 18.6 64 12.8 2.1 2 1-4 A2 270 54 127 25.4 52 10.4 51 10.2 1.8 1 1-4 A3 226 45.2 143 28.6 79 15.8 52 10.4 1.9 2 1-4 A4 258 51.6 120 24 64 12.8 58 11.6 1.8 1 1-4 A5 364 72.8 87 17.4 33 6.6 16 3.2 1.4 1 1-4 A6 272 54.4 122 24.4 60 12 46 9.2 1.8 1 1-4 A7 257 51.4 128 25.6 56 11.2 59 11.8 1.8 1 1-4 A8 175 35 115 23 107 21.4 103 20.6 2.3 2 1-4 A9 124 24.8 150 30 126 25.2 100 20 2.4 2 1-4 A10 200 40 132 26.4 96 19.2 72 14.4 2.1 2 1-4 A11 185 37 138 27.6 105 21 72 14.4 2.1 2 1-4 A12 240 48 137 27.4 64 12.8 59 11.8 1.9 2 1-4 A13 203 40.6 146 29.2 73 14.6 78 15.6 2.1 2 1-4 A14 276 55.2 141 28.2 53 10.6 30 6 1.7 1 1-4 S1 101 20.2 165 33 141 28.2 93 18.6 2.5 2 1-4 S2 92 18.4 176 35.2 116 23.2 116 23.2 2.5 2 1-4 99 S3 81 16.2 150 30 134 26.8 135 27 2.6 3 1-4 S4 125 25 163 32.6 132 26.4 80 16 2.3 2 1-4 S5 91 18.2 167 33.4 137 27.4 105 21 2.5 2 1-4 S6 90 18 159 31.8 139 27.8 112 22.4 2.5 3 1-4 S7 105 21 184 36.8 119 23.8 92 18.4 2.4 2 1-4 S8 53 10.6 202 40.4 125 25 120 24 2.6 2 1-4 S9 103 20.6 175 35 125 25 97 19.4 2.4 2 1-4 S10 73 14.6 185 37 114 22.8 128 25.6 2.6 2 1-4 S11 123 24.6 177 35.4 118 23.6 82 16.4 2.3 2 1-4 S12 133 26.6 184 36.8 113 22.6 70 14 2.2 2 1-4 S13 94 18.8 178 35.6 117 23.4 111 22.2 2.5 2 1-4 S14 119 23.8 189 37.8 111 22.2 81 16.2 2.3 2 1-4 Note: Response 1 = Did not apply to me at all, Response 2 = Applied to me to some degree, or some of the time, Response 3 = Applied to me to a considerable degree, or a good part of the time, Response 4 = Applied to me very much, or most of the time 100 Appendix F: DASS Coding of Dependent Clause Types Table 12 provides the presence/absence coding for all of the DASS-42 items for each dependent clause type. The totals represent the overall presence of each clause type across the 42 items. Table 12 Presence/Absence Coding of DASS Items Item # Item Fin advl Fin comp Fin rel Nf comp Nf rel A1 I was aware of dryness of my mouth. 0 0 0 0 0 A2 I experienced breathing difficulty (eg, excessively rapid breathing, breathlessness in the absence of physical exertion). 0 0 0 0 0 A3 I had a feeling of shakiness (eg, legs going to give way). 0 0 0 0 1 A4 I found myself in situations that made me so anxious I was most relieved when they ended. 1 0 1 0 0 A5 I had a feeling of faintness. 0 0 0 0 0 A6 I perspired noticeably (eg, hands sweaty) in the absence of high temperatures or physical exertion. 0 0 0 0 0 A7 I felt scared without any good reason. 0 0 0 0 0 A8 I had difficulty in swallowing. 0 0 0 0 0 A9 I was aware of the action of my heart in the absence of physical exertion (eg, sense of heart rate increase, heart missing a beat). 0 0 0 0 1 A10 I felt I was close to panic. 0 1 0 0 0 A11 I feared that I would be “thrown” by some trivial but unfamiliar task. 0 1 0 0 0 A12 I felt terrified. 0 0 0 0 0 A13 I was worried about situations in which I might panic and make a fool of myself. 0 0 1 0 0 A14 I experienced trembling (eg, in the hands). 0 0 0 1 0 101 Item # Item Fin advl Fin comp Fin rel Nf comp Nf rel D1 I couldn't seem to experience any positive feeling at all. 0 0 0 1 0 D2 I just couldn’t seem to get going. 0 0 0 1 0 D3 I felt that I had nothing to look forward to. 0 1 0 1 0 D4 I felt sad and depressed. 0 0 0 0 0 D5 I felt that I had lost interest in just about everything. 0 1 0 0 0 D6 I felt I wasn’t worth much as a person. 0 1 0 0 0 D7 I felt that life wasn’t worthwhile. 0 1 0 0 0 D8 I couldn’t seem to get any enjoyment out of the things I did. 0 0 1 1 0 D9 I felt down-hearted and blue. 0 0 0 0 0 D10 I was unable to become enthusiastic about anything. 0 0 0 1 0 D11 I felt I was pretty worthless. 0 1 0 0 0 D12 I could see nothing in the future to be hopeful about. 0 0 0 1 0 D13 I felt that life was meaningless. 0 1 0 0 0 D14 I found it difficult to work up the initiative to do things. 0 0 0 1 0 S1 I found myself getting upset by quite trivial things. 0 0 0 1 0 S2 I tended to over-react to situations. 0 0 0 1 0 S3 I found it difficult to relax. 0 0 0 1 0 S4 I found myself getting upset rather easily. 0 0 0 1 0 S5 I felt that I was using a lot of nervous energy. 0 1 0 0 0 S6 I found myself getting impatient when I was delayed in any way (eg, elevators, traffic lights, being kept waiting). 1 0 0 1 0 S7 I felt that I was rather touchy. 0 1 0 0 0 S8 I found it hard to wind down. 0 0 0 1 0 S9 I found that I was very irritable. 0 1 0 0 0 S10 I found it hard to calm down after something upset me. 1 0 0 1 0 102 Item # Item Fin advl Fin comp Fin rel Nf comp Nf rel S11 I found it difficult to tolerate interruptions to what I was doing. 0 1 0 1 0 S12 I was in a state of nervous tension. 0 0 0 0 0 S13 I was intolerant of anything that kept me from getting on with what I was doing. 0 1 1 1 0 S14 I found myself getting agitated. 0 0 0 1 0 Total 3 13 4 18 2 103 Appendix G: Sample R Code #### Load required packages library(haven) library(ordinal) library(dplyr) library(tidyr) library(marginaleffects) library(tidyverse) library(effects) library(car) library(ggeffects) library(RVAideMemoire) library(emmeans) library(r2mlm) # for R-squared values library(performance) # for ICC library(rcompanion) library(DHARMa) library(lmtest) library(modelsummary) library(MuMIn) library(ggplot2) library(car) library(MASS) library(sjstats) library(easystats) library(apaTables) library(sjPlot) library(officer) library(flextable) library(lme4) library(misty) ################################################################### #### Read in the data data <- read_csv() ######################################################################## ## convert *explanatory variables to factors* for the CLMM command data$items42<- as.factor(data$items42) data$person<- as.factor(data$ID) ## convert *DVs to ordinal* for the CLMM command data$Response <- as.ordered(data$Response) ######################################################################## # 1. Group-Center Level-1 Predictors 104 # Group-center the level-1 predictors (fin_comp, fin_advl, fin_rel, nf_comp, nf_rel) using person as the clustering variable data$fin_comp_gc <- center(data$fin_comp, cluster = data$person, type = "CWC") data$fin_advl_gc <- center(data$fin_advl, cluster = data$person, type = "CWC") data$fin_rel_gc <- center(data$fin_rel, cluster = data$person, type = "CWC") data$nf_comp_gc <- center(data$nf_comp, cluster = data$person, type = "CWC") data$nf_rel_gc <- center(data$nf_rel, cluster = data$person, type = "CWC") # 2. Grand-Mean Center Level-2 Predictor # Grand-mean-center the level-2 predictor Vocab_Total data$Vocab_Total_gm <- center(data$Vocab_Total, type = "CGM") # 3. Verify Centering # To confirm that the group-centering and grand-mean-centering were applied correctly: # For group-centered variables: The mean of the centered variable within each person should be approximately zero: aggregate(cbind(fin_comp_gc, fin_advl_gc, fin_rel_gc, nf_comp_gc, nf_rel_gc) ~ person, data, mean) # For the grand-mean-centered variable: The overall mean should be close to zero: mean(data$Vocab_Total_gm) summary(Vocab_Total_gm) ######################################################################## # Step 1: Baseline model with no predictors. Used to calculate the baseline ICC and DEFF, # and as a reference for a Likelihood Ratio Test (LRT) # Probit mixed-effects model m0 <- clmm(Response ~ 1 + (1|person), link = "probit", data = data) logLik(m0) summary(m0) # A design effect (DEFF) was computed for each outcome variable based on the # average cluster size. A DEFF greater than 2 indicates that the data's clustering needs # to be considered during estimation (Muthén & Satorra, 1995). # By design, the average cluster size is 42 for this data. # library(performance) #ICCmodel <- performance::icc(m0) #ICCmodel #design_effect(42, icc = ICCmodel) m0a <- clmm(Response ~ 1 + (1|person) + (1|items42), link = "probit", data = data) 105 logLik(m0a) summary(m0a) # Summary of conditional random effects summary_random <- random_parameters(m0a, component = "conditional") print(summary_random) lrtest(m0, m0a) ################################################ ## The following models are used later in this code to test the ## statistical significance of a model (1) over-and-above the most ## basic model with only the grand mean, and (2) over-and-above the ## basic random effects structure. ## A model comparison to (1) is similar to the ANOVA F-test of a model ## in regression or linear modeling. # Fit the null model (without ANY effects, only the grand mean) null_model0 <- clm(Response ~ 1 , link = "probit", data = data) logLik(null_model0) summary(null_model0) # Fit the null model (without fixed effects) null_model <- m0a #m0a <- clmm(Response ~ 1 + # (1|person) + # (1|items42), # link = "probit", # data = data) logLik(null_model) summary(null_model) ######################################################################## #### LANGUAGE FEATURE: nf_comp ### SECTION #1: QUANTIFYING AND TESTING THE SIGNIFICANCE OF ### SECTION #1: THE RELATIONSHIP BETWEEN THE VECTOR OF ITEM ### SECTION #1: RESPONSES AND THE LANGUAGE FEATURE ## In the model below the random effect of the person is a proxy for the total DASS score # Random Intercept and Slope model, with a correlation of intercept and slope m1_nf_comp <- clmm(Response ~ 1 + nf_comp_gc + (1+nf_comp_gc|person) + (1|items42), link = "probit", data = data) 106 logLik(m1_nf_comp) summary(m1_nf_comp) # To test the overall statistical significance of the model, # compare the full and null models using likelihood ratio test. lrtest(null_model0,m1_nf_comp) # Similar to the F-test test of a model, Anova.clmm(m1_nf_comp) # To test the overall statistical significance of the fixed effects model, # compare the full and null models using likelihood ratio test. lrtest(null_model,m1_nf_comp) # statistical significance of all fixed effects. ################################################################# ## Post-hoc test if the main effect of Lang Feature is stat. signif. # Frequency table of nf_comp_gc frequency_table <- table(data$nf_comp_gc) # Print the frequency table print(frequency_table) # Compute emmeans for the fixed effect emmeans_results <- emmeans(m1_nf_comp, ~ nf_comp_gc, at = list(nf_comp_gc = c(-0.428571428571429, 0.571428571428571))) # Specify value # Display the results emmeans_results # Back-transform emmeans to the response scale emmeans_original <- emmeans(m1_nf_comp, ~ nf_comp_gc, type = "response") # Display the back-transformed emmeans print(emmeans_original) pairwise_comparisons <- contrast(emmeans_results, method = "pairwise") pairwise_comparisons summary(data$Response) # On probit scale emmeans_probit <- emmeans(m1_nf_comp, ~ nf_comp_gc) # Back-transformed to original scale emmeans_response <- emmeans(m1_nf_comp, ~ nf_comp_gc, type = "response") # Compare list(Probit = emmeans_probit, Response = emmeans_response) 107 library(ggplot2) ggplot(data, aes(x = nf_comp_gc, fill = as.factor(Response))) + geom_bar(position = "fill") + labs(fill = "Response") ####################################################################### #### INVESTIGATION OF THE CROSS-LEVEL INTERACTION of VOCAB_TOTAL ### SECTION #2: A CHECK ON THE IMPLIED INTERACTION OF VOCAB ### AND nf_comp ABOVE # The analysis reported above investigating the association of the LF at # different levels of Vocab_Total implies an interaction of Vocab_Total and the LF m1_nf_comp_interaction <- clmm(Response ~ 1 + Vocab_Total_gm + nf_comp_gc + (Vocab_Total_gm*nf_comp_gc) + (1 + nf_comp_gc | person) + (1 | items42), link = "probit", data = data) logLik(m1_nf_comp_interaction) summary(m1_nf_comp_interaction) m1_nf_comp_Vocab <- clmm(Response ~ 1 + Vocab_Total_gm + nf_comp_gc + (1 + nf_comp_gc | person) + (1 | items42), link = "probit", data = data) logLik(m1_nf_comp_Vocab) summary(m1_nf_comp_Vocab) lrtest(m1_nf_comp_Vocab, m1_nf_comp_interaction) # Similar to the F-test test of a model, ################################################################### ## Post-hoc if the interaction is stat. signif. ################################################################### # Get estimated marginal means on the metric of responding 1-4 summary(Vocab_Total_gm) ## The summary statistics for Vocab_Total, ## Min. 1st Qu. Median Mean 3rd Qu. Max. ## 4.00 9.00 9.00 9.68 11.00 13.00 ## Min. 1st Qu. Median Mean 3rd Qu. Max. ## -5.68, -0.68, -0.68, 0.00, 1.32, 3.32 108 ######################################### # Specify levels of vocab_total for simple effects emm_interaction2 <- emmeans(m1_nf_comp_interaction, ~ nf_comp_gc | Vocab_Total_gm, type = "link", at = list(Vocab_Total_gm = c(-5.68, -0.68, 0.00, 1.32, 3.32))) emm_interaction3 <- emmeans(m1_nf_comp_interaction, ~ nf_comp_gc | Vocab_Total_gm, mode = "mean.class", at = list(Vocab_Total_gm = c(-5.68, -0.68, 0.00, 1.32, 3.32))) # Print the marginal means print(emm_interaction3) # Pairwise comparisons for nf_comp at each level of Vocab_Total pairwise_results3 <- contrast(emm_interaction3, method = "pairwise", adjust = "BH") summary(pairwise_results3) plot(emm_interaction3, CIs=TRUE, comparisons=TRUE) # Print the marginal means print(emm_interaction2) # Pairwise comparisons for nf_comp at each level of Vocab_Total pairwise_results2 <- contrast(emm_interaction2, method = "pairwise", adjust = "BH") summary(pairwise_results2) #print(pairwise_results2) ## Interaction plot with the Dependent variable being the original response scale (1,2,3,4) ## Vocab_Total plots for the 25th, 50th, and 75th percentiles ## (i.e., 1st quartile, median, and 3rd quartiles) (emm_int2 <- emmeans(m2, ~ LF2 | Vocab_Total, mode = "mean.class", at = list(Vocab_Total = c(4,9,11)))) plot(emm_int2, CIs=TRUE, comparisons=TRUE) + coord_flip() ######################################### # Specify levels of vocab_total for simple effects emm_interaction1 <- emmeans(m1_nf_comp_interaction, ~ Vocab_Total_gm | nf_comp_gc, type = "link", at = list(Vocab_Total_gm = c(-5.68, -0.68, 0.00, 1.32, 3.32))) #Print the marginal means print(emm_interaction1) # Pairwise comparisons for fin_advl at each level of Vocab_Total pairwise_results1 <- contrast(emm_interaction1, method = "pairwise", adjust = "BH") summary(pairwise_results1) 109 ######################################################################## # Visualize the Marginal Effects # You can plot the marginal effects of the language feature at levels of the # Vocab for better interpretation: # emmeans: Estimated Marginal Means, and given that the probit link is invoked, the # results are on the scale of the *UNDERLYING LATENT RESPONSE VARIABLE* N(0,1), Z-scores. # The language feature is, in essence, in the Cohen d metric. #### Plot of the language feature effect at the underlying variable ### summary(m1_nf_comp_interaction) # What follows also produces the means on the latent response scale. # The contrast provides what is, in essence, the Cohen's d effect size. #(emm_wt <- emmeans(m1_nf_comp_interaction, specs=pairwise~nf_comp_gc)) m1_nf_comp_interaction$data$nf_comp_gc <- as.factor(m1_nf_comp_interaction$data$nf_comp_gc) m1_nf_comp_interaction$data$Vocab_Total_gm <- as.factor(m1_nf_comp_interaction$data$Vocab_Total_gm) ####### ## nf_comp by Vocab level (Vocab on the X-axis) emm_wt2 <- emmeans(m1_nf_comp_interaction, ~ nf_comp_gc | Vocab_Total_gm, # Switch the focus type = "link", at = list(Vocab_Total_gm = c(-5.68, -0.68, 0.00, 1.32, 3.32)), infer = c(TRUE, TRUE)) # Ensure CIs are computed contrast_results <- pairs(emm_wt2, adjust = "none") # Two group per contrast to no adjment needed. print(contrast_results) emm_df <- as.data.frame(emm_wt2) attach(emm_df) view(emm_df) #emm_df <- as.data.frame(contrast_results) print(colnames(emm_df)) # Check for CI columns # ---- Save emm_df to a CSV file that can be opened with Excel ---- write.csv(emm_df, "emm_df.csv", row.names = FALSE) # Open the .csv file emm_df and change the values of the "nc_comp_gc" and the "Vocab_Total_gm" 110 emm_df <- read_csv("emm_df.csv") attach(emm_df) view(emm_df) print(colnames(emm_df)) # Check for CI columns ggplot(emm_df, aes(x = Vocab_Total_gm, y = emmean, color = as.factor(nf_comp_gc))) + geom_point(size = 3) + geom_line(aes(group = nf_comp_gc)) + geom_errorbar(aes(ymin = asymp.LCL, ymax = asymp.UCL), width = 0.2) + # Use asymp.LCL/UCL labs( title = "Marginal Effect of Vocabulary", x = "Vocabulary Score (range 3-13)", y = "Estimated Marginal Means - Z-score metric" ) + scale_color_manual( name = "Nonfinite Complement", values = c("black", "blue", "grey", "red") # Custom colors ) + theme_bw() ########################################################################
Thesis/Dissertation
eng
Measurement, Evaluation and Research Methodology
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Zumbo, Bruno D.
Unpacking the grammar of tests and measures : the hidden complexity in psychological test items and its association with item responses
Text