Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Improving the factor structure of psychological scales : the Expanded format as the alternative to the… Zhang, Xijuan 2015

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
24-ubc_2015_september_zhang_xijuan.pdf [ 3.98MB ]
Metadata
JSON: 24-1.0166371.json
JSON-LD: 24-1.0166371-ld.json
RDF/XML (Pretty): 24-1.0166371-rdf.xml
RDF/JSON: 24-1.0166371-rdf.json
Turtle: 24-1.0166371-turtle.txt
N-Triples: 24-1.0166371-rdf-ntriples.txt
Original Record: 24-1.0166371-source.json
Full Text
24-1.0166371-fulltext.txt
Citation
24-1.0166371.ris

Full Text

 IMPROVING THE FACTOR STRUCTURE OF PSYCHOLOGICAL SCALES: THE EXPANDED FORMAT AS THE ALTERNATIVE TO THE LIKERT SCALE FORMAT by Xijuan Zhang B.A., The University of British Columbia, 2015 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS in The Faculty of Graduate and Postdoctoral Studies (Psychology)  THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver)   July 2015   © Xijuan Zhang, 2015 ii Abstract Many psychological scales written in the Likert format include reverse worded (RW) items in order to control acquiescence bias. However, studies have shown that RW items often contaminate the factor structure of the scale by creating one or more method factors. The present study examines an alternative scale format, called the Expanded format, which replaces each response option in the Likert scale with a full sentence. We hypothesized that this format would result in a cleaner factor structure as compared to the Likert format. We tested this hypothesis on three popular psychological scales: the Rosenberg Self-Esteem scale, the Conscientiousness subscale of the Big Five Inventory, and the Beck Depression Inventory II. Scales in both formats showed comparable reliabilities and convergent validities. However, scales in the Expanded format had better (i.e., lower and more theoretically defensible) dimensionalities than scales in the Likert format, as assessed by both exploratory factor analyses and confirmatory factor analyses. We encourage further study and wider use of the Expanded format, particularly when the dimensionality of a scale is of theoretical interest.    iii Preface This thesis is based on the research conducted by my supervisor, Dr. Victoria Savalei and myself. Equations in section 1.2 are mainly based on unpublished research notes written by Dr. Carl Falk and Dr. Victoria Savalei, and their published paper, Recovering substantive factor loadings in the presence of acquiescence bias: A comparison of three approaches, in the Multivariate Behavioral Research journal in 2014. This research has not been published. UBC Research Ethics Board approval was required and obtained to conduct all stages of this research. The certificate number is H13-02870.    iv Table of Contents  Abstract ..................................................................................................................................... ii Preface ...................................................................................................................................... iii Table of Contents ..................................................................................................................... iv List of Tables ............................................................................................................................ vi List of Figures ........................................................................................................................ viii Acknowledgements ................................................................................................................... ix Dedication .................................................................................................................................. x Chapter  1: Introduction .................................................................................................................... 1 1.1 Definition and Purpose of Reverse Worded Items ................................................................. 1 1.2 Why Reverse Worded Items Do Not Control Acquiescence Bias ......................................... 2 1.3 Other Problems with Reverse Worded Items ......................................................................... 4 1.4 Models for Removing Method Effects and Acquiescence Bias ............................................. 6 1.5 An Alternative Scale Format: The Expanded Format ............................................................ 8 1.6 Goals of the Present Research .............................................................................................. 10 Chapter  2: Method .......................................................................................................................... 12 2.1 Participants ........................................................................................................................... 12 2.2 Procedure .............................................................................................................................. 12 2.3 Measures ............................................................................................................................... 13 2.3.1 Rosenberg Self-Esteem Scale ...................................................................................................... 13 2.3.2 Conscientiousness Scale .............................................................................................................. 13 2.3.3 Beck Depression Inventory .......................................................................................................... 14 2.4 Analytic Method ................................................................................................................... 15 Chapter  3: Results ........................................................................................................................... 18 3.1 Descriptive Statistics ............................................................................................................ 18 3.2 Exploratory Factor Analysis ................................................................................................. 18 3.3 Confirmatory Factor Analysis .............................................................................................. 21 3.3.1 Model Fit ...................................................................................................................................... 21 3.3.2 Standardized Solution .................................................................................................................. 23 3.4 Reliability and Convergent Validity ..................................................................................... 24 Chapter  4: Discussion and Conclusion .......................................................................................... 26 4.1 Limitations and Future Directions ........................................................................................ 29 4.2 Recommendations for Making Scales in the Expanded Format ........................................... 32  v 4.3 Conclusion ............................................................................................................................ 33 Bibliography ............................................................................................................................ 35 Appendices ............................................................................................................................... 44 Appendix A Rosenberg Self-Esteem Scale ................................................................................ 44 A.1 Likert Version .............................................................................................................. 44 A.2 Expanded Version ........................................................................................................ 45 Appendix B Conscientious Scale ................................................................................................ 47 B.1 Likert Version .............................................................................................................. 47 B.2 Expanded Version ........................................................................................................ 48 Appendix C Beck Depression Inventory .................................................................................... 50 C.1 Likert Version I ............................................................................................................ 50 C.2 Likert Version II .......................................................................................................... 52 Appendix D Figures for Introduction ......................................................................................... 54 Appendix E Figure for Method ................................................................................................... 58 Appendix F Figures for Results .................................................................................................. 59 Appendix G Tables for Results ................................................................................................... 61   vi List of Tables Table 1: Covariance Matrix for the Likert Rosenberg Self-Esteem Scale ...................................................... 61!Table 2: Covariance Matrix for the Expanded Rosenberg Self-Esteem Scale ............................................... 62!Table 3: Covariance Matrix for the Likert Conscientiousness Scale ............................................................. 63!Table 4: Covariance Matrix for the Expanded Conscientiousness Scale ....................................................... 64!Table 5: Covariance Matrix of the Likert Version I Beck Depression Inventor ............................................. 65!Table 6: Covariance Matrix of the Likert Version II Beck Depression Inventory ......................................... 66!Table 7: Covariance Matrix of the Expanded Beck Depression Inventory (a.k.a. BDI-II) ............................ 67!Table 8: Means and Standard Deviations for Two Versions of the Rosenberg Self-Esteem Scale ................ 68!Table 9: Means and Standard Deviations for Two Versions of the Conscientiousness Scale ........................ 69!Table 10: Means and Standard Deviations for All Versions of the Beck Depression Inventory .................... 70!Table 11: Item Endorsement Proportion for the Likert Rosenberg Self-Esteem Scale .................................. 71!Table 12: Item Endorsement Proportion for the Expanded Rosenberg Self-Esteem Scale ............................ 72!Table 13: Item Endorsement Proportion for the Likert Conscientiousness Scale .......................................... 73!Table 14: Item Endorsement Proportion for the Expanded Conscientiousness Scale ................................... 74!Table 15: Item Endorsement Proportion for the Likert Version I Beck Depression Inventory ...................... 75!Table 16: Item Endorsement Proportion for the Likert Version II Beck Depression Inventory .................... 76!Table 17: Item Endorsement Proportion for the Expanded Beck Depression Inventory (a.k.a BDI-II) ........ 77!Table 18: Two-Factor Results for Exploratory Factor Analysis (EFA) for the Rosenberg Self-Esteem Scale ........................................................................................................................................................................ 78!Table 19: Two-Factor Results for EFA for the Conscientiousness Scale ....................................................... 79!Table 20:  Four-Factor Results for EFA for the Likert Version I Beck Depression Inventory ...................... 80!Table 21: Four-Factor Results for EFA for the Likert Version II Beck Depression Inventory ...................... 82!Table 22: Four-Factor Results for EFA for the Expanded Version Beck Depression Inventory (a.k.a. BDI-II) .................................................................................................................................................................... 84!Table 23: Two-Factor Results for EFA for the Likert Version I Beck Depression Inventory ........................ 86!Table 24: Two-Factor Results for EFA for the Likert Version II Beck Depression Inventory ....................... 87! vii Table 25: Two-Factor Results for EFA for the Expanded Version Beck Depression Inventory (a.k.a. BDI-II) ........................................................................................................................................................................ 88!Table 26Summary of the CFA fit statistics for the Rosenberg Self-Esteem Scale and Conscientiousness Scale ................................................................................................................................................................ 89!Table 27Summary of the CFI fit statistics for Beck Depression Inventory ................................................ 90!Table 28: Standardized factor loadings for CFA models for the Rosenberg Self-Esteem Scale .................... 91!Table 29: Standardized factor loadings for CFA models for the Conscientiousness Scale ........................... 93!Table 30: Standardized factor loadings for CFA Model 1 for the Beck Depression Inventory ..................... 95!Table 31: Standardized factor loadings for CFA Model 2 for the Beck Depression Inventory ..................... 96!Table 32: Standardized factor loadings for CFA Model 3 for the Beck Depression Inventory ..................... 98!Table 33: Model-Based Reliabilities for All Versions of the Three Scales .................................................. 100!Table 34: Correlation Matrices for the Rosenberg Self-Esteem Scale and the Conscientiousness Scale and for the Rosenberg Self-Esteem Scale and the Beck Depression Inventory ................................................... 101!             viii List of Figures  Figure 1: Factor Model of Three Items with Acquiescence Bias ................................................................... 54!Figure 2: Correlated trait, correlated uniqueness (CTCU) models ............................................................... 55!Figure 3: Correlated trait, correlated method (CTCM) models .................................................................... 56!Figure 4: Bi-dimensional Model ..................................................................................................................... 57!Figure 5: The Three Factor Structure Models of the Study ........................................................................... 58!Figure 6: Parallel Scree Plots for All Versions of the Rosenberg Self-Esteem Scale and the Conscientiousness Scale ................................................................................................................................. 59!Figure 7: Parallel Scree Plots for All Versions of the Beck Depression Inventory ....................................... 60! ix Acknowledgements I thank Dr. Savalei for her mentorship, for teaching me structural equation modeling and for guiding me to a better understanding of the field of quantitative psychology. I also thank Dr. Biesanz and Dr. Hewitt for serving on my committee and for helping me in the process of my Master’s studies. I would also thank my past research assistants, Ramsha Noor and Kurtis Stewart, for helping me hold debriefing sessions for this research.  Finally, I would like to thank my family and friends, who have supported me throughout my educational experience. x Dedication I dedicate this work to my high school teacher, Mr. Morphett. He is the most loving, caring and understanding teacher I have ever met. Without his encouragement and support, I would not have made some of the important achievements in my life.  1 Chapter  1: Introduction 1.1 Definition and Purpose of Reverse Worded Items Many psychological scales in the Likert format contain both positively worded (PW) and reverse worded (RW) items. The PW items are phrased in the direction of the construct, while RW items are phrased in the opposite direction. There are two kinds of RW items: 1) negation RW items created by adding negative particles such as not or no or by adding affixal negation such as un- or -less (e.g., I’m unhappy on a scale measuring happiness); and 2) polar opposite RW items created by using words with an opposite meaning (e.g., I’m sad on a scale measuring happiness). According to an analysis on nearly 2000 Likert items by Swain, Weathers and Niedrich (2008), about 80% of the RW items are negation RW items. The main reason for including RW items on scales is to control for acquiescence bias, or the tendency for respondents to endorse the item regardless of its content (Ray, 1983; Watson, 1992). This response style is to be distinguished from carelessness and/or confusion, in that the respondent is presumed to have understood the meaning of the item. When acquiescence bias is operating, it is assumed that for a balanced scale in which half of the items are PW and the other half of the items are RW, the participant would agree with both PW and RW items, leading to the cancellation of the bias in the sum/average score. After the RW items have been reverse-coded, the total scale score is then presumed to be bias-free. However, there is an emerging consensus among methodologists that these items might be doing more harm than good. Specifically, RW items contaminate the covariance structure of the scale in unanticipated ways by creating method effects (Distefano & Motl, 2006; Rodebaugh, Woods, Heimberg, Liebowitz & Schneier, 2005; Lindwall et al., 2012).    2 1.2 Why Reverse Worded Items Do Not Control Acquiescence Bias The wisdom of introducing RW items into scales has recently been questioned (e.g., Lindwall et al., 2012; Rodebaugh, Woods & Heimberg, 2007; Roszkowski & Soven, 2010; Sonderen, Sanderman & Coyne, 2013). To the extent that the tendency to acquiesce is an individual difference variable (Couch & Keniston, 1960; Ray, 1983; Weijters, Geuens, & Schillewaert, 2010), acquiescence bias will also contaminate the covariance structure of the data (Savalei & Falk, 2014). If the covariance structure of the data is of direct interest (e.g., if the researcher is interested in conducting factor analyses, reliability analyses, and full structural equation modeling), then the introduction of RW items into scales does little to fix the problem of acquiescence bias. This problem can be shown numerically. If we assume that acquiescence bias is a random variable3 (i.e., it differs across people), then person j’s scores on item i can be written as:  xij = µi + aij +λi f j + eij = µi +γ i +bi !a j +λi f j + eij                            (1) where µi  is the mean of the substantive construct the scale measures for item , λi is the loading of the substantive construct for item , f j is a random variable of the substantive construct across individuals j, eij is the residual for item which is also a random variable across individuals j, and aij = γ i +bi !a j  is the part modeling acquiescence bias, in which !a j  is a random variable of the acquiescence bias across individuals j with a loading of a bi  for item i and a nonzero item mean of γ i .                                                  3 It is a reasonable assumption according because acquiescence bias may be related to dialectical thinking (Spencer-Rodgers, Peng, Wang & Hou, 2004), in which individuals may see “truth” in both PW and RW items, and it is more characteristics of those from East Asian cultural backgrounds. ii i 3 If we make the assumptions that E( f j ) = E(eij ) = E( !a j ) = 0  4, the expected value of each item across individuals j is E(xi ) = µi +γ i . Therefore, the acquiescence bias for each item biases the item mean: it either inflates or deflates the item mean by the value of γ i . The expected value of the total scale score across individuals is E(xtotal ) = µii=1p∑ + γ ii=1p∑ . For a balanced scale in which half of the items are PW and the other half of the items are RW, the expected total score is unbiased. This can be shown if we assume that the acquiescence bias is equal across all items. Suppose that the scale contains p items, of which / 2k p=  are RW items. For each PW item of each person, letγ i = γ and for each RW item of each person, letγ i = −γ , it follows that the expected value of total score for a balanced scale across individuals j is: E(xtotal ) = µii=1p∑ + kγ − kγ = µii=1p∑                                     (2) However, the covariance structure is affected. The covariance between any two items (i.e., item i and item t) is cov(xi ,xt ) = cov(bi !a j +λi f j + eij ,bi !a j +λt f j + etj ) = bibt +λiλt , under the assumption that acquiescence is unrelated to the substantive construct the scale measures. The variance of any item is var(xi ) = bi2 +λi2 + var(ei ) = bi2 +λi2 +ψii . Thus, both variances and covariances get inflated.5 To understand the covariance structure of the inflated measure, consider a scale with three items as an example (see Figure 1 in Appendix D for the structural                                                 4 These are standard assumptions that means of latent variables and errors are zero. 5 This holds assuming cov(ai,at ) = bibt  is positive, otherwise some covariances could be deflated. If bi  were positive for PW items and negative for RW items (prior to reverse coding), then this could occur. However, it is difficult to conceive of this occurring for the way we have conceptualized acquiescence bias, and this is inconsistent with the typical pattern that is observed in data, i.e., weaker loadings for RW items.  4 equation modeling diagram for these three items). Let represent ΣACQ the covariance matrix of these items with acquiescence bias and let Σ  represent the covariance matrix of these items in absence of acquiescence bias. Then: ΣACQ = Σ+ b12b1b2 b22b1b3 b2b3 b32"#$$$$ %&''''= λ12 +ψ11λ1λ2 λ22 +ψ22λ1λ3 λ2λ3 λ32 +ψ33"#$$$$ %&''''+ b12b1b2 b22b1b3 b2b3 b32"#$$$$ %&'''' (3)6 The covariance matrix of items given acquiescence bias is distorted in an arbitrary way, even for balanced scales. If we do not model acquiescence bias, factor loadings estimates for the substantive factor and residual variance estimates will be biased.  1.3 Other Problems with Reverse Worded Items  In addition to the problem that RW items actually do not remove acquiescence bias in the covariance structure of data, the inclusion of negation RW items may also cause confusion and lead to errors due to carelessness among some respondents. Two kinds of carelessness usually exist in responding to psychological scales. The first kind of carelessness takes place when respondents answer one item and assume that all other items are similar and thus pick the same response anchor for all items. This kind of carelessness may be more prominent in longer scales and/or scales with similar items (Sonderen et al., 2013). The second kind of carelessness takes place when a respondent misses the presence of a negative particle (e.g., misread I am not happy as I am happy) or an affixal negation (e.g., misread I am unhappy as I am happy).  The negation RW items in a Likert scale will affect this kind of carelessness (Sonderen et al., 2013). Swain et al. (2008) have also demonstrated that negation RW items                                                 6 Note that ΣACQ can also be obtained using Wright (1934)’s rules of tracing for Figure 1.   5 that do describe the respondent’s actual state and PW items that do not describe the respondent’s actual state will increase respondent’s difficulty in interpreting the items. In addition, Schmitt and Stuits (1985) and Woods (2006) showed that the confusion and carelessness created by RW items might cause the emergence of a method factor common to all of the RW items. Specifically, their simulation studies showed that if at least 10% of respondents are careless, a clearly identifiable method component emerges from principal component analysis (PCA) (Schmitt & Stuits, 1985), and the fit of one-factor model through confirmatory factor analysis (CFA) is considerably worse (Woods, 2006).  Another problem associated with the inclusion of RW items is that it may also create method effects that are not related to acquiescence bias nor confusion or carelessness but instead represent a consistent behavioral trait, such as fear of negative evaluation, self-consciousness, or approach and avoidance behavior tendencies (DiStefano & Molt, 2006; Quilty, Oakman, & Risko, 2006). That is, participants may be responding differently to the sets of PW and RW items for idiosyncratic reasons. These reasons may or may not be related to the construct being measured.  The emergence of method effects in Likert scales containing PW and RW items can influence the assessment of the factor structure of the scale and distort parameter estimates (DiStefano & Molt, 2006; Sonderen et al., 2013). For instance, Sonderen et al. (2013) showed that the correlations between pairs of items are often higher when they are worded in the same direction than when they measure the same construct. Consistent with these expectations, research finds that the inclusion of RW items in Likert scales frequently results in lower validity (e.g., Rodebaugh, et al., 2011; Schriescheim & Hill, 1981), reduced reliability (e.g., Roszkowski & Soven, 2010), and the emergence of multiple factors for a scale that is  6 supposed to measure one underlying construct (e.g., DiStefano & Molt, 2006; Rodebaugh et al., 2005; Lindwall et al., 2012). For example, the emergence of a two-factor or two-component solution differentiating PW and RW items from exploratory factor analysis (EFA) or from PCA has been documented for the Rosenberg Self-Esteem Scale (e.g., Carmines & Zeller, 1979; Hensley & Roberts, 1976; Kaplan & Pokorny, 1969), Beck’s Hopelessness Scale (e.g., Steed, 2001) and the trait scale in the State-Trait Anxiety Inventory (Bieling, Antony & Swinson, 1998), to name a few.  1.4 Models for Removing Method Effects and Acquiescence Bias One way to remove the impact of acquiescence bias or method effects on the covariance structure of a Likert scale with RW items is to explicitly model acquiescence bias (e.g., Billiet & McClendon, 2000; Maydeu-Olivares & Coffman, 2006; Savalei & Falk, 2014) or method factors (e.g., DiStefano & Molt, 2006; Lindwall et al., 2012; Marsh, 1996) using CFA.  The standard way to model method effects in Likert scales with RW items is to use multitrait-multimethod (MTMM) models. Past researchers (e.g., DiStenfano & Motl, 2006; Lindwall et al., 2012; Marsh 1996; Wu, 2008) have proposed two main types of MTMM models to separate the substantive factor (a.k.a content factor) that a scale measures from the method effects caused by the RW items in the scale. The first type is the correlated trait, correlated uniqueness (CTCU) model. The second type is the correlated trait, correlated methods (CTCM) model. The CTCU model includes correlations among the residuals or uniqueness of the PW items (CTCU-PW) or RW items (CTCU-RW) or both (CTCU-PW/RW; see Figure 2 in Appendix D for an example with 3-item). In contrast, the CTCM model introduces one or two latent method effect factors. The CTCM model can consist of one  7 method factor which all PW items (CTCM-PW) or all RW items (CTCM-RW) load on; or it can consist of two method factors with all PW items loading on one factor and all RW items loading on the other (CTCM-PW/RW; see Figure 3 in Appendix D). The CTCM models belong to a particular type of model called the bifactor model in which the substantive construct is the general factor and the method factors are the subfactors (Weijters et al., 2013). It is important to note that the CTCU-PW model and CTCU-RW model are nested in the CTCU-PW/RW model (i.e., Model (a) and (b) are nested in Model (c) in Figure 2 in Appendix D); the CTCM-PW model and CTCM-RW model are nested in the CTCM-PW/RW (i.e., Model (a) and (b) are nested in Model (c) in Figure 3 in Appendix D).  However, several problems exist with the MTMM modeling approach. Firstly, it is unclear what causes the method factors in the Likert scales. Even though several potential causes of method factors such as fear of negative evaluation, self-consciousness, or approach and avoidance behavior tendencies (DiStefano & Molt, 2006; Quilty et al., 2006) have been proposed in the literature, none of them has received much support. Secondly, there is little consensus on which MTMM model is the best. Although the CTCU-PW/RW model has consistently been shown to have a better fit than all the other MTMM models (Lindwall et al., 2012; Marsh 1996; Quilty et al., 2006), this model often causes identification problems (Marsh 1996; Wu, 2008).  Furthermore, the CTCU-PW/RW model has the least constraints than all other MTMM models; thus, it is unclear whether the CTCU-PW/RW model is the best model when the parsimony of the model is taken into account. Finally, the MTMM models do not model acquiescence bias but instead assume unique response biases for either the PW items, the RW items, or both.  In order to actually model and remove acquiescence bias, several authors have  8 proposed a model with a single substantive factor and a single acquiescence factor which both PW items and RW items load on (Maydeu-Olivares & Coffman, 2006; Savalei & Falk, 2014; Welkenhuysen-Gybels, Billiet, & Cambre 2003; see Figure 4 in Appendix D). In order for this model to be identified, the loadings on the acquiescence factor are set to one and the variance of the factor is freely estimated. This model has various names in the literature, including random intercept item factor model (Maydeu-Olivares & Coffman, 2006) and one-content factor and one-style factor model (Cambre, Welkbenhuysen-Gybels & Billiet, 2002). In this thesis, I will call it the bi-dimensional model, as per Savalei and Falk (2014). The bi-dimensional model is more consistent with the model presented in Equation (1) and Figure 17. In other words, this model is the most consistent with the conceptualization of acquiescence bias. This bi-dimensional modeling approach has rarely been compared to the MTMM modeling approach; therefore it is unclear which approach is better. Furthermore, a CFA model cannot contain both acquiescence factor and method factors because such a model will not be identified. In summary, due to the various problems listed above, removing acquiescence bias and/or method effects in Likert scales using any of the modeling approaches is problematic.  1.5 An Alternative Scale Format: The Expanded Format In order to simultaneously minimize acquiescence bias and to solve the problems caused by RW items, several authors (e.g., Brown & Maydeu-Olivares, 2011; Swain et al., 2008; White & Mackay, 1973) have advocated alternative scale formats. One such format is the forced-choice format (e.g, Javeline, 1999; Schuman & Presser, 1981). In the forced-choice                                                 7 In order to get the bi-dimensional model from the model Equation (1)/Figure 1, two more assumptions need to be made: 1) the tendency to engage in acquiescence bias for each person is constant across items; 2) the correlation between the tendencies to engage in acquiescence bias for any two different items is one. These assumptions need to be made in order to identify the model with the acquiescence bias factor.   9 format, the respondent is required to choose between two substantive response options rather than simply agree or disagree with a statement. For example, Schuman and Presser (1981) compared Likert format questions such as Individuals are more to blame than social conditions for crime and lawlessness in this country; agree or disagree? 8 to forced-choice format questions such as Which in your opinion is more to blame for crime and lawlessness in this country – individuals or social conditions?. They argued that in the forced-choice format, acquiescence response bias should be eliminated because the response task requires a choice between two assertions rather than an endorsement of a single assertion. Additionally, because this format removes the very concept of PW and RW items, the factor structure of a scale should no longer be contaminated by method factors elicited by the direction of the item. Forced-choice format scales have shown good validity and reliability (Javeline, 1999; Schuman & Presser 1981). One disadvantage of the forced-choice format as originally defined, however, is that the items are dichotomous, making it hard to compare this format to the Likert format, which usually involves four or more response options.  In the present study, we examined an alternative scale format, which is an extension of the forced-choice format including more response options. We refer to this format as the Expanded format. To our knowledge, this format has only been used in the Beck Depression Inventory (Beck, Ward, Mendelson, Mock & Erbaugh, 1961) and in the Oxford Happiness Inventory (OHI, Argyle, Martin, & Crossland, 1989). To convert a Likert item to the Expanded format, each response option would be replaced by a full sentence. For instance, an item from the Rosenberg Self-Esteem Scale that reads On the whole, I am satisfied with myself. and that has four response options (with anchors strongly agree, somewhat agree,                                                 8 This study was conducted over the phone, and thus item wording was adapted for the item to be delivered orally.   10 somewhat disagree, and strongly disagree) can be written in the Expanded format as follows: • On the whole, I am very satisfied with myself. • On the whole, I am satisfied with myself. • On the whole, I am disappointed with myself. • On the whole, I am very disappointed with myself. Because both PW and RW items are presented for each item, acquiescence bias and unique method effects due to the direction of the item are theoretically eliminated. In addition, this format forces participants to pay more attention to the content of the item and to notice the subtle differences between options, thus potentially reduces confusion and careless responding. This format has the potential to effectively address the methodological issues associated with RW items in particular and with Likert scales more generally.  When the number of response options in the Expanded format and in the Likert format is the same, the factor structure of scales in the Expanded format and the Likert format can be compared. However, research comparing these two formats is virtually nonexistent. The only study is that of Hills and Argyle (2002), who compared a happiness scale in the Likert and the Expanded formats. The scale in the Likert format contained both PW and RW items. But only EFA analyses were conducted and the outdated “eigenvalue greater than one” rule was used to judge dimensionality. Nonetheless, the dimensionality of the scale in the Expanded format was found to be one factor less, suggesting that the Likert format did in fact introduce additional variance contamination into the data.  1.6 Goals of the Present Research The goal of the present study is to investigate the effectiveness of the Expanded format and to compare it to the Likert format. In order to conduct this comparison, we changed two  11 popular Likert scales—the Rosenberg Self-Esteem scale (RSES; Rosenberg, 1965) and the Conscientiousness scale (CS) from the Big Five Inventory (John, Naumann, & Soto, 2008)—into the Expanded format. We also changed the Beck Depression Inventory II (BDI-II), which is already available in the Expanded format, into the Likert format in two different ways. For each scale, the factor structures of the data in the two formats were compared using both EFA and CFA. We hypothesized that scales in the Expanded format would have better (i.e., lower and more theoretically defensible) dimensionalities than scales in the Likert format when evaluated using EFA. We further hypothesized that when the one-factor CFA model is fit to data, scales in the Expanded format would have a better fit by the chi-square test and indices of approximate fit than scales in the Likert format. Further, we expected that the fit of the Likert scales would improve substantially when two substantive factors were modeled or when a method factor was added to the one-factor model, whereas the fit of the scales in the Expanded format would stay relatively the same. Finally, we also examined the model-based reliabilities and convergent validities of the scales in both formats, although specific hypotheses were not formed because contamination due to item wording can cause higher or lower correlations as well as affect the variances of the observed variables.    12 Chapter  2: Method 2.1 Participants  Participants were undergraduate students enrolled in psychology courses at the University of British Columbia. A between-subject design was used, so that each participant would only see one version of each scale. There were 641, 621 and 763 participants who completed one version of the Rosenberg Self-Esteem Scale (RSES), Conscientiousness Scale (CS) and Beck Depression Inventory (BDI), respectively. The mean age of participants was 20 years (SD= 2.75), and 21% were male. The ethnicities of participants were mainly Caucasian (27%) and East Asian (53%). 2.2 Procedure Participants completed an anonymous survey online. The study took each participant about 20 minutes to complete. For two of the scales (RSES and CS), participants were randomly assigned to complete either the Likert version or the Expanded format version. For the BDI scale, participants were randomly assigned to complete the Likert Version I, Likert Version II or the Expanded format version. Informed consent was obtained prior to the start of the survey.  The majority of participants (n=472) were asked to complete one version of the RSES, CS and BDI; however some participants were asked to complete either one version of the BDI (n=307) or one version of each of the RSES and CS9(n=169). It should be noted that                                                 9 The data for the different versions of the RSES and CS were collected at the same time; most of data for the different versions of the BDI was collected together with the other two scales. Specifically, 641 participants were asked to complete one version of the RSES and one version of the CS. However, 20 participant’s responses on one version of the CS were deleted due to missing data. Out of the 641 participants, 472 participants were also asked completed one version of the BDI. An additional 307 participants’ data were collected for the versions of the BDI. Out of 779 participants who were asked to complete the one version of the BDI, 16 participants were deleted due to missing data.  13 participants were also asked to complete several other psychological scales for other research projects. These scales included the Need for Cognition scale (Cacioppo & Petty, 1982), Subjective Happiness Scale (Lyubomirsky & Lepper, 1999), and Self-Competence and Self-Liking Scale (Tafarodi & Swann, 1995). The last portion of the survey consisted of questions relating to demographic information. After completing the survey, all participants were required to attend an oral debriefing session during which they received course credit for participation.  2.3 Measures 2.3.1 Rosenberg Self-Esteem Scale  The 10-item Likert version of Rosenberg Self-Esteem Scale (RSES) (n=319) is the original RSES scale (Rosenberg, 1965) containing five PW items and five RW items, all measured on a four-point scale, where 4 corresponds to Strongly agree and 1 corresponds to Strongly disagree (see A.1 in Appendix A). In the corresponding 10-item Expanded version of RSES (n=322), each item consists of four sentences to choose from, and all items are always arranged from the highest to the lowest self-esteem. For each item, participants were asked to select one of the four options that best describes them. The text of the corresponding Likert version item was used to create the four options for each item in the Expanded format (see A.2 in Appendix A). For most items in the Expanded version, the original Likert item was included as one of the four options; for the rest of the items, the options were created by adding a modifier to the original Likert item. 2.3.2 Conscientiousness Scale The Likert version of the Conscientiousness Scale (CS) (n= 314) was taken from the Big Five Inventory (John et al., 2008). It contains five PW items and four RW items. The  14 original items were on a five-point scale; however, we used a four-point scale from Disagree Strongly to Agree Strongly (see B.1 in Appendix B). This was done in order to match the number of response options to the four options in the Expanded format version. While it is possible to create the Expanded format version with five response options, it may be too many for the respondents to process. The comparison between Expanded formats with four versus five response options will be the subject of future research. The Expanded version (n=307) contains 9 items, with response options for each item ranging from the indication of the lowest conscientiousness to the highest conscientiousness (see B.2 in Appendix B). For each Expanded format item, the text of the corresponding Likert version item was used to create the four options. For eight of the nine items in the Expanded version, the original Likert item was included as one of the four options; for the remaining one item, the options were created by adding a different modifier for the original Likert item. 2.3.3 Beck Depression Inventory The Expanded version (n=256) is the original 21-item Beck Depression Inventory II10 (BDI-II; Beck, Steer, & Brown, 1996). Each item contains four options ranging in depression intensity (from absence of symptoms to frequent or intense symptoms) and participants were asked to pick the option that best described how they had been feeling in the last two weeks prior to study participation. The original BDI-II was designed to measure only the presence and the degree of depression. For every item on the scale, the first option was written to indicate that a particular symptom is absent (e.g., I am not discouraged about the future). Thus, the first option for each item could be viewed as a negation RW item for depression. However, a typical Likert scale usually contains both negation RW items and polar opposite                                                 10 Since BDI-II is a licensed scale, it is not presented in this paper.   15 RW items. For this reason, we created two Likert versions of the BDI. The Likert version I (n=254) was created by taking either the first or the last option in the original BDI-II as the corresponding Likert item (see C.1 in Appendix C). In other words, all RW items in the Likert version I were negation RW items. The advantage of this version is that the wording of each item is identical to the wording of one of the response options for that item in the original BDI-II. The Likert version II (n=253) was created by replacing some of the negation RW items in Likert version I with newly written polar opposite RW items (see C.2 in Appendix C). The advantage of this version is that it better resembles a typical Likert scale. For both Likert versions of the BDI, there were 11 PW items and 10 RW items, all measured on a four-point scale from “4” =Strongly agree to “1”= Strongly disagree. 2.4 Analytic Method For exploratory factor analysis (EFA), the psych package (Revelle, 2014) in R was used. Parallel analysis (Horn, 1965), available within the psych package, was used to determine the number of factors to be extracted. Parallel analysis is an improved version of the scree plot that incorporates sampling variability into the analysis (Zwick & Velicer, 1986). It does so by comparing the scree plot obtained from the data to an average scree plot obtained from simulated datasets of the same dimension as the original dataset but generated from a population where all variables are uncorrelated (e.g., Hayton, Allen, & Scarpello, 2004; Horn, 1965). Following this method, the recommended number of factors to be extracted is the number of original data’s eigenvalues that are greater than the corresponding simulated data eigenvalues. Because all of the items were measured on a four-point scale, we treated the data as categorical (Rhemtulla, Brosseau-Liard, & Savalei, 2012). The polychoric correlation matrix was used in the parallel analyses and in the subsequent EFA analyses. The EFA  16 extraction method was least squares (a.k.a. minres), followed by an oblimin rotation.  For confirmatory factory analysis (CFA), the lavaan package (Rosseel, 2012) in R was used. Because the data were treated as categorical, the diagonally weighted least squares estimator with robust corrections was used (i.e., estimator=”WLSMV” in lavaan). Three different models were fit to the data from each scale (see Figure 5 in Appendix E). These models are commonly used to study method effects in Likert scales (e.g., Lindwall et al., 2012; Marsh, Scalas, & Nagengast, 2010; Quilty et al., 2006). Model 1 posits a single substantive factor with no method effects. Model 2 posits two oblique factors with all PW items loading on one factor and all RW items loading on the other. Model 1 is nested within Model 2. Model 3 posits a single substantive factor and an orthogonal method factor for the RW items. In the Expanded format, there is no distinction between the PW and RW items, and the factors for Models 2 and 3 are formed according to whether the corresponding item was RW or PW in the Likert format11. Model 1 is also nested within Model 3. Models 1-3 are fit to all datasets regardless of the number of factors suggested by the EFA. Thus, CFA analyses were treated as quite distinct from the EFA analyses: their main purpose was to investigate whether a unidimensional model (Model 1) would fit the data better under the Expanded format than under the Likert format, and whether the Likert format data would suggest more strongly than the Expanded format data that the construct being measured by the scale is multidimensional.  The following criteria were used to evaluate model fit: 1) the test of exact fit using the robust (mean-and-variance adjusted) chi-square statistic; 2) the comparative fit index (CFI; Bentler, 1990), with the value of 0.95 or greater indicating a well-fitting model (Hu & Bentler,                                                 11For the Expanded format BDI, factors were formed in two different ways: according to whether the corresponding item is RW or PW in both the Likert version I and version II results were very similar.   17 1999); 3) the root mean square error of approximation (RMSEA; Steiger, 1980), with the value of 0.08 or less indicating a reasonable fit (Browne & Cudeck, 1993); 4) the chi-square difference tests between Model 1 and Model 2, and between Model 1 and Model 3, using the method suggested by Satorra (2000) and implemented in lavaan; and 5) the tests of small differences in fit between Model 1 and Model 2, and between Model 1 and Model 3, using RMSEAA = 0.09 and RMSEAB= 0.08, testing  the null hypothesis that the difference in fit between a pair of models is equal to or less than the difference in model fit that would change the RMSEA value from 0.08 to 0.09 (see MacCallum, Browne, & Cai, 2006 for more detail). The tests of close fit (4 and 6), based on smallhypothesized values of the RMSEA, are of more practical importance than the tests of exact fit (1 and 5), because most models in psychological research are at best approximations to reality (MacCallum et al., 2006). We also examined reliabilities for each scale in each of the formats. To allow for comparison across formats, model-based reliabilities (e.g., Bentler, 2009; Raykov, 1997) were computed under Model 1 (the one-factor CFA model) and Model 3 (the model with one substantive factor and one method factor). For the model-based reliability computed under Model 3, the variance due to the method factor was treated as error. This was an exploratory comparison, and no predictions were made. Finally, previous research showed that self-esteem and conscientiousness are moderately positively correlated (e.g., Pullmann & Allik, 2000) and that self-esteem and depression are largely inversely correlated (e.g., Greenberger, Chen, Dmitrieva, & Farruggia, 2002). To see how scale format affects these relationships and convergent validity coefficients of the scales in general, correlations between different versions of the RSES and CS, and between versions of the RSES and BDI were examined.   18 Chapter  3: Results  3.1 Descriptive Statistics  Covariance matrices, item means, standard deviations and item endorsement for all three scales in different formats are shown in Table 1 to 17 in Appendix G. Scales in the Expanded version consistently had smaller standard deviations than the corresponding scales in the Likert version, possibly due to the reduction in variance contamination due to the absence of RW method effects. For the RSES and CS, the average means across items were very similar with an average mean difference of 0.04 between the two versions of RSES and an average mean difference of 0.09 for those of CS). However, for the BDI, the average means were significantly different among the three versions, F(2,63) = 35.25, p < 0.001; the Likert version II had the highest average mean (average M=2.14), followed by the Likert version I (average M=1.95), followed by the Expanded version (average M=1.56). These results illustrate that it is relatively difficult to create a Likert version of the BDI that would map well onto the original BDI.  3.2 Exploratory Factor Analysis Figures 6 and 7 in Appendix F show the results for the parallel analyses of all three scales in different formats. Overall, as expected, parallel analysis suggested fewer factors for the Expanded version of each scale than for the corresponding Likert version(s). Specifically, for the RSES and CS’s Likert versions, the scree plots from parallel analysis clearly indicated two factors; however, for the Expanded version, the second factor was much weaker, and the corresponding eigenvalue was close to the cutoff line obtained from the simulated data, suggesting a single factor. For the two Likert versions of the BDI, the scree plots from parallel analysis indicated that four eigenvalues were above the sampling fluctuations observed in that  19 simulated data, suggesting at most four factors, whereas the scree plot for the Expanded BDI indicated at most two factors, with the second factor being very close to the cutoff. These results were consistent with the previous findings that the BDI is not a unidimensional scale (see literature review conducted by Manian, Schmidt, Bornstein, & Martinez, 2013).  EFA analyses were performed next (see Table 18 to 25 in Appendix G). For the RSES and CS scales in both formats, we extracted two factors (see Table 18 and 19 in Appendix G). This was the number suggested by the parallel analyses for the Likert version of these scales. Even though the number of factors suggested for the Expanded versions was one, the same number of factors was extracted for both formats in order to compare the results across the formats. The initial extracted solution was obliquely rotated. As expected, for the Likert versions of the RSES and CS, the pattern of loadings indicated that the two factors should be formed based on whether the item was PW or RW. These results replicated previous findings that the inclusion of RW items in Likert scales causes the emergence of a two-factor solution based on the wording of the items (e.g., Carmines & Zeller, 1979; Hensley & Roberts, 1976). On the other hand, the RSES and CS scales in the Expanded version did not produce very clear two-factor solutions. Most items loaded highly on the first factor; the few items that loaded highly on the second factor had similar item content. For example, Items 1 and 4 in the RSES were both about social comparison between oneself and others. Items 1 and 3 in CS were both related to being a good worker. Thus, when a two-factor solution was forced on the data in the Expanded format, the factors formed based on the content and not a method effect due to the item wording. Additionally, the factor correlations in the Expanded RSES (r=0.72) and CS (r=0.77) were higher than those in the Likert RSES (r=0.63) and Likert CS (r=0.49) respectively, supporting the conclusions from parallel analysis that these  20 scales in the Expanded format can be modeled using a single factor. For the BDI, parallel analysis suggested a 2-factor solution for the Expanded version and a 4-factor solution for the Likert versions I and II (see Table 20 to 25 in Appendix G). To compare the versions on the same number of factors, we performed EFA analyses extracting two and four factors for all versions. For the 4-factor solutions of the two Likert versions, the factors were formed based on both item content and item wording direction. For the Likert version I, all the PW items loaded highest on Factor 1 or 2 and most of the RW items loaded highest on Factor 3 or 4. In terms of item content, Factors 1 and 2 appeared to be the cognitive-affective factor and the somatic factor of depression among the PW items respectively, whereas Factors 3 and 4 seemed to be the cognitive factor and somatic-affective factor of depression among the RW items respectively. For the Likert version II, all the PW items loaded highest on either Factor 2, 3 or 4, and eight out of the ten RW items loaded highest on Factor 1. In terms of item content, Factor 2 and Factor 3/4 corresponded to items tapping into the cognitive factor and somatic-affective factor among the PW items, respectively. Factor 1 seemed to tap into all aspects of depression among the RW items. For the Expanded version, the factors were formed based on item content. Factor 1 was the cognitive factor for depression whereas Factor 2 and 3 were most related to the somatic-affective aspect of depression. Factor 4 seemed to be a spurious factor with only two items. Consistent with the results from the parallel analysis, this solution resulted in over-extracted factors for the Expanded version of the scale.  When the two Likert scales were forced into a 2-factor solution, the Likert version II solution was formed based on the item wording direction, with the PW items primarily loading on Factor 2, and the RW items primarily loading on Factor 1. Interestingly, the 2- 21 factor solution in the Likert version I was formed based on item content more so than on item wording: Factor 1 was most related to the somatic items and Factor 2 was most related to the cognitive-affective items. These results suggest that the substantive multidimensionality of the BDI (as captured by this scale) is stronger than the dimensionality due to method effects (item wording). However, this 2-factor solution was not very clear. Six items loaded poorly on both factors (standardized loadings below 0.4), and only five items had standardized loadings greater than 0.6 on either factor. In contrast, the 2-factor solution for the Expanded version of the BDI scale was formed based on item content and was much clearer than the solution produced by the Likert II format data. Factor 1 was again a cognitive-affective factor and Factor 2 was a somatic factor. Only two items loaded poorly on both factors (with loading sizes below 0.4), and 11 items had standardized loadings greater than 0.6 on one of the factors. This 2-factor solution was a good replication of previous EFA analyses from other studies using community samples (e.g., Beck et al., 1996; Steer & Clark, 1997). Consistent with the results of the RSES and CI, the factor correlation for the Expanded BDI (r=0.65) was higher than the factor correlations for the Likert version I (r=0.54) and version II (r=0.62). However, BDI is not a theoretically unidimensional scale, and it is reasonable that the factor correlation for the Expanded BDI is not as high as those for the Expanded RSES and CI.   3.3 Confirmatory Factor Analysis  3.3.1 Model Fit The fit indexes for the three CFA models tested (see Figure 5 in Appendix E) for different versions of the three scales are shown in Table 26 and 27 in Appendix G. As expected, for the Likert versions of all three scales, Model 1, positing a single factor, did not fit the data well according to the CFI and the RMSEA. On the contrary, Model 1 had a  22 relatively good fit for the Expanded versions of all three scales according to the CFI and the RMSEA. Specifically, the CFI values were all equal or above 0.95 and the RMSEA values were all equal or below 0.08.  Model 2, which is a CFA model with two oblique factors (one among PW items and one among RW items), produced significantly better fit relative to Model 1 for the RSES and CS Likert scales, according to the exact chi-square difference test but not the test of small difference (see Table 26 and 27 in Appendix G). However, Model 2 did not produce significantly better fit than Model 1 for the two versions of BDI Likert scale according to both the exact chi-square difference best and the test of small difference (see Table 27 in Appendix G). While the fit indices did improve for Model 2 relative to Model 1 for the Likert versions of the three scales, the fit indices did not always reach acceptable values, particularly for the BDI, with Likert version I fitting worse than Likert version II. This result is not unexpected given that parallel analyses revealed a 4-factor structure for the Likert versions of the BDI scale.     Model 2 did not result in a significant improvement in fit over Model 1 for the Expanded versions of all three scales, according to both the chi-square difference test and the test of small difference. To understand this result for the BDI, it is important to remember that in the tested 2-factor CFA model, the factors were formed based on the item wording direction in the corresponding Likert versions (see Figure 5 in Appendix E). The RMSEA, the CFI, and the test of close fit yielded very similar results for Models 1 and 2 for all three scales in the Expanded format.  Model 3, which posited two orthogonal factors, a substantive factor common to all items and a method factor for the RW items, produced very similar results to Model 2 for all  23 scales in all versions. Model 3 generally produced significantly better fit for the Likert versions of the three scales in comparison with the fit of Model 1, although the BDI Likert version II did not produce significantly better fit than Model 1 according to the test of small difference. In contrast, Model 3 did not produce significantly better fit relative to Model 1 for the Expanded versions of the three scales, with only the exact chi-square difference test of the CS Expanded scale being an exception.  In summary, for the Likert scales, Model 1, which posits unidimensionality, did not fit the data well, but Models 2 and 3, which allow for multidimensionality, resulted in better fit than Model 1 by multiple fit criteria. These results are consistent with our hypothesis that the RW items of the three scales in the Likert format contaminate the factor structure of the scales. On the other hand, Model 1 had acceptable approximate fit for all three scales in the Expanded version, and including the additional factors did not improve fit much. These CFA results are also consistent with the EFA results, which demonstrated that the factor structures of the Expanded versions of the three scales had lower dimensionality. It is important to note that we did not test multidimensional models based on item content, only based on the direction of item wording. Allowing for multiple factors based on content could result in a better-fitting model even for the Expanded version of the scales. However, the goal of this article is to demonstrate the impact of scale format on fit. The scales’ structure approximated unidimensionality to a much higher degree when contamination due to the direction of item wording was not an issue.  3.3.2 Standardized Solution  Standardized factor loadings for each factor in all models of the three scales are shown in Tables 28 to 32 in Appendix G. Several interesting patterns emerged. First, for all  24 three scales, average standardized factor loadings for Model 1 were similar for the Likert version(s) and the Expanded version. In other words, under both scale formats, the same percentage of reliable variance was attributed to the substantive factor when Model 1 was fit to data. Second, for all three scales, the factor correlation in Model 2 was much lower for the Likert version(s) than for the Expanded version; in fact, they were so high for the Expanded versions that most researchers would likely conclude from such data that the two factors should be collapsed into one. The two-dimensional solution sometimes found for these scales in the Likert format (e.g., “positive” and “negative self-esteem”; Marsh, 1996) was thus likely due to item wording and not the existence of two substantive dimensions. Finally, for each scale, the average item loading value on the RW method factor was much greater in the Likert versions than in the Expanded versions, consistent with the hypothesis that the modeled method effects were not present in the Expanded versions of the scales.  3.4 Reliability and Convergent Validity All versions of all scales had very good model-based reliability coefficients (computed under Model 1 and Model 3; see Table 33 in Appendix G). When the model-based reliability was computed under Model 1, the Likert and Expanded versions of each scale had very similar reliability coefficients with a maximum difference of 0.03. When the model-based reliability was computed under Model 3, the reliability of the Expanded version of each scale was very close to the one computed under Model 1; however, the reliability of the Likert version of each scale was less than the one computed under Model 1. As a result, for the reliability computed under Model 3, the Expanded version of each scale had consistently better reliabilities with an average difference of 0.085 between the Expanded version and the Likert version.   25 The correlation coefficients between different versions of the RSES and CS, and between the RSES and BDI were computed to examine how the scale format affected the convergent validities of the scales (see Table 34 in Appendix G). As previously mentioned, self-esteem and conscientiousness are usually moderately positively correlated (e.g., Pullmann & Allik, 2000), whereas self-esteem and depression are usually inversely correlated (e.g., Greenberger et al., 2002). The correlations between the RSES and CS were very similar across formats, ranging from 0.32 to 0.40. Correlations between the RSES and BDI varied more across the formats, ranging from -0.55 to -0.82, but they all indicated a large inverse relationship between self-esteem and depression. The highest correlation was between the RSES Likert version and the BDI Likert version II (r=-0.82). This high correlation was probably caused by the fact that some of newly created items on the BDI Likert version II had considerable item content overlap with the items on the RSES Likert version.      26 Chapter  4: Discussion and Conclusion The debate of whether researchers should use RW items in Likert scales has gone on for decades (e.g., Ahlawat, 1985; Burk 1999; Roszkowski & Soven, 2010; Schrieheim & Hill, 1981; Sonderen et al., 2013).  One of the biggest problems with RW items is that they often cause the emergence of spurious method factors for Likert scales (e.g., Bors, Vigneau, & Lalande, 2006; Brown, 2003; DiStefano & Molt, 2006; Lindwall et al., 2012; Marsh, 1996) This issue has contributed to the proliferation of the various factor structures of the Rosenberg Self-Esteem Scale (RSES) to model the method factor(s) in the scale (e.g., DiStefano & Molt, 2006; Lindwall et al., 2012; Marsh, 1996). In our study, we proposed the use of an alternative scale format, called the Expanded format, to solve this problem of RW items. The main distinction of the Expanded format from the Likert format is that the former asks participants to choose among response options that are complete sentences, rather than to simply indicate their level of agreement or disagreement with a statement. This more elaborate format eliminates the very concept of PW and RW items because each item contains sentence options of both types, and it also forces participants to pay more attention to the questions, potentially reducing carelessness. We examined the impact of changing existing scales in the Likert format to the Expanded format on the factor structure of the scales. We hypothesized that scales in the Expanded format would show more parsimonious factor structures compared to the same scales in the Likert format, as assessed by EFA and CFA analyses, due to the elimination of method variance associated with item wording and the reduction of the number of careless or confused responses.   The results confirmed our predictions. The scales in the Likert format tended to produce 2-factor EFA solutions differentiating PW and RW items (for the BDI, a higher factor  27 solution was necessary), and to require two factors to produce acceptable approximate fit in the CFA analyses. These findings are consistent with previous studies (e.g., Carmines & Zeller, 1979; DiStefano & Molt, 2006). In contrast, the scales in the Expanded format did not show similar problems. The EFA analyses suggested fewer factors for the scales in the Expanded format. Specifically, for two out of the three scales, the suggested number of factors was one. The factors in the Expanded format scales were also more consistent with the theoretical structure of the scales. These results were consistent with the findings of Hills and Argyle (2002)’s, who conducted principal components analyses on the Oxford Happiness Inventory (OHI), which was in the Expanded format, and the Oxford Happiness Questionnaire (OHQ) which was in the Likert format. Hills and Argyle (2002) found that the OHI extracted less factors and the extracted factors made some theoretical sense; however, the OHQ extracted more factors and “the extracted factors could not plausible be interpreted” based on the item content. In addition, our CFA analyses revealed that the 1-factor model fit the data of the Expanded format scale substantially better than the data of the Likert format scale, and adding factors did not significantly improve fit. This finding is quite impressive considering the similarities between the corresponding items in the two formats. For example, to create the Expanded version of Conscientiousness Scale (CS), we created new response options for most items simply by adding an adjective modifier such as somewhat or by replacing a word with an antonym (e.g., changing useful to useless; see B.2 in Appendix B).  The main features of the Expanded format item that we believe are responsible for this improvement in the factor structure are 1) that it is non-directional; 2) that it does not require participants to agree or disagree with the item (i.e., participants only need to pick a response option); and 3) that both PW and RW item wordings are included as response  28 options in the item. The first feature eliminates method effects due to item keying. The second feature minimizes acquiescence bias, which theoretically only occurs when the scale requires respondents to agree or disagree with the items. We believe that the third feature reduces carelessness and confusion, although this point is difficult to demonstrate conclusively. When participants are presented with both PW and RW item wordings as response options to the same item, they might be more likely to notice the difference between the response options and thus less likely to misread the item.  In addition, changing existing scales to the Expanded format did not result in decreased reliability and validity. The reliabilities computed under Model 3 were even higher for the scales in the Expanded format, although the reliabilities computed under Model 1 were similar between the scales in the Likert and Expanded formats. This finding may indicate that when a Likert scale is modeled under a one-factor model (i.e., Model 1), some of the variance due to the method factor may be forced to go through the substantive factor, thus resulting an inflated reliability; however, when the method factor is included in the model as in Model 3, the variance due to the method factor was able to be accounted for, resulting less biased reliability. In contrast, the scales in the Expanded format are not affected by the method factor; thus modeling method factor would not affect their reliabilities. The convergent validities of the Likert scales and the corresponding Expanded scales were very similar. This may seem to contradict some of the past research showing that the removal of RW items or their replacement with PW items resulted in better scale validity (e.g., Rodebaugh et al., 2011). However, increasing validity after removing or replacing RW items usually happens for unbalanced scales. For balanced scales, the effects due to the method factors and the acquiescence bias may be balanced out across the items for sum or average scores (see Section  29 1.1 and 1.2 in Chapter 1; Ray, 1983; Savalei & Falk, 2014). The three scales used in this study are either perfectly balanced (i.e., the Likert RSES has equal number of PW and RW items) or almost perfectly balanced (i.e., the number of RW items in the Likert CS or BDI is only one less that the number of PW items). Therefore, the sum or average scores of the three scales, which we used for calculating convergent validity coefficients, may not be affected by the method factors or the acquiescence bias.  4.1 Limitations and Future Directions Despite considerable advantages, the Expanded format may nonetheless have important limitations, and the impact of these limitations should be explored in future research. Firstly, scales in the Expanded format may take longer to complete because they involve more reading and force participants to pay attention to the text of the item more carefully. This may, however, be the inevitable cost of obtaining higher quality data free of method effects.  Secondly, changing Likert items into Expanded items can be tricky at times. For example, in the original Likert CS, one item is I am someone who can be somewhat careless. For a respondent who is strongly agree with this item, it is unclear whether he means that he is very careless, or he is careless most of the time, or he definitely can be somewhat careless at times. In such case, the researcher has to pick one particular interpretation to create the options for the corresponding Expanded item. This kind of difficulty in changing some items in the Likert format to the Expanded format demonstrates the great ambiguity of the response anchors in a Likert scale. When researchers make such ambiguous Likert items into Expanded items, respondents will have a clearer understanding of the response options of the items. Of course, if a researcher creates a brand new scale in the Expanded format, the researcher would  30 not worry about how to change Likert items into Expanded items. Future research should explore different ways of changing Likert items into Expanded items and examine how to create better Expanded items.  Thirdly, while the method effects due to PW and RW items are theoretically eliminated, there may still be order effects for items in the Expanded format. Previous research on the forced-choice format found both primacy and recency effects (e.g., Krosnick & Alwin, 1987; McClendon, 1984, 1991). Interestingly, McClendon (1991) found that when items on the RSES were translated into the forced-choice format, no order effects emerged. Since most research on the forced-choice format was conducted in political science and sociology, it is possible that participants are prone to order effects only when they do not have strong preexisting opinions on the question being asked and thus must construct an opinion at the time of the question; this problem would be reduced for questions about the self (McClendon, 1991). We also note that in the current study, the order of response options varied across scales. For instance, the RSES scale in the Expanded format had response options ordered from high self-esteem to low self-esteem, while the CS scale in the Expanded format had response options ordered from low conscientiousness to high conscientiousness. Yet both these scales produced item means that were very similar to the corresponding items in the Likert version (see Table 8 to 10 in Appendix G), suggesting that neither order was superior. Nonetheless, future studies should investigate whether the Expanded format scales are affected by order effects in a more systematic fashion.  Finally, items on the Expanded format scales must necessarily have fewer response options than the typical Likert scale items. In this study, items in the Expanded format had four response options (and thus the data were treated as categorical). In contrast, Likert scales  31 often have 5-7 categories, and the data from such scales can be more safely treated as continuous (Rhemtulla et al., 2012). This difference may be deemed by some practitioners as a limitation; however, categorical methods for data analysis have become widely available, and it is straightforward to treat the data as categorical for EFA and CFA analyses, as was done in the present study (e.g., Bock & Lieberman, 1970; Muthen, 1984; Muthen & Asparouhov, 2002). When it comes to average or sum scores, these can be safely treated as continuous even when the item data only have four categories. Nonetheless, future research shall address whether the Expanded format can be extended to include more response options: that is, whether participants can choose meaningfully among five (or more) sentences. In a sense, the Expanded format makes explicit what is also true of the Likert format: participants may have a hard time differentiating among too many response options.  In addition, there are a few other limitations specific for this study. First, in this study, only perfectly balanced or almost perfectly balanced scales were used. However, many psychological scales are unbalanced. As mentioned previously, unbalanced Likert scales tend to show increased validity (e.g., Rodebaugh et al., 2011) after removing or replacing RW items. Therefore, it is possible that making the unbalanced Likert scales into the Expanded format would also increase the scales’ validity. Future studies should examine how changing unbalanced Likert scales into the Expanded format would impact the factor structures, reliability and validity of the scales.  Second, in this research, only undergraduate student participants were used. This is a meaningful starting point as researchers in many areas of psychology often use undergraduate student samples. However, examining whether the same pattern of results manifested our study would still hold true in other samples is important and should be pursued in future  32 research. In addition, the potential advantage of the Expanded format in reducing carelessness and confusion may be particularly important for certain populations such as children. Due to children’s limited cognitive and language ability, they may experience difficulties in understanding how to respond to negation RW items (Marsh, 1986). For example, it may be hard for preadolescent children to understand that the item I am not good at mathematics requires them to disagree with it to indicate that I am good at mathematics (Marsh, 1986). The Expanded format scales state each of the response options in full sentences, thus may facilitate children to understand the items better. Future studies should explore this potential benefit of the Expanded format scales for children. Finally, our research contained only a few criterion measures to assess the validity of the scales in the Expanded format. A more thorough investigation of the validity of the measures in the new format may be warranted in some cases.  4.2 Recommendations for Making Scales in the Expanded Format For researchers who want to create scales in the Expanded format or to change scales in the Likert format to the Expanded format, we offer a few recommendations based on our experience in creating the Expanded format scales in this study12. Firstly, when changing an item in the Likert format to the Expanded format, researchers can think about what the respondents would mean when they select a specific response option in the Likert scale. For example, for the Item 1 of the Likert CS, I am someone who does a thorough job (see B.2 in Appendix B), researchers can think what the respondents usually mean if they pick the option, strongly agree, somewhat agree, somewhat disagree, or strongly disagree. This may provide researchers with some ideas about how to create the response options for the corresponding                                                 12 These recommendations have not been tested empirically.   33 item in the Expanded format.  Secondly, when changing an item in the Likert format to the Expanded format, researchers should try to keep the original wording of the Likert item in the response options of the corresponding items in the Expanded format in order to minimize changes in item content. However, due to the ambiguity of wordings in some Likert items, researchers may need to change the wording of the items in order to change the items into the Expanded format. For example, for Item 9 of the Likert RSES, I wish I could have more respect for myself (see A.1 in Appendix A), it is hard to predict what a respondent exactly means when he/she picks strongly disagree for this item. The respondent may mean that I really don’t wish I could have more respect for myself or I wish I could have less respect for myself.   In this case, to change the item into the Expanded format, we simplified the item wording without changing the content of the item too much, and created the response options, I have a lot of respect for myself, I have some respect for myself, I have little respect for myself and I have no respect for myself (see A.2 in Appendix A). Finally, when creating an item in the Expanded format, researchers minimize the differences between the response options. This can be achieved by varying just a few words or one phrase between the options. For example, for the Item 2 of the CS in the Expanded format, we created the options by switching between the words, careful and careless, and by adding or deleting the word somewhat (see B.2 in Appendix B). Making the differences between the options obvious may reduce confusion for the respondents.  4.3 Conclusion In summary, our study showed that in comparison to the scales in the Likert format, the scales in the Expanded format had better dimensionalities, and comparable reliability and  34 validity. We recommend that researchers in the social sciences, especially those who employ analyses such as factor analysis, reliability analyses and general structural equation modeling that involve the use of covariance structure of the data, to consider creating and using scales in the Expanded format. Specifically, for general structural equation modeling, researchers must first specify CFA measurement models for scales used in the research, and then build a path model with the latent variables measured in the scales. Any misspecification of the CFA measurement models will cause poor fit for the overall SEM model. A traditional Likert scale with RW items often does not follow the theoretical model of the scale due to acquiescence bias and possible method effects, making it difficult for researchers to know which measurement model to fit for each scale. As shown in our study, a typical Likert scale can be fairly easily converted to the Expanded format, although re-wording challenges do arise. By reducing the impact of acquiescence response bias and method effects on the structure of the Likert scale, the Expanded format may lead to better dimensionality and better fit under the scale’s theoretical model, without reducing reliability or compromising validity.     35 Bibliography Ahlawat, K. S. (1985). On the negative valence items in self-report measures. Journal of General Psychology, 112, 89-99. doi: 10.1080/00221309.1985.9710992 Argyle, M., Martin, M., & Crossland, J. (1989). Happiness as a function of personality and social encounters. In J. P. Forgas, & J. M. Innes (Eds.,), Recent advances in social psychology: An international perspective (pp. 189-203). North-Holland: Elsevier.  Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Manual for the Beck Depression Inventory-II. San Antonio, TX: Psychological Corp.  Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561-571. doi: 10.1001/archpsyc.1961.01710120031004 Bentler, P.M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238-246. doi: 10.1037/0033-2909.107.2.238 Bentler, P. M. (2009). Alpha, dimension-free, and model-based internal consistency reliability. Psychometrika, 74, 137-143. doi: 10.1007/s11336-008-9100-1 Bieling, P. J., Antony, M. M., & Swinson, R. P. (1998). The State-Trait Anxiety Inventory, Trait version: structure and content re-examined. Behaviour Research and Therapy, 36, 777-788. doi: 10.1016/S0005-7967(98)00023-0 Billiet, J. B., & McClendon, M. J. (2000). Modeling acquiescence in measurement models for two balanced sets of items. Structural Equation Modeling, 7, 608-628. doi: 10.1207/S15328007SEM0704_5 Bock, R. D., & Lieberman, M. (1970). Fitting a response model for n dichotomously scored items. Psychometrika, 35, 179-197.doi: 10.1007/BF02291262  36 Bors, D. A., Vigneau. F., & Lalande, F. (2006). Measuring the need for cognition: Item polarity, dimensionality, and the relation with ability. Personality and Individual Differences, 40, 819-828. doi: 10.1509/jmkr.38.3.143.18840 Browne, M.W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136-162). Thousand Oaks, CA: Sage.  Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaire. Educational and Psychological Measurement, 71, 460-502. doi: 10.1177/0013164410375112  Brown, T. A. (2003). Confirmatory factor analysis of the Penn State Worry Questionnaire: Multiple factors or method effects? Behaviour Research and Therapy, 41, 1411-1426. doi: 10.1016/S0005-7967(03)00059-7 Burke, B. G. (1999). Item reversals and response validity in the Job Diagnostic Survey. Psychological Reports, 85, 213-219. doi: 10.2466/pr0.1999.85.1.213 Cacioppo, J.T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116–131. doi:10.1037/0022-3514.42.1.116 Cambre, B., Welkenhuysen-Gybels, J., & Billiet, J. (2002). Is it content or style? An evaluation of two competitive measurement models applied to a balanced set of ethnocentrism items. International Journal of Comparative Sociology, 43, 1-20. doi: 10. 1177/002071520204300101 Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. Beverly Hills, CA: Sage.  Couch, A., & Keniston, K. (1960). Yessayers and naysayers: Agreeing response set as a  37 personality variable. Journal of Abnormal and Social Psychology, 60, 151-174. doi:10.1037/h0040372 DiStenfano, C., & Motl, R. W. (2006). Further investigating method effects associated with negatively worded items on self-report surveys. Structural Equation Modeling, 13(3), 440-464. doi: 10.1207/s15328007sem1303_6 Greenberger, E., Chen, C., Dmitrieva, J., & Farruggia, S. P. (2002). Item-wording and the dimensionality of the Rosenberg Self-Esteem Scale: do they matter? Personality and Individual Differences, 35, 1241-1254. doi: 10.1016/S0191-8869(02)00331-8 Hayton, J. C., Allen, D. G., & Scarpello. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7, 191-205. doi: 10.1177/1094428104263675 Hensley, W.E., & Roberts, M. K. (1976). Dimensions of Rosenburg’s Self-Esteem Scale. Psychological Reports, 38, 583-584. doi: 10.2466/pr0.1976.28.2.582 Hills, P., & Argyle, M. (2002). The Oxford Happiness Questionnaire: a compact scale for the measurement of psychological well-being. Personality and Individual Differences, 33, 1073-1082. doi: 10.1016/S0191-8869(01)00213-6 Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 32, 179-185. doi: 10.1007/BF02289447 Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55. doi: 10.1080/10705519909540118 Jackson, D. M., & Messick, S. (1961). Acquiescence and desirability as response determinants on the MMPI. Educational and Psychological Measurement, 21, 771-790. doi:  38 10.1177/001316446102100402  Javeline, D. (1999). Response effects in polite cultures: A test of acquiescence in Kazakhstan. The Public Opinion Quarterly, 63, 1-28. doi:10.1086/297701 John, O. P., Naumann, L. P., & Soto, C. J. (2008). Paradigm shift to the integrative Big Five trait taxonomy: History, measurement, and conceptual issues. In O. P. John, R.W. Robins, & L.A. Pervin (Eds), Handbook of personality: Theory and research (pp. 114-158). New York, NY: Guilford Press.  Kaplan, H. B., & Pokorny, A. D. (1969). Self-derogation and psychosocial adjustment. The Journal of Nervous and Mental Disease, 149, 421-434.  Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51, 201-219. doi: 10.1086/269029 Lindwall, M., Barkoukis, V., Grano, C., Lucidi, F., Raudsepp, L., Liukkonen, J, & Thogersen-Ntoumani, C. (2012). Method effects: the problem with negatively versus positively keyed items. Journal of Personality Assessment, 94(2), 196-204. doi: 10.1080/00223891.2011.645936 Lyubomirsky, S., & Lepper, H. (1999). A measure of subjective happiness: Preliminary  reliability and construct validation. Social Indicators Research, 46, 137-155. doi: 10.1023/A:1006824100041 MacCallum, R. C., Browne, M. W., & Cai, L. (2006). Testing differences between nested covariance structure models: Power analysis and null hypotheses. Psychological Methods, 11, 19-33. doi: 10.1037/1082-989X.11.1.19 Manian, N., Schmidt, E., Bornstein, M. H., & Martinez, P. (2013). Factor structure and  39 clinical utility of BDI-II factor scores in postpartum women. Journal of Affective Disorders, 149, 259-268. doi: 10.1016/j.jad.2013.01.039 Marsh, H. W. (1986). Negative item bias in ratings scales for preadolescent children: A cognitive-developmental phenomenon. Development Psychology, 22, 37-49. doi:10.1037/0012-1649.22.1.37  Marsh, H. W. (1996). Positive and negative global self-esteem: A substantive meaningful distinction or artifactors? Journal of Personality and Social Psychology, 70, 810-819. doi: 10.1037/0022-3514.70.4.810  Marsh, H. W., Scalas, L. F., & Nagengast, B. (2010). Longitudinal tests of competing factor structures for the Rosenberg Self-Esteem Scale: Traits, ephemeral artifacts, and stable response styles. Psychological Assessment, 22, 361-381. doi: 10.1037/a0019225 Maydeu-Olivares, A., & Coffman, D. L. (2006). Random intercept item factor analysis. Psychological Methods, 11, 344-362. doi: 10.1037/1082-989X.11.4.344 McClendon, M. J. (1986). Response-order effects for dichotomous questions. Social Science Quarterly, 67, 205-211.  McClendon, M. J. (1991). Acquiescence and recency response-order effects in interview surveys. Sociological Methods & Research, 20, 60-103. doi: 10.1177/0049124191020001003 Muthen, B. O. (1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49, 115-132. doi: 10.1007/BF02294210 Muthen, B. O., Asparouhov, T. (2002). Latent variable analysis with categorical outcomes: Multiple-group and growth modeling in Mplus. Retrieved from  40 http://www.statmodel.com/download/webnotes /CatMGLong.pdf on June 10, 2015.  Pullmann, H., & Allik, J. (2000). The Rosenberg Self-Esteem Scale: Its dimensionality, stability and personality correlates in Estonian. Personality and Individual Differences, 28, 701-715. doi: 10.1016/S0191-8869(99)00132-4  Quilty, L. C., Oakman, J. M., & Risko, E. (2006). Correlates of the Rosenberg Self-Esteem Scale method effects. Structural Equation Modeling, 13, 99-117. doi: 10.1207/s15328007sem1301_5 Ray, J. J. (1983). Reviving the problem of acquiescent response bias. Journal of Social Psychology. 121, 81-96. doi: 10.1080/00224545.1983.9924470 Raykov, T. (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21(2), 173-184. doi: 10.1177/ 01466216970212006 Revelle, W. (2014) psych: Procedures for Personality and Psychological Research, Northwestern University, Evanston, Illinois, USA. Retrieved from  http://CRAN.R-project.org/package=psych on Dec 20, 2014.   Rodebaugh, T. L., Heimberg, R. G., Brown, P.J., Fernandez, K. C., Blanco, C., Schneier, F. R., & Liebowitz, M. R. (2011). More reasons to be straightforward: Findings and norms for two scales relevant to social anxiety. Journal of Anxiety Disorders, 25, 623-630. doi: 10.1016/j.janxdis.2011.02.002 Rodebaugh, T. L., Woods, C. M., Heimberg, R. G., Liebowitz, M. R., & Schneier, F. R. (2006). The factor structure and screening utility of the Social Interaction Anxiety Scale. Psychological Assessment, 18, 231-237. doi: 10.1037/1040-3590.18.2.231 Rosenberg, M. (1965). Society and the Adolescent Self-image. Princeton, NJ: Princeton University Press.   41 Rosseel, Y (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical software, 48(2), 1-36. Rhemtulla, M., Brosseau-Liard, P. E., & Savalei, V. (2012). How many categories is enough to treat data as continuous? A comparison of robust continuous and categorical SEM estimation method under range of non-ideal situations. Psychological Methods, 17, 354-474. doi: 10.1037/a0029315 Roszkowski, M. J., & Soven (2009). Shifting gears: Consequences of including two negatively worded items in the middle of a positively worded questionnaire. Assessment & Evaluation in Higher Education, 35, 113-130. doi: 10.1080/02602930802618344 Satorra, A. (2000). Scaled and adjusted restricted tests in multi-sample analysis of moment structures. In Heijman, R.D.H., Pollock, D.S.G. & Satorra, A. (eds.), Innovations in multivariate statistical analysis. A Festschrift for Heinz Neudecker (pp. 233-247). London: Kluwer Academic Publishers.  Savalei, V., & Falk, C. (2014). Recovering substantive factor loadings in the presence of acquiescence bias: A comparison of three approaches. Multivariate Behavioral Research, 49, 407-424. doi: 10.1080/00273171.2014.931800 Schmitt, N., & Stuits, D. M. (1985). Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement, 9, 367-373. doi: 10.1177/014662168500900405 Schriesheim, C. A., & Hill, K. D. (1981). Controlling acquiescence response bias by item reversals: The effect on questionnaire validity. Educational and Psychological Measurement, 41, 1101-14. doi: 10.1177/001316448104100420  42 Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys: Experiments on question form, wording and context. Orlando, FL: Academic Press.  Sonderen E. V., Sanderman, R., & Coyne, J. C. (2013). Ineffectiveness of reverse wording of questionnaire items: Let’s learn from cows in the rain. PloS ONE, 8(7). doi:10.1371/journal.pone.0068967.  Spencer-Rodgers, J., Peng, K., Wang, L., & Hou, Y. (2004) Dialectical self-esteem and East-West differences in psychological well-being. Personality and Social Psychology Bulletin, 30, 1416-1432. doi: 10.1177/0146167204264243 Steed (2001). Further validity and reliability evidence for Beck Hopelessness scale scores in a nonclinical sample. Educational and Psychological Measurement, 61, 303-316. doi: 10.1177/00131640121971121.  Steer, R. A., & Clark, D. A. (1997). Psychometric characteristics of the Beck Depression Inventory-II with college students. Measurement and Evaluation in Counseling and Development, 30, 128-136. doi:10.1080/13651500510014800 Steiger, J. H., & Lind, J. C. (1980, May). Statistically based tests for the number of factors. Paper presented at the annual meeting of the Psychometric Society, Iowa City, IA.  Swain, S. D., Weathers, D., & Niedrich, R. W. (2008). Assessing three sources of misresponse to reversed Likert items. Journal of Marking Research, 45, 116-131. doi: 10.1509/jmkr.45.1.116 Tafarodi, R. W., & Swann, W. B. Jr. (1995). Self-liking and self-competence as dimensions of global self-esteem: initial validation of a measure. Journal of Personality Assessment, 65, 322-342. doi: 10.1207/s15327752jpa6502_8 Watson, D. (1992). Correcting for acquiescent response bias in the absence of a balanced  43 scale: An application to class consciousness. Sociological Methods & Research, 21, 52-88. doi: 10.1177/0049124192021001003 Weijters, B., Geuens, M., & Schillewaert, N. (2010). The stability of individual response styles. Psychological Methods, 15, 96-110. doi: 10.1037/a0018721 White, R. T., & Mackay, L. D. (1973). A note on a possible alternative to Likert scales. Research in Science Education, 3(1), 75-76. doi: 10.1007/BF02558560 Woods, C. M. (2006). Careless responding to reverse worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 189-194. doi: 10.1007/s10862-005-9004-7  Wright, S. (1934). The method of path coefficients. The Annals of Mathematical Statistics, 5, 161-215. doi: 10.1214/aoms/1177732676 Wu (2008). An examination of the wording effect in Rosenberg Self-Esteem Scale among culturally Chinese people. The Journal of Social Psychology, 148, 535-551.  doi: 10.3200/SOCP.148.5.535-552 Zwick, W.R., & Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-442. doi: 10.1007/BF02289447  44 Appendices Appendix A   Rosenberg Self-Esteem Scale  A.1 Likert Version For each of the following, please pick the one that corresponds with the answer that best describes how strongly you agree or disagree with the statement about yourself now.    Strongly agree  Somewhat agree  Somewhat disagree  Strongly disagree  1) I feel that I am a person of worth, at least on an equal plane with others.  !  !  !  !  2) I feel that I have a number of good qualities.  !  !  !  !  3) All in all, I’m inclined to feel that I am a failure.  !  !  !  !  4) I am able to do things as well as most other people.  !  !  !  !  5) I feel I do not have much to be proud of.  !  !  !  !  6) I take a positive attitude towards myself.  !  !  !  !  7) On the whole, I am satisfied with myself.  !  !  !  !  8) I certainly feel useless at times.  !  !  !  !  9) I wish I could have more respect for myself.  !  !  !  !  10) At times, I think I am no good at all.  !  !  !  !   Note: Item 3, 5, 8, 9 and 10 are reverse worded item.  45 A.2 Expanded Version This questionnaire consists of 10 groups of statements. Please read each group of statements carefully, and then pick out the one statement in each group that best describes your attitude towards yourself. Be sure that you do not choose more than one statement for any group.   1) Select the one that best describes your attitude towards yourself.  ! I feel that I’m a person of great worth, more so than other people.  ! I feel that I’m a person of worth, at least on an equal basis with others.  ! I feel that I’m a person of little worth, not on an equal basis with others.  ! I feel that I’m a person of little worth, less so than other people.   2) Select the one that best describes your attitude towards yourself.  ! I feel that I have a great number of good qualities.  ! I feel that I have some good qualities.  ! I feel that I don’t have many good qualities.  ! I feel that I have very few good qualities.   3) Select the one that best describes your attitude towards yourself.  ! All in all, I think I am a success.  ! All in all, I think I am somewhat a success.  ! All in all, I think I am somewhat a failure.  ! All in all, I am inclined to think I’m a failure.   4) Select the one that best describes your attitude towards yourself.  ! I am able to do things much better than most other people.  ! I am able to do things as well as most other people.  ! I am unable to do things as well as most other people.  ! I usually do things much worse than other people.   5) Select the one that best describes your attitude towards yourself. ! I feel I have a lot to be proud of.  ! I feel I have some qualities that I feel proud of.  ! I feel I do not have much to be proud of.  ! I feel I have nothing to be proud of.  6) Select the one that best describes your attitude towards yourself.  ! I take a very positive attitude towards myself.  ! I take a somewhat positive attitude towards myself.  ! I take a somewhat negative attitude towards myself.  ! I take a very negative attitude towards myself.   46  7) Select the one that best describes your attitude towards yourself.  ! On the whole, I am very satisfied with myself.  ! On the whole, I am satisfied with myself.  ! On the whole, I am disappointed with myself.  ! On the whole, I am very disappointed with myself.   8) Select the one that best describes your attitude towards yourself.  ! I feel useful most of the time.  ! I certainly feel useful at times.  ! I certainly feel useless at times.  ! I feel useless most of the time.   9) Select the one that best describes your attitude towards yourself.  ! I have a lot of respect for myself.  ! I have some respect for myself.  ! I have little respect for myself.  ! I have no respect for myself.   10) Select the one that best describes your attitude towards yourself.  ! Most of the time, I think I’m very good.  ! At times, I think I’m fairly good.  ! At times, I think I’m no good at all.  ! Most of the time, I think I’m no good at all.    47 Appendix B  Conscientious Scale B.1 Likert Version  Here are a number of characteristics that may or may not apply to you. Please rate the extent to which you agree or disagree with that statement.   Disagree strongly Disagree a little  Agree a little  Agree strongly  1) I am someone who does a thorough job.  !  !  !  !  2) I am someone who can be somewhat careless.  !  !  !  !  3) I am someone who is a reliable worker.  !  !  !  !  4) I am someone who tends to be disorganized.  !  !  !  !  5) I am someone who tends to be lazy.  !  !  !  !  6) I am someone who perseveres until the task is finished.  !  !  !  !  7) I am someone who does things efficiently.  !  !  !  !  8) I am someone who makes plans and follows through on them.  !  !  !  !  9) I am someone who is easily distracted.  !  !  !  !   Note: Item 2, 4, 5 and 9 are reverse worded items.   48 B.2 Expanded Version This questionnaire consists of 9 groups of statements. Please read each group of statements carefully, and then pick out the one statement in each group that best describes you. Be sure that you do not choose more than one statement for any group.   1) Select the one of the following options that best describes you ! I am someone who does a sloppy job.  ! I am someone who does a somewhat sloppy job.  ! I am someone who does a somewhat thorough job.  ! I am someone who does a thorough job.   2) Select the one of the following options that best describes you ! I am someone who can be careless.  ! I am someone who can be somewhat careless.  ! I am someone who can be somewhat careful.  ! I am someone who can be careful.   3) Select the one of the following options that best describes you ! I am someone who is an unreliable worker.  ! I am someone who is a somewhat unreliable worker.  ! I am someone who is a somewhat reliable worker.  ! I am someone who is a reliable worker.   4) Select the one of the following options that best describes you ! I am someone who tends to be disorganized.  ! I am someone who tends to be somewhat disorganized.  ! I am someone who tends to be somewhat organized.  ! I am someone who tends to be organized.   5) Select the one of the following options that best describes you ! I am someone who tends to be very lazy.  ! I am someone who tends to be lazy.  ! I am someone who tends to be diligent.  ! I am someone who tends to be very diligent.   6) Select the one of the following options that best describes you ! I am someone who often gives up before the task is finished.  ! I am someone who gives up before the task is finished.  ! I am someone who perseveres until the task is finished.  ! I am someone who often perseveres until the task is finished.        49 7) Select the one of the following options that best describes you ! I am someone who does things very inefficiently.  ! I am someone who does things inefficiently.  ! I am someone who does things efficiently.  ! I am someone who does things very efficiently.   8) Select the one of the following options that best describes you ! I am someone who makes plans but rarely follows through on them.  ! I am someone who makes plans but sometimes does not follow through on them.  ! I am someone who makes plans and usually follows through on them.  ! I am someone who makes plans and always follows through on them.   9) Select the one of the following options that best describes you ! I am someone who can be very easily distracted.  ! I am someone who can be easily distracted.  ! I am someone who can easily stay focused.  ! I am someone who can very easily stay focused.    50 Appendix C  Beck Depression Inventory C.1 Likert Version I For each item, select the point that best describes the way you have being feeling in the past two weeks, including today.  Strongly Agree  Somewhat Agree  Somewhat Disagree  Strongly Disagree  1) I am so sad or unhappy that I can’t stand it.  !  !  !  !  2) I am not discouraged about my future.  !  !  !  !  3) I feel I am a total failure as a person.  !  !  !  !  4) I can’t get any pleasure from the things I used to enjoy.  !  !  !  !  5) I don’t feel particularly guilty.  !  !  !  !  6) I don’t feel I am being punished.  !  !  !  !  7) I dislike myself.  !  !  !  !  8) I blame myself for everything bad that happens.  !  !  !  !  9) I don’t have thoughts of killing myself.  !  !  !  !  10) I don’t cry anymore than I used to.  !  !  !  !  11) I am no more restless or wound up than usual.  !  !  !  !  12) It’s hard to get interested in anything.  !  !  !  !  13) I make decisions about as well as ever.  !  !  !  !  14) I do not feel I am worthless.  !  !  !  !  15) I don’t have enough energy to do anything.  !  !  !  !  16) I wake up 1-2 hours early and can’t get back to sleep.  !  !  !  !   51  Strongly Agree Somewhat Agree Somewhat Disagree Somewhat Disagree 17) I am irritable all the time.  !  !  !  !  18) I have not experienced any change in my appetite.  !  !  !  !  19) I can concentrate as well as ever.  !  !  !  !  20) I am too tired or fatigued to do most of the things I used to do.  !  !  !  !  21) I have lost interest in sex completely.  !  !  !  !   Note: Item 2, 5, 6, 9, 10, 11, 13, 14, 18 and 19 are reverse worded items.  52 C.2 Likert Version II For each item, select the point that best describes the way you have being feeling in the past two weeks, including today.  Strongly Agree Somewhat Agree  Somewhat Disagree  Strongly Disagree  1) I feel sad much of the time.  !  !  !  !  2) I feel optimistic about the future.  !  !  !  !  3) I have failed more than I should have.  !  !  !  !  4) I get little pleasure from the things I used to enjoy.  !  !  !  !  5) I rarely feel guilty.  !  !  !  !  6) I don’t feel I am being punished.  !  !  !  !  7) I am disappointed in myself.  !  !  !  !  8) I criticize myself for a lot of my faults.  !  !  !  !  9) I rarely have thoughts of killing myself.  !  !  !  !  10) I have been crying a lot recently.  !  !  !  !  11) I am usually calm and laid-back.  !  !  !  !  12) I have lost most of my interest in other people or things.  !  !  !  !  13) I make decisions about as well as ever.  !  !  !  !  14) I feel I am a person of worth.  !  !  !  !  15) I have a lot of energy to do many tasks.  !  !  !  !  16) I have not been sleeping well lately.  !  !  !  !  17) I am irritable most of the time.  !  !  !  !   53  Strongly Agree Somewhat Agree Somewhat Disagree Somewhat Disagree 18) I have not experienced any change in my appetite.  !  !  !  !  19) I can concentrate well.  !  !  !  !  20) I am too tired or fatigue to do a lot of the things.  !  !  !  !  21) I am much less interested in sex now.  !  !  !  !   Note: Item 2, 5, 6, 9, 11, 13, 14, 15, 18 and 19 are reverse worded items.  54 Appendix D  Figures for Introduction Figure 1: Factor Model of Three Items with Acquiescence Bias                55  Figure 2: Correlated trait, correlated uniqueness (CTCU) models  (a)  (b)  (c) Note. Model (a) is a CTCU model in which correlations are among the residuals of the PW items. Model (b) is a CTCU model in which correlations are among the residuals of the RW items. Model (c) is a CTCU model in which correlations are among the residuals of the PW items and the residuals of the RW items. P=PW item; R=RW item.  56 Figure 3: Correlated trait, correlated method (CTCM) models  (a)   (b)  (c) Note.  Model (a) is a CTCM model that consists of one method factor which all PW items load on. Model (b) is a CTCM model that consists of one method factor which all RW items load on. Model (c) is a CTCM model that consists of two method factors with all PW items loading on one factor and all RW items loading on the other. P=PW item; R=RW items.   57 Figure 4: Bi-dimensional Model     Note. ACQ=acquiescence factor. P=PW item; R=RW item. 58 Appendix E  Figure for Method Figure 5: The Three Factor Structure Models of the Study     Model 1    Model 2  Model 3 Note. P=Positively worded item; R=Reverse worded item.   59 Appendix F  Figures for Results Figure 6: Parallel Scree Plots for All Versions of the Rosenberg Self-Esteem Scale and the Conscientiousness Scale   Note. Plot (a) is for the RSES Likert version; plot (b) is for the RSES Expanded; plot (c) is for the CS Likert version; plot (d) is for the CS Expanded version.    0 2 4 6 8 100123456(a)Factor Number Eigenvalues of Principle FactorsFA  Actual Data FA  Simulated Data0 2 4 6 8 100123456(b)Factor Number Eigenvalues of Principle FactorsFA  Actual Data FA  Simulated Data0 2 4 6 8 1001234(c)Factor Number Eigenvalues of Principle FactorsFA  Actual Data FA  Simulated Data0 2 4 6 8 1001234(d)Factor Number Eigenvalues of Principle FactorsFA  Actual Data FA  Simulated Data 60 Figure 7: Parallel Scree Plots for All Versions of the Beck Depression Inventory    Note. Plot (a) is for the Likert Version I; plot (b) is for Likert Version II; plot is (c) for Expanded version0 5 10 15 20 2502468(a)Factor Number Eigenvalues of Principle FactorsFA  Actual Data FA  Simulated Data0 5 10 15 20 2502468(b)Factor Number Eigenvalues of Principle FactorsFA  Actual Data FA  Simulated Data0 5 10 15 20 250246810(c)Factor Number Eigenvalues of Principle FactorsFA  Actual Data FA  Simulated Data 61 Appendix G  Tables for Results Table 1: Covariance Matrix for the Likert Rosenberg Self-Esteem Scale    Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 Item 1 0.3882 0.2633 0.2288 0.2197 0.2057 0.2489 0.2829 0.1740 0.1917 0.2771 Item 2 0.2633 0.4075 0.2203 0.2078 0.2305 0.2393 0.2397 0.1538 0.1582 0.2389 Item 3 0.2288 0.2203 0.6524 0.1912 0.3482 0.2727 0.2716 0.2930 0.2478 0.3661 Item 4 0.2197 0.2078 0.1912 0.4257 0.1725 0.2434 0.2406 0.1416 0.1105 0.2194 Item 5 0.2057 0.2305 0.3482 0.1725 0.7552 0.2407 0.2639 0.2700 0.2531 0.3448 Item 6 0.2489 0.2393 0.2727 0.2434 0.2407 0.5168 0.3613 0.2043 0.2632 0.3266 Item 7 0.2829 0.2397 0.2716 0.2406 0.2639 0.3613 0.5788 0.2423 0.2914 0.3455 Item 8 0.1740 0.1538 0.2930 0.1416 0.2700 0.2043 0.2423 0.6845 0.3030 0.5060 Item 9 0.1917 0.1582 0.2478 0.1105 0.2531 0.2632 0.2914 0.3030 0.8063 0.4082 Item 10 0.2771 0.2389 0.3661 0.2194 0.3448 0.3266 0.3455 0.5060 0.4082 0.8815  Note. The reverse worded items are reverse-keyed.  62 Table 2: Covariance Matrix for the Expanded Rosenberg Self-Esteem Scale    Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 Item 1 0.2467 0.1652 0.1298 0.1047 0.0946 0.1089 0.1147 0.1055 0.1068 0.1339 Item 2 0.1652 0.4278 0.2089 0.1561 0.1711 0.1969 0.1959 0.1936 0.1757 0.1931 Item 3 0.1298 0.2089 0.4024 0.1021 0.1898 0.2075 0.2192 0.2260 0.1514 0.1958 Item 4 0.1047 0.1561 0.1021 0.2969 0.0865 0.0994 0.0942 0.1178 0.0889 0.1076 Item 5 0.0946 0.1711 0.1898 0.0865 0.3414 0.1736 0.1727 0.1879 0.1593 0.1538 Item 6 0.1089 0.1969 0.2075 0.0994 0.1736 0.5384 0.2526 0.2569 0.2207 0.2206 Item 7 0.1147 0.1959 0.2192 0.0942 0.1727 0.2526 0.3902 0.2235 0.2042 0.1979 Item 8 0.1055 0.1936 0.2260 0.1178 0.1879 0.2569 0.2235 0.4458 0.1582 0.2159 Item 9 0.1068 0.1757 0.1514 0.0889 0.1593 0.2207 0.2042 0.1582 0.3969 0.1781 Item 10 0.1339 0.1931 0.1958 0.1076 0.1538 0.2206 0.1979 0.2159 0.1781 0.3895  63 Table 3: Covariance Matrix for the Likert Conscientiousness Scale    Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 1 0.5379 0.1701 0.2803 0.1966 0.1941 0.2758 0.2739 0.2697 0.0630 Item 2 0.1701 0.7597 0.1326 0.4414 0.2654 0.1121 0.2006 0.2094 0.2202 Item 3 0.2803 0.1326 0.4061 0.1355 0.1147 0.2502 0.2444 0.1871 0.0463 Item 4 0.1966 0.4414 0.1355 0.8566 0.3703 0.1259 0.2273 0.2837 0.2740 Item 5 0.1941 0.2654 0.1147 0.3703 0.8123 0.1625 0.1632 0.2509 0.3238 Item 6 0.2758 0.1121 0.2502 0.1259 0.1625 0.6035 0.2655 0.2528 0.1101 Item 7 0.2739 0.2006 0.2444 0.2273 0.1632 0.2655 0.6654 0.2845 0.1643 Item 8 0.2697 0.2094 0.1871 0.2837 0.2509 0.2528 0.2845 0.6805 0.1477 Item 9 0.0630 0.2202 0.0463 0.2740 0.3238 0.1101 0.1643 0.1477 0.6744  Note. The reverse worded items are reverse-keyed.   64 Table 4: Covariance Matrix for the Expanded Conscientiousness Scale    Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 1 0.4229 0.2637 0.1896 0.2239 0.2343 0.1680 0.1515 0.1727 0.1679 Item 2 0.2637 0.8562 0.2002 0.3663 0.2825 0.1650 0.1568 0.1553 0.2409 Item 3 0.1896 0.2002 0.3510 0.1921 0.1547 0.1122 0.1164 0.1193 0.0788 Item 4 0.2239 0.3663 0.1921 0.8119 0.3126 0.1233 0.1537 0.1930 0.2169 Item 5 0.2343 0.2825 0.1547 0.3126 0.6075 0.2045 0.2050 0.2119 0.2338 Item 6 0.1680 0.1650 0.1122 0.1233 0.2045 0.3911 0.1446 0.1178 0.1100 Item 7 0.1515 0.1568 0.1164 0.1537 0.2050 0.1446 0.4738 0.2084 0.1678 Item 8 0.1727 0.1553 0.1193 0.1930 0.2119 0.1178 0.2084 0.5200 0.1652 Item 9 0.1679 0.2409 0.0788 0.2169 0.2338 0.1100 0.1678 0.1652 0.5500  65 Table 5: Covariance Matrix of the Likert Version I Beck Depression Inventor    Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 Item 11 Item 12 Item 13 Item 14 Item 15 Item 16 Item 17 Item 18 Item 19 Item 20 Item 21 Item 1 0.71 0.23 0.39 0.34 0.21 0.20 0.38 0.26 0.21 0.27 0.25 0.32 0.14 0.31 0.20 0.14 0.31 0.14 0.25 0.22 0.12 Item 2 0.23 0.68 0.20 0.19 0.22 0.17 0.18 0.07 0.06 0.10 0.16 0.16 0.15 0.24 0.12 0.13 0.21 0.08 0.18 0.14 0.14 Item 3 0.39 0.20 0.54 0.32 0.17 0.13 0.39 0.29 0.17 0.15 0.14 0.25 0.13 0.32 0.18 0.12 0.25 0.10 0.23 0.18 0.10 Item 4 0.34 0.19 0.32 0.73 0.18 0.19 0.38 0.21 0.16 0.10 0.18 0.39 0.12 0.26 0.33 0.12 0.25 0.16 0.29 0.32 0.15 Item 5 0.21 0.22 0.17 0.18 0.83 0.51 0.23 0.12 0.17 0.11 0.08 0.15 0.20 0.26 0.11 0.08 0.13 0.20 0.21 0.15 0.02 Item 6 0.20 0.17 0.13 0.19 0.51 0.92 0.17 0.11 0.17 0.09 0.08 0.13 0.20 0.29 0.05 0.02 0.12 0.19 0.17 0.09 0.05 Item 7 0.38 0.18 0.39 0.38 0.23 0.17 0.71 0.35 0.23 0.13 0.16 0.27 0.14 0.41 0.22 0.10 0.25 0.09 0.26 0.22 0.11 Item 8 0.26 0.07 0.29 0.21 0.12 0.11 0.35 0.73 0.07 0.11 0.09 0.17 0.06 0.26 0.14 0.23 0.21 -0.02 0.16 0.19 0.06 Item 9 0.21 0.06 0.17 0.16 0.17 0.17 0.23 0.07 0.75 0.16 0.08 0.11 0.12 0.19 0.09 0.11 0.11 0.14 0.22 0.06 0.06 Item 10 0.27 0.10 0.15 0.10 0.11 0.09 0.13 0.11 0.16 0.77 0.29 0.16 0.19 0.22 0.12 0.13 0.15 0.16 0.15 0.09 0.03 Item 11 0.25 0.16 0.14 0.18 0.08 0.08 0.16 0.09 0.08 0.29 0.74 0.25 0.17 0.23 0.20 0.16 0.21 0.21 0.23 0.15 0.05 Item 12 0.32 0.16 0.25 0.39 0.15 0.13 0.27 0.17 0.11 0.16 0.25 0.78 0.15 0.23 0.41 0.16 0.29 0.15 0.23 0.37 0.14 Item 13 0.14 0.15 0.13 0.12 0.20 0.20 0.14 0.06 0.12 0.19 0.17 0.15 0.51 0.23 0.18 0.03 0.13 0.21 0.29 0.16 0.09 Item 14 0.31 0.24 0.32 0.26 0.26 0.29 0.41 0.26 0.19 0.22 0.23 0.23 0.23 0.67 0.15 0.11 0.24 0.15 0.27 0.15 0.09 Item 15 0.20 0.12 0.18 0.33 0.11 0.05 0.22 0.14 0.09 0.12 0.20 0.41 0.18 0.15 0.70 0.15 0.23 0.11 0.28 0.47 0.14 Item 16 0.14 0.13 0.12 0.12 0.08 0.02 0.10 0.23 0.11 0.13 0.16 0.16 0.03 0.11 0.15 0.88 0.11 -0.02 -0.01 0.21 0.12 Item 17 0.31 0.21 0.25 0.25 0.13 0.12 0.25 0.21 0.11 0.15 0.21 0.29 0.13 0.24 0.23 0.11 0.65 0.07 0.17 0.22 0.15 Item 18 0.14 0.08 0.10 0.16 0.20 0.19 0.09 -0.02 0.14 0.16 0.21 0.15 0.21 0.15 0.11 -0.02 0.07 0.88 0.28 0.12 0.03 Item 19 0.25 0.18 0.23 0.29 0.21 0.17 0.26 0.16 0.22 0.15 0.23 0.23 0.29 0.27 0.28 -0.01 0.17 0.28 0.79 0.28 0.13 Item 20 0.22 0.14 0.18 0.32 0.15 0.09 0.22 0.19 0.06 0.09 0.15 0.37 0.16 0.15 0.47 0.21 0.22 0.12 0.28 0.77 0.15 Item 21 0.12 0.14 0.10 0.15 0.02 0.05 0.11 0.06 0.06 0.03 0.05 0.14 0.09 0.09 0.14 0.12 0.15 0.03 0.13 0.15 0.54  Note. The reverse worded items are reverse-keyed.  66 Table 6: Covariance Matrix of the Likert Version II Beck Depression Inventory   Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 Item 11 Item 12 Item 13 Item 14 Item 15 Item 16 Item 17 Item 18 Item 19 Item 20 Item 21 Item 1 0.80 0.29 0.35 0.30 0.16 0.21 0.47 0.25 0.26 0.33 0.28 0.39 0.31 0.33 0.40 0.25 0.37 0.28 0.34 0.38 0.17 Item 2 0.29 0.64 0.21 0.19 0.07 0.17 0.31 0.21 0.12 0.10 0.18 0.21 0.28 0.31 0.34 0.11 0.17 0.15 0.23 0.20 0.13 Item 3 0.35 0.21 0.86 0.21 0.22 0.16 0.53 0.30 0.07 0.19 0.12 0.27 0.18 0.26 0.28 0.21 0.25 0.13 0.27 0.24 0.11 Item 4 0.30 0.19 0.21 0.78 0.04 0.15 0.24 0.18 0.15 0.13 0.07 0.40 0.25 0.21 0.22 0.34 0.30 0.29 0.19 0.32 0.24 Item 5 0.16 0.07 0.22 0.04 0.70 0.20 0.23 0.21 0.08 0.03 0.17 0.10 0.13 0.10 0.14 0.14 0.10 0.07 0.15 0.12 0.01 Item 6 0.21 0.17 0.16 0.15 0.20 0.76 0.29 0.18 0.19 0.14 0.12 0.24 0.23 0.23 0.21 0.15 0.21 0.16 0.21 0.22 0.18 Item 7 0.47 0.31 0.53 0.24 0.23 0.29 0.89 0.45 0.15 0.22 0.24 0.35 0.27 0.33 0.39 0.23 0.28 0.19 0.38 0.32 0.25 Item 8 0.25 0.21 0.30 0.18 0.21 0.18 0.45 0.73 0.03 0.10 0.13 0.18 0.17 0.17 0.26 0.25 0.21 0.08 0.17 0.24 0.11 Item 9 0.26 0.12 0.07 0.15 0.08 0.19 0.15 0.03 0.83 0.10 0.20 0.16 0.20 0.19 0.18 0.04 0.09 0.18 0.14 0.19 0.07 Item 10 0.33 0.10 0.19 0.13 0.03 0.14 0.22 0.10 0.10 0.76 0.16 0.18 0.13 0.13 0.09 0.23 0.26 0.17 0.13 0.18 0.12 Item 11 0.28 0.18 0.12 0.07 0.17 0.12 0.24 0.13 0.20 0.16 0.77 0.13 0.24 0.19 0.22 0.06 0.17 0.11 0.17 0.13 0.09 Item 12 0.39 0.21 0.27 0.40 0.10 0.24 0.35 0.18 0.16 0.18 0.13 0.78 0.22 0.24 0.27 0.22 0.38 0.25 0.25 0.27 0.19 Item 13 0.31 0.28 0.18 0.25 0.13 0.23 0.27 0.17 0.20 0.13 0.24 0.22 0.57 0.30 0.35 0.16 0.19 0.22 0.25 0.21 0.18 Item 14 0.33 0.31 0.26 0.21 0.10 0.23 0.33 0.17 0.19 0.13 0.19 0.24 0.30 0.49 0.36 0.10 0.18 0.15 0.23 0.19 0.20 Item 15 0.40 0.34 0.28 0.22 0.14 0.21 0.39 0.26 0.18 0.09 0.22 0.27 0.35 0.36 0.76 0.20 0.21 0.17 0.37 0.39 0.23 Item 16 0.25 0.11 0.21 0.34 0.14 0.15 0.23 0.25 0.04 0.23 0.06 0.22 0.16 0.10 0.20 0.99 0.32 0.29 0.26 0.33 0.20 Item 17 0.37 0.17 0.25 0.30 0.10 0.21 0.28 0.21 0.09 0.26 0.17 0.38 0.19 0.18 0.21 0.32 0.65 0.24 0.24 0.31 0.13 Item 18 0.28 0.15 0.13 0.29 0.07 0.16 0.19 0.08 0.18 0.17 0.11 0.25 0.22 0.15 0.17 0.29 0.24 0.80 0.26 0.23 0.17 Item 19 0.34 0.23 0.27 0.19 0.15 0.21 0.38 0.17 0.14 0.13 0.17 0.25 0.25 0.23 0.37 0.26 0.24 0.26 0.68 0.30 0.19 Item 20 0.38 0.20 0.24 0.32 0.12 0.22 0.32 0.24 0.19 0.18 0.13 0.27 0.21 0.19 0.39 0.33 0.31 0.23 0.30 0.77 0.25 Item 21 0.17 0.13 0.11 0.24 0.01 0.18 0.25 0.11 0.07 0.12 0.09 0.19 0.18 0.20 0.23 0.20 0.13 0.17 0.19 0.25 0.77  Note. The reverse worded items are reverse-keyed.  67 Table 7: Covariance Matrix of the Expanded Beck Depression Inventory (a.k.a. BDI-II)   Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 Item 11 Item 12 Item 13 Item 14 Item 15 Item 16 Item 17 Item 18 Item 19 Item 20 Item 21 Item 1 0.45 0.17 0.20 0.16 0.17 0.16 0.23 0.17 0.14 0.13 0.16 0.19 0.26 0.21 0.18 0.09 0.15 0.15 0.21 0.23 0.14 Item 2 0.17 0.38 0.17 0.18 0.12 0.12 0.23 0.21 0.12 0.12 0.11 0.17 0.22 0.18 0.16 0.07 0.13 0.09 0.18 0.15 0.10 Item 3 0.20 0.17 0.53 0.16 0.19 0.13 0.32 0.20 0.14 0.14 0.19 0.14 0.27 0.26 0.16 0.11 0.15 0.12 0.19 0.19 0.12 Item 4 0.16 0.18 0.16 0.45 0.16 0.13 0.33 0.19 0.13 0.14 0.17 0.30 0.27 0.26 0.20 0.13 0.19 0.14 0.25 0.21 0.17 Item 5 0.17 0.12 0.19 0.16 0.47 0.19 0.22 0.22 0.15 0.12 0.14 0.14 0.19 0.22 0.13 0.06 0.14 0.14 0.20 0.17 0.15 Item 6 0.16 0.12 0.13 0.13 0.19 0.36 0.22 0.18 0.11 0.11 0.09 0.12 0.20 0.19 0.11 0.10 0.14 0.14 0.20 0.15 0.08 Item 7 0.23 0.23 0.32 0.33 0.22 0.22 0.73 0.31 0.17 0.18 0.24 0.30 0.40 0.38 0.20 0.12 0.26 0.17 0.30 0.23 0.20 Item 8 0.17 0.21 0.20 0.19 0.22 0.18 0.31 0.58 0.13 0.15 0.14 0.22 0.30 0.27 0.18 0.09 0.19 0.14 0.25 0.20 0.15 Item 9 0.14 0.12 0.14 0.13 0.15 0.11 0.17 0.13 0.44 0.10 0.15 0.17 0.16 0.16 0.13 0.14 0.20 0.13 0.15 0.19 0.10 Item 10 0.13 0.12 0.14 0.14 0.12 0.11 0.18 0.15 0.10 0.22 0.14 0.15 0.16 0.16 0.11 0.04 0.12 0.12 0.15 0.13 0.13 Item 11 0.16 0.11 0.19 0.17 0.14 0.09 0.24 0.14 0.15 0.14 0.59 0.17 0.23 0.20 0.19 0.10 0.16 0.15 0.17 0.15 0.12 Item 12 0.19 0.17 0.14 0.30 0.14 0.12 0.30 0.22 0.17 0.15 0.17 0.52 0.34 0.27 0.26 0.14 0.24 0.14 0.27 0.28 0.18 Item 13 0.26 0.22 0.27 0.27 0.19 0.20 0.40 0.30 0.16 0.16 0.23 0.34 0.83 0.40 0.28 0.17 0.27 0.16 0.34 0.30 0.23 Item 14 0.21 0.18 0.26 0.26 0.22 0.19 0.38 0.27 0.16 0.16 0.20 0.27 0.40 0.54 0.18 0.11 0.23 0.15 0.27 0.19 0.15 Item 15 0.18 0.16 0.16 0.20 0.13 0.11 0.20 0.18 0.13 0.11 0.19 0.26 0.28 0.18 0.50 0.23 0.24 0.14 0.28 0.31 0.15 Item 16 0.09 0.07 0.11 0.13 0.06 0.10 0.12 0.09 0.14 0.04 0.10 0.14 0.17 0.11 0.23 0.57 0.17 0.11 0.17 0.23 0.07 Item 17 0.15 0.13 0.15 0.19 0.14 0.14 0.26 0.19 0.20 0.12 0.16 0.24 0.27 0.23 0.24 0.17 0.51 0.18 0.30 0.23 0.14 Item 18 0.15 0.09 0.12 0.14 0.14 0.14 0.17 0.14 0.13 0.12 0.15 0.14 0.16 0.15 0.14 0.11 0.18 0.46 0.19 0.18 0.16 Item 19 0.21 0.18 0.19 0.25 0.20 0.20 0.30 0.25 0.15 0.15 0.17 0.27 0.34 0.27 0.28 0.17 0.30 0.19 0.67 0.33 0.18 Item 20 0.23 0.15 0.19 0.21 0.17 0.15 0.23 0.20 0.19 0.13 0.15 0.28 0.30 0.19 0.31 0.23 0.23 0.18 0.33 0.53 0.15 Item 21 0.14 0.10 0.12 0.17 0.15 0.08 0.20 0.15 0.10 0.13 0.12 0.18 0.23 0.15 0.15 0.07 0.14 0.16 0.18 0.15 0.36   68 Table 8: Means and Standard Deviations for Two Versions of the Rosenberg Self-Esteem Scale    Note. n=320 for Likert Version; n=324 for Expanded Version. Scores are ranging from 1 to 4 for all items. Higher values indicate higher self-esteem Likert Version  Expanded Version  M SD  M SD Item 1 3.52 0.62  2.97 0.50 Item 2 3.50 0.64  3.23 0.65 Item 3 3.22 0.81  2.99 0.63 Item 4 3.20 0.65  2.98 0.54 Item 5 2.98 0.87  3.15 0.58 Item 6 3.16 0.72  3.02 0.73 Item 7 3.03 0.76  2.90 0.62 Item 8 2.43 0.83  3.01 0.67 Item 9 2.55 0.90  3.45 0.63 Item 10 2.80 0.94  3.13 0.62 Average 3.04 0.77  3.08 0.62   69 Table 9: Means and Standard Deviations for Two Versions of the Conscientiousness Scale  Likert Version  Expanded Version  M SD  M SD Item 1 3.42 0.73  3.34 0.66 Item 2 2.55 0.87  3.00 0.92 Item 3 3.57 0.64  3.63 0.60 Item 4 2.79 0.93  3.05 0.90 Item 5 2.53 0.90  2.79 0.78 Item 6 3.13 0.78  3.25 0.62 Item 7 3.05 0.82  2.99 0.68 Item 8 3.00 0.82  2.76 0.73 Item 9 2.24 0.82  2.36 0.75 Average 2.92 0.81  3.01 0.74  Note. n=314 for Likert Version; n=307 for Expanded Version. Scores ranging from 1 to 4 for all items. Higher values indicate higher conscientiousness.    70 Table 10: Means and Standard Deviations for All Versions of the Beck Depression Inventory   Likert Version I  Likert Version II  Expanded Version  M SD  M SD  M SD Item 1 1.72 0.84  2.03 0.88  1.53 0.68 Item 2 2.32 0.84  2.01 0.80  1.66 0.63 Item 3 1.57 0.73  2.27 0.92  1.70 0.74 Item 4 1.85 0.85  2.16 0.88  1.51 0.68 Item 5 2.17 0.91  2.67 0.85  1.52 0.69 Item 6 1.94 0.95  1.93 0.87  1.28 0.61 Item 7 1.73 0.84  2.28 0.94  1.65 0.86 Item 8 2.08 0.86  2.76 0.86  1.73 0.77 Item 9 1.49 0.86  1.58 0.91  1.49 0.66 Item 10 1.81 0.87  1.69 0.87  1.20 0.50 Item 11 2.15 0.86  2.15 0.87  1.43 0.77 Item 12 2.04 0.88  1.97 0.88  1.54 0.74 Item 13 2.04 0.72  2.08 0.75  1.67 0.92 Item 14 1.74 0.81  1.71 0.69  1.44 0.75 Item 15 2.11 0.84  2.27 0.87  1.73 0.73 Item 16 1.85 0.93  2.45 0.99  1.78 0.77 Item 17 1.94 0.81  2.10 0.81  1.52 0.73 Item 18 2.16 0.94  2.10 0.90  1.43 0.70 Item 19 2.36 0.89  2.31 0.82  1.80 0.82 Item 20 2.08 0.87  2.36 0.88  1.81 0.72 Item 21 1.71 0.74  2.03 0.88  1.27 0.61 Average 1.95 0.85  2.14 0.86  1.56 0.72  Note. n=254 for Likert Version I; n=253 for Likert Version II; n=256 for Expanded Version. Scores ranging from 1 to 4 for all items. Higher values indicate higher depression.    71 Table 11: Item Endorsement Proportion for the Likert Rosenberg Self-Esteem Scale  Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 Strongly Agree 0.58 0.57 0.42 0.33 0.32 0.33 0.28 0.11 0.17 0.28 Somewhat Agree 0.37 0.38 0.42 0.56 0.41 0.51 0.52 0.33 0.31 0.33 Somewhat Disagree 0.04 0.04 0.13 0.10 0.23 0.14 0.18 0.45 0.41 0.31 Strongly Disagree 0.01 0.01 0.04 0.01 0.05 0.02 0.03 0.11 0.11 0.08   72  Table 12: Item Endorsement Proportion for the Expanded Rosenberg Self-Esteem Scale  Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 1st Response Option  0.09 0.33 0.18 0.13 0.26 0.26 0.14 0.22 0.52 0.25 2nd Response Option  0.82 0.60 0.65 0.72 0.65 0.52 0.64 0.58 0.42 0.63 3rd Response Option  0.07 0.04 0.16 0.14 0.10 0.20 0.21 0.19 0.06 0.10 4th Response Option 0.02 0.03 0.02 0.01 0.00 0.02 0.01 0.01 0.01 0.01  Note. The response options ranged from indication of highest self-esteem to indication of lowest self-esteem.   73 Table 13: Item Endorsement Proportion for the Likert Conscientiousness Scale   Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Disagree strongly 0.02 0.08 0.01 0.07 0.11 0.03 0.04 0.03 0.17 Disagree a little  0.10 0.46 0.05 0.34 0.42 0.16 0.19 0.25 0.50 Agree a little  0.34 0.29 0.30 0.31 0.30 0.47 0.45 0.42 0.25 Agree strongly  0.55 0.17 0.64 0.27 0.17 0.34 0.32 0.31 0.08   74 Table 14: Item Endorsement Proportion for the Expanded Conscientiousness Scale  Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 1st Response Option  0.00 0.05 0.01 0.07 0.05 0.01 0.02 0.04 0.11 2nd Response Option  0.09 0.26 0.03 0.18 0.33 0.07 0.17 0.28 0.49 3rd Response Option  0.46 0.32 0.28 0.39 0.47 0.58 0.60 0.54 0.35 4th Response Option 0.44 0.37 0.68 0.37 0.15 0.34 0.21 0.13 0.06  Note.  The response options ranged from indication of lowest conscientiousness to indication of highest conscientiousness    75 Table 15: Item Endorsement Proportion for the Likert Version I Beck Depression Inventory   I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 I11 I12 I13 I14 I15 I16 I17 I18 I19 I20 I21 Strongly Agree 0.55 0.42 0.45 0.58 0.57 0.78 0.57 0.44 0.58 0.83 0.70 0.58 0.56 0.70 0.41 0.40 0.60 0.68 0.42 0.33 0.80                       Somewhat Agree 0.38 0.52 0.42 0.33 0.36 0.18 0.27 0.42 0.37 0.14 0.23 0.32 0.28 0.19 0.48 0.44 0.31 0.23 0.42 0.57 0.14                       Somewhat Disagree 0.04 0.05 0.11 0.07 0.05 0.02 0.12 0.11 0.03 0.02 0.02 0.07 0.08 0.10 0.08 0.13 0.07 0.08 0.13 0.05 0.04                       Strongly Disagree 0.02 0.01 0.02 0.01 0.02 0.02 0.05 0.03 0.02 0.01 0.05 0.03 0.08 0.02 0.03 0.03 0.02 0.01 0.04 0.04 0.02  Note. “I1” =Item 1.                                                                            76 Table 16: Item Endorsement Proportion for the Likert Version II Beck Depression Inventory  I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 I11 I12 I13 I14 I15 I16 I17 I18 I19 I20 I21 Strongly Agree 0.33 0.25 0.23 0.23 0.08 0.39 0.27 0.08 0.66 0.54 0.24 0.37 0.19 0.41 0.15 0.16 0.22 0.30 0.14 0.15 0.28                       Somewhat Agree 0.39 0.50 0.39 0.42 0.32 0.42 0.33 0.23 0.19 0.27 0.44 0.37 0.58 0.49 0.43 0.39 0.47 0.38 0.47 0.46 0.47                       Somewhat Disagree 0.23 0.22 0.32 0.28 0.46 0.15 0.34 0.51 0.08 0.15 0.23 0.21 0.20 0.10 0.33 0.26 0.27 0.29 0.31 0.25 0.19                       Strongly Disagree 0.05 0.03 0.06 0.06 0.15 0.04 0.06 0.17 0.06 0.04 0.09 0.05 0.03 0.00 0.09 0.18 0.04 0.03 0.08 0.14 0.06  Note. “I1” =Item 1.                          77 Table 17: Item Endorsement Proportion for the Expanded Beck Depression Inventory (a.k.a BDI-II)  I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 I11 I12 I13 I14 I15 I16 I17 I18 I19 I20 I21 1st Response Option  0.55 0.42 0.45 0.58 0.57 0.78 0.57 0.44 0.58 0.83 0.70 0.58 0.56 0.70 0.41 0.40 0.60 0.68 0.42 0.33 0.80                       2nd Response Option  0.38 0.52 0.42 0.33 0.36 0.18 0.27 0.42 0.37 0.14 0.23 0.32 0.28 0.19 0.48 0.44 0.31 0.23 0.42 0.57 0.14                       3rd Response Option  0.04 0.05 0.11 0.07 0.05 0.02 0.12 0.11 0.03 0.02 0.02 0.07 0.08 0.10 0.08 0.13 0.07 0.08 0.13 0.05 0.04                       4th Response Option 0.02 0.01 0.02 0.01 0.02 0.02 0.05 0.03 0.02 0.01 0.05 0.03 0.08 0.02 0.03 0.03 0.02 0.01 0.04 0.04 0.02  Note. The response options ranged from indication of absence of depression to indication of severe depression.   78 Table 18: Two-Factor Results for Exploratory Factor Analysis (EFA) for the Rosenberg Self-Esteem Scale   Likert Version Expanded Version  Factor 1 Factor 2 Factor 1 Factor 2 Item 1 (PW) 0.90 0.02 0.08 0.71 Item 2 (PW) 0.88 -0.05 0.22 0.69 Item 4 (PW) 0.82 -0.11 -0.10 0.76 Item 6 (PW) 0.73 0.12 0.87 -0.13 Item 7 (PW) 0.71 0.15 0.89 -0.03 Item 3 (RW) 0.39 0.40 0.65 0.19  Item 5 (RW) 0.35 0.34 0.62 0.16 Item 8 (RW) -0.09 0.89 0.75 0.04 Item 9 (RW) 0.17 0.50 0.70 0.03 Item 10 (RW) 0.11 0.81 0.59 0.26 Factor correlation  0.63 0.72  Note. For each item, the highest loading was shaded. PW=positively worded item; RW=reverse worded item.     79 Table 19: Two-Factor Results for EFA for the Conscientiousness Scale   Likert Version Expanded Version  Factor 1 Factor 2 Factor 1 Factor 2 Item 1 (PW) 0.82 0.05 0.06 0.76 Item 3 (PW) 0.90 -0.06 -0.08 0.83 Item 6 (PW) 0.74 -0.06 0.39 0.24 Item 7 (PW) 0.62 0.14 0.66 -0.01 Item 8 (PW) 0.46 0.30 0.62 0.01 Item 2 (RW) 0.04 0.69 0.07 0.60  Item 4 (RW) 0.00 0.82 0.41 0.26 Item 5 (RW) 0.07 0.62 0.57 0.23 Item 9 (RW) -0.10 0.63 0.67 -0.06 Factor correlation  0.49 0.77  Note. For each item, the highest loading was shaded. PW=positively worded item; RW=reverse worded item.     80 Table 20:  Four-Factor Results for EFA for the Likert Version I Beck Depression Inventory    Factor 1 (F1) Factor 2 (F2) Factor 3 (F3) Factor 4 (F4) Item 1 (PW): Sadness 0.65 0.04 0.03 0.27 Item 3 (PW): Past failure 0.86 0.02 -0.02 0.06 Item 4 (PW): Loss of pleasure 0.47 0.42 0.11 -0.07 Item 7 (PW): Self-dislike 0.83 0.07 0.10 -0.09 Item 8 (PW): Self-criticalness 0.73 -0.02 -0.10 -0.10 Item 12 (PW): Loss of interest 0.21 0.57 -0.03 0.12 Item 15 (PW): Loss of energy -0.03 0.91 -0.04 0.02 Item 16 (PW): Changes in sleeping pattern 0.19 0.19 -0.11 0.12 Item 17 (PW): Irritability 0.44 0.20 -0.03 0.14 Item 20 (PW): Tiredness or fatigue 0.01 0.81 0.04 -0.05 Item 21 (PW): Loss of interest in sex 0.17 0.29 -0.02 0.01 Item 2 (RW): Pessimism  0.24 0.07 0.21 0.16 Item 5 (RW): Guilty feelings 0.07 0.03 0.76 -0.05 Item 6 (RW): Punishment feelings 0.02 -0.05 0.82 -0.03 Item 9 (RW): Suicidal thoughts or wishes 0.33 -0.05 0.21 0.17 Item 10 (RW): Crying 0.12 -0.08 -0.03 0.68   81  Factor 1 (F1) Factor 2 (F2) Factor 3 (F3) Factor 4 (F4) Item 11 (RW): Agitation 0.05 0.13 -0.08 0.65 Item 13 (RW): Indecisiveness -0.09 0.22 0.38 0.40 Item 14 (RW): Worthlessness 0.56 -0.07 0.28 0.23 Item 18 (RW): Changes in appetite -0.17 0.11 0.31 0.37 Item 19 (RW): Concentration difficulty 0.12 0.31 0.22 0.24 Factor correlation matrix   F1 F2 F3 F4 F1 1.00    F2 0.48   1.00   F3 0.42 0.25 1.00  F4 0.40 0.35 0.31 1.00   Note. For each item, the highest loading was shaded.    82 Table 21: Four-Factor Results for EFA for the Likert Version II Beck Depression Inventory    Factor 1 (F1) Factor 2 (F2) Factor 3 (F3) Factor 4 (F4) Item 1 (PW): Sadness 0.26 0.21 0.15 0.50 Item 3 (PW): Past failure 0.01 0.69 0.03 0.04 Item 4 (PW): Loss of pleasure 0.18 -0.06 0.74 -0.10 Item 7 (PW): Self-dislike 0.10 0.86 -0.03 0.05 Item 8 (PW): Self-criticalness -0.04 0.72 0.08 -0.09 Item 10 (PW):  Crying -0.09 0.10 0.19 0.53 Item 12 (PW): Loss of interest 0.13 0.13 0.49 0.15 Item 16 (PW): Changes in sleeping pattern -0.15 0.11 0.66 0.00 Item 17 (PW): Irritability -0.06 0.13 0.58 0.32 Item 20 (PW): Tiredness or fatigue 0.16 0.14 0.46 0.08 Item 21 (PW): Loss of interest in sex 0.32 0.05 0.28 -0.18 Item 2 (RW): Pessimism  0.64 0.13 0.01 -0.05 Item 5 (RW): Guilty feelings 0.04 0.37 -0.09 0.09 Item 6 (RW): Punishment feelings 0.31 0.15 0.14 0.04 Item 9 (RW): Suicidal thoughts or wishes 0.44 -0.17 -0.01 0.36 Item 11 (RW): Agitation 0.34 0.07 -0.16 0.36   83  Factor 1 (F1) Factor 2 (F2) Factor 3 (F3) Factor 4 (F4) Item 13 (RW): Indecisiveness 0.77 -0.08 0.12 0.04 Item 14 (RW): Worthlessness 0.83 0.11 -0.02 0.03 Item 15 (RW): Loss of energy 0.68 0.15 0.06 -0.02 Item 18 (RW): Changes in appetite 0.16 -0.11 0.45 0.16 Item 19 (RW): Concentration difficulty 0.32 0.25 0.18 0.08 Factor correlation matrix  F1 F2 F3 F4 F1 1.00    F2 0.56 1.00   F3 0.47 0.48 1.00  F4 0.40 0.36 0.38 1.00   Note. For each item, the highest loading was shaded.    84 Table 22: Four-Factor Results for EFA for the Expanded Version Beck Depression Inventory (a.k.a. BDI-II)   Factor 1 (F1) Factor 2 (F2) Factor 3 (F3) Factor 4 (F4) Item 1: Sadness 0.32 0.24 0.31 -0.09 Item 2: Pessimism  0.32 0.05 0.37 0.06 Item 3: Past failure 0.54 0.06 0.24 -0.18 Item 4: Loss of pleasure 0.41 0.11 0.17 0.43 Item 5: Guilty feelings 0.60 0.07 0.12 -0.26 Item 6: Punishment feelings 0.67 0.10 0.11 -0.26 Item 7: Self-dislike 0.74 -0.02 0.09 0.14 Item 8: Self-criticalness 0.46 0.02 0.28 0.02 Item 9: Suicidal thoughts or wishes  0.26 0.35 0.05 -0.02 Item 10: Crying 0.07 0.02 0.87 0.05 Item 11: Agitation 0.14 0.14 0.44 0.02 Item 12: Loss of interest 0.17 0.33 0.21 0.53 Item 13: Indecisiveness  0.59 0.28 -0.10 0.13 Item 14: Worthlessness 0.87 -0.05 0.03 0.17 Item 15: Loss of energy -0.08 0.77 0.16 0.12 Item 16: Changes in sleeping pattern 0.12 0.71 -0.28 -0.04 Item 17: Irritability 0.23 0.45 0.10 0.11 Item 18: Changes in appetite 0.07 0.32 0.39 -0.10 Item 19: Concentration difficulty  0.31 0.45 0.07 0.02   85  Factor 1 (F1) Factor 2 (F2) Factor 3 (F3) Factor 4 (F5) Item 20: Tiredness or fatigue -0.04 0.89 0.08 -0.01 Item 21: Loss of interest in sex 0.14 0.22 0.39 0.08 Factor correlation   F1 F2 F3 F4 F1 1.00    F2 0.57 1.00   F3 0.71 0.52 1.00  F4 0.26 0.28 0.24 1.00   Note. For each item, the highest loading was shaded.    86 Table 23: Two-Factor Results for EFA for the Likert Version I Beck Depression Inventory    Factor 1 Factor 2 Item 1 (PW): Sadness 0.75 0.09 Item 3 (PW): Past failure  0.82 0.04 Item 4 (PW): Loss of pleasure 0.47 0.40 Item 7 (PW): Self-dislike 0.81 0.03 Item 8 (PW): Self-criticalness 0.56 -0.01 Item 12 (PW): Loss of interest 0.21 0.61 Item 15 (PW): Loss of energy -0.05 0.89 Item 16 (PW): Changes in sleeping pattern 0.14 0.24 Item 17 (PW): Irritability  0.48 0.20 Item 20 (PW): Tiredness or fatigue -0.01 0.80 Item 21 (PW): Loss of interest in sex 0.14 0.31 Item 2 (RW): Pessimism  0.46 0.06 Item 5 (RW): Guilty feelings 0.57 -0.09 Item 6 (RW): Punishment feelings 0.57 -0.17 Item 9 (RW): Suicidal thoughts or wishes 0.57 -0.08 Item 10 (RW): Crying 0.43 0.02 Item 11 (RW): Agitation  0.32 0.23 Item 13 (RW): Indecisiveness  0.39 0.19 Item 14 (RW): Worthlessness  0.86 -0.10 Item 18 (RW): Changes in appetite  0.25 0.10 Item 19 (RW): Concentration difficulty 0.40 0.29 Factor correlation  0.54  Note. For each item, the highest loading was shaded.   87 Table 24: Two-Factor Results for EFA for the Likert Version II Beck Depression Inventory    Factor 1 Factor 2 Item 1 (PW): Sadness 0.42 0.49 Item 3 (PW): Past failure  0.32 0.33 Item 4 (PW): Loss of pleasure 0.09 0.59 Item 7 (PW): Self-dislike 0.49 0.34 Item 8 (PW): Self-criticalness 0.26 0.33 Item 10 (PW): Crying -0.01 0.53 Item 12 (PW): Loss of interest 0.15 0.61 Item 16 (PW): Changes in sleeping pattern -0.18 0.71 Item 17 (PW): Irritability  -0.05 0.83 Item 20 (PW): Tiredness or fatigue 0.19 0.54 Item 21 (PW): Loss of interest in sex 0.30 0.16 Item 2 (RW): Pessimism  0.73 -0.04 Item 5 (RW): Guilty feelings 0.22 0.13 Item 6 (RW): Punishment feelings 0.36 0.21 Item 9 (RW): Suicidal thoughts or wishes 0.43 0.07 Item 11 (RW): Agitation 0.44 0.05 Item 13 (RW): Indecisiveness 0.75 0.03 Item 14 (RW): Worthlessness  0.93 -0.06 Item 15 (RW): Loss of energy 0.78 0.03 Item 18 (RW): Changes in appetite  0.08 0.48 Item 19 (RW): Concentration difficulty 0.42 0.31 Factor correlation  0.62  Note. For each item, the highest loading was shaded.  88 Table 25: Two-Factor Results for EFA for the Expanded Version Beck Depression Inventory (a.k.a. BDI-II)    Factor 1 Factor 2 Item 1: Sadness 0.58 0.18 Item 2: Pessimism  0.66 0.04 Item 3: Past failure 0.71 -0.02 Item 4: Loss of pleasure 0.67 0.16 Item 5: Guilty feelings 0.67 -0.06 Item 6: Punishment feelings 0.73 -0.05 Item 7: Self-dislike 0.91 -0.09 Item 8: Self-criticalness 0.72 -0.02 Item 9: Suicidal thoughts or wishes  0.32 0.29 Item 10: Crying 0.76 0.09 Item 11: Agitation 0.49 0.17 Item 12: Loss of interest 0.47 0.41 Item 13: Indecisiveness  0.55 0.24 Item 14: Worthlessness 0.94 -0.08 Item 15: Loss of energy 0.03 0.85 Item 16: Changes in sleeping pattern -0.14 0.69 Item 17: Irritability 0.32 0.48 Item 18: Changes in appetite 0.35 0.31 Item 19: Concentration difficulty  0.38 0.43 Item 20: Tiredness or fatigue 0.03 0.86 Item 21: Loss of interest in sex 0.48 0.24 Factor correlation  0.65  Note. For each item, the highest loading was shaded.   89 Table 26Summary of the CFA fit statistics for the Rosenberg Self-Esteem Scale and Conscientiousness Scale   Likert Version  Expanded Version  Rosenberg Self-Esteem  χ2 df CFI RMSEA   χ2 df CFI RMSEA Model 1 296.73 35 0.93 0.15   106.12 35 0.98 0.08  Model 2 141.32 34 0.97 0.10   103.76 34 0.98 0.08  Model 3  95.08 30 0.98 0.08   95.25 30 0.98 0.08  Δχ2 m1-m2 (df=1) (χ2 critical, p-value for test of small difference) 6.13 (p=0.01) (38.72, 0.98)  0.11 (p=0.74)  (39.17, 1.00) Δχ2 m1-m3 (df=5) (χ2 critical, p-value for test of small difference) 167.00 (p=0.00)  (54.12, 0.00)  13.46 (p=0.02)  (54.47, 0.99)  Conscientiousness Scale  χ2 df CFI RMSEA   χ2 df CFI RMSEA Model 1 245.61 27 0.86 0.16   75.32 27 0.97 0.08 Model 2 87.74 23 0.96 0.09  63.81 26 0.98 0.07 Model 3 87.74 23 0.96 0.09  60.08 23 0.98 0.07 Δχ2 m1-m2 (df=1) {χ2 critical, p-value for test of small difference} 11.33 (p=0.00) (32.39, 0.75)  0.64 (p=0.42) (31.87, 1.00) Δχ2 m1-m3 (df=5) {χ2 critical, p-value for test of small difference} 137.89 (p=0.00) (44.13, 0.00)  16.10 (p=0.00) (43.45, 0.84)  Note. CFI= comparative fit index; RMSEA=root mean square error of approximation. Model 1: Single substantive factor; Model 2: Two oblique factors, one among positively worded (PW) item and one among reverse worded (RW) items; Model 3: Two factors, substantive and method effect among RW items. In Expanded version, there is no distinction between PW and RW; the factors were formed according to whether the corresponding item is RW or PW in the Likert version. Δχ2 was calculated according to Satorra (2000)’s suggestion for calculating chi-square difference test using the Satorra-Bentler scaled chi-square; and Δχ2 was used to conduct both chi-square difference and test of small difference.     90 Table 27Summary of the CFI fit statistics for Beck Depression Inventory   Likert Version I  Likert Version II  Expanded Version (BDI-II)  χ2 df CFI RMSEA  χ2 df CFI RMSEA   χ2 df CFI RMSEA  Model 1 (M1) 775.46 189 0.84 0.11   561.57 189 0.91 0.09   412.01 189 0.95 0.07   Model 2 (M2) 672.54 188 0.87 0.10    481.45 188 0.93 0.08    410.84 188 0.95 0.06   Model 3 (M3) 584.42 179 0.89 0.09    462.52 179 0.93 0.08    397.13 179 0.95 0.06   Δχ2 m1-m2 (df=1) (χ2 critical, p-value for test of small difference) 1.48 (p=0.22) (115.57, 1.00)  0.87 (p=0.35) (115.18, 1.00)  0.01 (p=0.92) (116.34, 1.00) Δχ2 m1-m3 (df=10) (χ2 critical, p-value for test of small difference) 182.41 (p=0.00) (142.39, 0.00)  112.03 (p=0.00) (141.94, 0.39)  0.01 (p=0.91) (143.28, 1.00)  Note. CFI= comparative fit index; RMSEA=root mean square error of approximation. Model 1: Single substantive factor; Model 2: Two oblique factors, one among PW items and one among RW items; Model 3: Two factors, substantive and method effect among RW items. In Expanded version, there is no distinction between PW and RW. For the results of the Expanded version presented in this table, the factors were formed according to whether the corresponding item is RW or PW in the Likert version I9. Δχ2 was calculated according to Satorra (2000)’s suggestion for calculating chi-square difference test using the Satorra-Bentler scaled chi-square; and Δχ2 was used to conduct both chi-square difference and test of small difference.   91 Table 28: Standardized factor loadings for CFA models for the Rosenberg Self-Esteem Scale   Model 1  Likert Version Expanded Version  GSE GSE Item 1 (PW 1) 0.88 0.69 Item 2 (PW 2) 0.80 0.78 Item 4 (PW 3) 0.68 0.57 Item 6 (PW 4) 0.83 0.74 Item 7 (PW 5) 0.82 0.84 Item 3 (RW 1) 0.71 0.81 Item 5 (RW 2) 0.62 0.75 Item 8 (RW 3) 0.69 0.77 Item 9 (RW 4) 0.59 0.72 Item 10 (RW 5) 0.79 0.80 Average 0.74 0.75     Model 2  Likert Version Expanded Version  PSE NSE PSE NSE Item 1 (PW 1) 0.91  0.70  Item 2 (PW 2) 0.83  0.79  Item 4 (PW 3) 0.71  0.57  Item 6 (PW 4) 0.86  0.74  Item 7 (PW 5) 0.85  0.85  Item 3 (RW 1)  0.77  0.81 Item 5 (RW 2)  0.67  0.75 Item 8 (RW 3)  0.73  0.78 Item 9 (RW 4)  0.62  0.72 Item 10 (RW 5)  0.85  0.80 Average  0.83 0.73 0.73 0.77 Factor correlation 0.78 0.97     92  Model 3  Likert Version Expanded Version  GSE RME GSE RME Item 1 (PW 1) 0.90  0.69  Item 2 (PW 2) 0.83  0.79  Item 4 (PW 3) 0.71  0.57  Item 6 (PW 4) 0.86  0.74  Item 7 (PW 5) 0.85  0.84  Item 3 (RW 1) 0.65 0.32 0.79 0.36  Item 5 (RW 2) 0.57 0.28 0.84 0.15 ns Item 8 (RW 3) 0.48 0.70 0.76 0.26 ns Item 9 (RW 4) 0.51 0.33 0.75 -0.22 ns Item 10 (RW 5) 0.63 0.61 0.80 0.05 ns Average  0.70 0.45 0.75 0.21  Note. All factor loadings except the ones with ns are statistically significant at p < 0.01. Average loading sizes were calculated using absolute values of the loading values. GSE=Global self-esteem; PSE=positive self-esteem; NSE=negative self-esteem; PW=positively worded item; RW=reverse worded item; RME=reverse worded item method factor.   93 Table 29: Standardized factor loadings for CFA models for the Conscientiousness Scale  Model 1  Likert Version Expanded Version  GC GC Item 1 (PW) 0.80 0.80 Item 3 (PW) 0.79 0.73 Item 6 (PW) 0.64 0.60 Item 7 (PW) 0.67 0.61 Item 8 (PW) 0.65 0.59 Item 2 (RW) 0.59 0.65 Item 4 (RW) 0.67 0.63 Item 5 (RW) 0.58 0.75 Item 9 (RW) 0.45 0.58 Average 0.65 0.66  Model 2  Likert Version Expanded Version  PC NC PC NC Item 1 (PW) 0.84  0.82  Item 3 (PW) 0.82  0.74  Item 6 (PW) 0.69  0.62  Item 7 (PW) 0.72  0.62  Item 8 (PW) 0.69  0.60  Item 2 (RW)  0.71  0.67 Item 4 (RW)  0.80  0.65 Item 5 (RW)  0.70  0.78 Item 9 (RW)  0.56  0.60 Average  0.75 0.70 0.68 0.68 Factor correlation 0.57 0.89  Model 3  Likert Version Expanded Version  GC RME GC RME Item 1 (PW) 0.84  0.82  Item 3 (PW) 0.82  0.74  Item 6 (PW) 0.69  0.62    94  Model 3  Likert Version Expanded Version  GC RME GC RME Item 7 (PW) 0.72  0.62  Item 8 (PW) 0.69  0.60  Item 2 (RW) 0.41 0.58 0.60 0.30  Item 4 (RW) 0.45 0.67 0.55 0.52 Item 5 (RW) 0.42 0.53 0.71 0.23  Item 9 (RW) 0.28 0.53 0.55 0.22 ns Average  0.59 0.58 0.65 0.32  Note. All factor loadings except the ones with ns are statistically significant at p < 0.01. Average loading sizes were calculated using absolute values of the loading values. GC=Global conscientiousness; PC=positive conscientiousness; NC=negative conscientiousness; PW=positively worded item; RW=reverse worded item; RME=reverse worded item method factor.  95 Table 30: Standardized factor loadings for CFA Model 1 for the Beck Depression Inventory    Likert Version I Likert Version II Expanded Version  GD GD GD Item 1 0.78 0.81 0.69 Item 2 0.47 0.64 0.64 Item 3 0.82 0.61 0.64 Item 4 0.73 0.61 0.78 Item 5 0.53 0.34 0.61 Item 6 0.50 0.53 0.69 Item 7 0.80 0.78 0.80 Item 8 0.51 0.56 0.67 Item 9 0.48 0.45 0.59 Item 10 0.46 0.46 0.79 Item 11 0.51 0.43 0.60 Item 12 0.68 0.68 0.79 Item 13 0.56 0.72 0.75 Item 14 0.76 0.80 0.85 Item 15 0.67 0.75 0.73 Item 16 0.31 0.47 0.45 Item 17 0.61 0.67 0.70 Item 18 0.34 0.50 0.57 Item 19 0.61 0.67 0.72 Item 20 0.65 0.66 0.75 Item 21 0.36 0.44 0.67 Average  0.58 0.60 0.69  Note. All factor loadings except the ones with ns are statistically significant at p < 0.01. GD=Global depression.  96 Table 31: Standardized factor loadings for CFA Model 2 for the Beck Depression Inventory   Model 2: Factors Formed Based on Likert Version I  Likert Version I Expanded Version  PD ND PD ND Item 1 (PW) 0.80  0.69  Item 3 (PW) 0.84  0.64  Item 4 (PW) 0.75  0.78  Item 7 (PW) 0.83  0.81  Item 8 (PW) 0.53  0.67  Item 12 (PW) 0.71  0.79  Item 15 (PW) 0.69  0.73  Item 16 (PW) 0.32  0.45  Item 17 (PW) 0.63  0.71  Item 20 (PW) 0.68  0.75  Item 21 (PW) 0.37  0.67  Item 2 (RW)  0.51  0.66 Item 5 (RW)  0.60  0.62 Item 6 (RW)  0.56  0.69 Item 9 (RW)  0.54  0.60 Item 10 (RW)  0.50  0.80 Item 11 (RW)  0.55  0.61 Item 13 (RW)  0.62  0.76 Item 14 (RW)  0.86  0.85 Item 18 (RW)  0.39  0.58 Item 19 (RW)  0.67  0.72 Average  0.65 0.58 0.70 0.69 Factor correlation 0.76 0.98  Model 2: Factors Formed Based on Likert Version II  Likert Version II Expanded Version  GD RD GD RD Item 1 (PW) 0.83  0.69  Item 3 (PW) 0.63  0.64  Item 4 (PW) 0.63  0.78    97  Model 2: Factors Formed Based on Likert Version II  Likert Version II Expanded Version  GD RD GD RD Item 7 (PW) 0.82  0.80  Item 8 (PW) 0.58  0.66  Item 10 (PW) 0.48  0.79  Item 12 (PW) 0.71  0.78  Item 16 (PW) 0.50  0.44  Item 17 (PW) 0.70  0.70  Item 20 (PW) 0.69  0.75  Item 21 (PW) 0.46  0.66  Item 2 (RW)  0.67  0.65  Item 5 (RW)  0.36  0.61 Item 6 (RW)  0.55  0.68 Item 9 (RW)  0.48  0.59 Item 11 (RW)  0.46  0.60 Item 13 (RW)  0.76  0.75 Item 14 (RW)  0.84  0.84 Item 15 (RW)  0.79  0.72 Item 18 (RW)  0.53  0.57 Item 19 (RW)  0.70  0.72 Average  0.64 0.61 0.70 0.67 Factor correlation 0.83 1.00  Note. All factor loadings except the ones with ns are statistically significant at p < 0.01. Average loading sizes were calculated using absolute values of the loading values. PD=positive depression; ND=negative depression; PW=positively worded item; RW=reverse worded item; RME=reverse worded item method factor.  98 Table 32: Standardized factor loadings for CFA Model 3 for the Beck Depression Inventory   Model 3: Factors Formed Based on Likert Version I  Likert Version I Expanded Version  PD ND PD ND Item 1 (PW) 0.80  0.69  Item 3 (PW) 0.84  0.64  Item 4 (PW) 0.75  0.78  Item 7 (PW) 0.83  0.81  Item 8 (PW) 0.53  0.67  Item 12 (PW) 0.70  0.79  Item 15 (PW) 0.69  0.73  Item 16 (PW) 0.33  0.45  Item 17 (PW) 0.63  0.71  Item 20 (PW) 0.67  0.75  Item 21 (PW) 0.37  0.67  Item 2 (RW) 0.43 0.21 0.65 0.07 ns Item 5 (RW) 0.34 0.70 0.58 0.34 ns Item 6 (RW) 0.29 0.72 0.63 0.78  Item 9 (RW) 0.45 0.23 0.59 0.04 ns Item 10 (RW) 0.42 0.21 0.79 0.09 ns Item 11 (RW) 0.48 0.17 0.61 -0.06 ns Item 13 (RW) 0.46 0.47 0.75 0.04 ns Item 14 (RW) 0.71 0.32 0.84 0.13 ns Item 18 (RW) 0.26 0.37 0.56 0.14 ns Item 19 (RW) 0.56 0.29 0.72 0.06 ns Average  0.55 0.37 0.69 0.16  Model 3:Factors Formed Based on Likert Version II Likert Version II Expanded Version  GD RD GD RD Item 1 (PW) 0.83  0.69  Item 3 (PW) 0.63  0.64  Item 4 (PW) 0.63  0.78  Item 7 (PW) 0.81  0.80    99  Model 3:Factors Formed Based on Likert Version II  Likert Version II Expanded Version  GD RD GD RD Item 8 (PW) 0.58  0.67  Item 10 (PW) 0.47  0.79  Item 12 (PW) 0.71  0.79  Item 16 (PW) 0.50  0.45  Item 17 (PW) 0.70  0.70  Item 20 (PW) 0.68  0.75  Item 21 (PW) 0.45  0.67  Item 2 (RW) 0.52 0.47 0.65 0.06 ns Item 5 (RW) 0.34 0.07 0.60 0.40 Item 6 (RW) 0.49 0.20 0.67 0.46 Item 9 (RW) 0.39 0.30 0.59 0.08 ns Item 11 (RW) 0.36 0.31 0.79 -0.05 ns Item 13 (RW) 0.60 0.53 0.75 0.06 ns Item 14 (RW) 0.66 0.60 0.84 0.24  Item 15 (RW) 0.65 0.45 0.76 -0.42  Item 18 (RW) 0.51 0.03 0.57 0.07 ns Item 19 (RW) 0.64 0.21 0.72 -0.03 ns Average  0.58 0.32 0.69 0.19  Note. All factor loadings except the ones with ns are statistically significant at p < 0.01. Average loading sizes were calculated using absolute values of the loading values. GD=global depression; PW=positively worded item; RW=reverse worded item; RME=reverse worded item method factor.    100 Table 33: Model-Based Reliabilities for All Versions of the Three Scales  RSES  CS  BDI  Likert version  Expanded version  Likert version  Expanded version  Likert version I Likert version II Expanded version Model-based reliability based on Model 1 0.93   0.93   0.88 0.87  0.92 0.92 0.95           Model-based reliability based on Model 3 0.84 0.92  0.75 0.84  0.84 0.87 0.94 Note. RSES=Rosenberg Self-Esteem Scale; CS=Conscientiousness Scale; BDI=Beck Depression Inventory. The model-based reliability was calculated based on Model 1 in Figure 5 in Appendix E.  Model-based reliabilities based on Model 3 were calculated by partialling out the variance due to the method factor as error.   101 Table 34: Correlation Matrices for the Rosenberg Self-Esteem Scale and the Conscientiousness Scale and for the Rosenberg Self-Esteem Scale and the Beck Depression Inventory    CS   Likert Version  (Original scale) Expanded Version RSES Likert version  (Original scale) 0.33 (n=160) 0.40 (n=152) Expanded version 0.32 (n=154) 0.32 (n=155)   RSES   Likert Version (Original Scale) Expanded Version BDI Likert version I -0.70 (n=74) -0.55 (n=81) Likert version II -0.82 (n=67) -0.60 (n=87) Expanded version (Original Scale) -0.60 (n=88) -0.76 (n=66)   Note. All conditions were between-subject. Sample size for each condition is shown in parentheses. All correlations are significant at .0001 level. RSES=Rosenberg Self-Esteem Scale; CS=Conscientiousness Scale; BDI=Beck Depression Inventory   

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.24.1-0166371/manifest

Comment

Related Items