UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Inferential procedures for multifaceted coefficients of generalizability Schroeder, Marsha Lynn 1986

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-UBC_1986_A1 S37.pdf [ 4.76MB ]
Metadata
JSON: 831-1.0097301.json
JSON-LD: 831-1.0097301-ld.json
RDF/XML (Pretty): 831-1.0097301-rdf.xml
RDF/JSON: 831-1.0097301-rdf.json
Turtle: 831-1.0097301-turtle.txt
N-Triples: 831-1.0097301-rdf-ntriples.txt
Original Record: 831-1.0097301-source.json
Full Text
831-1.0097301-fulltext.txt
Citation
831-1.0097301.ris

Full Text

INFERENTIAL PROCEDURES FOR MULTIFACETED COEFFICIENTS OF GENERALIZABILITY by MARSHA LYNN SCHROEDER A., The U n i v e r s i t y of B r i t i s h Columbia, 1982 THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE STUDIES Department of Psychology We accept t h i s t h e s i s as conforming to the r e q u i r e d standard THE UNIVERSITY OF BRITISH COLUMBIA September 1986 © Marsha Lynn Schroeder, 1986 In p r e s e n t i n g t h i s t h e s i s i n p a r t i a l f u l f i l m e n t o f the requirements f o r an advanced degree a t the U n i v e r s i t y o f B r i t i s h Columbia, I agree t h a t the L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r r e f e r e n c e and study. I f u r t h e r agree t h a t p e r m i s s i o n f o r e x t e n s i v e copying o f t h i s t h e s i s f o r s c h o l a r l y purposes may be granted by the head o f my department o r by h i s or her r e p r e s e n t a t i v e s . I t i s understood t h a t copying or p u b l i c a t i o n o f t h i s t h e s i s f o r f i n a n c i a l g a i n s h a l l not be allowed without my w r i t t e n p e r m i s s i o n . Department of PSYCHOLOGY  The U n i v e r s i t y o f B r i t i s h Columbia 1956 Main Mall Vancouver, Canada V6T 1Y3 Date S E P T « 2 9> 1 9 8 6 DE-6 C3/81) INFERENTIAL PROCEDURES FOR MULTIFACETED COEFFICIENTS OF GENERALIZABILITY ABSTRACT G e n e r a l i z a b i l i t y theory was developed by Cronbach as an a l t e r n a t i v e to c l a s s i c a l t e s t score r e l i a b i l i t y theory. G e n e r a l i z a b i l i t y uses an experimental design approach to r e l i a b i l i t y that permits the systematic e v a l u a t i o n of s e v e r a l sources of e r r o r simultaneously. The c o e f f i c i e n t of g e n e r a l i z a b i l i t y (CG) i s a s i n g l e number summarizing the d e p e n d a b i l i t y of the measurement process. In the present study a normalizing transformation was f i r s t a p p l i e d to a f u n c t i o n of the CG. The d e l t a method was a p p l i e d to the transformed CGs f o r four d i f f e r e n t two-facet experimental design models to develop asymptotic variance expressions f o r the CGs. The accuracy of the variance expressions was t e s t e d v i a Monte Carlo simulations. In .these simulations the Type I e r r o r c o n t r o l was i n v e s t i g a t e d . The maj o r i t y of the simulations were conducted using a two-facet f u l l y random experimental design, corresponding to a three-way random e f f e c t s a n a l y s i s of variance. A t o t a l of 81 combinations of sample s i z e , f a c e t c o n d i t i o n s , and population CG values were i n v e s t i g a t e d . The r e s u l t s suggested that the procedure g e n e r a l l y was p r e c i s e i n i t s c o n t r o l of Type I e r r o r . The r e s u l t s w ere so^twhat l e s s p r e c i s e when only two facet c o n d i t i o n s were sampled. Fi v e other side studies were conducted. Three of these used other two-facet models: a design with one f i x e d f a c e t , a design with a f i n i t e f a c e t , and a design with a nested f a c e t . The r e s u l t s of these studies were s i m i l a r to those found i n the l a r g e r study; g e n e r a l l y good Type I e r r o r c o n t r o l was r e a l i z e d . An a d d i t i o n a l study looked at the performance of the variance expression i n the presence of negative variance component estimates. Results i n t h i s s e c t i o n of the study suggested that such negative component estimates d i d not adversely a f f e c t Type I e r r o r c o n t r o l . The f i n a l study i n v e s t i g a t e d the performance of the variance expression with dichotomous data. The r e s u l t s i n d i c a t e d that Type I e r r o r c o n t r o l was not as p r e c i s e with two facet c o n d i t i o n s as i t was with f i v e or eight c o n d i t i o n s . In these l a t t e r cases good e r r o r c o n t r o l was r e a l i z e d . i v Table of Contents Page Ab s t r a c t i i L i s t of Tables v i L i s t of F i g u r e s ix Acknowledgements x Chapter 1 - I n t r o d u c t i o n 1 The Object of Measurement 3 Universes 3 Components of Variance 4 G and D Studi e s 6 Two Kinds of D e c i s i o n s ; Two Kinds of E r r o r Variance 7 The C o e f f i c i e n t of G e n e r a l i z a b i l i t y 9 E s t i m a t i n g the C G 12 G e n e r a l i z a b i l i t y Research and A p p l i c a t i o n 13 Purpose of the Present Study 1 8 Chapter 2 - Mathematical Development 20 The Approximate D i s t r i b u t i o n of the CG 20 The N o r m a l i z a t i o n of a Function of the CG 25 The D e r i v a t i o n of an Asymptotic Variance E x p r e s s i o n f o r the CG 26 Variance E x p r e s s i o n s f o r CGs A r i s i n g from Other Experimental Designs 29 Chapter 3 - Method 36 Data Generation 36 Overview of the Study f o r Design VII with Both Facets Random 38 Three Other Models 39 A S p e c i a l C o n d i t i o n : Zero Variance Components 41 A S p e c i a l C o n d i t i o n : Dichotomous Data 43 T e s t i n g the Adequacy of the Variance E x p r e s s i o n 44 Chapter 4 - R e s u l t s 49 Design VII with Both Facets Random 49 Design VII with F i x e d Item Facet 57 Design VII with F i n i t e Random Rater Facet 57 Design V-B with Both Facets Random 60 The Treatment of Negative Variance Components 61 Design VII with Dichotomous Data 62 Chapter 5 - Conclus i o n s and a Worked Example of the Procedures 67 Some Observations about the E m p i r i c a l R e s u l t s 67 I m p l i c a t i o n s of the Study for the Use of G e n e r a l i z a b i l i t y Theory 70 L i m i t a t i o n s of the Present Study 72 Suggestions f o r Future Research 73 A Worked Example Using the Present Procedure 75 References 77 Appendix A 84 L i s t of Tables v i Page Table 1 Expected Mean Squares f o r Design VII with both Facets Random 22 Table 2 A n a l y s i s of Variance R e s u l t s and Estimated Variance Components f o r Psychopathy Data 23 Table 3 Summary of Designs and C o n d i t i o n s for the E m p i r i c a l I n v e s t i g a t i o n 45 Table 4 P r o p o r t i o n of the Standardized Estimates F a l l i n g below S e l e c t e d P e r c e n t i l e Points of the Unit Normal D i s t r i b u t i o n and A c t u a l P r o p o r t i o n of Type I E r r o r s f o r p 2 = .50 (Design V I I , both Facets Random) 51 Table 5 P r o p o r t i o n of the Standardized Estimates F a l l i n g below S e l e c t e d P e r c e n t i l e Points of the Unit Normal D i s t r i b u t i o n and Actu a l P r o p o r t i o n of Type I E r r o r s f o r p 2 = .70 (Design V I I , both Facets Random) 52 Table 6 P r o p o r t i o n of the Standardized Estimates F a l l i n g below S e l e c t e d P e r c e n t i l e Points of the Unit Normal D i s t r i b u t i o n and A c t u a l P r o p o r t i o n of Type I E r r o r s f o r p 2 = .90 (Design V I I , both Facets Random) 53 V 1 1 Table 7 Mean Values and O v e r a l l Chi-square Values by Parameter Values and Facet C o n d i t i o n s (Design V I I , both Facets Random) f o r S e l e c t e d P e r c e n t i l e P o i n t s of the Unit Normal D i s t r i b u t i o n and Three L e v e l s of Type I E r r o r 54 Table 8 P r o p o r t i o n of the Standardized Estimates F a l l i n g below S e l e c t e d P e r c e n t i l e P o i n t s of the Unit Normal D i s t r i b u t i o n and A c t u a l P r o p o r t i o n of Type I E r r o r s (Design V I I, Item Facet Fixed) 58 Table 9 P r o p o r t i o n of the Standardized Estimates F a l l i n g below S e l e c t e d P e r c e n t i l e P o i n t s of the Unit Normal D i s t r i b u t i o n and A c t u a l P r o p o r t i o n of Type I E r r o r s (Design V I I , Rater Facet F i n i t e ) 59 Table 10 P r o p o r t i o n of the Standardized Estimates F a l l i n g below S e l e c t e d P e r c e n t i l e P o i n t s of the Unit Normal D i s t r i b u t i o n and A c t u a l P r o p o r t i o n of Type I E r r o r s (Design V-B, both Facets Random) 61 v i i i Table 11 P r o p o r t i o n of the Standardized Estimates F a l l i n g below S e l e c t e d P e r c e n t i l e P o i n t s of the Unit Normal D i s t r i b u t i o n and A c t u a l P r o p o r t i o n of Type I E r r o r s f o r Two Treatments of Negative Variance Component Estimates (Design V I I , Both Facets Random) 63 Table 12 P r o p o r t i o n of the Standardized Estimates F a l l i n g below S e l e c t e d P e r c e n t i l e P o i n t s of the Unit Normal D i s t r i b u t i o n and A c t u a l P r o p o r t i o n of Type I E r r o r s f o r Dichotomized Data and Comparable C o n d i t i o n s with Continuous Data (Design V I I , both Facets Random) 66 L i s t o f F i g u r e s P a g e F i g u r e 1 L a y o u t o f D a t a f o r D e s i g n V I I A n a l y s i s o f V a r i a n c e 37 \ Acknowledgements x I would l i k e to thank my t h e s i s committee f o r t h e i r support and encouragement throughout the course of t h i s p r o j e c t . I would l i k e to thank Demetri Papageorgis f o r consenting to serve on my committee. I a l s o wish to express my a p p r e c i a t i o n to Jim S t e i g e r f o r h i s many h e l p f u l comments and suggestions. I am a l s o indebted to Todd Rogers for i n t r o d u c i n g me to G e n e r a l i z a b i l i t y theory; I a l s o wish to thank Todd for h i s c a r e f u l reading of and t h o u g h t f u l comments on previous d r a f t s of t h i s t h e s i s . I e s p e c i a l l y would l i k e to thank Ralph Hakstian f o r c h a i r i n g my committee. His p a t i e n t a s s i s t a n c e and encouragement were inst r u m e n t a l to the completion of t h i s t h e s i s . F i n a l l y , I would l i k e to thank Klaus Schroeder f o r h i s a s s i s t a n c e with text p r e p a r a t i o n ; I would a l s o l i k e to thank Klaus for h i s c o n s i d e r a b l e support throughout t h i s p r o j e c t . 1 Chapter 1 I n t r o d u c t i o n G e n e r a l i z a b i l i t y theory was proposed by Cronbach (Cronbach, G l e s e r , Nanda, & Rajaratnam, 1972; Cronbach, Rajaratnam, & G l e s e r , 1963; G l e s e r , Cronbach, & Rajaratnam, 1965; Rajaratnam, Cronbach, & G l e s e r , 1965) as a l i b e r a l i z a t i o n of c l a s s i c a l t e s t score r e l i a b i l i t y theory. C l a s s i c a l t e s t score theory i s based on the assumption of p a r a l l e l measurements; g e n e r a l i z a b i l i t y theory does not r e l y on t h i s r e s t r i c t i v e assumption. Measurement e r r o r w i t h i n the c l a s s i c a l model i s regarded as an amorphous q u a n t i t y ; in g e n e r a l i z a b i l i t y theory measurement e r r o r i s examined s y s t e m a t i c a l l y . The view of measurement e r r o r as an amorphous q u a n t i t y has been questioned by a number of w r i t e r s (e.g., Cronbach, 1947, 1970; G u l l i k s e n , 1936). E r r o r s a r i s i n g from d i f f e r e n t sources may not be e q u i v a l e n t ; f o r example, e r r o r a r i s i n g from repeated t e s t i n g with the same form of an instrument l i k e l y d i f f e r s from e r r o r a r i s i n g when p a r a l l e l forms of the instrument are used. G e n e r a l i z a b i l i t y theory permits the simultaneous e v a l u a t i o n of s e v e r a l sources of e r r o r by using an experimental design approach to measurement. The next s e c t i o n s of t h i s t h e s i s w i l l i n t r o d u c e g e n e r a l i z a b i l i t y theory in some d e t a i l . F o l l o w i n g t h i s p r e s e n t a t i o n , i n f e r e n t i a l procedures p e r t a i n i n g to g e n e r a l i z a b i l i t y theory w i l l be developed. 2 To introduce the n o t i o n of an experimental design approach to measurement, consider the f o l l o w i n g example taken from a study by Schroeder, Schroeder, and Hare (1983). In t h i s study the g e n e r a l i z a b i l i t y ( r e l i a b i l i t y ) of a c h e c k l i s t used to measure psychopathy i n p r i s o n inmates was assessed. The c h e c k l i s t i s composed of 22 items tapping aspects of psychopathy. I t i s t y p i c a l l y used by two or more t r a i n e d r a t e r s who judge the a p p l i c a b i l i t y of each item to a p a r t i c u l a r inmate. In each of f i v e years a number of inmates were evaluated on the 22 items by two r a t e r s . In what f o l l o w s we w i l l be concerned with the data f o r a s i n g l e year. If r e l i a b i l i t y were assessed using the c l a s s i c a l t e s t score model, a number of d i f f e r e n t indexes of r e l i a b i l i t y would be c a l c u l a t e d . For example an index of i n t e r n a l c o n s i s t e n c y f o r each r a t e r , a c o e f f i c i e n t of i n t e r r a t e r agreement, the s i g n i f i c a n c e of d i f f e r e n c e s between r a t e r s on the means and v a r i a n c e s f o r t o t a l t e s t scores ( i . e . , the sum of the r a t i n g s given over the 22 items), and i n d i v i d u a l item s t a t i s t i c s c o u l d be e v a l u a t e d . Using g e n e r a l i z a b i l i t y theory, we would c o n c e p t u a l i z e these data as a three-way random e f f e c t s a n a l y s i s of v a r i a n c e (ANOVA) and conduct the corresponding Persons ( i . e . , inmates) X Items X Raters f u l l y - c r o s s e d ANOVA. The r e s u l t i n g mean squares are used to c a l c u l a t e the seven separate v a r i a n c e component estimates. These v a r i a n c e estimates are the key elements of g e n e r a l i z a b i l i t y theory; they guide the researcher i n the design of sound data c o l l e c t i o n procedures. They are a l s o 3 used to compute the value of the c o e f f i c i e n t of g e n e r a l i z a b i l i t y (CG), a s i n g l e number summarizing the r e l i a b i l i t y or d e p e n d a b i l i t y of the measurement p r o c e s s . At t h i s p o i n t some g e n e r a l i z a b i l i t y theory concepts and terminology are presented to he l p c l a r i f y the m a t e r i a l that f o l l o w s . The Object of Measurement The o b j e c t of measurement i n g e n e r a l i z a b i l i t y theory i s the element of the study about which one wishes t o make judgments. In the Schroeder et a l . (1983) study the o b j e c t s of measurement were inmates. Throughout t h i s t h e s i s the ob j e c t s of measurement w i l l be r e f e r r e d to as persons. The ob j e c t s of measurement i n a g e n e r a l i z a b i l i t y study, however, c o u l d be any p o p u l a t i o n of organisms or o b j e c t s , t o be evalu a t e d . For example, the o b j e c t s of measurement c o u l d be p a i n t i n g s or animals. Universes The emphasis i n both c l a s s i c a l t e s t s c o r e theory and g e n e r a l i z a b i l i t y theory i s p l a c e d on g e n e r a l i z i n g beyond the c o n d i t i o n s used i n the study to a l a r g e r s et of c o n d i t i o n s . Any o b s e r v a t i o n i s made under a set of c o n d i t i o n s . Facets are composed of conditions (the term f a c e t i s analogous to the ANOVA term factor; the term c o n d i t i o n to level). An o b s e r v a t i o n can be c l a s s i f i e d a c c o r d i n g t o the 4 c o n d i t i o n s of the f a c e t s under which i t was made. The universe of admissible observations c o n s i s t s of a l l c o n d i t i o n s of a f a c e t that t h e o r e t i c a l l y c o u l d be in c l u d e d i n a study. The universe of generali zat i on r e p r e s e n t s a l l c o n d i t i o n s of the f a c e t over which one wishes to g e n e r a l i z e . These two terms may be synonymous; however, i n some s i t u a t i o n s the universe of g e n e r a l i z a t i o n i s a subset of the univ e r s e of ad m i s s i b l e o b s e r v a t i o n s . In a g e n e r a l i z a b i l i t y study, f a c e t s over which the research e r would l i k e to g e n e r a l i z e are i d e n t i f i e d . Two or more c o n d i t i o n s of each f a c e t are sampled f o r i n c l u s i o n in the study. Facets can be f i x e d , random, or f i n i t e e f f e c t s . When a f a c e t i s f i x e d , a l l c o n d i t i o n s of the uni v e r s e of ad m i s s i b l e o b s e r v a t i o n s are i n c l u d e d i n the study. When a fa c e t i s random, a number of c o n d i t i o n s are sampled randomly from the t h e o r e t i c a l l y i n f i n i t e u n i v e r s e of a d m i s s i b l e o b s e r v a t i o n s . When a f i n i t e f a c e t i s in c l u d e d i n a study, a number of c o n d i t i o n s are sampled at random from a f i n i t e u n i v e r s e of ad m i s s i b l e o b s e r v a t i o n s . R e l a t e d to the concept of u n i v e r s e s i s the concept of univ e r s e score. The universe score i s l i k e the c l a s s i c a l t e s t score n o t i o n of true score. Components of Variance The v a r i a n c e components are the b u i l d i n g blocks of g e n e r a l i z a b i l i t y theory. Consider, f o r example, a matrix c o n t a i n i n g data (persons' scores on items) obtained from the 5 a d m i n i s t r a t i o n of a t e s t . T h i s design i s d e s c r i b e d in g e n e r a l i z a b i l i t y theory as a one-facet (Items) design. The o b j e c t of measurement, persons, i s not c o n s i d e r e d a facet even though, as shown next, i t i s i n c l u d e d as a f a c t o r in the corresponding a n a l y s i s of v a r i a n c e . In a b a s i c one-facet design where each person r a t e s him/herself (or i s scored, on the same set of items and where Items i s co n s i d e r e d as a random f a c e t ) the t o t a l observed score v a r i a n c e can be expressed as the sum of three v a r i a n c e components: ( 1 ) a2 (X . ) = a2 + a 2 + a2 . p i p l p i , e In t h i s model a 2 , the un i v e r s e score v a r i a n c e , i s variance P due to d i f f e r e n c e s among respondents, the o b j e c t s of measurement; t h i s v a r i a n c e i s l i k e true score variance i n c l a s s i c a l t e s t score theory. The v a r i a n c e a s s o c i a t e d with items, a 2 , r e f l e c t s d i f f e r e n c e s among the items; item h e t e r o g e n e i t y can i n d i c a t e d i f f e r i n g l e v e l s of item d i f f i c u l t y or s o c i a l d e s i r a b i l i t y . The term a 2 . i s the p i , e r e s i d u a l v a r i a n c e which i s composed of the two-way Person by Item i n t e r a c t i o n p l u s e r r o r due to f a c e t s other than those e x p l i c i t l y c o n s i d e r e d f a c e t s and random f l u c t u a t i o n . The Schroeder et a l . (1983) design i s an items by r a t e r s f u l l y - c r o s s e d two-facet d e s i g n . For t h i s design the 6 t o t a l observed score v a r i a n c e can be expressed as: a 2 ( X . ) = a2 + a 2 + a2 + a2. + a 2 p i r p 1 r p i pr (2) + a? +a 2. i r p i r , e The v a r i a n c e components i n equations (1) and (2) are estimated using from s t a t i s t i c a l theory the e x p r e s s i o n s f o r the ANOVA expected mean squares. The o b t a i n e d mean squares are unbiased estimates of the expected mean squares. By s u b s t i t u t i n g the v a l u e s f o r the mean squares i n the equations f o r the expected mean squares, e s t i m a t e s of the v a r i a n c e components can be c a l c u l a t e d . For example, the estimate f o r a2 i s given by P a2 = (MS - MS . - MS + MS . ) P P P i pr p i r , e f o r the two-facet random e f f e c t s model. The form of the ex p r e s s i o n s f o r the expected mean squares depends on whether the f a c e t s are f i x e d , random, or f i n i t e . Millman and G l a s s (1967) pr o v i d e d r u l e s f o r w r i t i n g these e x p r e s s i o n s ( a l s o see Cronbach et a l . , 1972, Ch. 2 ) . G and D S t u d i e s Cronbach et a l . (1972) d i s t i n g u i s h e d between two types of study. The f i r s t , the g e n e r a l i z a b i l i t y or G study, i s conducted to o b t a i n e s t i m a t e s of the v a r i a n c e components to be used t o pl a n the second, the d e c i s i o n or D study. The purpose of the D study i s to make d e c i s i o n s about the o b j e c t 7 of measurement. The D study i s planned using information p r o v i d e d i n the G study v a r i a n c e component estimates. Cronbach et a l . viewed the G and D s t u d i e s as being i d e a l l y d i s t i n c t . They b e l i e v e d that l a r g e sample G s t u d i e s should be conducted with the r e s u l t a n t v a r i a n c e components being used to design other s t u d i e s . In p r a c t i c e , however, the same data u s u a l l y are used f o r both the G and D s t u d i e s . Two Kinds of D e c i s i o n ; Two Kinds of E r r o r Variance In the D study one of two kinds of d e c i s i o n i s made. An absolute decision i s made when an i n d i v i d u a l ' s universe score estimate i s compared to a c r i t e r i o n or c u t t i n g score. In the Schroeder et a l . (1983) study i f an inmate were c l a s s i f i e d as a psychopath when he r e c e i v e d a mean r a t i n g of at l e a s t 2.8 out of a maximum of 3, an absolute d e c i s i o n would have been made. The d e c i s i o n i s absolute i n the sense that no d i r e c t comparisons among the inmates t e s t e d were made. A relative decision i s made when the i n d i v i d u a l s are rank ordered a c c o r d i n g to t h e i r estimated universe score. The r e l a t i v e standing of an i n d i v i d u a l i s then used to make a d e c i s i o n about her or h i s f u t u r e treatment. In the Schroeder et a l . study i f the hig h e s t s c o r i n g 20% of the sample had been designated psychopaths, a r e l a t i v e d e c i s i o n would have been made. Jus t as the two kinds of d e c i s i o n d i f f e r , so too do the estimates of e r r o r v a r i a n c e a s s o c i a t e d with the estimates of un i v e r s e score upon which the d e c i s i o n s are based. In the 8 case of ab s o l u t e d e c i s i o n s , e r r o r v a r i a n c e a r i s e s from a l l sources of v a r i a n c e except the source due to the object of measurement. Cronbach et a l . (1972) l a b e l l e d t h i s e r r o r v a r i a n c e a 2 ( A ) . In the case of r e l a t i v e d e c i s i o n s , l a b e l l e d by Cronbach et a l . as a 2 ( 6 ) , some sources of v a r i a n c e other than that due to the ob j e c t of measurement do not enter i n t o the e x p r e s s i o n f o r the e r r o r v a r i a n c e . Variance due to Items in the Schroeder et a l . (1983) study, f o r example, r e f l e c t s item h e t e r o g e n e i t y . T h i s source of v a r i a n c e i s considered constant across a l l inmates ( p o s s i b l e item-inmate i n t e r a c t i o n i s accounted for in the item-person i n t e r a c t i o n v a r i a n c e component). When a l l f a c e t s of the design are random, a 2 ( 6 ) i n c o r p o r a t e s a l l v a r i a n c e components i n v o l v i n g i n t e r a c t i o n with the ob j e c t of measurement. In c o n t r a s t , in the case of a design i n c o r p o r a t i n g a f i x e d f a c e t , i n t e r a c t i o n of the ob j e c t of measurement with a f i x e d f acet i s not co n s i d e r e d to be e r r o r v a r i a n c e . V a r i a n c e due to i n t e r a c t i o n s between the object of measurement and a f i n i t e f a c e t i s c o n s i d e r e d e r r o r . For both kinds of e r r o r v a r i a n c e , each c o n s t i t u e n t v a r i a n c e component i s d i v i d e d by the t o t a l number of c o n d i t i o n s w i t h i n the ob j e c t of measurement upon which the v a r i a n c e i s based. T h i s procedure r e f l e c t s the f a c t that the un i v e r s e score estimate i s the average value of the o b s e r v a t i o n s w i t h i n the ob j e c t of measurement. T h i s procedure i s e q u i v a l e n t to using the Spearman-Brown c o r r e c t i o n f o r t e s t l e n g t h (Cronbach et a l . , 1972, p. 82). 9 In the Schroeder et a l . study each inmate's universe score estimate i s the average value taken over 44 observ a t i o n s (22 items f o r each of two r a t e r s ) . In the present r e s e a r c h , emphasis i s p l a c e d on the e r r o r v a r i a n c e o 2 ( 6 ) . T h i s c o n c e p t u a l i z a t i o n of e r r o r v a r i a n c e i s c l o s e r to the c l a s s i c a l t e s t score notion of e r r o r v a r i a n c e than i s the a 2(A) e r r o r v a r i a n c e . T h i s e r r o r i s a l s o of relevance i n the study of i n d i v i d u a l d i f f e r e n c e s where comparisons among persons are made. The C o e f f i c i e n t of G e n e r a l i z a b i l i t y (CG) The c o e f f i c i e n t of g e n e r a l i z a b i l i t y , CG, i s an i n t r a c l a s s c o r r e l a t i o n c o e f f i c i e n t summarizing the adequacy of the measurement procedure in the case of r e l a t i v e d e c i s i o n s . The i n t r a c l a s s c o r r e l a t i o n c o e f f i c i e n t i s the c o r r e l a t i o n between exchangeable measurements obtained on the same o b j e c t (Cronbach et a l . , 1972, p. 17; Shrout & F l e i s s , 1979). A n e g a t i v e l y b i a s e d but c o n s i s t e n t s t a t i s t i c (Lahey, Downey, & S a a l , 1983), the maximum value of the i n t r a c l a s s c o r r e l a t i o n c o e f f i c i e n t i s one; the t h e o r e t i c a l lower l i m i t i s zero. Although the standard e r r o r of measurement i s o f t e n used to summarize the adequacy of measurement, the CG prese n t s the advantage of having a s t a n d a r d i z e d m e t r i c . As an example, c o n s i d e r again a simple one-facet design i n which a number of persons are t e s t e d or evaluated on a number of items measuring some a t t r i b u t e . F u r t h e r , suppose 1 0 that the n. items were sampled at random from a t h e o r e t i c a l l y i n f i n i t e l y l a r g e item pool f o r each i n d i v i d u a l ( i . e . , each i n d i v i d u a l r e c e i v e s a d i f f e r e n t set of items). These data would be analysed as a one-way random e f f e c t s ANOVA with Persons as the Between f a c t o r i n the d e s i g n . The i n t r a c l a s s c o r r e l a t i o n c o e f f i c i e n t f o r t h i s design i s the r a t i o of the between c l a s s v a r i a n c e (here person variance) to the t o t a l v a r i a n c e (Haggard, 1958, p. 4). The expression for the p o p u l a t i o n value i s : R = a2/ [o2 + a2 ], P P i :p where a2 i s the v a r i a n c e due to Persons, and o2 i s the P i :p v a r i a n c e due to Items nested w i t h i n Persons. In Haggard's treatment of the i n t r a c l a s s c o r r e l a t i o n c o e f f i c i e n t a t t e n t i o n i s focused on the i n d i v i d u a l item or o b s e r v a t i o n as the u n i t of a n a l y s i s ( h i s treatment of the t o p i c encompasses more than psychometric a p p l i c a t i o n s ) . Cronbach et a l . ' s treatment focuses on scores averaged over the c o n d i t i o n s of f a c e t s ; here the i n t r a c l a s s c o r r e l a t i o n c o e f f i c i e n t ( i . e . , the CG) i s p 2 = a2/ [o2 + o2 / n . ] . P P i : p i -Thus, Haggard's c o e f f i c i e n t , R, r e p r e s e n t s the average r e l i a b i l i t y of a s i n g l e item while Cronbach et a l . ' s c o e f f i c i e n t , p 2 , represents the r e l i a b i l i t y of the aggregated items. 11 Cronbach et a l . (1972, p. 17) d e f i n e d the CG as the r a t i o of universe score v a r i a n c e to expected observed score v a r i a n c e . T h i s value i s approximately equal to the squared c o r r e l a t i o n between the observed score and the universe score (Cronbach et a l . , p. 82), and as such i s a c o e f f i c i e n t of d e t e r m i n a t i o n . The CG can be expressed more g e n e r a l l y as P 2 = O 2 ( T ) / [ o 2 ( r ) + a 2 ( 6 ) ] where C T 2 ( T ) i s universe score v a r i a n c e and a 2 ( 6 ) i s , as d e f i n e d p r e v i o u s l y , e r r o r v a r i a n c e a s s o c i a t e d with r e l a t i v e d e c i s i o n s . The CG i s comparable to c l a s s i c a l t e s t score r e l i a b i l i t y c o e f f i c i e n t s (Cronbach et a l . , 1972, p. 84) and i s i n t e r p r e t e d the same way. The magnitude of the c o e f f i c i e n t , p 2, i n d i c a t e s how r e l i a b l y the o b j e c t of measurement (e.g., the n persons) can be rank ordered with P the f a c e t s and the numbers of f a c e t c o n d i t i o n s used in the d e s i g n . For example, in the Schroeder et a l . (1983) study, the obtained value of p 2 = .86 ( f o r a sample of 71 inmates) i n d i c a t e s that inmates can be r e l i a b l y ranked using the average r a t i n g a s s i g n e d by two r a t e r s on the 22 item c h e c k l i s t . F o l l o w i n g Nunnally's (1978, pp. 245-246) g u i d e l i n e s f o r r e l i a b i l i t y , a c o e f f i c i e n t of at l e a s t .7 i s a c c e p t a b l e f o r r e s e a r c h purposes where group averages are the l o c u s of i n t e r e s t . In a p p l i e d s e t t i n g s , where d e c i s i o n s w i l l be made about i n d i v i d u a l s , r e l i a b i l i t y ( g e n e r a l i z a b i l i t y ) should be as high as p o s s i b l e . 12 E s t i m a t i n g the CG In p r a c t i c e , the CG i s estimated from the varia n c e components c a l c u l a t e d i n the G study. The c a l c u l a t e d CG giv e s the researcher an estimate of the g e n e r a l i z a b i l i t y of a proposed D-study data c o l l e c t i o n procedure. From the va r i a n c e component estimates the researcher can evaluate the r e l a t i v e adequacy of a v a r i e t y of designs f o r data c o l l e c t i o n p r i o r to a c t u a l l y conducting the study. This i s accomplished by v a r y i n g the number of fac e t c o n d i t i o n s used in the computation of the CG. The magnitude of the c o e f f i c i e n t s that w i l l c o n s t i t u t e a 2(6) guides the research e r i n determining whether more or fewer c o n d i t i o n s of a p a r t i c u l a r f a c e t should be sampled. If the estimate of a v a r i a n c e component contained w i t h i n a 2(6) i s sm a l l , i t w i l l c o n t r i b u t e l i t t l e to e r r o r v a r i a n c e . In t h i s case only a few l e v e l s of the fa c e t need to be sampled. In c o n t r a s t , i f a p a r t i c u l a r component estimate accounts f o r a large p r o p o r t i o n of the e r r o r v a r i a n c e , i t s impact'can be dimi n i s h e d by planning a D study with a l a r g e number of c o n d i t i o n s or seeking ways to d i m i n i s h the value of the var i a n c e i t s e l f . For example, i n the Schroeder et a l . (1983) example, the general form of the estimated CG i s expressed as: p 2 = a2/[o2 + d 2 . / n . +a2 /n + a 2 . /n.n ] p p p i I pr r p i r , e I r 13 If i t i s found that the component a2 , corresponding to the pr person by r a t e r i n t e r a c t i o n , i s l a r g e , the impact of t h i s component can be minimized by employing a l a r g e number of r a t e r s i n the D study. If t h i s approach i s not p r a c t i c a l , then steps should be taken to improve the q u a l i t y of the data by improving the t r a i n i n g procedures, thereby d e c r e a s i n g the value of a2 . p r I t should be noted here that the choice of f a c e t s s t u d i e d depends upon the purpose of the study. The Schroeder et a l . (1983) example focused on the equivalence of r a t e r s . In other t e s t i n g s i t u a t i o n s the s t a b i l i t y of measurement over time may be of i n t e r e s t . In such a case an Occasions f a c e t would be i n c l u d e d i n the g e n e r a l i z a b i l i t y study. Also i t should be noted that s e v e r a l f a c e t s can be i n c l u d e d in the same desi g n . F u r t h e r , as suggested in an e a r l i e r example, some f a c e t s can be nested, e i t h e r because the study was so designed or because f a c e t s are n a t u r a l l y nested w i t h i n some other f a c e t - - s u c h as C l a s s e s nested w i t h i n S c h o o l s . G e n e r a l i z a b i l i t y Theory Research and A p p l i c a t i o n The use of g e n e r a l i z a b i l i t y theory has been advocated by a number of authors w i t h i n the f i e l d of psychology (e.g., Jackson & Paunonen, 1980; M i t c h e l l , 1979; Wiggins, 1973). These authors s t r e s s e d the advantages of a m u l t i f a c e t e d approach to r e l i a b i l i t y e s t i m a t i o n over the t r a d i t i o n a l c l a s s i c a l t e s t score approach. 1 4 Pedagogical a r t i c l e s demonstrating the a p p l i c a t i o n of the g e n e r a l i z a b i l i t y approach have appeared i n the s o c i a l s c i e n c e l i t e r a t u r e . C a r d i n e t , Tourneur, and A l l a l (1976, 1981) and Rentz (1980) presented d e t a i l e d demonstrations of the a p p l i c a t i o n of the technique to e d u c a t i o n a l measurement. Brennan and Kane (1979) presented a summary of the e s s e n t i a l concepts and f e a t u r e s of g e n e r a l i z a b i l i t y theory along with an example of a two-facet design a p p l i c a t i o n . A p p l i c a t i o n s of g e n e r a l i z a b i l i t y theory in research s e t t i n g s appear in the s o c i a l s c i e n c e l i t e r a t u r e . In the p s y c h o l o g i c a l l i t e r a t u r e a number of authors (e.g., Chalmers & Knight, 1985; F a r r e l l , M a r i o t t o , Conger, Curran, & Wallander, 1979; Hansen, T i s d e l l e , & O'Dell 1985; Wallander, Conger, & Conger, 1985) have used a g e n e r a l i z a b i l i t y approach to estimate r e l i a b i l i t y i n m u l t i f a c e t e d instrument development s t u d i e s . G e n e r a l i z a b i l i t y theory a p p l i c a t i o n s a l s o are found i n the e d u c a t i o n a l (e.g., G i l l m o r e , Kane, & Naccarato, 1978; Kane & Brennan, 1977; Nussbaum, 198,4) and o r g a n i z a t i o n a l behaviour (e.g., Cain & Green, 1983; Doverspike & B a r r e t t , 1984; F r a s e r , Cronshaw, & Alexander, 1984) l i t e r a t u r e . Although l i t t l e r e s e a r c h has focused on the s t a t i s t i c a l p r o p e r t i e s of the g e n e r a l i z a b i l i t y technique, some work has been d i r e c t e d toward examining the sampling p r o p e r t i e s of the v a r i a n c e component es t i m a t e s . Shavelson and Webb (1981, p. 138) r e f e r r e d to these components as the " A c h i l l e s h e e l " 15 of g e n e r a l i z a b i l i t y theory because of t h e i r sampling p r o p e r t i e s . Smith (1978) e m p i r i c a l l y i n v e s t i g a t e d the sampling e r r o r s of v a r i a n c e component estimates under two design models. His r e s u l t s i n d i c a t e d that v a r i a n c e component estimates based on small numbers of f a c e t c o n d i t i o n s are u n s t a b l e . G s t u d i e s i n v o l v i n g small numbers of c o n d i t i o n s thus do not provide good estimates from which to p r e d i c t the adequacy of subsequent D s t u d i e s . Leone and Nelson (1966) performed a s i m i l a r study using h i e r a r c h i c a l (completely nested) designs. T h e i r r e s u l t s a l s o i n d i c a t e d a high degree of sampling v a r i a b i l i t y in the estimates that were based on l i n e a r combinations of mean squares. Very l i t t l e r e s e a r c h has been d i r e c t e d toward i n v e s t i g a t i n g the sampling e r r o r of the CG. One reason for t h i s n e g l e c t l i k e l y i s that Cronbach et a l . ( l 9 7 2 ) placed r e l a t i v e l y l i t t l e emphasis on the c o e f f i c i e n t ; they s t r e s s e d the g r e a t e r importance of the v a r i a n c e components. They suggested that l a r g e s c a l e G s t u d i e s should be conducted to provide s t a b l e v a r i a n c e component estimates. Such s t u d i e s would i n v o l v e sampling l a r g e numbers of f a c e t c o n d i t i o n s . The v a r i a n c e components c a l c u l a t e d from such s t u d i e s would then be used to estimate the g e n e r a l i z a b i l i t y of the proposed d e s i g n . Because component estimates based on l a r g e numbers of f a c e t c o n d i t i o n s l i k e l y are s t a b l e , the CGs c a l c u l a t e d with these components l i k e l y would a l s o be s t a b l e . 16 However, i n many t e s t i n g s i t u a t i o n s s u f f i c i e n t resources are not a v a i l a b l e to conduct a p r e l i m i n a r y G study. Nor, g e n e r a l l y , are s u i t a b l e p u b l i s h e d G study r e s u l t s a v a i l a b l e . Even i f G study r e s u l t s were a v a i l a b l e , r e s e a r c h e r s would be advised to e x e r c i s e c a u t i o n i n using other r e s e a r c h e r s ' v a r i a n c e component e s t i m a t e s . Such estimates may be unstable ( c f . Smith, 1978). F u r t h e r , hidden or unmeasured f a c e t s , such as geographic l o c a t i o n or occasion of t e s t i n g , c o u l d have an impact on the D study. Most p u b l i s h e d r e s e a r c h using g e n e r a l i z a b i l i t y theory i s based on a s i n g l e data c o l l e c t i o n that serves as both the D and G study. Researchers tend to place g r e a t e r emphasis on the CG than they do on the v a r i a n c e component estimates. T h i s probably r e f l e c t s the long-standing psychometric t r a d i t i o n of r e p o r t i n g r e l i a b i l i t y and v a l i d i t y c o e f f i c i e n t s in instrument development s t u d i e s . In a d d i t i o n , c o e f f i c i e n t s summarizing the adequacy of measurement o f t e n are needed i n s t u d i e s in which a new instrument has been developed f o r experimental purposes. Researchers (e.g., Doverspike, C a r l i s i , B a r r e t t , & Alexander, 1983; Kane, G i l l m o r e , & Crooks, 1976) o f t e n r e p o r t the estimated CG f o r the number of c o n d i t i o n s used in the study as we l l as estimates f o r other numbers of c o n d i t i o n s . Some authors have tended to t r e a t these estimated CGs as i f they were parameter v a l u e s , rather than e s t i m a t e s . Doverspike et a l . , f o r example, s t a t e d that 17 " . . . r e l i a b i l i t y dropped only s l i g h t l y when the number of r a t e r s was reduced from 10 to 1" (p. 481). However, no a t t e n t i o n was given to the presence of sampling e r r o r in the estimates c o n s i d e r e d . To c l a r i f y such c o n c l u s i o n s , i t i s important to report a confidence i n t e r v a l or the standard e r r o r f o r each c o e f f i c i e n t , thereby i n d i c a t i n g a measure of u n c e r t a i n t y due to sampling. A high value of an estimated CG does not n e c e s s a r i l y guarantee that the measurement procedure i s adequately r e l i a b l e or that a s i m i l a r l y f a v o u r a b l e c o e f f i c i e n t would be found upon r e p l i c a t i o n of the D study. But, by r e p o r t i n g say the 90% confidence i n t e r v a l , r e s e a r c h e r s would be p r o v i d e d with the l i k e l y range of the p o p u l a t i o n parameter. Some resea r c h has been d i r e c t e d toward examining the i n f e r e n t i a l p r o p e r t i e s of CGs computed for s i n g l e f a c e t s t u d i e s . These s t u d i e s have not d e a l t with g e n e r a l i z a b i l i t y theory, per se; they have i n v e s t i g a t e d the p r o p e r t i e s of i n t r a c l a s s c o r r e l a t i o n c o e f f i c i e n t s and c o e f f i c i e n t a l p h a — b o t h of which are e q u i v a l e n t to CGs. F e l d t (1965, 1969) presented i n f e r e n t i a l techniques f o r c o n s t r u c t i n g single-sample c o n f i d e n c e i n t e r v a l s and f o r making two independent sample comparisons f o r c o e f f i c i e n t alpha (a one-facet c r o s s e d design i n g e n e r a l i z a b i l i t y t e r m i n o l o g y ) . T h i s work was extended by Hakstian and Whalen (1976) who developed a k independent sample s i g n i f i c a n c e t e s t f o r alpha c o e f f i c i e n t s . F l e i s s and Shrout (1978; Shrout & F l e i s s , 1979) developed approximate c o n f i d e n c e i n t e r v a l s for s i x i n t r a c l a s s c o r r e l a t i o n c o e f f i c i e n t s . More r e c e n t l y , Hakstian and L i n d (1982) d e r i v e d approximate v a r i a n c e and co v a r i a n c e e x p r e s s i o n s for c o e f f i c i e n t a l p h a . These authors used the exp r e s s i o n s to develop i n f e r e n t i a l procedures f o r m u l t i p l e dependent sample alpha c o e f f i c i e n t s . Purpose of the Present Study As suggested above, i n f e r e n t i a l procedures f o r CGs c a l c u l a t e d from designs i n v o l v i n g more than one f a c e t have not been developed. In the absence of such developments, i t i s not p o s s i b l e to comment on the p r e c i s i o n of a s i n g l e c o e f f i c i e n t or to make comparisons among two or more c o e f f i c i e n t s . Thus, the major focus of the present study was the development of i n f e r e n t i a l procedures f o r the CG estimate. The two-facet design, with both f a c e t s random, • r w i l l be used to i l l u s t r a t e , i n the next chapter, these developments, g r e a t e s t d e t a i l . T h i s design, r e f e r r e d to as Design VII (see Cronbach et a l . , 1972, Ch. 2), has a broad range of a p p l i c a t i o n to p s y c h o l o g i c a l and e d u c a t i o n a l measurement problems; i t i s a p p r o p r i a t e f o r s t u d i e s i n v o l v i n g m u l t i p l e r a t e r s , observers, or o b s e r v a t i o n p e r i o d s . F u r t h e r , t h i s design i s r e l a t i v e l y uncomplicated, a l l o w i n g the e x p l i c a t i o n of the development of the i n f e r e n t i a l procedure without e x t e n s i v e n o t a t i o n . Extension of the g e n e r a l i z a b i l i t y approach to designs with more cr o s s e d f a c e t s i s s t r a i g h t f o r w a r d . 1 9 The random f a c e t model was chosen as the focus of t h i s study because the author b e l i e v e s that most f a c e t s s t u d i e d are, as a s s e r t e d by Shavelson and Webb (1981) with the n o t i o n of e x c h a n g e a b i l i t y , sampled from a l a r g e r u n i v e r s e . In t h e i r view, i f f a c e t c o n d i t i o n s can be exchanged with other p o t e n t i a l c o n d i t i o n s of the f a c e t , the f a c e t should be t r e a t e d as random. The present r e s e a r c h used a method s i m i l a r to that employed by Hakstian and L i n d (1982) to develop an approximate v a r i a n c e e x p r e s s i o n f o r estimated CGs under d i f f e r e n t experimental design and sampling models. F i r s t , a n o r m a l i z i n g t r a n s f o r m a t i o n was a p p l i e d to the e x p r e s s i o n for the sample c o e f f i c i e n t . Then the d e l t a method was used to d e r i v e the v a r i a n c e e x p r e s s i o n s . D e t a i l s of these procedures are presented i n the f o l l o w i n g chapter. Chapter 2 Mathematical Development The development of the asymptotic v a r i a n c e expressions and t h e i r use in c o n s t r u c t i n g confidence i n t e r v a l s w i l l be i l l u s t r a t e d f o r the f u l l y - c r o s s e d two-facet design--Cronbach et a l . ' s (1972, p. 38) Design V I I — w i t h both f a c e t s random. T h i s chapter begins with a d i s c u s s i o n of the approximate d i s t r i b u t i o n of the CG and then proceeds with the a p p l i c a t i o n of a n o r m a l i z i n g t r a n s f o r m a t i o n . F o l l o w i n g t h i s step, the d e l t a method i s a p p l i e d to the transformed c o e f f i c i e n t to obt a i n the v a r i a n c e e x p r e s s i o n . The development of v a r i a n c e expressions f o r other s e l e c t e d two-facet designs f o l l o w s the same st e p s . The Approximate D i s t r i b u t i o n of the CG The Schroeder et a l . (1983) study d i s c u s s e d p r e v i o u s l y i s an example of a Design VII G Study. In t h i s case each of 71 inmates was rated on 22 items by each of two r a t e r s . T h i s layout can be c h a r a c t e r i z e d as a between-within s u b j e c t s balanced ANOVA desig n . The data f o r such a design are analysed as a three-way f a c t o r i a l design without r e p l i c a t i o n s ( i . e . , there i s only one o b s e r v a t i o n per c e l l ) . The l i n e a r model u n d e r l y i n g these data i s expressed as X . = M + a + b . + c + a b . + a c + be. + abc . , p i r p I r p i pr i r p i r , e 21 where u i s the grand mean, a i s the e f f e c t due to person p, P b. i s the e f f e c t due to item i , and c i s the e f f e c t due to 1 r r a t e r r . The remaining terms i n the model represent the e f f e c t s due to i n t e r a c t i o n s among the f a c t o r s . The e p s i l o n i n the s u b s c r i p t f o r the three-way i n t e r a c t i o n i n d i c a t e s t h at random e r r o r i s confounded with t h i s i n t e r a c t i o n . The formulas f o r the expected mean squares f o r the f u l l y random ANOVA model are presented in Table 1 . The mean squares and va r i a n c e component estimates are presented i n Table 2 . Throughout these d e r i v a t i o n s i t i s co n s i d e r e d that the assumptions of the ANOVA model are te n a b l e : independence of o b s e r v a t i o n s , homogeneity of v a r i a n c e , and u n d e r l y i n g normal d i s t r i b u t i o n . The p o p u l a t i o n CG f o r t h i s design i s (3) p 2 = a2/[a2 + a 2./n. +a 2-/n + a 2 . /n.n ]. 1 p p pa l pr r p i r , e l r The s u b s c r i p t 1 i s used to d i s t i n g u i s h the CG f o r the f u l l y random model from the CG f o r other models. The sample estimate f o r t h i s q u a n t i t y , expressed i n terms of observed mean squares, i s (4) p 2 = 1 - [(MS . + MS - MS . )/MS ]. 1 p i pr p i r , e p F i r s t c o n s i d e r the numerator of the q u a n t i t y (1 - P 2 ) / MS . + MS - MS . p i pr p i r , e T h i s e x p r e s s i o n i s a l i n e a r combination of independent Table 1 Expected Mean Squares f o r Design VII with both Facets Random E ( M S p ) = a p i r , e + n r a p > i + n i a P r + n i n r p p E ( M S i ) = aP i r , e + n r ° p i n p a i r + n p n r p i E ( M S r ) = a p i r , e + n i a P r + n p a i r + n p n i ^ r E ( M S p i ) = a F > i r , € + n r a P i E ( M S p r ) = a p i r , e + n i a P r E ( M S i r ) = a p i r , e + n p a i r E ( M S p i r f € ) = a 2 i r > e 23 Table 2 A n a l y s i s of Variance R e s u l t s and Estimated Variance  Components for Psychopathy Data Source Sum of Squares df Mean Square o2 Persons 363.4225 70 5.1918 .001 5 I terns 115.4882 21 5.4994 .031 1 Raters .7686 1 .7686 .0001 P X I 1021 .9437 1 470 .6952 .2604 P X R 14.2996 70 .2043 .0014 I X R 11.9286 21 .5680 .0055 P X I X R 256.5032 1 470 .1745 .1745 v a r i a n c e e s t i m a t e s . The exact form of the d i s t r i b u t i o n of such a combination i s too complicated to be of p r a c t i c a l u t i l i t y (see F l e i s s , 1971). However, S a t t e r t h w a i t e (1941, 1946) developed an approximation to the d i s t r i b u t i o n of such a l i n e a r combination. If g i s a l i n e a r combination of v a r i a n c e e s timates, g = a s 2 + a s 2 + ... + a s 2 , * 1 1 2 2 k k where the sample v a r i a n c e s 2 has expected value a 2 , r. I i i degrees of freedom, and weighting f a c t o r a^, then the q u a n t i t y rg/E(g) i s approximately chi-square d i s t r i b u t e d with r degrees of freedom, (a a2 + a o 2 + ... + a, a 2 ) 2 1 1 2 2 k k r = ( S a t t e r t h w a i t e , 1941). C o n s i d e r i n g the denominator of (1 - P 2 ) / note that the q u a n t i t y (n - 1)MS /E(MS ), P P P i s c h i - s q u a r e d i s t r i b u t e d with n p ~ 1 degrees of freedom. Hence, the q u a n t i t y (1 - p 2) i s approximately F d i s t r i b u t e d with degrees of freedom c, and v2 given by V , = (MS . + MS - MS . ) 2 p i pr p i r , e MS2 . P 1 + MS2 P_r + MS2 . p i r , e (n - D ( n . - l ) P i (n - D ( n -1) P r (n -1)(n.-1)(n -1 ) P i • . r and v2 = (n P 1 ) . The N o r m a l i z a t i o n of a Function of the CG The d i s t r i b u t i o n a l p r o p e r t i e s of the q u a n t i t y (1 - p 2 ) noted above could be used to develop an approximate confidence i n t e r v a l in a s i m i l a r manner to the technique used by F e l d t (1965). However, the development of a variance e x p r e s s i o n permits not only the s e t t i n g of confidence i n t e r v a l s ; i t can a l s o be used to develop i n f e r e n t i a l techniques f o r hypothesis t e s t i n g . In t h i s l a t t e r a p p l i c a t i o n , having a c o e f f i c i e n t with an u n d e r l y i n g normal d i s t r i b u t i o n permits the a p p l i c a t i o n of a v a r i e t y of s t a t i s t i c a l procedures based on normal theory. For t h i s reason, Paulson's (1942) n o r m a l i z i n g t r a n s f o r m a t i o n was a p p l i e d to a f u n c t i o n of the CG. Paulson (1942) extended Wilson and H i l f e r t y ' s (1931) work on a n o r m a l i z i n g t r a n s f o r m a t i o n f o r x 2 v a r i a b l e s . Paulson developed a very c l o s e n o r m a l i z a t i o n f o r F - d i s t r i b u t e d random v a r i a b l e s , (cF 1/3 - n) a ~ N(0,1), where u = 1 - 2/(9»,), a 2 = [ 2 / ( 9 ^ 2 ) ] F 2 / 3 + 2 / ( 9 1 ; , ) , c = 1 - 2/(9v 2) and the q u a n t i t i e s and v2 are the degrees of freedom a s s o c i a t e d with the F r a t i o . Hakstian and Whalen (1976) used Paulson's t r a n s f o r m a t i o n i n t h e i r development of a t e s t of s i g n i f i c a n c e for m u l t i p l e independent alpha c o e f f i c i e n t s (to s i m p l i f y the e x p r e s s i o n they set the c term equal to one). T h e i r e m p i r i c a l r e s u l t s i n d i c a t e d that a p p l i c a t i o n of Paulson's r e s u l t y i e l d e d a t e s t with good Type I e r r o r • c o n t r o l . The value of the c term, which c o r r e c t s f o r b i a s i n the cube root transformed F v a r i a t e , was c l o s e to u n i t y i n the present i n v e s t i g a t i o n ; i t was set equal to one f o r a l l a n a l y s e s . With the s m a l l e s t value of v2 (n = 25) c o n s i d e r e d P i n the present study, the a c t u a l value of c i s .9907. With the l a r g e r samples c o n s i d e r e d , the value of c i s even c l o s e r to u n i t y . F u r t h e r , p r e l i m i n a r y a n a l y s i s of the adequacy of the transformed CG with and without the s c a l i n g f a c t o r i n d i c a t e d that the i n c l u s i o n of the f a c t o r had v i r t u a l l y no impact on the c o n t r o l of Type I e r r o r . The D e r i v a t i o n of an Asymptotic Variance Expression f o r the CG In the present r e s e a r c h , the d e l t a method (Rao, 1973, p. 387) was used to develop the asymptotic v a r i a n c e e x p r e s s i o n for the CG. The d e l t a method i s a technique for d e r i v i n g an asymptotic v a r i a n c e e x p r e s s i o n f o r a s t a t i s t i c that i s expressed as a f u n c t i o n of s t a t i s t i c s with a known co v a r i a n c e s t r u c t u r e . We begin with a v e c t o r , t, which c o n t a i n s unbiased and c o n s i s t e n t estimates of the elements of the v e c t o r 6. The s t a t i s t i c s in the vecto r t have known covar i a n c e matrix I . By asymptotic theory, the vect o r ( n ) 1 / / 2 U - 0) ~ MVN( 0 , I ) , where n i s the sample s i z e . 1 /3 The s t a t i s t i c f ( t ) = (1 - p 2) estimates tie). By the d e l t a method, [f (t) - f(6) ] ~ N ( 0 , a 2) where a 2 = <j>'L4>. The vecto r <p c o n t a i n s the p a r t i a l d e r i v a t i v e s , 9 f ( t ) / 9 t | . Thus, a 2 i s the asymptotic t — & v a r i a n c e of the s t a t i s t i c f ( t ) ; i n p r a c t i c e , the value of the v a r i a n c e i s estimated by s u b s t i t u t i n g sample values for parameter values i n the v a r i a n c e e x p r e s s i o n . In the present a p p l i c a t i o n of the d e l t a method, the ve c t o r t c o n t a i n s the four mean squares which compose the sample CG: t ' = (MS , MS ., MS , MS . ) . P p i pr p i r r e These are the unbiased and c o n s i s t e n t estimates of the expected mean squares c o n t a i n e d i n 8. The q u a n t i t y 1/3 f ( t ) = (1 - p 2) i s the c o n s i s t e n t estimator of 1 1/3 f(6) = (1 - p 2) . The matrix E c o n t a i n s the v a r i a n c e s and co v a r i a n c e s of the four mean squares. In a balanced ANOVA design the mean squares are independent; thus, Z i s a d i a g o n a l matrix. A mean square i s the v a r i a n c e of a normally d i s t r i b u t e d random v a r i a b l e . The v a r i a n c e of the sample mean square i s Var(MS) = 2 [ E ( M S ) ] 2 / f , where v i s the degrees of freedom a s s o c i a t e d with the mean square ( S c h e f f e , 1959, Ch. 7). In t h i s a p p l i c a t i o n , the m a t r i x l i s : I = 2[E(MS ) ] 2 P (n -1) P 0 2[E(MS . ) ] 2 P i (n - 1 ) ( n . - l ) P I' 2[E(MS ) ] 2 Pr (n - D ( n -1) P r 0 2[E(MS . ) ] 2 p i r , e (n - 1 ) ( n . - 1 ) ( n -1) P i r The p a r t i a l d e r i v a t i v e s of f ( t ) contained i n # are 3 f ( t ) / 9 M S | n = 1/3(1 - p 2 ) l / 3 / E ( M S ) p t=e 1 p 9f(t)/dMS .1 = 1/3(1 - p 2 ) " 2 / 3 / E ( M S ) p i t=6 1 p 9f(t)/9MS | = 1/3(1 - p 2 ) " 2 / 3 / E ( M S ) pr t=d 1 p 9f(t)/9MS . | = -1/3(1 - p 2 ) " 2 / 3 / E ( M S ). pir, e ' t = 0 1 p The asymptotic v a r i a n c e of the the transformed CG i s given by the q u a n t i t y (5) V a r O - P 2 ) 1 / 3 = *'Ztf = 2(1 - p 2 ) ~ 4 / 3 ^1(1 - p 2 ) 2 9(n -1) P + [E(MS . ) ] 2 + [E(MS ) ] 2  p i pr [E(MS ) ] 2 ( n . - 1 ) [E(MS ) ] 2 ( n -1) p i p r [E(MS . ) ] 2  P i r , e [E(MS ) ] 2 ( n . - D ( n - D f P i r In p r a c t i c e , where the expected mean squares are not known, t h e i r unbiased e s t i m a t e s , the mean squares are s u b s t i t u t e d . The sample c o e f f i c i e n t , p 2 , i s s u b s t i t u t e d f o r the p o p u l a t i o n v a l u e . T h i s sample v a r i a n c e estimate i s denoted v t r O - p 2 ) , / 3 . V a r i a n c e E x p r e s s i o n s f o r CGs A r i s i n g from other Experimental  Designs The d e l t a method was used to d e r i v e asymptotic v a r i a n c e e x p r e s s i o n s f o r transformed c o e f f i c i e n t s a r i s i n g from three other two-facet d e s i g n s . In each case the CG was expressed i n terms of mean squares, and the p a r t i a l d e r i v a t i v e s of the transformed c o e f f i c i e n t were taken. The p a r t i a l d e r i v a t i v e s and the covar i a n c e matrix f o r the mean squares were then used to develop the asymptotic v a r i a n c e e x p r e s s i o n for the transformed CG. Design VII with a f i x e d f a c e t . In some s i t u a t i o n s a l l c o n d i t i o n s of a p a r t i c u l a r f a c e t may be i n c l u d e d i n the resear c h d e s i g n . We may have a design l i k e that employed by Schroeder et a l . where each person i s eva l u a t e d on each item by each r a t e r . I t i s p o s s i b l e that one of these f a c e t s cannot be con s i d e r e d random. T h i s would be the case when, fo r example, s p e c i f i c tasks ( i . e . , items) had been developed f o r t e s t i n g purposes or when r a t e r s have c o n s i d e r a b l e e x p e r t i s e and cannot be viewed as exchangeable with other s i m i l a r r a t e r s . In these cases the f a c e t d e s c r i b e d would be con s i d e r e d f i x e d . T h i s has two i m p l i c a t i o n s . F i r s t , the exp r e s s i o n s f o r the expected mean squares are a l t e r e d (see Winer, 1971, p. 344 f o r general formulas f o r the three-way f a c t o r i a l ANOVA). Second, the ex p r e s s i o n f o r the CG changes. The v a r i a n c e component f o r the i n t e r a c t i o n of the object of measurement with the f i x e d f a c e t no longer i s considered to be e r r o r v a r i a n c e ; t h i s v a r i a n c e i s t r e a t e d as universe score v a r i a n c e (Cronbach et a l . , 1972, Ch. 4). For example, i n the f o l l o w i n g development the Item component i s 31 c o n s i d e r e d to be a f i x e d f a c e t . The CG f o r Design VII with one f i x e d facet i s (6) 2 = [ a 2 + o2./n.]/[o2 + a 2./n. + (a2 /n 2 p p i I p p i I pr r + o2. /n.n ) ], p i r , e l r where the s u b s c r i p t 2 i n d i c a t e s that the CG i s f o r the f i x e d f a c e t model. In terms of the observed mean squares, the CG i s estimated by pi = 1 - [(MS + MS . )/ (MS + MS . ) ] . 2 pr p i r , e p p i The corresponding asymptotic v a r i a n c e e x p r e s s i o n of 1 /3 (1 - p 2) , d e r i v e d using the d e l t a method, i s V a r ( l - p 2 ) 1 7 3 = 2(1 - p 2 ) " 4 / 3 9[E(MS ) + E(MS .) ] 2 ( n -1) P P i P (1 - p 2 ) 2 [ E ( M S ) ] 2 2 P + (1 - p 2 ) 2 [E(MS . ) ] 2 + [E(MS ) ] 2 2 p i pr (n.-1 ) (n -1) i r + [E(MS . Pi ( n.-1)(n -1) ( r J 1/3 The sample estimate i s denoted Var(1 - p 2) Design VII with a f i n i t e f a c e t . Another v a r i a n t on Design VII i s a design i n which the c o n d i t i o n s of a fa c e t are sampled at random from a f i n i t e u n i v e r s e . For example, a subset of four r a t e r s i s drawn from a pool of 40 p o t e n t i a l r a t e r s or four of twenty machines are s e l e c t e d f o r t e s t i n g purposes. The i n t e r a c t i o n of the f i n i t e f a c e t (here, r a t e r s ) with the ob j e c t of measurement i s co n s i d e r e d e r r o r v a r i a n c e — i . e . , we wish to g e n e r a l i z e beyond the r a t e r s used in the study to the f i n i t e p o p u l a t i o n of p o t e n t i a l from which they were randomly sampled. Thus, Raters becomes a f i n i t e f a c e t . As i n the case of the f i x e d facet design, the expre s s i o n s f o r the expected mean squares are a l t e r e d . The formula f o r the p o p u l a t i o n CG, expressed i n terms of var i a n c e components, i s the same as that f o r the a l l f a c e t s random model (eq. 3). In terms of observed mean squares, the estimated CG (the s u b s c r i p t 3 i n d i c a t e s CG f o r the f i n i t e model) can be expressed as p 2 = [(MS - MS . - kMS + kMS . )/(MS + (1 - k)MS )] 3 p p i pr p i r , e p pr where k = (1 - n /N ) and n /N , the sampling f r a c t i o n , i s r R r R the p r o p o r t i o n of the p o p u l a t i o n of c o n d i t i o n s s e l e c t e d for i n c l u s i o n i n the study. As the f r a c t i o n decreases, the value of p 2 tends to the value of the CG f o r the f u l l y random 3 model. A p p l i c a t i o n of the d e l t a method y i e l d s the asymptotic v a r i a n c e e x p r e s s i o n Var (1 9[E(MS ) + (k-1)E(MS ) ] 2 ( n -1) P pr p (n -1) r + k 2[E(MS . ) ] 2 p i r , e (n.-1)(n -1 ) I r 1/3 The sample v a r i a n c e estimate i s denoted by Var(1 - p 2 ) Design V-B with a l l f a c e t s random. In some circumstances, i t i s i m p r a c t i c a l or impossible to conduct a f u l l y c r o s s e d D study. Sometimes a f a c e t i s n a t u r a l l y nested w i t h i n another f a c e t . In other circumstances i t would be i n e f f i c i e n t to conduct a f u l l y c r o s s e d D study. As an example of a study with a nested f a c e t , Kane and Brennan (1977) i n v e s t i g a t e d the g e n e r a l i z a b i l i t y of course e v a l u a t i o n means. C l a s s e s were the o b j e c t s of measurement. Students ( i . e . , Raters) are nested w i t h i n the o b j e c t of measurement. Another example of a nested.design i s a s i t u a t i o n i n which t r a i n e e s ' performance i s r a t e d on a s e r i e s of dimensions by a panel of s u p e r v i s o r s . In t h i s example, each t r a i n e e has h i s / h e r own panel of s u p e r v i s o r s . Both of these designs are examples of what Cronbach et a l . (1972, p. 38) termed Design V-B. The t o t a l observed score v a r i a n c e can be expressed as: o 2(X . ) = a2 + o2 + a2 + a 2 . + a 2 p i r p 1 r:p p i i r : p , e The v a r i a n c e component a2 confounds the sources of r :p v a r i a n c e a2 and a2 from Design V I I . S i m i l a r l y , the r e s i d u a l r pr v a r i a n c e , a2 , confounds a2 and a 2 . . From these ir:p, e it p i r , e r e l a t i o n s h i p s i t i s p o s s i b l e to estimate the magnitude of the CG f o r a V-B design from the va r i a n c e component estimates d e r i v e d from a Design VII G study (the confounded v a r i a n c e components from Design VII are summed to y i e l d the esimates f o r the V-B components). The p o p u l a t i o n value of the Design V-B CG with a l l f a c e t s random i s : p 2 = a 2 / [ a 2 + a 2./n. + a 2 /n +a'r /n. n ] . 4 p p p i 1 r:p r i r : p , e I r The s u b s c r i p t 4 i n d i c a t e s that the CG p e r t a i n s to the Design V-B model. In terms of mean squares, t h i s q u a n t i t y i s estimated by: p 2 = 1 - [(MS . + MS - MS. )/MS ]. 4 p i r:p i r : p , e p 3 5 The asymptotic v a r i a n c e e x p r e s s i o n f o r the normalized CG i s : V a r O - p 2 ) 1 / 3 = 2 ( 1 - p 2 ) " 4 / 3 I ( 1 - p 2 ) 2 4 4 7 4_ (n - 1 ) P [E(MS . ) ] 2  P i [E(MS ) 3 2 ( n -1)(n.-1) P P i + [E(MS ) ] 2  r:p [E(MS ) ] 2 n (n - 1 ) P P r + [E(MS. ) ] 2 i r : p , e  [E(MS ) ] 2 n ( n . - l ) ( n - 1 ) p p l r j ^ \ 1 / 3 The sample estimate i s denoted by V a r ( l - p 2) Chapter 3 Method Data Generation Data were generated a c c c o r d i n g to the model presented i n Equation (2) for the three Design VII c o n d i t i o n s . Each o b s e r v a t i o n was composed of seven independent components with an u n d e r l y i n g normal d i s t r i b u t i o n . The model f o r any given o b s e r v a t i o n can be expressed as: (7) X . = x + x. + x + x . + x +x. + x . p i r p l r p i pr i r p i r , e The q u a n t i t i e s on the r i g h t hand sid e of the equation are d e v i a t i o n s c o r e s . For Person p, an o b s e r v a t i o n x was P generated from a N ( 0 , < 7 2 ) d i s t r i b u t i o n (p = 1 , 2 , . . . , n ). The ' P P x. ( i = 1 , 2 , . . . , n ) and x (r = 1 , 2,...,n ) observations I /' r T were generated i n a s i m i l a r manner. For Person p, n. independent o b s e r v a t i o n s were generated from a N(0,o 2.) P i d i s t r i b u t i o n (p = 1 , 2,...,n ). The x ob s e r v a t i o n s were P P r generated s i m i l a r l y . For item i , n independent observations were generated from a N(0,a 2 ) d i s t r i b u t i o n ( i = 1 , i r 2,...,n ). The f i n a l o b s e r v a t i o n s x . were generated for i p i r . e each person by generating n times n random ob s e r v a t i o n s from a N(0,a 2. ) d i s t r i b u t i o n . T h i s process y i e l d s a three p i r , e dimensional Persons by Items by Raters data array as shown in F i g u r e 1 . 37 F i g u r e 1 . L a y o u t o f d a t a f o r D e s i g n V I I A n a l y s i s o f V a r i a n c e The normal v a r i a t e s were generated with the RANDN generator implemented on the U n i v e r s i t y of B r i t i s h Columbia's Amdahl 580 computer. The RANDN a l g o r i t h m begins by g e n e r a t i n g o b s e r v a t i o n s from a uniform (0,1) d i s t r i b u t i o n . These o b s e r v a t i o n s are transformed to normally d i s t r i b u t e d (0,1) random samples by a p p l i c a t i o n of M a r s a g l i a ' s r e c t a n g u l a r wedge-tail method (Knuth, 1968). For the present study the observed score components were given the d e s i r e d v a r i a n c e by m u l t i p l y i n g them by the square root of the p o p u l a t i o n v a r i a n c e component (see below). The mean square components f o r the three-way ANOVA were then c a l c u l a t e d from the data with a double p r e c i s i o n F o r t r a n r o u t i n e . These components were used to c a l c u l a t e the sample CG and the estimated v a r i a n c e of the transformed c o e f f i c i e n t . The procedure f o r generating the data for the Design V-B model d i f f e r e d from t h i s method only i n that f i v e r a t her than seven v a r i a n c e components were i n v o l v e d i n the exp r e s s i o n for t o t a l v a r i a n c e . A n a l y s i s of these data y i e l d s f i v e mean square components. Overview of the study f o r Design VII with Both Facets Random As p r e v i o u s l y noted, a l a r g e r number of s i t u a t i o n s were examined f o r Design VII with both f a c e t s random. A t o t a l of 81 combinations of f a c e t c o n d i t i o n s and sample s i z e were were i n v e s t i g a t e d f o r t h i s model. Three sample s i z e s (n = 25, 75, 150), three c o n d i t i o n s of Items (« = 10, 20, P ' • ' 30), three c o n d i t i o n s of Raters {n = 2, 5, 8), and three r v a l u e s of the p o p u l a t i o n CG ( p 2 = .5, .7, .9) were combined i n a f u l l y c r o s s e d f a c t o r i a l d e s i g n . P o p u l a t i o n CGs were c o n s t r u c t e d by s p e c i f y i n g s u i t a b l e v a r i a n c e component v a l u e s . For the 81 c o n d i t i o n s the f o l l o w i n g values were held c o n s t a n t : a 2 = 1, a 2 = .6, a 2 = .5, a 2 = .4, and p i r i r a 2 . = .3. The values of o2. , a 2 , and a2 do not enter i n t o p i r , e l r i r the computation of the mean squares used to c o n s t r u c t the CG estimate f o r any of the s i t u a t i o n s examined i n t h i s study. Thus, the values f o r these v a r i a n c e s were chosen a r b i t r a r i l y . The values of a 2 , and a2 were set equal, and p i pr t h i s value was determined to give the d e s i r e d p o p u l a t i o n CG fo r a p a r t i c u l a r c o n d i t i o n . P r e l i m i n a r y s i m u l a t i o n s in which the magnitude of the p o p u l a t i o n v a r i a n c e components were v a r i e d i n d i c a t e d that the magnitudes of the components have l i t t l e i f any impact on Type I e r r o r c o n t r o l . Three Other Models The Design VII f i x e d f a c e t and the Design V-B models were each i n v e s t i g a t e d in nine combinations of c o n d i t i o n and sample s i z e s . Three s e t s of values (n = 25, n = 20, P * n = 8), (n = 75, n - 30, n = 2), and in = 150, n = 10, r p i r p i n = 5 ) were c r o s s e d with the same three p o p u l a t i o n CG value s used above. The sets of values were chosen such that each set con t a i n e d a low, a moderate, and a high value. For the f i n i t e model, two p o p u l a t i o n sampling f r a c t i o n s were i n v e s t i g a t e d : n /N = .10 and .20. These values were cros s e d r R with the s e t s ' o f c o n d i t i o n s used with the Design VII f i x e d f a c e t model and the Design V-B models, y i e l d i n g 18 c o n d i t i o n s . The data generation method f o r the v a r i a n t s on Design VII ( f i n i t e and f i x e d f a c e t models) d i f f e r e d somewhat from that used f o r the random f a c e t s model. For each set of c o n d i t i o n s , p o p u l a t i o n mean square terms were c o n s t r u c t e d that would y i e l d the d e s i r e d value of the CG f o r the model under study. Note that the computational procedures and the r e s u l t a n t mean squares are the same f o r a given set of data, r e g a r d l e s s of which ANOVA model ( f i x e d , random, or f i n i t e ) i s a p p r o p r i a t e f o r the data. Variance component estimates are c a l c u l a t e d from the mean squares a c c o r d i n g to the assumed u n d e r l y i n g model. To generate data with the required p o p u l a t i o n mean square v a l u e s , the p o p u l a t i o n mean squares were t r e a t e d as i f they had a r i s e n from the f u l l y random Design VII model. The random e f f e c t s v a r i a n c e components were used to generate o b s e r v a t i o n s that were summed ac c o r d i n g to the model presented i n Equation 7. A n a l y s i s of the r e s u l t i n g data gave r i s e to mean squares that behaved l i k e sample estimates of the p o p u l a t i o n mean squares a p p r o p r i a t e to the model under study. For example, c o n s i d e r the case where p* = .5 f o r the Design VII model with the Item f a c e t f i x e d and n = 25, n P ' 20, and n =8, and the p o p u l a t i o n expected mean squares r E(MS ) = 99.96, E(MS .) = 5, E(MS ) = 49.48, and E(MS . ) p p i pr p i r , e = 3. S o l v i n g f o r the v a r i a n c e components from Equations 8a through 8d give s the v a r i a n c e components f o r the Design V I I 41 model with the item f a c e t f i x e d : a 2 = .3155, a 2 . = .25, P P 1 a 2 = 2.474, and a 2 . = 3 . S u b s t i t u t i n g these values i n t o pr p i r , e Equation 6 giv e s the d e s i r e d value of p 2 = .5. (8a) E(MS ) = n.a 2 + n.n a2 p I pr I r p (8b) E(MS .) = a2 . + n a 2 . p i p i r , e r p i (Be). E(MS ) = n.a 2 pr I pr (8d) E(MS . ) = a 2 . p i r , e p i r , e Note, however, that the a2 values g e n e r a l l y are not equal to the values that would be c a l c u l a t e d i f a f u l l y random e f f e c t s model were assumed to u n d e r l i e the data. In order to use the Equation 7 model f o r data generation i t was necessary to determine the p o p u l a t i o n v a r i a n c e components fo r the Design VII f u l l y random model. These component values are then used to generate data i n the same manner as used f o r the Design VII f u l l y random model. The r e s u l t a n t mean square estimates behave as i f they were sampled from a po p u l a t i o n with the r e q u i r e d mean square v a l u e s . The va r i a n c e components used to generate the data f o r Design VII with a f i x e d f a c e t , with a f i n i t e f a c e t , and Design V-B are presented i n Appendix A. A S p e c i a l C o n d i t i o n : Zero V a r i a n c e Components In a l l of the above c o n d i t i o n s , the sample CG was c a l c u l a t e d from the v a r i a n c e component estimates computed from observed mean squares. Any va r i a n c e component estimate that i n v o l v e s s u b t r a c t i o n of mean square estimates can y i e l d a negative v a l u e . Negative estimates occur, f o r example, when the value of MS . i s grea t e r than the value of MS . p i r , e p i or MS . Cronbach et a l . (1972, p. 57) suggested that pr negative v a r i a n c e estimates are due to sampling e r r o r and recommended that such components be set equal to z e r o — t h e i r l i k e l y p o p u l a t i o n v a l u e . These zero values then should be used i n the estimate of higher v a r i a n c e components (e.g., i f a 2 , i s set to zero, the estimate of a 2 , i n the formula for p i p i a2 should a l s o be set equal to z e r o ) . T h i s zero s u b s t i t u t i o n P r e s u l t s i n a CG value that i s lower than that obtained from the usual mean square formula. The zero s u b s t i t u t i o n e f f e c t i v e l y d e l e t e s mean square terms from the CG e x p r e s s i o n . For example, i n the case where the a2, estimate P 1 i s negative (and i s r e p l a c e d by a zero value) the formula f o r the estimated CG, expressed i n terms of mean squares, becomes: p2 = (MS - MS )/MS . 1 p pr p The estimate of the v a r i a n c e of the transformed c o e f f i c i e n t thus a l s o changes from that c a l c u l a t e d when the zero estimate i s ignored. In the present study, negative v a r i a n c e estimates were d e a l t with two ways. F i r s t , the CG and the estimated v a r i a n c e were c a l c u l a t e d from the mean squares without any adjustment f o r negative v a l u e s . Second, the CG was c a l c u l a t e d with adjustment made f o r negative estimates. The formula f o r the v a r i a n c e estimate was a l s o a d j u s t e d to allow f o r the d e l e t i o n of mean square terms i n the estimated CG. These two treatments of zero components were compared in a s e r i e s of 12 combinations of parameter values and numbers of c o n d i t i o n s . E i g h t c o n d i t i o n s used values of n = 25, P n = 10, n = 2 with CG values of .5 and .9. The remaining / r four c o n d i t i o n s used values of n = 75, n = 20, n = 5 for P i r the same CG va l u e s . In h a l f of these c o n d i t i o n s a2, was set P 1 to zero; i n the other h a l f a2 was set to zero. The pr p o p u l a t i o n v a r i a n c e components used i n t h i s p o r t i o n of the study are presented i n Appendix A. The purpose of t h i s i n v e s t i g a t i o n was to determine which, i f e i t h e r , treatment of zero components performed b e t t e r when a p p l i e d in c o n j u n c t i o n with the i n f e r e n t i a l procedure proposed in t h i s t h e s i s . A S p e c i a l C o n d i t i o n : Dichotomous Data The c o n d i t i o n s d e s c r i b e d to t h i s p o i n t a l l i n v o l v e continuous dependent random v a r i a b l e s . In many instances where persons are r a t i n g themselves or are being t e s t e d , the items cannot be viewed as continuous. Many p e r s o n a l i t y i n v e n t o r i e s are composed of t r u e - f a l s e items; most achievement and a p t i t u d e t e s t items are scored as c o r r e c t or i n c o r r e c t . Because of the prevalence of t h i s type of item i n p s y c h o l o g i c a l t e s t s , a set of c o n d i t i o n s i n which the i n d i v i d u a l item scores were dichotomous were i n v e s t i g a t e d . The data, based on the Design VII random f a c e t s model, were generated i n the same manner as i n the continuous data case. The o b s e r v a t i o n s were then assigned the value 1 i f they were g r e a t e r than or equal to the p o p u l a t i o n mean of zero and the value 0 i f they were l e s s than the mean. T h i s procedure y i e l d s dichotomous items with a p o p u l a t i o n d i f f i c u l t y l e v e l of .5. For t h i s set of analyses, e i g h t c o n d i t i o n s were examined: (n = 25, n = 10, n = 2 ) and (« = 75, n = 20, P i r p i n = 5 ) f o r each of the three values of the CG f o r r continuous data ( p 2 = .5, .7, .9) and (« = 25, n = 10, P 1 n = 5) and f o r two values of the CG ( p 2 = .5, .9). r Dichotomizing the data a l t e r s the value of the p o p u l a t i o n CG. Because of the complicated r e l a t i o n s h i p between the p o p u l a t i o n v a r i a n c e components f o r the continuous and dichotomized data, and the absence of a mathematically t r a c t a b l e a n a l y t i c adjustment f o r d i c h o t o m i z a t i o n , i t was necessary to estimate the p o p u l a t i o n CG values e m p i r i c a l l y . Separate runs were conducted to provide estimates of the parameter values f o r the dichotomized c o n d i t i o n s . For the c o n d i t i o n s with n = 25, P 10,000 r e p l i c a t i o n s were simulated; 5,000 r e p l i c a t i o n s were run f o r c o n d i t i o n s with n = 75. These values were then used P as the p o p u l a t i o n values f o r the t e s t s t a t i s t i c . T e s t i n g the Adequacy of the Asymptotic Variance Expression For each of the c o n d i t i o n s o u t l i n e d above, and summarized in Table 3, 2,500 r e p l i c a t i o n s of the design were simulated. In each r e p l i c a t i o n the transformed sample CG and the v a r i a n c e estimate were computed. The t e s t s t a t i s t i c k = [(1 - p 2 ) 1 7 3 - (1 - p 2 ) l / 3 ] / [ Y a r d - P 2 ) ] 1 / 2 Table 3 Summary of Designs and C o n d i t i o n s for the E m p i r i c a l  Invest i q a t ion 1. Design VII--Both Facets Random P2 -= .5 2 nr = 5 "r = 8 25 75 = 150 ni •• ni 8 = 10 20 = 10 20 = 10 20 30 30 30 10 20 10 20 10 20 30 30 30 10 20 10 20 10 20 30 30 30 P2 -= .7 nr = 2 nr = 5 nr = 8 25 = 75 = 150 ni -ni •• ni = 10 20 = 10 20 = 10 20 30 30 30 10 20 10 20 10 20 30 30 30 10 20 10 20 10 20 30 30 30 P 2 • = .9 nr = 2 nr = 5 nr = 8 > -nP -25 = 75 = 150 ni •• ni •• ni •• = 10 20 = 10 20 = 10 20 30 30 30 10 20 10 20 10 20 30 30 30 10 20 10 20 10 20 30 30 30 Table 3 (continued) 2. Design VII--Item Facet F i x e d P 2 = .5 P 2 = • 7 P 2 = .9 nP ni nr ni nr ni nr 25 20 8 25 20 8 25 20 8 75 30 2 75 30 2 75 30 2 1 50 10 5 1 50 10 5 1 50 10 5 3. Design V I I - -Rater Facet F i n i t e P 2 = .5 P 2 = . 7 P 2 = .9 Sampling F r a c t i o n = = .10 nP ni nr nP ni "r nP ni 25 20 8 25 20 8 25 20 8 75 30 2 75 30 2 75 30 2 1 50 10 5 1 50 10 5 1 50 1 0 5 Sampling F r a c t i o n = = .20 nP ni nr nP ni nr nP " i "r 25 20 8 25 20 8 25 20 8 75 30 2 75 30 2 75 30 2 1 50 1 0 5 1 50 1 0 5 1 50 1 0 5 4. Design V-B Both i F acets Random P 2 = .5 P 2 = • 7 P 2 = .9 nP ni nr nP ni nr "P "/ 25 20 8 25 20 8 25 20 8 75 30 2 75 30 2 75 30 2 1 50 10 5 1 50 10 5 1 50 10 5 Table 3 (continued) 5. Design VII with Zero Variance Component 'hi = 0 = 0 p2 = .5 p2 = .9 P 2 = .5 P 2 _ .9 np nt nr np nt nr nP ni nr np 1 1 / nr 25 10 2 25 10 2 25 10 2 25 10 2 75 20 5 75 20 5 75 20 5 75 20 5 6. Design VII with Dichotomous : Data P 2 = .5 p 2 = .7 P 2 = .9 np ni "r np ni nr nP ni nr 25 20 8 25 20 8 25 20 8 25 10 5 25 1 0 5 75 20 5 75 20 5 75 20 5 was c a l c u l a t e d ; the r e s u l t i n g value of k was compared to p o i n t s of the normal d i s t r i b u t i o n corresponding to the a = .10, .05, .01 t w o - t a i l e d l e v e l s of s i g n i f i c a n c e . Chapter 4 R e s u l t s In t h i s s e c t i o n the r e s u l t s of the Monte C a r l o study are presented. These r e s u l t s are presented f o r the d i f f e r e n t experimental designs i n the order i n which the v a r i a n c e e x p r e s s i o n s were developed i n Chapter Two. Design VII with Both Facets Random The r e s u l t s of the study were examined f o r main e f f e c t s due to the f a c t o r s : sample s i z e , c o n d i t i o n s of Items, c o n d i t i o n s of Raters, and p o p u l a t i o n c o e f f i c i e n t v a l u e s . A d d i t i o n a l l y , the r e s u l t s were examined f o r the presence of two-way i n t e r a c t i o n s among these f a c t o r s . R e s u l t s for the 81 c o n d i t i o n s of Design VII with both f a c e t s random are presented i n Tables 4, 5, and 6. These r e s u l t s are organized a c c o r d i n g to the value of the p o p u l a t i o n c o e f f i c i e n t of g e n e r a l i z a b i l i t y . Summary s t a t i s t i c s - - m e a n p r o p o r t i o n s and o v e r a l l c h i-square v a l u e s - - f o r the main e f f e c t s are presented i n Table 7. No evidence of two-way i n t e r a c t i o n i n the r e s u l t s was seen. The o v e r a l l c h i-square goodness of f i t s t a t i s t i c (with 27 degrees of freedom) i s i n c l u d e d i n these t a b l e s to a s s i s t the reader i n e v a l u a t i n g the r e s u l t s . The s t a t i s t i c i s 27 X 2 = 2500 Z ( p . - p ) 2 / [ p ( 1 - p ) ] , i = 1 1 or 27 X 2 = 2500 I ( a . - a ) 2 / [ a ( 1 - a ) ] , i=1 1 where p. i s the e m p i r i c a l p r o p o r t i o n f o r combination i of sample s i z e , f a c e t c o n d i t i o n s , and p o p u l a t i o n CG, i s the e m p i r i c a l alpha l e v e l , and p i s the mean of the p for a p a r t i c u l a r p r o p o r t i o n averaged over 27 cases; a i s defined s i m i l a r l y . The c r i t i c a l values of the c h i - s q u a r e d i s t r i b u t i o n f o r the .05 and .01 s i g n i f i c a n c e l e v e l s are 40.11 and 46.96 r e s p e c t i v e l y . For t h i s s t a t i s t i c , the nominal alpha or p r o p o r t i o n i s s u b t r a c t e d from the e m p i r i c a l value and i s d i v i d e d by a c o n s i s t e n t e s t i m a t o r of the p o p u l a t i o n standard d e v i a t i o n , given a true n u l l hypothesis. The r e s u l t a n t value i s a normally d i s t r i b u t e d random v a r i a b l e . Squaring and summing these f o r the cases y i e l d s a chi-square random v a r i a b l e with 27 degrees of freedom. It should be noted that these t e s t s are very powerful because of the l a r g e number of r e p l i c a t i o n s used to estimate the sample p r o p o r t i o n . T h e r e f o r e , l a r g e t e s t s t a t i s t i c values, even though s i g n i f i c a n t , are not n e c e s s a r i l y i n d i c a t i v e of poor f i t . These values are i n c l u d e d f o r comparative purposes. The e m p i r i c a l alpha l e v e l s f o r the Design VII r e s u l t s are c l o s e to t h e i r corresponding nominal v a l u e s . From Table 4 f o r p 2 = .5 i t can be seen that a l l 27 observed values f e l l w i t h i n two binomial standard e r r o r s : 1 /2 2[(.10 X .90)/2500] = .012, of the nominal alpha l e v e l of .10. For nominal alpha of .05, Table 4 Proportion of the Standardized Estimates Falling below Selected Percentile Points of the Unit Normal Distribution and Actual Proportion of Type I Errors for p2 = .50 (Design VII, both~Facets  Random) H p n i n r 005 025 050 950 975 995 a: 10 05 01 25 10 2 013 038 060 960 981 997 100 057 016 25 20 2 014 038 062 966 985 1000 096 053 014 25 30 2 012 045 068 965 990 999 102 056 013 75 10 2 012 036 064 953 982 1000 110 055 012 75 20 2 009 032 060 954 980 998 106 052 011 75 30 2 010 039 066 959 984 998 106 055 012 150 10 2 009 029 055 955 978 998 100 051 011 150 20 2 004 029 052 956 978 997 095 051 007 150 30 2 007 027 052 958 981 998 094 046 009 25 10 5 004 022 045 944 974 997 101 049 007 25 20 5 007 025 044 940 971 993 104 054 014 25 30 5 006 025 042 944 976 995 098 048 011 75 10 5 004 027 049 942 972 993 107 055 011 75 20 5 005 024 044 942 972 996 102 053 009 75 30 5 008 027 053 940 975 996 112 052 012 150 10 5 004 024 053 948 975 996 106 049 008 150 20 5 005 023 049 945 973 994 104 050 011 150 30 5 005 022 044 946 972 995 099 050 010 25 10 8 004 019 036 941 972 993 095 047 011 25 20 8 004 028 046 948 972 995 098 056 009 25 30 8 004 022 040 946 972 997 093 050 007 75 10 8 003 024 048 945 974 996 103 050 007 75 20 8 004 020 041 946 975 996 095 046 008 75 30 8 004 021 044 950 978 996 094 043 008 150 10 8 002 017 042 954 976 997 089 041 005 150 20 8 003 021 048 949 975 996 099 046 007 150 30 8 005 021 050 947 978 997 103 043 008 Mean F>i or % 006 027 050 950 977 996 100 050 010 Overall Chi -square (df = 27) 144.1 131.1 91.7 75.8 71.0 91.6 23.0 25.2 45.1 Note. Decimal points omitted in alphas and proportions. The standardized estimate is the value [ ( 1 _ £ 2 ) 1 / 3 . ( i - p 2 ) 1 / 3 ] / [ ^ ( l . 3 2 ) 1 / 3 ] 1 / 2 . Table 5 Proportion of the Standardized Estimates Falling below Selected  Percentile Points of the Unit Normal Distribution and Actual  Proportion of Type I Errors for p z = .70 (Design VII, both Facets  Random) "p n i n f p: 005 025 050 950 975 995 a: 10 05 01 25 10 2 013 045 070 964 986 998 106 058 015 25 20 2 014 043 073 958 982 996 114 060 018 25 30 2 018 046 070 956 979 999 114 067 019 75 10 2 006 038 065 963 984 998 102 054 008 75 20 2 006 024 055 962 983 998 093 041 008 75 30 2 014 042 070 958 982 997 112 060 017 150 10 2 005 031 062 959 984 996 103 048 009 150 20 2 007 031 053 964 980 996 090 050 011 150 30 2 008 030 054 950 974 995 104 056 013 25 10 5 004 022 040 946 973 994 094 049 010 25 20 5 004 020 044 951 977 997 092 043 007 25 30 5 005 022 045 937 971 994 108 051 011 75 10 5 004 024 054 944 970 994 110 054 010 75 20 5 004 025 048 951 974 998 097 052 006 75 30 5 007 026 052 953 978 998 098 048 009 150 10 5 003 021 047 952 974 994 095 048 009 150 20 5 006 030 056 955 979 "994 100 051 012 150 30 5 005 025 044 945 973 994 100 052 011 25 10 8 005 023 048 933 970 993 115 053 012 25 20 8 005 022 046 939 972 996 107 050 009 25 30 8 006 024 044 949 974 994 094 050 012 75 10 8 006 023 043 951 974 994 092 049 012 75 20 8 004 022 049 944 968 994 105 053 010 75 30 8 003 021 047 943 976 997 104 045 006 150 10 8 004 022 044 941 968 994 103 054 010 150 20 8 004 025 050 944 974 996 107 051 008 150 30 8 005 021 043 947 976 997 096 045 008 Mean p i o r a i 006 028 052 . 950 976 996 102 052 011 Overall Chi -square (df = 27) 174.5 169.4 132.4 97.8 76.5 60.2 42.1 44.5 66.8 Note. Decimal points omitted in alphas and proportions. The standardized estimate is the value [ ( 1 - p 2 ) 1 / 3 - ( 1 - p 2 ) 1 / 3 ] / [ V a r ( 1 - p 2 ) 1 / 3 ] 1 / 2 . Table 6 Proportion of the Standardized Estimates Falling below Selected Percentile Points of the Unit Normal Distribution and Actual Proportion of Type I Errors for p * = .90 (Design VII , both Facets Random) n p nL n r p: 005 025 050 950 975 995 a: 10 05 01 25 10 2 016 047 078 969 987 1000 109 060 016 25 20 2 014 043 074 962 986 999 113 058 015 25 30 2 018 054 084 964 986 1000 120 068 018 75 10 2 010 033 060 959 979 998 101 054 012 75 20 2 009 034 060 958 980 998 102 054 011 75 30 2 006 026 047 956 984 996 090 042 010 150 10 2 009 031 060 958 980 996 102 051 013 150 20 2 006 028 054 952 979 997 102 049 009 150 30 2 011 035 060 958 981 997 102 054 013 25 10 5 004 022 041 945 973 994 096 049 010 25 20 5 006 027 048 943 974 995 105 053 011 25 30 5 008 028 046 951 978 995 095 050 013 75 10 5 004 022 043 936 970 996 106 053 008 75 20 5 004 022 052 943 972 994 108 050 010 75 30 5 006 030 058 948 972 996 109 057 010 150 10 5 006 022 045 947 975 994 098 047 012 150 20 5 007 021 048 956 976 997 092 045 101 150 30 5 008 026 054 946 976 997 108 050 011 25 10 8 003 024 043 935 966 992 108 058 011 25 20 8 004 022 042 940 968 994 102 054 010 25 30 8 006 021 040 941 970 994 100 050 012 75 . 10 8 003 022 037 945 976 995 092 046 008 75 20 8 002 023 047 941 969 993 106 054 009 75 30 8 005 025 048 940 975 995 108 050 010 150 10 8 004 020 046 949 974 996 097 046 008 150 20 8 008 024 052 951 980 997 100 045 011 150 30 8 007 026 040 946 972 996 094 054 011 Mean p^ or a i 007 028 052 950 976 996 102 052 011 Overall Chi -square (df = 27) 189.2 192.0 188.4 106.3 95.4 80.3 38.4 43.2 40.9 Note. Decimal points omitted in alphas and proportions. The standardized estimate is the value [ ( l - p 2 ) l / 3 _ ( i - p 2 ) 1 / 3 ] / [ ^ a N r ( l - ? 2 ) 1 / 3 ] 1 / 2 < 54 Table 7 Mean Values and Overall Chi-square Values by Parameter Values and  Facet Conditions (Design VII, both Facets Random) for Selected Percentile Points of the Unit Normal Distribution and Three Levels of Type I Error p: 005 025 050 950 975 995 a: 10 05 01 Coefficient Values p ^ . 5 Mean 006 X2(27) 144.1 027 131.1 050 91.7 950 75.8 977 71.0 996 91.6 100 23.0 050 25.2 010 45.1 p 2=. 7 Mean 006 X2(27) 174.5 028 169.4 052 132.4 950 97.8 976 76.5 996 60.2 102 42.1 052 44.4 011 66.8 p 2=. 9 Mean 007 X2(27) 182.9 028 192.0 052 188.4 950 106.3 976 95.4 996 80.3 102 38.4 052 43.2 011 40.9 Number of Persons n=25 Mean 008 X2(27) 276.6 030 319.1 052 273.6 950 156.1 976 130.6 996 111.1 103 47.8 054 60.4 012 86.3 np=75 Mean 006 X2(27) 110.3 027 103.0 052 96.6 949 76.6 976 ' 68.3 996 78.4 103 36.2 051 31.1 010 34.5 n =150 Mean 006 X2(27) 53.4 025 52.9 050 44.2 951 46.4 976 41.2 996 42.7 099 18.8 049 19.9 010 25.7 Number of Items ni=10 Mean 006 X2(27) 162.1 027 162.1 051 152.6 950 115.8 976 87.4 996 82.8 101 31.4 051 28.4 010 51.4 ^=20 Mean 006 X2(27) 130.0 027 112.3 052 94.6 950 88.7 976 72.3 996 73.4 101 29.4 051 27.3 010 45.2 ni=30 Mean 008 X2(27) 180.2 029 215.6 052 168.3 950 75.4 977 83.7 996 75.9 102 42.7 052 57.3 011 65.2 Table 7 (Continued) 005 025 050 950 975 995 a: 10 05 01 Number of Raters nr=2 Mean 010 036 062 959 982 998 103 054 013 X2(27) 292.6 344.6 266.5 178.5 232.1 318.1 48.3 71.2 45.2 nr=5 Mean 005 024 048 946 974 995 102 050 010 X2(27) 27.1 20.9 40.0 53.8 19.7 28.6 25.8 12.7 21.4 nr=8 Mean 004 022 045 945 973 995 100 049 009 X2(27) 35.2 38.2 70.4 67.9 38.9 29.6 29.0 27.5 30.4 Note. Decimal points omitted in alphas and proportions. The standardized estimate is the value [ ( 1 _ 3 2 ) 1 / 3 _ ( 1 - p 2 ) l / 3 ] / [ ^ ( 1 _ ^ 2 ) 1 / 3 ] 1 / 2 e 26 of 27 observed values f e l l w i t h i n two standard e r r o r s : 1 /2 2[(.05 X .95)/2500] ' = .0087. For nominal alpha of .01, 23 of 27 observed values f e l l w i t h i n two standard e r r o r s : 2[(.01 X . 9 9 ) / 2 5 0 0 ] 1 / 2 = .0039. For the c o n d i t i o n s i n which p2 = .7 (Table 5), 24 of 27 observed values f e l l w i t h i n two standard e r r o r s of nominal alpha of .10, 23 of 27 f e l l w i t h i n two standard e r r o r s of .05, and 21 of 27 f e l l w i t h i n two standard e r r o r s of .01. For c o e f f i c i e n t values of p2 = .9 (Table 6), 25 of 27 observed values f e l l w i t h i n two standard e r r o r s of nominal alpha l e v e l of .10; 25 of 27 f e l l w i t h i n two standard e r r o r s of nominal .05 alpha, and 24 of 27 f e l l w i t h i n two standard e r r o r s of nominal .01 alpha. From i n s p e c t i o n of Table 7 i t can be seen that the e m p i r i c a l p r o p o r t i o n of s t a n d a r d i z e d estimates f a l l i n g below any given p e r c e n t i l e value g e n e r a l l y was very c l o s e to the corresponding nominal v a l u e . The set of r e s u l t s showing the g r e a t e s t d i s c r e p a n c y from nominal p e r c e n t i l e values i s that based on two r a t e r s (n = 2 ) . Although the o v e r a l l e m p i r i c a l alpha l e v e l s tended to be reasonably c l o s e to t h e i r nominal v a l u e s , the e m p i r i c a l p r o p o r t i o n s i n d i c a t e asymmetry in the r e s u l t s . More of the s t a n d a r d i z e d estimates f e l l i n t o the lower t a i l of the normal d i s t r i b u t i o n than would be expected by chance. In a d d i t i o n , the c h i - s q u a r e s t a t i s t i c s f o r the n = 2 c o n d i t i o n s are l a r g e r than those f o r other c o n d i t i o n s . T h i s r e s u l t suggests that there was g r e a t e r b i a s i n these r e s u l t s than was seen f o r other values of n . The reason for t h i s r b i a s i s not c l e a r . I t i s p o s s i b l e that b i a s i n the s t a n d a r d i z e d estimate i s r e s p o n s i b l e f o r the asymmetry in the r e s u l t s . Another p o s s i b i l i t y i s that f o r cases with small numbers of f a c e t c o n d i t i o n s the normal d i s t r i b u t i o n i s not a good model f o r the d i s t r i b u t i o n of s t a n d a r d i z e d e s t i m a t e s . I t i s a l s o p o s s i b l e that the s t a t i s t i c has not reached i t s asymptotic value i n these c o n d i t i o n s . To i n v e s t i g a t e t h i s problem the d i s t r i b u t i o n of the normalized c o e f f i c i e n t s and t h e i r v a r i a n c e s would need to be examined. Design VII with F i x e d Item Facet The r e s u l t s f o r the nine c o n d i t i o n s i n which the Item f a c e t was f i x e d are presented i n Table 8. These r e s u l t s g e n e r a l l y i n d i c a t e reasonably good maintenance of Type I e r r o r c o n t r o l . However, these e m p i r i c a l r e s u l t s do not appear to be q u i t e as c l o s e to nominal l e v e l s as were those obtained f o r Design VII with both f a c e t s random. The c o n d i t i o n s with n =2 d e v i a t e d most from the nominal r p r o p o r t i o n s . Design VII with F i n i t e Rater Facet The r e s u l t s f o r the 18 c o n d i t i o n s i n which Raters was a f i n i t e f a c e t are presented i n Table 9. O v e r a l l , these r e s u l t s i n d i c a t e good c o n t r o l of Type I e r r o r f o r both 58 Table 8 Proportion of the Standardized Estimates Falling below Selected  Percentile Points of the Unit Normal Distribution and Actual  Proportion of Type I Errors (Design VII, Item Facet Fixed) "p n i n r P2 p: 005 025 050 950 975 995 a: 10 05 01 25 20 8 .5 008 026 049 941 968 994 108 057 014 25 20 8 .7 004 024 042 942 969 993 100 055 011 25 20 8 .9 007 023 052 937 970 992 115 053 014 75 30 2 .5 008 038 069 962 980 996 106 058 012 75 30 2 .7 012 042 069 962 979 . 998 106 063 014 75 30 2 .9 006 027 049 966 983 998 084 044 008 150 10 5 .5 006 028 055 942 970 994 113 057 012 150 10 5 .7 002 017 046 943 970 992 102 047 010 150 10 5 .9 004 024 045 943 970 992 102 053 012 Note. Decimal points omitted. The standardized estimate is the value [ ( 1 - p 2 ) 1 / 3 - ( 1 - p 2 ) 1 / 3 ] / [ V a " r ( l - p 2 ) 1 / 3 ] 1 / 2 . Table 9 Proportion of the-Standardized Estimates Falling below Selected  Percentile Points of the Unit Normal Distribution and Actual  Proportion of Type I Errors (Design VII, Rater Facet Finite) rip nL n r p 2 p: 005 025 050 950 975 995 c: 10 05 01 Sampling Fraction = .10 25 20 8 .5 005 024 045 944 970 992 101 054 011 25 20 8 .7 005 022 047 944 968 992 102 055 013 25 20 8 .9 002 020 042 940 970 993 104 050 009 75 30 2 .5 009 033 055 958 981 998 097 052 011 75 30 2 .7 007 030 056 950 973 996 106 058 011 75 30 2 .9 011 034 063 953 980 999 110 055 012 150 10 5 .5 008 030 050 946 974 994 104 057 015 150 10 5 .7 005 029 058 951 969 993 107 060 012 150 10 5 .9 005 022 041 954 974 996 088 048 009 Sampling Fraction = .20 25 20 8 .5 005 024 048 939 967 992 109 058 012 25 20 8 .7 006 020 041 940 967 993 101 054 014 25 20 8 .9 004 024 044 937 971 995 107 053 009 75 30 2 .5 006 027 054 956 980 996 098 048 010 75 30 2 .7 008 031 059 948 975 996 111 056 012 75 30 2 .9 007 035 059 956 980 994 102 056 013 150 10 5 .5 004 022 047 950 971 994 096 052 010 150 10 5 . .7 004 020 045 938 968 994 107 052 010 150 10 5 .9 006 025 056 946 975 996 110 050 010 Note. Decimal points omitted. The standardized estimate is the value [ < 1 - p 2 ) 1 / 3 - ( 1 - p 2 ) 1 / 3 ] / [ V a > r ( 1 - p 2 ) 1 / 3 ] 1 / 2 . sampling f r a c t i o n s used i n the study. The s i z e of the sampling f r a c t i o n does not seem to be i n f l u e n t i a l . Again, in these r e s u l t s , the c o n d i t i o n s with n = 2 d e v i a t e d most from r the nominal p r o p o r t i o n s . Design V-B with Both Facets Random R e s u l t s f o r the nine c o n d i t i o n s of Design V-B i n which the Rater f a c e t was nested w i t h i n persons are presented in Table 10. These r e s u l t s again i n d i c a t e f a i r l y good c o n t r o l of Type I e r r o r . The c o n d i t i o n s i n which n =2 demonstrated r the same s o r t of asymmetry as was seen in other r e s u l t s . The Treatment of Negative Variance Components Negative v a r i a n c e component estimates can a r i s e , because of sampling e r r o r , when a v a r i a n c e component i s equal to zero in the p o p u l a t i o n . Two ways of d e a l i n g with negative v a r i a n c e component estimates were i n v e s t i g a t e d in t h i s t h e s i s . For the f i r s t approach no adjustment was made fo r n e g a tive e s t i m a t e s ; the CG and the a s s o c i a t e d variance estimate were c a l c u l a t e d from the estimated mean squares. For the second approach negative estimates of v a r i a n c e components were set to zero. The zero components were i n c o r p o r a t e d i n the formulas f o r other v a r i a n c e component e s t i m a t e s . As a r e s u l t of t h i s procedure, when a negative estimate was encountered, both the CG and i t s estimated v a r i a n c e d i f f e r e d from those estimated by the f i r s t approach. When no negative components are encountered the Table 10 Proportion of the Standardized Estimates Falling below Selected  Percentile Points of the Unit Normal Distribution and Actual  Proportion of Type I Errors (Design V-B, both Facets Random) "p n i n r P 2 p: 005 025 050 950 975 995 a: 10 05 01 25 20 8 .5 005 022 042 935 972 995 106 050 010 25 20 8 .7 002 020 043 948 972 993 095 048 010 25 20 8 .9 006 022 043 930 967 994 113 055 012 ' 75 30 2 .5 012 036 060 960 982 998 100 054 013 75 30 2 .7 010 038 066 954 980 999 111 058 011 75 30 2 .9 006 032 063 951 980 995 112 052 011 150 10 5 .5 006 026 051 941 969 995 110 058 011 150 10 5 .7 006 026 051 957 981 998 094 045 008 150 10 5 .9 006 023 047 951 975 993 096 048 012 Note. Decimal points omitted. The standardized estimate is the value [ ( 1 - p 2 ) 1 / 3 - ( 1 - p 2 ) 1 / 3 ] / [ V a X r ( l - p 2 ) 1 / 3 ] 1 / 2 > two approaches y i e l d i d e n t i c a l e s t i m a t e s . A t o t a l of e i g h t c o n d i t i o n s were simulated to compare the two approaches. In the four c o n d i t i o n s i n which n = 25 P two separate runs of 2,500 r e p l i c a t i o n s were conducted for each c o n d i t i o n . The runs d i f f e r e d only i n the magnitude of t h e i r p o p u l a t i o n v a r i a n c e components. Because the r e s u l t s f o r the two runs d i d not d i f f e r a p p r e c i a b l y the r e s u l t s were combined i n t o a s i n g l e c o n d i t i o n based on 5,000 r e p l i c a t i o n s . In h a l f of the c o n d i t i o n s i n v e s t i g a t e d the a 2 . p i component was set to zero; the component was set to zero in the remaining four c o n d i t i o n s . In a l l c o n d i t i o n s t h i s r e s u l t e d i n negative component estimates i n approximately h a l f of the r e p l i c a t i o n s . The r e s u l t s comparing the two approaches to the treatment of negative v a r i a n c e component estimates are presented i n Table 11. The two approaches d i f f e r e d very l i t t l e i n t h e i r q u a l i t y of Type I e r r o r c o n t r o l . There was some tendency f o r the zero s u b s t i t u t i o n approach (Approach 2) to y i e l d fewer r e j e c t i o n s i n the lower t a i l of the normal d i s t r i b u t i o n than were given by the f i r s t approach. However, the d i f f e r e n c e i s s l i g h t and l i k e l y i s of l i t t l e p r a c t i c a l consequence. Design VII with Dichotomous Data E i g h t c o n d i t i o n s were run i n which the item scores were converted to dichotomous data by a s s i g n i n g to scores below Table 11 Proportion of the Standardized Estimates Falling below Selected  Percentile Points of the Unit Normal Distribution and Actual  Proportion of Type I Errors for Two Treatments of Negative Variance  Component Estimates (Design VII, Both Facets Random) n p nL n r p 2 p: 005 025 050 950 975 995 a: 10 05 01 25 10 2 .5 ff£i-0. Approach 1 017 048 077 962 986 999 114 062 018 Approach 2 017 048 075 962 985 999 113 062 018 25 10 2 .5 o=,r=0 Approach 1 004 021 044 948 975 995 096 046 009 Approach 2 003 019 038 942 971 994 096 047 009 25 10 2 .9 0-^ =0 Approach 1 015 040 071 967 986 999 104 054 016 Approach 2 014 038 066 964 985 998 103 052 016 25 10 2 .9 opr=0 Approach 1 007 028 053 947 977 996 108 051 011 Approach 2 004 020 039 937 969 996 102 051 009 Table 11 (continued) np n i n r p 2 p: 005 025 050 950 975 995 c: 10 05 01 75 20 5 .5 opi=0 Approach 1 Approach 2 006 006 024 024 052 050 944 943 975 975 994 994 108 107 049 049 011 012 75 20 5 .5 o r^=0 Approach 1 Approach 2 004 004 026 024 043 042 937 936 968 967 991 990 106 106 058 057 013 013 75 20 5 .9 opi=0 Approach 1 Approach 2 008 008 024 024 050 050 951 951 976 975 996 996 100 099 048 049 012 012 75 20 5 .9 opr=0 Approach 1 Approach 2 003 000 028 017 052 030 949 930 974 966 994 993 102 100 054 051 009 007 Note. Approach 1: no adjustment for negative variance component estimates; Approach 2: zero substitution for negative estimates. Conditions where n p = 25 are based on 5000 replications. Decimal points omitted. The standardized estimate is the value [ ( 1 - p 2 ) 1 / 3 - ( 1 - p 2 ) 1 / 3 ] / [ v Q r ( 1 - p 2 ) l / 3 ] l / 2 . the p o p u l a t i o n mean the value zero and to scores at or above the mean the value one. The transformed p o p u l a t i o n c o e f f i c i e n t used i n c o n s t r u c t i n g the s t a n d a r d i z e d estimate was estimated by s i m u l a t i n g the c o e f f i c i e n t . The r e s u l t s for c o n d i t i o n s using dichotomous data are presented i n the upper p o r t i o n of Table 12. In the lower p o r t i o n of t h i s t a b l e , r e s u l t s f o r the corresponding c o n d i t i o n s using continuous data are presented f o r purpose of comparison. In the three c o n d i t i o n s where n = 2 5 and n =2 the P r obtained alpha l e v e l s are i n excess of the nominal l e v e l s . I n s p e c t i o n of the p r o p o r t i o n s f a l l i n g below p e r c e n t i l e p o i n t s i n each t a i l i n d i c a t e s marked asymmetry in the r e s u l t s . Many more st a n d a r d i z e d estimates f e l l i n t o the lower t a i l of the normal d i s t r i b u t i o n than f e l l i n t o the upper t a i l . When the number of r a t e r s was i n c r e a s e d to f i v e the r e s u l t s improved somewhat. The e m p i r i c a l alpha l e v e l s were c l o s e r to nominal alpha although asymmetry was s t i l l apparent i n these r e s u l t s . The r e s u l t s f o r n =75 and P n = 5 were s i m i l a r to those obtained f o r n =25 and r P n = 5 . T h i s f i n d i n g suggests that the s i z e of n p l a y s a r r more c r u c i a l r o l e i n determining the q u a l i t y of Type I e r r o r c o n t r o l than does the magnitude of n . T h i s p a t t e r n i s P c o n s i s t e n t with that found with continuous data. G e n e r a l l y these r e s u l t s are not as good as those obtained with continuous data. Table 12 Proportion of the Standardized Estimates Falling below Selected  Percentile Points of the Unit Normal Distribution and Actual  Proportion of Type I Errors for Dichotomized Data and Comparable  Conditions with Continuous Data (Design VII, both Facets Random) n p n r p 2 p: 005 025 050 950 975 995 a: 10 05 01 Dichotomized Data 2 5 1 0 2 . 4 0 4 4 0 2 0 0 5 1 0 8 6 9 6 8 9 8 8 9 9 9 1 1 8 0 6 3 0 2 1 2 5 1 0 2 . 5 9 9 5 0 1 9 0 4 8 0 7 6 9 7 5 9 9 4 1 0 0 0 1 0 1 0 5 5 0 1 9 2 5 1 0 2 . 7 9 5 6 0 2 3 0 6 0 0 8 9 9 5 9 9 8 2 9 9 8 1 3 0 0 7 8 0 2 6 2 5 1 0 5 . 4 1 5 5 0 1 0 0 4 0 0 6 7 9 5 5 9 7 9 9 9 8 1 1 2 0 6 1 0 1 2 2 5 1 0 5 . 8 4 3 1 0 0 8 0 2 7 0 5 6 9 7 2 9 8 5 9 9 8 0 8 4 0 4 2 0 1 0 7 5 2 0 5 . 4 5 8 2 0 0 9 0 3 4 0 6 3 9 6 0 9 8 5 9 9 8 1 0 4 0 4 9 0 1 2 7 5 2 0 5 . 6 5 9 3 0 0 4 0 3 0 0 5 8 9 6 1 9 7 9 9 9 7 0 9 8 0 5 1 0 0 7 7 5 2 0 5 . 8 6 5 6 0 0 5 0 2 8 0 5 4 9 5 8 9 8 6 9 9 7 0 9 6 0 4 2 0 0 8 Continuous Data 2 5 1 0 2 . 5 0 1 3 0 3 8 0 6 0 9 6 0 9 8 1 9 9 7 1 0 0 0 5 7 0 1 6 2 5 1 0 2 . 7 0 1 3 0 4 5 0 7 0 9 6 4 9 8 6 9 9 8 1 0 6 0 5 8 0 1 5 2 5 1 0 2 . 9 0 1 6 0 4 7 0 7 8 9 6 9 9 8 7 1 0 0 0 1 0 9 0 6 0 0 1 6 2 5 1 0 5 . 5 0 0 4 0 2 2 0 4 5 9 4 4 9 7 4 9 9 7 1 0 1 0 4 9 0 0 7 2 5 1 0 5 . 9 0 0 4 0 2 2 0 - i l 9 4 5 9 7 3 9 9 4 0 9 6 0 4 9 0 1 0 7 5 2 0 5 . 5 0 0 5 0 2 4 0 v 4 9 4 2 9 7 2 9 9 6 1 0 2 0 5 3 0 0 9 7 5 2 0 5 . 7 0 0 4 0 2 5 0 4 8 9 5 1 9 7 4 9 9 8 0 9 7 0 5 2 0 0 6 7 5 2 0 5 . 9 0 0 4 0 2 2 0 5 2 9 4 3 9 7 2 9 9 4 1 0 8 0 5 0 0 1 0 Note. Decimal points omitted. The standardized estimate is the value [ ( 1 - P 2 ) 1 / 3 - ( 1 - p 2 ) l / 3 ] / [ V a \ ( 1 - p 2 ) 1 / 3 ] 1 / 2 . Chapter 5 Conclusions and a Worked Example of the Procedures The f i n a l chapter of t h i s t h e s i s has two purposes. F i r s t , the i m p l i c a t i o n s of the e m p i r i c a l r e s u l t s and d i r e c t i o n s f o r f u t u r e research are presented. The chapter then concludes with an example of how the asymptotic v a r i a n c e e x p r e s s i o n can be a p p l i e d to data. Some Observations about the E m p i r i c a l R e s u l t s The asymptotic v a r i a n c e e x p r e s s i o n . The major purpose of t h i s r e s e a r c h was to develop asymptotic v a r i a n c e e x p r e s s i o n s for CGs under a v a r i e t y of r e a l i s t i c experimental designs. The value of such an e x p r e s s i o n i s twofold. F i r s t , the v a r i a n c e e x p r e s s i o n can be used to c o n s t r u c t a confidence i n t e r v a l f o r the c o e f f i c i e n t . Second, the v a r i a n c e can be used to develop procedures f o r making s t a t i s t i c a l i n f e r e n c e s about sample c o e f f i c i e n t s . The r e s u l t s of the e m p i r i c a l s i m u l a t i o n s performed in t h i s t h e s i s i n d i c a t e that the v a r i a n c e expressions for the transformed CGs performed w e l l i n c o n t r o l l i n g Type I e r r o r in the m a j o r i t y of c o n d i t i o n s examined. The r e s u l t s suggest that the v a r i a n c e expressions can be used to set confidence i n t e r v a l s f o r designs l i k e those used i n the e m p i r i c a l study. Although the r e s u l t s were not as p r e c i s e f o r cases with n^ = 2 and with dichotomous data as they were for other c o n d i t i o n s , the approximate c o n f i d e n c e i n t e r v a l s i n these l e s s optimal c o n d i t i o n s s t i l l p r o v i d e a good estimate of the range of the parameter v a l u e . A more c a u t i o u s approach l i k e l y i s warranted i n using the asymptotic v a r i a n c e e x p r e s s i o n f o r hypothesis t e s t i n g . R e s u l t s f o r c o n d i t i o n s with dichotomous data and f o r a facet r e p resented by two c o n d i t i o n s i n d i c a t e d unequal r e j e c t i o n r a t e s in the two t a i l s of the normal d i s t r i b u t i o n when hypotheses of the form H 0: p 2 = p 2 are t e s t e d with the s t a t i s t i c k = [(1 - p 2 ) ] / 2 - (1 - p 2 , ) 1 7 3 ] / [vTrd - p 2 ) ] 1 / 2 . The t e s t s t a t i s t i c i s more l i k e l y to r e j e c t a true n u l l h y p o t hesis when the estimated CG i s l a r g e r than the hypothesized p o p u l a t i o n parameter than when the estimated value i s smaller than the hypothesized parameter (a l a r g e s t a n d a r d i z e d value i s a s s o c i a t e d with a small c o e f f i c i e n t v a l u e ) . To i l l u s t r a t e , c o n s i d e r the case where n = 25, P n = 30, n =2, and p 2 = .9. T e s t i n g the hypothesis i r 1 H 0: p2 = .9 when p 2 > p 2 , , the e m p i r i c a l r e s u l t s i n d i c a t e t h a t the true n u l l h ypothesis w i l l be r e j e c t e d 10.8% of the time at the nominal 5% alpha l e v e l . In c o n t r a s t , when t e s t i n g the hypothesis H 0: p 2 = .9 when p 2 < .9 , the true n u l l h y p o thesis w i l l be r e j e c t e d only 2.8% of the time at the nominal 5% l e v e l . ( I t should be kept i n mind, i n connection with the above, t h a t , i f we c o n s i d e r t h i s as a standard t w o - t a i l e d t e s t and do not s p e c i f y the d i r e c t i o n of the estimate r e l a t i v e to the hypothesized parameter the a c t u a l o v e r a l l Type I e r r o r , at the nominal .05 l e v e l , i s .068.) Treatment of negative v a r i a n c e component estimates. A side study was conducted to i n v e s t i g a t e the e f f e c t s of two treatments of negative v a r i a n c e component e s t i m a t e s . The two methods r e s u l t e d i n very s i m i l a r Type I e r r o r c o n t r o l . This f i n d i n g suggests that the technique f o r d e r i v i n g the asymptotic v a r i a n c e expression f o r the transformed c o e f f i c i e n t can be a p p l i e d with e i t h e r method of d e a l i n g with negative v a r i a n c e component e s t i m a t e s . However, p r a c t i c a l concerns i n d i c a t e that negative v a r i a n c e component estimates should be l e f t n e g a t i v e . When a negative estimate i s encountered i n p r a c t i c e and r e p l a c e d with a zero value, the e x p r e s s i o n f o r e s t i m a t i n g the CG changes. Then the exp r e s s i o n f o r the asymptotic v a r i a n c e estimate, d e r i v e d by a p p l i c a t i o n of the d e l t a method, i s d i f f e r e n t from that d e r i v e d i n the absence of a negative v a r i a n c e estimate. To f u r t h e r complicate matters, the form of the new vari a n c e e x p r e s s i o n depends upon which v a r i a n c e component i s nega t i v e . For the p r a c t i t i o n e r , i t i s more s t r a i g h t f o r w a r d to ignore negative v a r i a n c e component estimates and work only with the mean squares. 7 0 I m p l i c a t i o n s of the Study fo r the Use of G e n e r a l i z a b i l i t y  Theory The r e s u l t s of the e m p i r i c a l study i n v e s t i g a t i n g the adequacy of the proposed asymptotic v a r i a n c e expression have i m p l i c a t i o n s f o r p r a c t i c a l a p p l i c a t i o n of the technique. The p r e v i o u s l y d i s c u s s e d r e s u l t s f o r the treatment of negative v a r i a n c e component estimates have i m p l i c a t i o n s f o r p r a c t i c e . These r e s u l t s i n d i c a t e that negative estimates can be l e f t n egative without reducing the p r e c i s i o n of the technique. R e s u l t s obtained with small numbers of r a t e r s and with dichotomous data a l s o have i m p l i c a t i o n s f o r the a p p l i c a t i o n of the techniques developed i n t h i s t h e s i s . Small number of r a t e r s . I t was seen that the r e s u l t s f o r c o n d i t i o n s with n = 2 tended to be asymmetric. The n u l l h y p o t hesis was r e j e c t e d too o f t e n when the sample value of the CG was l a r g e r than the true p o p u l a t i o n parameter. C o e f f i c i e n t s lower than the true value, however, were not r e j e c t e d as o f t e n as would be expected. One s o l u t i o n to t h i s problem i s to plan s t u d i e s using more than two f a c e t c o n d i t i o n s . However, in many circumstances t h i s s o l u t i o n i s not a p r a c t i c a l one. For example, i t may be too expensive to i n c r e a s e the number of r a t e r s used i n a study beyond two. S i m i l a r l y , i f a study has an Occasion f a c e t , as i n a t e s t - r e t e s t design, i t may not be p o s s i b l e or d e s i r a b l e to have s u b j e c t s repeat the t e s t or q u e s t i o n n a i r e more than once. Of course, when i t i s p o s s i b l e to do so, s t u d i e s 71 should use more than two f a c e t c o n d i t i o n s . When t h i s i s not p r a c t i c a l , the researcher can use the technique and acknowledge the approximate nature of the r e s u l t s . A l t e r n a t i v e l y , she or he can a d j u s t the c r i t i c a l value for the s i g n i f i c a n c e t e s t a c c o r d i n g to whether the observed r e s u l t i s lower or higher than the hypothesized parameter v a l u e . For example, i f the observed CG i s l a r g e r than the hypothesized value, the researcher c o u l d t e s t the n u l l h ypothesis at the .035 l e v e l of s i g n i f i c a n c e i n s t e a d of at the .05 l e v e l . S i m i l a r l y , i f the observed CG i s smaller than the hypothesized value, a l e s s s t r i n g e n t alpha l e v e l can be used to t e s t the n u l l h y p o t h e s i s . Dichotomous data. The r e s u l t s f o r c o n d i t i o n s with dichotomous data a l s o i n d i c a t e d that r e s u l t s f a l l i n g i n t o the r e j e c t i o n r e g ion were not evenly d i v i d e d between the two t a i l s of the normal d i s t r i b u t i o n . F u r t h e r , with two r a t e r s , the Type I e r r o r c o n t r o l was not maintained. To overcome these shortcomings, the re s e a r c h e r , again, c o u l d adjust c r i t i c a l values a c c o r d i n g to whether the sample CG i s higher or lower than the hypothesized v a l u e . For example, where n = 25, n =10, n =2, and p 2 = .7956, the n u l l p i r 1 hypo t h e s i s would be r e j e c t e d approximately 12% of the time at the nominal 5% alpha l e v e l when the sample CG i s l a r g e r than the p o p u l a t i o n v a l u e . When the sample value i s smaller than the CG the true n u l l h y p o t h e s i s i s r e j e c t e d only 3.6% of the time. Again, i f p o s s i b l e , r e s e a r c h e r s should t r y to a v o i d using small numbers of f a c e t c o n d i t i o n s . However, t h i s approach to the problem i s not always f e a s i b l e and because of the prevalence of dichotomous items i n p s y c h o l o g i c a l r e s e a r c h , the researcher can deal with the asymmetry by a d j u s t i n g l e v e l s of s i g n i f i c a n c e a c c o r d i n g to the magnitude of the sample c o e f f i c i e n t . With a sample c o e f f i c i e n t l a r g e r than the hypothesized p o p u l a t i o n parameter, a 5% alpha l e v e l t e s t should be conducted at a more s t r i n g e n t l e v e l of s i g n i f i c a n c e . In the case of a c o e f f i c i e n t smaller than the hypothesized parameter, a l e s s s t r i n g e n t than nominal l e v e l of s i g n i f i c a n c e would be used. L i m i t a t i o n s of the Present Study As i n any e m p i r i c a l s i m u l a t i o n study, there are a number of l i m i t a t i o n s to t h i s i n v e s t i g a t i o n imposed by the c o n d i t i o n s s t u d i e d . One such l i m i t a t i o n i s that i t i s u n c e r t a i n what minimum number of f a c e t c o n d i t i o n s i s r e q u i r e d to produce r e a l l y good Type I e r r o r c o n t r o l . Two f a c e t c o n d i t i o n s tended to produce r e s u l t s that were asymmetric while the use of f i v e c o n d i t i o n s produced good e r r o r c o n t r o l i n both t a i l s of the d i s t r i b u t i o n . However, i t i s not known whether three or four c o n d i t i o n s would a l s o y i e l d a c c e p t a b l e r e s u l t s . I t seems r e l a t i v e l y c e r t a i n that three f a c e t c o n d i t i o n s would r e s u l t i n b e t t e r Type I e r r o r c o n t r o l than that r e a l i z e d with two c o n d i t i o n s . Another l i m i t a t i o n of the present study i s that i t focused on a small number of ANOVA designs. Numerous, more complicated, ANOVA designs are p o s s i b l e w i t h i n a measurement c o n t e x t . Although the r e s u l t s of t h i s study i n d i c a t e d that the proposed procedures perform w e l l with two f a c e t designs, i t i s u n c e r t a i n whether the technique would perform well with designs having l a r g e r numbers of f a c e t s . I t i s p o s s i b l e , f o r example, that the number of estimated mean squares needed f o r a three f a c e t design would produce large amounts of unsystematic e r r o r that would cause the s t a t i s t i c to behave e r r a t i c a l l y . One f u r t h e r r e s t r i c t i o n on the a p p l i c a t i o n of the present research r e s u l t s i s that the data used i n these s i m u l a t i o n s were sampled from p o p u l a t i o n s with u n d e r l y i n g normal d i s t r i b u t i o n s . Although the a n a l y s i s of v a r i a n c e i s robust with respect to v i o l a t i o n of the n o r m a l i t y assumption (see e.g., G l a s s , Peckham, & Sanders, 1972), i t i s not c e r t a i n that the same c l a i m can be made about the performance of the estimated CG and i t s a s s o c i a t e d variance e x p r e s s i o n . Further r e s e a r c h i s necessary to determine the e f f e c t of the v i o l a t i o n of the n o r m a l i t y assumption on the performance of these e s t i m a t e s . Suggestions f o r Future Research The above-mentioned l i m i t a t i o n s of the present study should be addressed i n f u t u r e r e s e a r c h . One such study should i n v e s t i g a t e the adequacy of the asymptotic variance e x p r e s s i o n with three and four f a c e t c o n d i t i o n s . Research i s a l s o needed to i n v e s t i g a t e the s u i t a b i l i t y of the procedure to more complicated ANOVA models. Studi e s should be undertaken i n which three and four f a c e t s (both nested and c r o s s e d designs) are used. Future r e s e a r c h should a l s o i n v e s t i g a t e the e f f e c t s of v i o l a t i n g the ANOVA assumption of the data having an u n d e r l y i n g normal d i s t r i b u t i o n . Experimental data sometimes do not meet t h i s requirement. I t would be d e s i r a b l e to know how the technique performs when the n o r m a l i t y assumption i s not met. S i m u l a t i o n s with data drawn from such d i s t r i b u t i o n s as the uniform, lognormal, and e x p o n e n t i a l would h e l p determine the importance of the no r m a l i t y assumption f o r a p p l i c a t i o n of the technique. The present r e s e a r c h was concerned with t e s t i n g a n u l l h y p o t h e s i s about the value of a s i n g l e p o p u l a t i o n CG. The technique can be extended t o develop t e s t s f o r two or more independent sample c o e f f i c i e n t s . For two c o e f f i c i e n t s , a t e s t analogous to the two independent group t - t e s t c o u l d be used; f o r m u l t i p l e c o e f f i c i e n t s the technique d e s c r i b e d by M a r a s c u i l o (1966) c o u l d be employed. The adequacy of these e x t e n s i o n s r e q u i r e s f u r t h e r i n v e s t i g a t i o n . Another extension of t h i s r e s e a r c h would be the development of a c o v a r i a n c e e x p r e s s i o n f o r two dependent sample c o e f f i c i e n t s . T h i s e x p r e s s i o n c o u l d then be used to t e s t hypotheses about c o e f f i c i e n t s based on data o b t a i n e d from dependent samples. For two dependent c o e f f i c i e n t s a technique analogous to the dependent sample t - t e s t c o u l d be used. M u l t i p l e dependent c o e f f i c i e n t s c o u l d be t e s t e d with a q u a d r a t i c form c h i - s q u a r e s t a t i s t i c . Again, such s t a t i s t i c s would r e q u i r e e m p i r i c a l v e r i f i c a t i o n before t h e i r use co u l d be recommended. As a f i n a l suggestion f o r f u t u r e reseach, a study could be undertaken to i n v e s t i g a t e power of the procedures. The present r e s e a r c h was concerned with c o n t r o l of Type I e r r o r under a v a r i e t y of s i t u a t i o n s . I n v e s t i g a t i o n s of the power of the technique would give r e s e a r c h e r s an i n d i c a t i o n of the sample s i z e s and c o n d i t i o n numbers necessary f o r adequate power when using the technique i n f e r e n t i a l l y . A Worked Example using the Present Procedure The a p p l i c a t i o n of one of the asymptotic v a r i a n c e e x p r e s s i o n s developed i n t h i s t h e s i s i s i l l u s t r a t e d in t h i s s e c t i o n using data from the Schroeder et a l . (1983) study. The random e f f e c t s ANOVA r e s u l t s were presented i n Table 2 . By s u b s t i t u t i n g the observed mean squares i n t o the formulas f o r the CG and the asymptotic v a r i a n c e e x p r e s s i o n s (eq. 4 and 5), the f o l l o w i n g values are obtained: p 2 = .860, 1/3 = .519, Var (1-p 2) 1/3 = .00197. The 95% confidence i n t e r v a l i s c o n s t r u c t e d as P r { [ ( 1 - p 2 ) 1/3 - 1.96a ] < (1-p 2) * 1/3 < [ ( l - p 2 ) 1 / 3 + 1.96a*]} = .95, * <s\ 1/3 1/2 where a = [ V a r ( l - p 2 ) ] A f t e r a l g e b r a i c m a n i p u l a t i o n , t h i s e x p r e s s i o n can be w r i t t e n as Pr{l - [ ( 1 - p 2 ) l / 3 + 1.965*] 3 < p 2 1/3 * 3 < 1 - [ ( 1 - p 2 ) 7 - 1.965 ] } = .95. S u b s t i t u t i o n of the observed values i n t o t h i s equation gives the i n t e r v a l Pr{.777 < p 2 < .919} = .95. In a d d i t i o n , a simple n u l l hypothesis of the form H 0: p 2 = pi (where p 2 i s some hypothesized value) can be t e s t e d with the s t a t i s t i c . T e s t i n g the n u l l hypothesis H 0: p 2 = .90 y i e l d s a value of k = [(1 - . 8 6 ) 1 / 3 - (1 - . 9 0 ) 1 / S ] / ( . 0 0 1 9 7 ) 1 / 2 = 1.235, which i s not s i g n i f i c a n t at the .10 l e v e l of s i g n i f i c a n c e ( t w o - t a i l e d t e s t ) . Note here that t h i s i s a c o n s e r v a t i v e t e s t ; n = 2 and the estimated CG i s s m a l l e r than the r hypothesized v a l u e . References Brennan, R.L. , & Kane, M.T. (1979). G e n e r a l i z a b i l i t y theory: A review of b a s i c concepts, i s s u e s , and procedures. In R.E. Traub (ed.), New d i r e c t i o n s i n t e s t i n g and  measurement. San F r a n c i s c o : Jossey-Bass. Cain, P.S., & Green, B.F. (1983). R e l i a b i l i t i e s of s e l e c t e d r a t i n g s a v a i l a b l e from the D i c t i o n a r y of Occupational  T i t l e s . J o u r n a l of A p p l i e d Psychology, 68, 155-165. C a r d i n e t , J . , Tourneur, Y., & A l l a l , L. (1976). The symmetry of g e n e r a l i z a b i l i t y theory: A p p l i c a t i o n s to e d u c a t i o n a l measurement. J o u r n a l of E d u c a t i o n a l Measurement, 13, 119-135. C a r d i n e t , J . , Tourneur, Y., & A l l a l , L. (1981). Extension of g e n e r a l i z a b i l i t y theory and i t s a p p l i c a t i o n s in e d u c a t i o n a l measurement. J o u r n a l of E d u c a t i o n a l  Measurement, 18, 183-204. Chalmers, D.J., & Knight, R.G. (1985). The r e l i a b i l i t y of r a t i n g s of the f a m i l i a r i t y of environmental s t i m u l i : A g e n e r a l i z a b i l i t y a n a l y s i s . Environment and Behavior, 17, 223-238. Cronbach, L . J . (1947). Test " r e l i a b i l i t y " : I t s meaning and d e t e r m i n a t i o n . Psychometrika, 12, 1-16. Cronbach, L . J . (1970). E s s e n t i a l s of p s y c h o l o g i c a l t e s t i n g (3rd e d . ) . New York: Harper & Row. Cronbach, L . J . , G l e s e r , G.C, Nanda, H., & Rajaratnam, N. (1972). The d e p e n d a b i l i t y of b e h a v i o r a l measurements. New York: Wiley. Cronbach, L . J . , Rajaratnam, N., & G l e s e r , G.C. (1963). Theory of g e n e r a l i z a b i l i t y : A l i b e r a l i z a t i o n of r e l i a b i l i t y theory. The B r i t i s h J o u r n a l of S t a t i s t i c a l Psychology, 16, 138-163. Doverspike, D., & B a r r e t t , G.V. (1984). An i n t e r n a l bias a n a l y s i s of a job e v a l u a t i o n instrument. J o u r n a l of A p p l i e d Psychology, 69, 648-662. Doverspike, D., C a r l i s i , A.M., B a r r e t t , G.V., & Alexander, R.A. (1983). G e n e r a l i z a b i l i t y a n a l y s i s of a point-method job e v a l u a t i o n instrument. J o u r n a l of A p p l i e d Psychology, 68, 476-483. F a r r e l l , A.D., M a r i o t t o , M.J., Conger, A.J., Curran, J.P., & Wallander, J.L. (1979). S e l f - r a t i n g s and judges' r a t i n g s of h e t e r o s e x u a l s o c i a l a n x i e t y and s k i l l : A g e n e r a l i z a b i l i t y study. J o u r n a l of C o n s u l t i n g and C l i n i c a l Psychology, 47, 164-175. F e l d t , L.S. (1965). The approximate sampling d i s t r i b u t i o n of Kuder-Richardson r e l i a b i l i t y c o e f f i c i e n t twenty. Psychometrika, 30, 357-370. F e l d t , L.S. (1969). A t e s t of the hypothesis that Cronbach's alpha or Kuder-Richardson c o e f f i c i e n t twenty i s the same fo r two t e s t s . Psychometrika, 34, 363-373. F l e i s s , J.L. ( 1 9 7 1 ) . On the d i s t r i b u t i o n of a l i n e a r combination of independent Chi squares. J o u r n a l of the American S t a t i s t i c a l A s s o c i a t i o n , 6 6 , 1 4 2 - 1 4 4 . F l e i s s , J.L., & Shrout, P.E. ( 1 9 7 8 ) . Approximate i n t e r v a l e s t i m a t i o n f o r a c e r t a i n i n t r a c l a s s c o r r e l a t i o n c o e f f i c i e n t . Psychometrika, 4 3 , 2 5 9 - 2 6 2 . F r a s e r , S.L., Cronshaw, S.F., & Alexander, R.A. ( 1 9 8 4 ) . G e n e r a l i z a b i l i t y a n a l y s i s of a p o i n t method job e v a l u a t i o n instrument: A f i e l d study. J o u r n a l of App l i e d Psychology, 6 9 , 6 4 3 - 6 4 7 . G i l l m o r e , G.M., Kane, M.T., & Naccarato, R.W. ( 1 9 7 8 ) . The g e n e r a l i z a b i l i t y of student r a t i n g s of i n s t r u c t i o n . J o u r n a l of E d u c a t i o n a l Measurement, 1 5 , 1 - 1 5 . G l a s s , G.V., Peckham, P.D., & Sanders, J.R. ( 1 9 7 2 ) . Consequences of f a i l u r e to meet assumptions un d e r l y i n g the f i x e d e f f e c t s a n a l y ses of v a r i a n c e and c o v a r i a n c e . J o u r n a l of E d u c a t i o n a l Research, 4 2 , 2 3 7 - 2 8 8 . G l e s e r , G.C., Cronbach, L . J . , & Rajaratnam, N. ( 1 9 6 5 ) . G e n e r a l i z a b i l i t y of s c o r e s i n f l u e n c e d by m u l t i p l e sources of v a r i a n c e . Psychometrika, 3 0 , 3 9 5 - 4 1 8 . G u l l i k s e n , H. ( 1 9 3 6 ) . The content r e l i a b i l i t y of a t e s t . Psychometrika, J _ , 1 8 9 - 1 9 4 . Haggard, E.A. ( 1 9 5 8 ) . I n t r a c l a s s c o r r e l a t i o n and the a n a l y s i s of v a r i a n c e . New York: Dryden. Hakstian, A.R., & L i n d , J.C. (1982, November). I n f e r e n t i a l procedures f o r alpha c o e f f i c i e n t s . Paper presented at the meeting of the S o c i e t y of M u l t i v a r i a t e Experimental Psychology, A t l a n t a . H akstian, A.R., & Whalen, T.E. (1976). A k-sample s i g n i f i c a n c e t e s t f o r independent alpha c o e f f i c i e n t s . Psychometrika, 41, 219-231. Hansen, D.J., T i s d e l l e , D.A., £< O ' D e l l , S.L. (1985). Audio recorded and d i r e c t l y observed p a r e n t - c h i l d i n t e r a c t i o n s : A comparison of o b s e r v a t i o n methods. B e h a v i o r a l Assessment, 7, 389-399. Jackson, D.N., & Paunonen, S.V. (1980). P e r s o n a l i t y s t r u c t u r e and assessment. In M.R. Rosenzweig & L.W. P o r t e r (Eds.) Annual Review of Psychology ( V o l . 31). Palo A l t o : Annual Reviews Inc. Kane, M.T., & Brennan, R.L. (1977). The g e n e r a l i z a b i l i t y of c l a s s means. Review of E d u c a t i o n a l Research, 47, 267-292. Kane, M.T., G i l l m o r e , G.M., & Crooks, T.J. (1976). Student e v a l u a t i o n s of t e a c h i n g : The g e n e r a l i z a b i l i t y of c l a s s means. J o u r n a l of E d u c a t i o n a l Measurement, 13, 171-183. Knuth, D.E. (1968). The a r t of computer programming ( V o l .  2): Seminumerical a l g o r i t h m s . Reading, MA: Addison-Wesley. Lahey, M.A., Downey, R.G., & S a a l , F.E. (1983). I n t r a c l a s s c o r r e l a t i o n s : There's more than meets the eye. P s y c h o l o g i c a l B u l l e t i n , 93, 586-595. Leone, F.C., & Nelson, L.S. (1966). Sampling d i s t r i b u t i o n s of v a r i a n c e components I. E m p i r i c a l s t u d i e s of balanced nested designs. Technometrics, 8 , 457-468. M a r a s c u i l o , L.A. (1966). Large sample m u l t i p l e comparisons. P s y c h o l o g i c a l B u l l e t i n , 65, 280-290. Millman, J . , & G l a s s , G.V (1967). Rules of thumb for w r i t i n g the ANOVA t a b l e . J o u r n a l of E d u c a t i o n a l Measurement, 4, 41-51. M i t c h e l l , S.K. (1979). I n t e r o b s e r v e r agreement, r e l i a b i l i t y , and g e n e r a l i z a b i l i t y of data c o l l e c t e d i n o b s e r v a t i o n a l s t u d i e s . P s y c h o l o g i c a l B u l l e t i n , 86, 376-390. Nunnally, J.C. (1978). Psychometric Theory (2nd ed.). New York: McGraw-Hill. Nussbaum, A. (1984). M u l t i v a r i a t e g e n e r a l i z a b i l i t y theory i n e d u c a t i o n a l measurement: An e m p i r i c a l study. A p p l i e d P s y c h o l o g i c a l Measurement, 8, 219-230. Paulson, E. (1942). An approximate n o r m a l i z a t i o n of the a n a l y s i s of v a r i a n c e d i s t r i b u t i o n . Annals of Mathematical S t a t i s t i c s , 13, 233-235. Rajaratnam, N., Cronbach, L . J . , & G l e s e r , G.C. (1965). G e n e r a l i z a b i l i t y of s t r a t i f i e d - p a r a l l e l t e s t s . Psychometrika, 30, 39-56. Rap, C.R. Li n e a r s t a t i s t i c a l i n f e r e n c e and i t s a p p l i c a t i o n s (2nd ed.). New York: Wiley. Rentz, R.R. (1980). Rules of thumb f o r e s t i m a t i n g r e l i a b i l i t y c o e f f i c i e n t s u s i n g g e n e r a l i z a b i l i t y theory. E d u c a t i o n a l and P s y c h o l o g i c a l Measurement, 40, 575-592. S a t t e r t h w a i t e , F.E. (1941). S y n t h e s i s of v a r i a n c e . Psychometrika, 6, 309-316. S a t t e r t h w a i t e , F.E. (1946). An approximate d i s t r i b u t i o n of estimates of v a r i a n c e components. B i o m e t r i c s B u l l e t i n , 2, 110-114. Sc h e f f e (1959). The a n a l y s i s of v a r i a n c e (2nd ed.). New York: Wiley. Schroeder, M.L., Schroeder, K.G., & Hare, R.D. (1983). G e n e r a l i z a b i l i t y of a c h e c k l i s t for assessment of psychopathy. J o u r n a l of C o n s u l t i n g and C l i n i c a l Psychology, 51, 511-516. Shavelson, R.J., & Webb, N.M. (1981). G e n e r a l i z a b i l i t y theory: 1973-1980. B r i t i s h J o u r n a l of Mathematical and S t a t i s t i c a l Psychology, 34, 133-166. Shrout, P.E., & F l e i s s , J.L. (1979). I n t r a c l a s s c o r r e l a t i o n s : Uses i n a s s e s s i n g r a t e r r e l i a b i l i t y . P s y c h o l o g i c a l B u l l e t i n , 86, 420-428. Smith, P.L. (1978). Sampling e r r o r s of v a r i a n c e components in small sample m u l t i f a c e t g e n e r a l i z a b i l i t y s t u d i e s . J o u r n a l of E d u c a t i o n a l S t a t i s t i c s , 3, 319-346. Wallender, J.L., Conger, A.J., & Conger, J.C. (1985). Development and e v a l u a t i o n of a b e h a v i o r a l l y referenced r a t i n g system f o r h e t e r o s o c i a l s k i l l s . B e h a v i o r a l Assessment, 7, 137-153. Wiggins, J.S. (1973). P e r s o n a l i t y and p r e d i c t i o n : P r i n c i p l e s of p e r s o n a l i t y assessment. Reading, MA: Addison-Wesley. Wilson, E.B., & H i l f e r t y , M.M. (1931). The d i s t r i b u t i o n of c h i - s q u a r e . Proceedings of the N a t i o n a l Academy of Science, 17, 684-688. Winer, B.J. (1971). S t a t i s t i c a l p r i n c i p l e s in experimental design (2nd ed.). New York: McGraw-Hill. Appendix A Values of the P o p u l a t i o n Variance Components \ 85 Values of P o p u l a t i o n Variance Components 1. Design VII with Item Facet F i x e d P 2 n p ni nr a p r a 2 . p i r , e .5 25 20 8 .303 .250 3.240 3 .000 .7 25 20 8 .434 .250 1 .275 3 .000 .9 25 20 8 .566 .250 .225 3 .000 .5 75 30 2 .733 1 .500 1 .467 3 .000 .7 75 30 2 .903 1 .500 .660 3 .000 .9 75 30 2 1 .790 1 .500 .220 3 .000 .5 150 10 5 .330 1 .000 2.250 2 .000 .7 150 10 5 .542 1 .000 1 . 190 2 .000 .9 150 10 5 .754 1 .000 . 1 30 2 .000 2. Design VII with Rater Facet F i n i t e P 2 n P nr • p i a p r o 2 . p i r Sampling F r a c t i o n = .10 .5 25 20 8 .964 6.340 5.735 .300 .7 25 20 8 .997 2.684 2.435 .300 .9 25 20 8 .984 .669 .585 .300 .5 75 30 2 .955 3.860 1 .823 .300 .7 75 30 2 .980 1 .096 .790 .300 .9 75 30 2 .990 .315 . 1 90 .300 .5 150 10 5 .968 3.724 3.270 .300 .7 1 50 10 5 .986 1 .538 1 .370 .300 .9 1 50 10 5 .989 .301 .370 .300 Sampling F r a c t i o n = .20 .5 25 20 8 .925 7.093 5.685 .300 .7 25 20 8 .976 2.785 2.435 .300 .9 25 20 8 1.011 .773 .585 .300 .5 75 30 2 .908 4.770 1 .857 .300 .7 75 30 2 .980 1 .620 .790 .300 .9 75 30 2 1 .004 .380 .190 .300 .5 150 10 5 .935 4.048 3.270 .300 .7 1 50 10 5 .978 1 .616 1 .370 .300 .9 150 10 5 1 .008 .324 .370 .300 3. Design V-B with both Facets Random P2 nP n r • P • p i a r : p " i r .5 25 20 8 .700 .600 5.345 .300 .7 25 20 8 .700 .600 2.145 .300 .9 25 20 8 .700 .600 .367 .300 .5 75 30 2 .700 .600 1 .350 .300 .7 75 30 2 .700 .600 .550 .300 .9 75 30 2 .700 .600 .106 .300 .5 150 10 5 .800 .600 3.670 .300 .7 150 10 5 .800 .600 1 .384 .300 .9 150 10 5 .800 .600 .114 .300 4. Design VII with a Zero Variance Component P 2 "p ni 12 r • P I a p r a P i r , e .5 25 10 2 1 .000 .000 1 .970 .300 .5 25 10 2 1 .000 .000 1.900 1 .000 .5 25 10 2 1 .000 9.500 .000 1 .000 .5 25 10 2 1 .000 8.000 .000 4 .000 .9 25 10 2 1 .000 .000 .122 1 .000 .9 25 10 2 1 .000 .000 .192 .300 .9 25 10 2 1 .000 .961 .000 .300 .9 25 1 0 2 1 .000 .611 .000 1 .000 .5 75 20 5 . 500 .000 2.500 5 .000 .5 75 20 5 .500 9.000 .000 5 .000 .9 75 20 5 1 .000 .000 .541 .300 .9 75 20 5 1 .000 .222 .000 10 .000 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0097301/manifest

Comment

Related Items