UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

A comparison of conventional and Rasch item analysis approaches applied to a grade four science test… Knodel, John William 1981

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.

Item Metadata

Download

Media
831-UBC_1981_A8 K66.pdf [ 7MB ]
Metadata
JSON: 831-1.0095022.json
JSON-LD: 831-1.0095022-ld.json
RDF/XML (Pretty): 831-1.0095022-rdf.xml
RDF/JSON: 831-1.0095022-rdf.json
Turtle: 831-1.0095022-turtle.txt
N-Triples: 831-1.0095022-rdf-ntriples.txt
Original Record: 831-1.0095022-source.json
Full Text
831-1.0095022-fulltext.txt
Citation
831-1.0095022.ris

Full Text

A COMPARISON OF CONVENTIONAL AND RASCH ITEM ANALYSIS APPROACHES APPLIED TO A GRADE FOUR SCIENCE TEST ITEM POOL by JOHN WILLIAM KNODEL B . S c , The U n i v e r s i t y o f B r i t i s h C o l u m b i a , 1968 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS i n THE FACULTY OF GRADUATE STUDIES ( E d u c a t i o n a l P s y c h o l o g y ) We a c c e p t t h i s t h e s i s as c o n f o r m i n g to the r e q u i r e d s t a n d a r d THE UNIVERSITY OF BRITISH COLUMBIA September, 1981 ©John W i11i am K n o d e l , 1981 In p r e s e n t i n g t h i s t h e s i s i n p a r t i a l f u l f i l m e n t o f the requirements f o r an advanced degree a t the U n i v e r s i t y o f B r i t i s h Columbia, I agree t h a t the L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r r e f e r e n c e and study. I f u r t h e r agree t h a t p e r m i s s i o n f o r e x t e n s i v e copying o f t h i s t h e s i s f o r s c h o l a r l y purposes may be granted by the head o f my department or by h i s o r her r e p r e s e n t a t i v e s . I t i s understood'that c o p y i n g o r p u b l i c a t i o n o f t h i s t h e s i s f o r f i n a n c i a l g a i n s h a l l not be allowed without my w r i t t e n p e r m i s s i o n . Department of E d u c a t i o n a l P s y c h o l o g y The U n i v e r s i t y o f B r i t i s h Columbia 2075 Wesbrook P l a c e Vancouver, Canada V6T 1W5 Date S e p t e m b e r 1 8 , 1 9 8 1 DE-6 (2/79) - i i -ABSTRACT Research S u p e r v i s o r : Dr. W. T. Rogers The purpose o f t h i s s t u d y was t o compare the r e s u l t s o f a p p l y i n g con-v e n t i o n a l and Rasch item a n a l y s i s approaches t o a grade f o u r s c i e n c e t e s t i t e m p o o l . A 76-'\tem m o d i f i e d v e r s i o n of the p i l o t t e s t s used t o c o n s t r u c t the B r i t i s h Columbia Grade k S c i e n c e Assessment T e s t a d m i n i s t e r e d i n S p r i n g 1978 was u t i l i z e d . T h i s item pool was a d m i n i s t e r e d t o 527 grade f o u r s t u -d ents a t t e n d i n g 15 s c h o o l s l o c a t e d i n t h r e e a d j a c e n t South Okanagan s c h o o l d i s t r i c t s . E l e v e n b o o k l e t s were e l i m i n a t e d t h r o u g h a p p l i c a t i o n of c r i t e r i a aimed at c o n t r o l l i n g f o r p o s s i b l e e f f e c t s of speededness. Item a n a l y s e s were o b t a i n e d u s i n g the 516 r e m a i n i n g b o o k l e t s . P r e l i m i n a r y i n v e s t i g a t i o n s o f the t e s t d a t a i n d i c a t e d t h a t i t would be b e s t t o l i m i t i t e m a n a l y s e s t o the 30-item Concepts and 32-item P r o c e s s e s s u b t e s t s i n the item p o o l . C o e f f i c i e n t a l p h a i n d i c e s and f a c t o r a n a l y s i s d a t a were used t o d e t e r m i n e the u n i d i m e n s i o n a 1 i t y o f t h e s u b t e s t s . C o e f f i -c i e n t a l p h a i n d i c e s i n d i c a t e d s t r o n g s u b t e s t homogeneity. For each o f the s u b t e s t s , however, more than one common f a c t o r was found on which t h e r e were s a l i e n t l o a d i n g s . Study o f c l u s t e r s o f items w i t h s a l i e n t p a t t e r n c o e f f i -c i e n t s , however, f a i l e d t o y i e l d u n i q u e d e f i n i t i o n s o f p o s s i b l e d i f f e r e n t t r a i t s b e i n g measured by the s u b t e s t s . It was d e c i d e d t h a t the s u b t e s t s were e s s e n t i a l l y u nidimensiona 1 and t h a t a p p l i c a t i o n o f the Rasch model was j u s t i f i ed. The LERTAP computer program was used f o r c o n v e n t i o n a l item a n a l y s i s . Four c r i t e r i a r e l a t i n g t o item d i f f i c u l t y , c o r r e c t e d i t e m - s u b t e s t p o i n t -b i s e r i a l c o r r e l a t i o n s , d i s t r a c t o r - s u b t e s t p o i n t - b i s e r i a 1 c o r r e l a t i o n s , and d i s t r a c t o r d i f f i c u l t y were a p p l i e d . The BICAL computer program was used f o r the Rasch item a n a l y s e s . - i i i -Rasch c r i t e r i a used r e l a t e d t o item mean square f i t , item d i s c r i m i n a t i o n , and i t e m d i f f i c u l t y . For Rasch Approaches I and IV a l l c r i t e r i a were used. In Rasch Approaches II and V, t h e i t e m d i f f i c u l t y c r i t e r i o n was e l i m i n a t e d . Rasch Approaches M i and VI used o n l y the mean s q u a r e f i t c r i t e r i o n . For Rasch Approaches I, I I , and I I I , P a n c h a p a k e s a n 1 s c o r r e c t i o n f o r g u e s s i n g f o r m u l a was used t o d e t e r m i n e s u b j e c t membership i n the c a l i b r a t i o n sample. The random g u e s s i n g leve1 -formu1 a was a p p l i e d i n Rasch Approaches IV, V, and V I. E i g h t comparisons were made on the s u b t e s t s r e s u l t i n g from the a p p l i -c a t i o n o f the C o n v e n t i o n a l Approach and the s i x Rasch a p p r o a c h e s . Four o f th e comparisons were aimed a t the item l e v e l . These i n c l u d e d t h e p e r c e n t a g e o f items r e j e c t e d by each a p p r o a c h , the e f f i c i e n c y o f t h e Rasch approaches in e l i m i n a t i n g items i l l u s t r a t i n g problems r e l a t e d t o c o n v e n t i o n a l c r i t e r i a , the p e r c e n t a g e o v e r l a p o f r e j e c t e d iterns among p a i r s o f d i f f e r e n t a p p r o a c h e s , and the p e r c e n t a g e o f items r e j e c t e d s o l e l y on the b a s i s o f Rasch c r i t e r i a i n t he Rasch approaches. Four comparisons f o c u s s e d on the s u b t e s t s as e n t i t i e s . The f i r s t i n v o l v e d comparisons o f numbers o f items i n each s u b t e s t , s u b t e s t means, s t a n d a r d d e v i a t i o n s , and s c o r e r a n g e s , as w e l l as Hoyt e s t i m a t e s o f i n t e r n a l c o n s i s t e n c y and s u b t e s t s t a n d a r d e r r o r s o f measurement. A second comparison i n v o l v e d c o r r e l a t i o n s o f s u b j e c t s ' s c o r e s among a l l Concepts s u b t e s t s and among a l l P r o c e s s e s s u b t e s t s . In a t h i r d c o m p a r i s o n , c o r r e l a t e d t - t e s t s were performed among a l l p o s s i b l e p a i r s o f Concepts s u b t e s t s and among a l l p o s s i b l e p a i r s o f P r o c e s s e s s u b t e s t s . The f i n a l c omparison i n v o l v e d the f i t o f items i n the s u b t e s t s t o the Concepts and P r o c e s s e s items used i n the f i n a l v e r s i o n o f the B r i t i s h Columbia Grade k S c i e n c e T e s t . The one c o n v e n t i o n a l and s i x Rasch approaches produced q u i t e d i f f e r e n t Concepts and P r o c e s s e s s u b t e s t s as regard s s p e c i f i c items s e l e c t e d by each. - i v -Numbers o f items i n t h e s u b t e s t s and c o n s e q u e n t l y s u b t e s t c h a r a c t e r i s t i c s , however, were more r e l a t e d t o the s t r i n g e n c y o f t h e c r i t e r i a a p p l i e d r a t h e r than the a p p roach--conventiona 1 o r Rasch--used to b u i l d the s u b t e s t s . The number o f items i n each s u b t e s t a f f e c t e d the r e l i a b i l i t y o f the i n s t r u m e n t . Use o f the Spearman-Brown prophecy f o r m u l a t o a d j u s t the Hoyt i n t e r n a l c o n s i s t e n c y e s t i m a t e s f o r the s u b t e s t s y i e l d e d n e a r l y e q u i v a l e n t r e l i a b i l i t i e s . The c o n t e n t s a m p l i n g o f the s u b t e s t s was a l s o a f u n c t i o n o f the s t r i n g e n c y o f the item a n a l y s i s c r i t e r i a used i n t h e i r c o n s t r u c t i o n . S h o r t e r s u b t e s t s p r o v i d e d a p o o r e r s a m p l i n g o f the c o n t e n t domain than d i d the l o n g e r s u b t e s t s c o n s t r u c t e d u s i n g more l e n i e n t approaches. W h i l e t h e r e appears t o be a s i m i l a r i t y between the c o n v e n t i o n a l £-value and the Rasch i t e m d i f f i c u l t y i n d e x , a p p l i c a t i o n o f Rasch c r i t e r i a i d e n t i -f i e d items w i t h c o n v e n t i o n a l i t e m - s u b t e s t p o i n t - b i s e r i a l problems l e s s e f f e c -t i v e l y . Rasch item a n a l y s i s proved t o be p a r t i c u l a r l y i n e f f i c i e n t i n i d e n t i -f y i n g items w i t h c o n v e n t i o n a l d i s t r a c t o r p roblems. A l t h o u g h Rasch approaches produced s u b t e s t s o f equal r e l i a b i l i t y com-pared t o t h o s e b u i l t u s i n g the c o n v e n t i o n a l a p p r o a c h , and Rasch s u b t e s t s o r d e r e d s u b j e c t s i n e s s e n t i a l l y t h e same f a s h i o n as s u b t e s t s b u i l t u s i n g c o n v e n t i o n a l methods, i t was c o n c l u d e d t h a t the c o n v e n t i o n a l approach t o itern a n a l y s i s s h o u l d remain the method of c h o i c e . The c o n v e n t i o n a l approach p r o v i d e s i n f o r m a t i o n not o n l y t o i d e n t i f y poor items but a l s o t o improve them. Rasch approaches p r o v i d e i n f o r m a t i o n r e l a t e d t o item q u a l i t y , but do not p r o v i d e i n s i g h t s f o r improvement o f poor i t e m s . Rasch approaches to i t e m a n a l y s i s s h o u l d t h e r e f o r e o n l y be a p p l i e d t o l a r g e item p o o l s where r e j e c t i o n o f items would not s e r i o u s l y a f f e c t t h e r e s u l t i n g i n s t r u m e n t s ' e f f e c t i v e n e s s i n s a m p l i n g the c o n t e n t domain. - v " TABLE OF CONTENTS A b s t r a c t i i L i s t o f T a b l e s v i ' ' L i s t o f F i g u r e s I X Acknowledgements x CHAPTER I THE PROBLEM 1 B r i t i s h Columbia Grade h S c i e n c e Assessment Test 3 T e s t Purpose. 3 T e s t P l a n n i n g and C o n s t r u c t i o n . b Goals and o b j e c t i v e s 5 T e s t i terns 7 P i l o t t e s t a d m i n i s t r a t i o n 8 I tern ana l y s i s 8 O r g a n i z a t i o n o f t h e P r e s e n t Study 10 CHAPTER II PRESENTATION OF BASIC PROCEDURES AND REVIEW OF RELEVANT LITERATURE 12 C o n v e n t i o n a l I tern A n a l y s i s . . . . . 12 I tern D i f f i c u l t y Index 12 Item D i s c r i m i n a t i o n I n d i c e s . 15 Po i nt-b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t . . . 16 B i s e r i a l c o r r e l a t i o n c o e f f i c i e n t 17 D i s t r a c t o r adequacy 18 C h o i c e o f i t e m d i s c r i m i n a t i o n i n d e x 19 C o r r e c t i o n o f i t e m - t o t a l c o r r e l a t i o n s f o r s p u r i o u s n e s s 22 T e s t E v a l u a t i o n . 25 I n t e r n a l C o n s i s t e n c y . . . . 25 Hoyt's ANOVA. . . 26 Cronbach's c o e f f i c i e n t a l p h a 27 S t a n d a r d E r r o r o f Measurement 28 The Rasch L o g i s t i c Model 30 General L a t e n t T r a i t Theory. 30 L a t e n t T r a i t Models , 30 The Rasch S i m p l e L o g i s t i c Model 31 I tern C h a r a c t e r i s t i c Curves f o r t h e Rasch Model J>h Assumptions and Robustness o f . the Rasch L o g i s t i c Model 3^  - v i -A d o p t i o n o f a p r o b a b i l i t y model 3 6 S t o c h a s t i c independence 3 6 The un i d i mens i ona 1 i t y assumption 3 7 The assumptions o f e q u a l item d i s c r i m i n a t i o n and l a c k o f q u e s s i n g ko E s t i m a t i o n o f I tern Parameters f o r the Rasch L o g i s t i c Mode 1 . . k3 S t a n d a r d e r r o r o f e s t i m a t e f o r item d i f f i c u l t i e s kS S t a n d a r d e r r o r o f e s t i m a t e f o r person a b i l i t i e s 4 6 T e s t i n g I tern F i t t o the Rasch Model.... ^7 The q u e s t i o n o f sample s i z e i n a p p l i c a t i o n o f the model 5 0 T e s t A n a l y s i s and Development U s i n g the Rasch Model 51 CHAPTER I I I DESIGN OF THE STUDY 5 5 I tern Pool . 5 5 The Samp 1 e. . 5 6 Sample Si ze. . . 56 O b t a i n i n g the Sample 5 7 T e s t i ng 5 9 Data P r e p a r a t i o n 53 Data A n a l y s i s 60 R e s t r i c t i o n o f A n a l y s e s t o Concepts and P r o c e s s e s S u b t e s t s 6 0 T e s t f o r U n i d i m e n s i o n a 1 i t y . 61 Cronbach's c o e f f i c i e n t a l p h a 61 F a c t o r a n a l y s i s 61 I tern A n a l y s i s and Tes t C o n s t r u c t i o n 6 3 C o r r e c t i o n o f i t e m - s u b t e s t p o i n t - b i s e r i a 1 c o r r e l a t i o n s 6 5 E l i m i n a t i o n o f s u b j e c t s from the Rasch a n a l y s e s 6 6 S u b t e s t Comparisons. 67 P e r c e n t a g e o f items r e j e c t e d by each approach 67 P e r c e n t a g e o v e r l a p o f r e j e c t e d items 68 Rasch e l i m i n a t i o n o f items not meeting c o n v e n t i o n a l c r i t e r i a . 6 8 Rasch s p e c i f i c items 69 S u b t e s t c h a r a c t e r i s t i c s 69 C o r r e l a t i o n s among s u b j e c t s ' s c o r e s on s u b t e s t s 69 Comparison o f s u b t e s t means 7 0 D e t e r m i n a t i o n o f f i t t o the o r i g i n a l assessment t e s t 7 0 CHAPTER IV RESULTS 72 F i n a l Sample 7 2 - v i i -I tern A n a l y s i s and T e s t C o n s t r u c t i o n . 73 S u b t e s t Comparisons 74 P e r c e n t a g e o f items r e j e c t e d by each approach 74 P e r c e n t a g e o v e r l a p o f r e j e c t e d items 74 Rasch e l i m i n a t i o n o f items not meeting c o n v e n t i o n a l c r i t e r i a Ik Rasch s p e c i f i c items . .. 75 S u b t e s t c h a r a c t e r i s t i c s 78 C o r r e l a t i o n s among s u b t e s t s ' s c o r e s on s u b t e s t s 78 Comparisons o f s u b t e s t means.... 78 D e t e r m i n a t i o n o f f i t t o o r i g i n a l assessment t e s t 78 CHAPTER V SUMMARY AND CONCLUSIONS 84 Summary 84 Purpose and P r o c e d u r e 84 D i s c u s s i o n o f R e s u l t s 86 P e r c e n t a g e o f items r e j e c t e d by each approach 86 P e r c e n t a g e o v e r l a p o f r e j e c t e d items 87 Rasch e l i m i n a t i o n o f items not meeting c o n v e n t i o n a l c r i t e r i a 90 Rasch s p e c i f i c items 91 S u b t e s t c h a r a c t e r i s t i c s 92 C o r r e l a t i o n s among s u b j e c t s ' s c o r e s on s u b t e s t s 96 Comparisons o f s u b t e s t means 96 D e t e r m i n a t i o n o f f i t t o o r i g i n a l assessment t e s t 97 C o n c l u s i o n s 97 L i m i t a t i o n s o f T h i s Study 100 D i r e c t i o n s f o r F u t u r e Research 100 BIBLIOGRAPHY 102 APPENDIX A T e s t and A d m i n i s t r a t i o n M a t e r i a l s 107 APPENDIX B Correspondence 148 APPENDIX C Item A n a l y s i s Data 153 - v i i i -LIST OF TABLES T. S e l e c t i o n o f I terns from the 1977 Grade h S c i e n c e P i l o t T e s t s 10 2. Assignment o f Items t o Goals 56 3. F a c t o r A n a l y s e s o f S u b t e s t s . . 63 k. Number of S u b j e c t s i n Each Item A n a l y s i s 67 5. I terns Used i n t h e F i n a l V e r s i o n o f the Grade A Assessment Te s t 71 6. C r i t e r i a Used i n t h e I tern A n a l y s e s 73 7. I tern A n a l y s i s C r i t e r i a lh 8. P e r c e n t a g e o f Items R e j e c t e d ••• 75 9. P e r c e n t a g e O v e r l a p o f R e j e c t e d I t e m s — C o n c e p t s S u b t e s t 76 10. P e r c e n t a g e O v e r l a p o f R e j e c t e d I t e m s - - P r o c e s s e s S u b t e s t 76 11. I terns R e j e c t e d — C o n v e n t i o n a l Approach 76 12. Rasch Approaches — Concepts S u b t e s t : P e r c e n t a g e o f I terns R e j e c t e d A c c o r d i n g t o C o n v e n t i o n a l C r i t e r i a 77 13. Rasch Approaches — P r o c e s s e s S u b t e s t : P e r c e n t a g e o f Items R e j e c t e d A c c o r d i n g t o C o n v e n t i o n a l C r i t e r i a . . . . 77 \h. P e r c e n t a g e o f Rasch-Spec i f i c I terns 77 15. T e s t C h a r a c t e r i s t i c s — Concepts S u b t e s t 80 16. T e s t C h a r a c t e r i s t i c s — P r o c e s s e s S u b t e s t 80 1.7. C o r r e l a t i o n s Between S u b j e c t s 1 S c o r e s — C o n c e p t s S u b t e s t 81 18. C o r r e l a t i o n s Between S u b j e c t s ' S c o r e s — P r o c e s s e s S u b t e s t 81 19- C o r r e l a t e d t - t e s t s — C o n c e p t s S u b t e s t . . . . . . . . 81 20. C o r r e l a t e d t - ' t e s t s — P r o c e s s e s S u b t e s t 82 21. I terns R e j e c t e d — C o n c e p t s Subtests..- 82 22. I terns Rej e c t e d — P r o c e s s e s S u b t e s t 83 23. P e r c e n t a g e o f I terns i n the F i n a l Assessment T e s t R e t a i n e d by Each Approach 83 2k. P e r c e n t a g e O v e r l a p o f R e j e c t e d I t e r n s — C o n v e n t i o n a 1 and Rasch Approaches 88 25. P e r c e n t a g e O v e r l a p o f R e j e c t e d I terns — P a r a l l e i Rasch Approaches 88 26. P e r c e n t a g e O v e r l a p o f R e j e c t e d Items — R a s c h Approaches W i t h i n and A c r o s s Methods o f D e t e r m i n i n g the C a l i b r a t i o n Sample 89 27. Stepped-up Hoyt Rel i a b i 1 i t y E s t i m a t e s 95 28. R e l a t i o n s h i p s Between S u b t e s t SEM's and / k_ ' s 96 - i x -LIST OF FIGURES T. Three ICC's T h e o r e t i c a l l y O b t a i n a b l e U s i n g the Rasch Model 3 5 2 . Raw Sco r e by Item M a t r i x I l l u s t r a t i n g U n i d i m e n s i o n a 1 i t y 3 8 3 . S u b t e s t Means (%) 9 4 - x -Acknowledgements I w i s h t o thank the chairman o f my committee, Dr. Todd Rogers, f o r s u g g e s t i n g the t o p i c f o r t h i s s t u d y and f o r hi s - s u p p o r t i n c a r r y i n g out the p r o j e c t . Dr. Rogers' c o u r s e s i n s t a t i s t i c s and t e s t t h e o r y have been a s o u r c e o f c h a l l e n g e and enjoyment t h r o u g h o u t my program. Thanks a l s o t o my committee members: Dr. Tom O'Shea, f o r h e l p i n under-s t a n d i n g and a p p l y i n g the Rasch model, and Dr. H a r o l d R a t z l a f f , whose c o u r s e i n e d u c a t i o n a l r e s e a r c h a i d e d me i n p l a n n i n g t h i s s t u d y , and who made many h e l p f u l s u g g e s t i o n s d u r i n g the i n i t i a l s t a g e s o f t h i s p r o j e c t . I would a l s o l i k e t o thank Dr. Gaalen E r i c k s o n f o r p r o v i d i n g me w i t h i m p o r t a n t u n p u b l i s h e d i n f o r m a t i o n r e l a t e d t o the B r i t i s h Columbia Grade 4 S c i e n c e Assessment T e s t development. Thanks t o the E d u c a t i o n a l Research I n s t i t u t e o f B r i t i s h Columbia which p r o v i d e d f i n a n c i a l a s s i s t a n c e f o r t h i s s t u d y i n the form o f a d i s c r e t i o n a r y g r a n t . L a s t l y , I w i s h t o acknowledge the i m p o r t a n t r o l e s p l a y e d by the su p e r -i n t e n d e n t s , p r i n c i p a l s , t e a c h e r s , and grade f o u r p u p i l s o f the South Okana-gan (#1.4), P e n t i c t o n (#15) and Summerland (#77) s c h o o l d i s t r i c t s , who made the 1979 d a t a c o l l e c t i o n f o r the p r o j e c t p o s s i b l e . - 1 -CHAPTER I THE PROBLEM T e s t c o n s t r u c t i o n has been f o r m a l i z e d i n t o s e v e r a l b a s i c s t e p s : 1 . T e s t p l a n n i n g . The c o n t e n t a r e a s and s k i l l c a t e g o r i e s t o be cov-e r e d i n the t e s t a r e s p e c i f i e d a c c o r d i n g t o the purpose t h e t e s t w i l l s e r v e . 2 . O b t a i n i n g i t e m s . P r e v i o u s l y d e v e l o p e d items a r e o b t a i n e d o r new items a r e w r i t t e n t o co v e r t h e a r e a s s p e c i f i e d i n t h e t e s t p l a n . More items than w i l l be needed i n the f i n a l v e r s i o n o f the t e s t a r e assembled so t h a t poor i t e m s , as r e v e a l e d by i t e m - a n a l y s i s t e c h n i q u e s , can be e i t h e r d i s c a r d e d o r improved. 3 . P i l o t t e s t a d m i n i s t r a t i o n . I terns t o be a n a l y z e d a r e c o m p i l e d i n t o a p i l o t t e s t w h i c h i s a d m i n i s t e r e d t o a sample r e p r e s e n t a t i v e o f t h e popu-l a t i o n f o r which the f i n a l v e r s i o n o f the t e s t i s i n t e n d e d . The p i l o t t e s t i s a d m i n i s t e r e d under c o n d i t i o n s which a r e u n i f o r m f o r a l l examinees. k. I tern a n a l y s i s . T e s t items a r e a n a l y z e d e m p l o y i n g an a c c e p t a b l e method of item a n a l y s i s . U n s a t i s f a c t o r y items a r e e i t h e r d i s c a r d e d o r im-proved upon where p o s s i b l e . A r e v i s e d t e s t i s c o n s t r u c t e d u s i n g a c c e p t a b l e o r r e f i n e d items. 5 . R e v i s e d t e s t e v a l u a t i o n . The r e v i s e d t e s t i s a d m i n i s t e r e d t o a r e p r e s e n t a t i v e sample of examinees. The t e s t i s then a n a l y z e d t o d e t e r m i n e whether i t has c e r t a i n d e s i r a b l e f e a t u r e s r e l a t i n g t o a d m i n i s t r a t i o n time and r e l i a b i l i t y . If the r e v i s e d t e s t meets d e s i r e d s p e c i f i c a t i o n s i t i s a c c e p t e d as the f i n a l v e r s i o n o f the t e s t . Steps 1 , 2 , 3 and 5 are s i m i l a r f o r a l l t y p e s o f t e s t c o n s t r u c t i o n . There i s , however, a c h o i c e o f methods a v a i l a b l e f o r use i n Step k--1 tern A n a l y s i s . - 2 -U n t i l r e c e n t l y , the c o n v e n t i o n a l method o f item a n a l y s i s , e m p l o y i n g d a t a r e l a t i n g t o item d i f f i c u l t y and i t e r n / t o t a 1 - t e s t s c o r e b i s e r i a l o r p o i n t -b i s e r i a l c o r r e l a t i o n s , has been the s t a n d a r d approach t o t e s t c o n s t r u c t i o n . High-speed computer programs e x i s t t o o b t a i n e s t i m a t e s o f item p a r a m e t e r s , and c o n s i d e r a b l e e x p e r t i s e has been ac c u m u l a t e d i n a p p l y i n g t h i s c o n v e n t i o n a l method t o t e s t c o n s t r u c t i o n problems. In r e c e n t y e a r s , t h e r e has been much i n t e r e s t i n the a p p l i c a t i o n o f l a t e n t t r a i t t h e o r y , a l s o r e f e r r e d t o as item response t h e o r y or item c h a r a c -t e r i s t i c c u r v e (ICC) t h e o r y , to item a n a l y s i s . For example, the 1 9 7 7 (Summer) i ssue o f the J o u r n a l o f E d u c a t i o n a l Measurement c o n t a i ned s i x papers on 1 a t -ent t r a i t t h e o r y and i t s a p p l i c a t i o n s . Much o f the i n t e r e s t i n l a t e n t t r a i t t h e o r y has been f o s t e r e d by L o r d ' s work begun i n t h e 1 9 5 0 ' s and t h e subse-quent p u b l i c a t i o n o f Lord and N o v i c k ' s S t a t i s t i c a l T h e o r i e s o f Mental T e s t  Scores ( 1 9 6 8 ) . T h i s t e x t b o o k i n c l u d e d f i v e c h a p t e r s ( f o u r o f them a u t h o r e d by A l l e n Birnbaum) on l a t e n t t r a i t t h e o r y . U n t i l r e c e n t l y , much o f the r e s e a r c h i n l a t e n t t r a i t t h e o r y was f o c u s s e d on the one-parameter Rasch l o g i s t i c model i n t r o d u c e d by Georg Rasch ( i 9 6 0 ; 1 9 6 6 ) . T h i s model p r o v i d e s d a t a r e l a t i n g t o a t e s t item's f i t t o the l a t e n t t r a i t model, i t e m d i f f i c u l t y , and item d i s c r i m i n a t i o n . Because o f i t s mathe-m a t i c a l t r a c t a b i 1 i t y , computer programs f o r a p p l i c a t i o n o f the Rasch model i n t e s t c o n s t r u c t i o n were d e v e l o p e d e a r l y . C o n s e q u e n t l y , r e s e a r c h r e l a t i n g t o the Rasch model has been predominant i n the l i t e r a t u r e . T h i s t r e n d i s chang-ing today as computer programs f o r t h e two- and t h r e e - parameter models have now been d e v e l o p e d . Given t h e dominant use of the c o n v e n t i o n a l and Rasch approaches t o item a n a l y s i s and t h e ap p a r e n t l a c k o f i n f o r m a t i o n r e g a r d i n g the c l e a r advantage of one approach o v e r the o t h e r , the purpose o f t h e p r e s e n t s t u d y was t o ob-j e c t i v e l y a p p l y c o n v e n t i o n a l and Rasch item a n a l y s i s p r o c e d u r e s t o a pool o f - 3 -t e s t items and t o compare t h e s p e c i f i c n a t u r e o f items s e 1 e c t e d / r e j e c t e d and the o v e r a l 1 c h a r a c t e r i s t i c s o f t e s t s r e s u l t i n g from t h e i r a p p l i c a t i o n . Two s e r i e s o f t e s t s , one c o n s t r u c t e d on t h e b a s i s o f c o n v e n t i o n a l i t e m a n a l y s i s , t h e o t h e r s e r i e s c o n s t r u c t e d on the b a s i s o f item a n a l y s i s u s i n g the Rasch model, were c o n s t r u c t e d . The same i t e m pool was used t o c o n s t r u c t t h e two s e r i e s o f t e s t s . For each o f t h e methods, t h e r e s p e c t i v e c r i t e r i a f o r item s e l e c t i o n / r e j e c t i o n were s e t a c c o r d i n g t o o p i n i o n i n c u r r e n t l i t e r a t u r e and s t r i c t l y adhered t o . S i n c e t h e c o n v e n t i o n a l a p p r o a c h t o item a n a l y s i s has proven t o be u s e f u l and i s predominant i n t e s t c o n s t r u c t i o n , t h e e f f i c i e n c y o f Rasch item a n a l y s i s i n i d e n t i f y i n g items showing problems i n d i c a t e d by c o n v e n t i o n a l item a n a l y s i s c r i t e r i a was a l s o d e t e r m i n e d . B r i t i s h C o l u m b i a Grade 4 S c i e n c e Assessment T e s t The item pool employed i n the p r e s e n t s t u d y c o n s i s t e d o f t h e items used i n the p i l o t phase o f t h e Grade k S c i e n c e Assessment t e s t a d m i n i s t e r e d as p a r t o f the g e n e r a l S c i e n c e Assessment c o n d u c t e d i n the P r o v i n c e o f B r i t i s h C olumbia i n t h e S p r i n g o f 1978. G i v e n t h e c e n t r a l i t y o f the B r i t i s h C olumbia S c i e n c e Assessment i t e m p o o l a t t h e Grade k l e v e l i n t h e p r e s e n t s t u d y , a d i s c u s s i o n o f t h e development and c o n s t r u c t i o n o f t h e Grade k S c i e n c e A s s e s s -ment t e s t i s p r o v i d e d below. T e s t Purpose The purpose o f t h e m u l t i p l e - c h o i c e s c i e n c e assessment t e s t s a d m i n i s t e r -ed t o a l l g r a d e 4, 8, and 12 s t u d e n t s i n t h e P r o v i n c e o f B r i t i s h C o l u m b i a was t o i n f o r m e d u c a t o r s and t h e p u b l i c o f s t r e n g t h s and weaknesses i n s p e c i -f i c a r e a s o f s c i e n c e l e a r n i n g , t o p r o v i d e p o s s i b l e i n s i g h t s t o e d u c a t o r s a t the c l a s s r o o m , d i s t r i c t , and p r o v i n c i a l l e v e l as t o how t o m a i n t a i n s t r e n g t h s and overcome i d e n t i f i e d w e a k n e s s e s , and t o p r o v i d e a d a t a b a s e l i n e f o r f u t u r e - k -m o n i t o r i n g o f s t u d e n t l e a r n i n g i n s c i e n c e ( B r i t i s h Co 1umbia' M i n i s t r y o f E d u c a t i o n , 1979)-The s c i e n c e t e s t s formed one f a c e t o f the p r o v i n c i a l s c i e n c e assessment program. In a d d i t i o n t o achievement t e s t s , i n s t r u m e n t s t o measure s t u d e n t s ' a t t i t u d e s towards s c i e n c e , s c i e n t i s t s , and s c i e n c e t e a c h i n g were employed. Q u e s t i o n n a i r e s were used t o o b t a i n t e a c h e r s ' o p i n i o n s about s c i e n c e t e a c h i n g , the s c i e n c e c u r r i c u l u m , t e x t b o o k s and sup p l e m e n t a r y m a t e r i a l s , s c i e n c e t e a c h -e r e d u c a t i o n , and i n s t r u c t i o n a l p r a c t i c e s i n s c i e n c e . The S c i e n c e A s s e s s -ment was p a r t o f the t o t a l P r o v i n c i a l L e a r n i n g Assessment Program which i n -c l u d e s most s u b j e c t a r e a s o f t h e p u b l i c s c h o o l c u r r i c u l u m i n B r i t i s h C olumbia. General g u i d e l i n e s o u t l i n i n g p r o c e d u r e s f o r t h e s e assessments a re p r e s e n t e d in Assessment P l a n n i n g : B.C. Assessment Program ( B r i t i s h C olumbia Department o f E d u c a t i o n , 1975)• T e s t P l a n n i n g and C o n s t r u c t i o n The group o f e d u c a t o r s , o r C o n t r a c t Team, r e s p o n s i b l e f o r the s c i e n c e assessment program was charged by the M i n i s t r y o f E d u c a t i o n w i t h t h e t a s k o f g a i n i n g s p e c i f i c i n s i g h t s r e g a r d i n g B r i t i s h Columbia s t u d e n t s ' g r a s p o f s c i -ence s k i l l s c o n s i d e r e d t o be i m p o r t a n t i n t h e p r o v i n c i a l s c i e n c e c u r r i c u l u m . The p r o c e d u r e f o l l o w e d by the C o n t r a c t Team i n v o l v e d (a) i d e n t i f i c a t i o n o f g o a l s and r e l a t e d o b j e c t i v e s c o n s i d e r e d t o be im p o r t a n t i n the s c i e n c e c u r -r i c u l u m , (b) c o n s t r u c t i o n o f t e s t s c o n s i s t i n g o f items r e l a t e d t o . t h e o b j e c -t i v e s , (c) a d m i n i s t r a t i o n o f t h e t e s t s , (d) s c o r i n g and a n a l y s i s o f t h e t e s t d a t a , and (e) d i s c r e p a n c y a n a l y s i s . The l a t t e r p r o c e s s i n v o l v e d an e v a l u a -t i o n o f the e x t e n t t o which s t u d e n t s were- a c h i e v i n g the o b j e c t i v e s h e l d t o be i m p o r t a n t ( R o g e r s , 1979)- The C o n t r a c t Team was a i d e d i n t h e i r work by a Management Committee, c o n s i s t i n g o f members r e p r e s e n t a t i v e : o f the educa-t i o n a l community, who p r o v i d e d g e n e r a l a d v i c e r e l a t i n g t o t h e assessment and - 5 -r e v i e w e d the assessment m a t e r i a l s produced by the C o n t r a c t Team. In a d d i t i o n , s e v e r a l ad hoc re v i e w committees were empanelled t o a i d i n r e v i e w i n g and r e -v i s i n g assessment m a t e r i a l s . G oals and o b j e c t i v e s . P r e l i m i n a r i e s t o t e s t c o n s t r u c t i o n began i n the S p r i n g o f 1977. S i n c e the f o c u s o f the assessment was on s p e c i f i c knowledge and o b j e c t i v e s , t o t a l t e s t s c o r e s were not t o be used as measures o f s c i e n c e achievement. Such g l o b a l s c o r e s would p r o v i d e i n f o r m a t i o n which was too gen-e r a l t o d e t e c t s p e c i f i c s t r e n g t h s and weaknesses i n s c i e n c e achievement. T h e r e f o r e , the t e s t was planned t o p r o v i d e m e a n i n g f u l s t a t e m e n t s r e g a r d i n g the s t a t e o f l e a r n i n g r e f e r e n c e d t o s e v e r a l s c i e n c e e d u c a t i o n g o a l s and r e -l a t e d s p e c i f i c o b j e c t i v e s . The C o n t r a c t Team and Management Committee c o o p e r a t e d i n dra w i n g up a framework o f g o a l s and r e l a t e d o b j e c t i v e s f o r the s e r i e s o f s c i e n c e t e s t s . G o als were d e f i n e d as s t a t e m e n t s o f b r o a d , g e n e r a l outcomes o f s c i e n c e l e a r n -i n g . O b j e c t i v e s were d e f i n e d as s p e c i f i c s t a t e m e n t s o f b e h a v i o r w h i c h were t o be c o n s i d e r e d as a c c e p t a b l e e v i d e n c e f o r h a v i n g a c h i e v e d p r o f i c i e n c y i n a p a r t i c u l a r g o a l . O b j e c t i v e s were s t a t e d i.n; such a manner t h a t s p e c i f i c t e s t items t o measure them c o u l d be chosen from e x i s t i n g s c i e n c e t e s t s o r c o u l d be newly c o n s t r u c t e d . L e v e l o f performance on t e s t items r e l a t e d t o o b j e c t i v e s subsumed under a p a r t i c u l a r g o a l was then t o be taken as e v i d e n c e o f mastery o f t h a t p a r t i c u l a r g o a l . The t e s t makers were co n c e r n e d t h a t the o b j e c t i v e s chosen f o r t e s t i n g s h o u l d r e f l e c t t h e s t a t e d g o a l s o f the B r i t i s h Columbia s c i e n c e c u r r i c u l u m . K l o p f e r ' s O v e r a l l T a b l e o f S p e c i f i c a t i o n s (Bloom, H a s t i n g s , & Madaus, 1971) was chosen as a b a s i s f o r t e s t c o n s t r u c t i o n . The g o a l s and o b j e c t i v e s l i s t e d i n t h i s t a b l e were f e l t t o a p p r o x i m a t e the i n t e n t i o n s o f the p r o v i n c i a l s c i -ence c u r r i c u l u m . K l o p f e r ' s t a b l e i n c l u d e d t h e c o r e c u r r i c u l u m g o a l s f o r s c i e n c e as s p e c i f i e d by the M i n i s t r y o f E d u c a t i o n . As w e l l as o b j e c t i v e s r e -- 6 -l a t e d t o s c i e n c e knowledge and c o m p r e h e n s i o n , the t a b l e a l s o emphasized ob-j e c t i v e s r e l a t e d t o the development o f s c i e n t i f i c l i t e r a c y and s t r e s s e d ob-j e c t i v e s r e l a t i n g t o t h e p r o c e s s o f s c i e n t i f i c i n q u i r y . To b e t t e r r e f l e c t the B r i t i s h Columbia s c i e n c e c u r r i c u l u m , the o b j e c -t i v e s were i n c o r p o r a t e d under f o u r o v e r a r c h i n g g o a l s which were h e l d t o r e f l e c t c u r r e n t c u r r i c u l u m t r e n d s i n the p r o v i n c e . Some o f t h e o b j e c t i v e s were r e f o r m u l a t e d so t h a t the t e r m i n o l o g y used was more i n k e e p i n g w i t h the terms used t o d e s c r i b e the B r i t i s h Columbia s c i e n c e c u r r i c u l u m . Both the o b j e c t i v e s and the c o n t e n t a r e a s t o be c o v e r e d were r a t e d by t h e C o n t r a c t Team i n terms o f p r i o r i t y f o r assessment purposes a t each o f t h e g r a d e k, 8, and 12 l e v e l s . The m o d i f i e d t a b l e o f t e s t s p e c i f i c a t i o n s was next s u b m i t t e d t o the Management Committee f o r e v a l u a t i o n . On the b a s i s o f recommendations from the Management Committee, the C o n t r a c t Team then proceeded t o p r e p a r e d r a f t s o f t a b l e s o f s p e c i f i c a t i o n s f o r m u l t i p l e - c h o i c e achievement t e s t s a t the t h r e e grade l e v e l s . . The s p e c i f i c a t i o n s t a b l e f o r the grade h t e s t c a l l e d f o r p r i m a r y emphasis i n t e s t i n g the p r o c e s s s k i l l s o f s c i e n c e — o b s e r v i ng , c l a s s -i f y i n g , m e a s u r i n g , c o m m u n i c a t i n g , and combined p r o c e s s e s . A s e c o n d a r y , but s t i l l i m p o r t a n t emphasis i n the t e s t was t o be on t h e u n d e r s t a n d i n g o f s c i e n -t i f i c f a c t s , c o n c e p t s , and p r i n c i p l e s . The next s t e p i n the t e s t p l a n n i n g p r o c e s s was t o convene Review P a n e l s o f e d u c a t o r s and members o f t h e p u b l i c . The purpose o f the Review P a n e l s was t o i n f o r m the p r o f e s s i o n a l and l a y p u b l i c of the purposes and d e s i g n o f t h e S c i e n c e Assessment and t o o b t a i n f e e d b a c k from t e a c h e r s and t h e p u b l i c as t o what t h e emphases o f the assessment s h o u l d be. The Review P a n e l s a l s o exam-ined and e v a l u a t e d t h e g o a l s and o b j e c t i v e s f o r each grade l e v e l w h i c h had been f o r m u l a t e d by the C o n t r a c t Team, and e v a l u a t e d sample items' f o r measur-ing the o b j e c t i v e s . The o b j e c t i v e s were r a t e d on a s c a l e from 0 (minimum - 7 -p r i o r i t y ) t o k (maximum p r i o r i t y ) , . The median r a t i n g of the Review P a n e l s was a p p l i e d as the f i n a l o v e r a l l r a t i n g f o r each o b j e c t i v e . The above s t a g e s i n t e s t c o n s t r u c t i o n l e d t o some minor m o d i f i c a t i o n s i n the t a b l e s o f s p e c i f i c a t i o n s . The Review P a n e l s had e x p r e s s e d c o n c e r n t h a t the p r o t o t y p e items r e l a t i n g t o t h e s c i e n t i f i c a p p l i c a t i o n g o a l d i d not seem t o r e l a t e t o p r a c t i c a l e v e r y d a y s i t u a t i o n s . They a l s o s u g g e s t e d t h a t items d e a l i n g w i t h the a b i l i t y t o read and i n t e r p r e t newspaper and magazine a r t i c l e s c o n c e r n i n g s c i e n c e - r e 1ated i s s u e s be i n c l u d e d under the g o a l o f s c i e n t i f i c l i t e r a c y . The f i n a l v e r s i o n o f the g o a l s t o be i n c l u d e d i n the s c i e n c e achievement t e s t s was as f o l l o w s : Goal 1--Understanding C o n c e p t s ; Goal 2 — S k i 11 i n the P r o c e s s e s o f S c i e n c e ; Goal 3 - - A p p l i c a t i o n ; Goal 4--S a f e t y ; Goal 5 - _ S c i e n t i f i c L i t e r a c y . T e s t i t e m s . S i n c e the g o a l s s e t t l e d upon i n t h e f i n a l v e r s i o n of the t e s t t a b l e of s p e c i f i c a t i o n s r e p r e s e n t e d a broad a r e a of l e a r n i n g o b j e c t i v e s and t h e t e s t i n g time was l i m i t e d , o n l y a s m a l l number o f t h e many p o s s i b l e o b j e c t i v e s r e l a t i n g t o a p a r t i c u l a r g o a l c o u l d be t e s t e d . The C o n t r a c t Team had a l r e a d y assembled a bank of m u l t i p l e - c h o i c e q u e s t i o n s from e s t a b l i s h e d s c i e n c e t e s t s . Where t h e r e was an i n s u f f i c i e n t number o f items t o t e s t a p a r t i c u l a r g o a l o r c o n t e n t c a t e g o r y , a d d i t i o n a l items were c o n s t r u c t e d by the Contract.Team. Thus, a more adequate s a m p l i n g o f the c o n t e n t domain r e l a t i n g t o a p a r t i c u l a r g o a l p r o v i d e d a f i r m e r b a s i s f o r i n t e r p r e t a t i o n o f a c h i e v e -ment r e l a t e d t o t h a t g o a l . Each item was c a t e g o r i z e d a c c o r d i n g t o g o a l , con-t e n t a r e a , and g r a d e l e v e l . The items used came from d i v e r s e s o u r c e s and had v a r y i n g numbers o f response c h o i c e s . Some i t e m s , e s p e c i a l l y those r e l e a s e d from the N a t i o n a l Assessment o f E d u c a t i o n a l P r o g r e s s , i n c l u d e d the 'I don't know 1 o p t i o n . The C o n t r a c t Team d e c i d e d t o i n c l u d e t h i s o p t i o n f o r a l l t h e items used. - 8 -P i l o t t e s t a d m i n i s t r a t i o n . The item pool f o r grade k i n i t i a l l y i n c l u d -ed 81 items. From t h i s item p o o l , a t w o - p a r t p i l o t t e s t c o n s i s t i n g o f 37 items i n S e c t i o n A and 38 items i n S e c t i o n B was c o n s t r u c t e d . D u r i n g the months o f November and December, 1977, P i l o t T e s t A was a d m i n i s t e r e d t o 172 grade k p u p i l s and P i l o t T e s t B was a d m i n i s t e r e d t o a second sample of 179 p u p i l s . Samples o f c o n v e n i e n c e , s e l e c t e d w i t h c o n s i d e r a t i o n o f t h e s c h o o l s ' g e o g r a p h i c l o c a t i o n s , s i z e s , and s o c i o e c o n o m i c e n v i r o n m e n t s were used. 11 em a n a l y s i s . The items i n t h e 1977 p i l o t t e s t s were s u b j e c t e d t o con-v e n t i o n a l item a n a l y s i s i n o r d e r t o d e l e t e u n s u i t a b l e items and t o improve items f o r r e t e n t i o n i n the f i n a l v e r s i o n o f t h e t e s t . The items r e l a t i n g t o each g o a l o r o b j e c t i v e were grouped t o g e t h e r and a measure of i n t e r n a l con-s i s t e n c y ( a l p h a ) was o b t a i n e d f o r each item c l u s t e r . S t a t i s t i c a l d a t a were a l s o o b t a i n e d f o r the i n d i v i d u a l items i n each c l u s t e r . I terns which were e a s i e r than e x p e c t e d were i n s p e c t e d f o r w o r d i n g which might have i n a d v e r t e n t l y g i v e n away the c o r r e c t answer. C o r r e s p o n d i n g -l y , items which were more d i f f i c u l t than e x p e c t e d were checked f o r p o s s i b l e a m b i g u i t i e s w h i c h may have c o n f u s e d the more c a p a b l e s t u d e n t s t a k i n g the t e s t . The p o i n t - b i s e r i a l c o r r e l a t i o n o f each o p t i o n w i t h the t o t a l s c o r e on each c l u s t e r o f items was a l s o o b t a i n e d f o r each item. The p o i n t - b i s e r i a l c o r r e l a t i o n o f the c o r r e c t o p t i o n was c o r r e c t e d f o r s p u r i o u s n e s s . I f the c o r r e c t o p t i o n o f an i t e m b e l i e v e d t o measure performance r e l a t e d t o a p a r t i -c u l a r g o a l had a n e g a t i v e p o i n t - b i s e r i a 1 c o r r e l a t i o n w i t h the items i n t h a t g o a l c l u s t e r , the item became a c a n d i d a t e f o r o m i s s i o n from the f i n a l v e r s i o n o f t h e t e s t . F o i l s h a v i n g a p o s i t i v e u n c o r r e c t e d p o i n t - b i s e r i a 1 c o r r e l a t i o n a l s o a l e r t e d the t e s t b u i l d e r s t o p o s s i b l e problems w i t h t h a t item. The p r i m a r y c o n s i d e r a t i o n i n c o n s t r u c t i n g the s c i e n c e assessment t e s t s was c o n t e n t v a l i d i t y . For t h i s r e a s o n , items were not n e c e s s a r i l y d i s c a r d e d - 9 -i f t he v a l u e s o f t h e i r c o n v e n t i o n a l item i n d i c e s , d i d not f a l l w i t h i n " a c c e p t -a b l e " l i m i t s . In some c a s e s , poor i t e m i n d i c e s a l e r t e d the t e s t d e v e l o p e r s t o problems i n the w o r d i n g o f t h e items w h i c h c o u l d be c o r r e c t e d . In o t h e r c a s e s , items w i t h poor i n d e x v a l u e s were r e t a i n e d i f i t was f e l t t h a t they p o s s e s s e d c o n t e n t v a l i d i t y . T h i s i s a common p r a c t i c e i n t e s t c o n s t r u c t i o n , e s p e c i a l l y i n cases where the number o f items t o measure performance., i n a p a r t i c u l a r goal i s l i m i t e d . Thus, e m p i r i c a l l y o b t a i n e d s t a t i s t i c a l i n f o r m a -t i o n r e l a t e d t o an item was g i v e n s e c o n d a r y importance t o the item's l o g i c -a l l y j u d g e d c o n t e n t v a l i d i t y . To i l l u s t r a t e the r o l e p l a y e d by human j u d g e -ment i n the t e s t c o n s t r u c t i o n , as opposed t o s t r i c t s t a t i s t i c a l c o n s i d e r a -t i o n s , i t s h o u l d be p o i n t e d out t h a t , even a f t e r p i l o t t e s t i n g , items were t r a n s f e r r e d from one g o a l c l u s t e r t o a n o t h e r i f i t was f e l t t h a t they r e l a t e d more c l o s e l y t o t h a t g o a l than t o the one o r i g i n a l l y i n t e n d e d . A l s o , one itern w hich was not p i l o t t e s t e d was i n c l u d e d i n the f i n a l v e r s i o n o f the t e s t . T a b l e 1 p r o v i d e s a summary o f the number of items i n each g o a l and ob-j e c t i v e c a t e g o r y o f p i l o t t e s t s A and B. The u n b r a c k e t e d numbers show the number o f items r e t a i n e d from each p i l o t t e s t i n the f i n a l v e r s i o n o f the Grade k S c i e n c e Assessment T e s t . B r a c k e t e d numbers i n d i c a t e the number o f items r e j e c t e d . I n s p e c t i o n o f the t e c h n i c a l d a t a r e s u l t i n g from the p i l o t t e s t i n g shows t h a t items were i n c l u d e d i n the f i n a l t e s t even though t h e i r d i f f i c u l t y l e v e l f e l l o u t s i d e the range o f .20 t o .80, o r t h e i r p o i n t - b i s e r i -a l c o r r e l a t i o n s w i t h the a p p r o p r i a t e item c l u s t e r was l e s s than z e r o ( In p i l o t t e s t A, t h r e e items had a d i f f i c u l t y i n d e x g r e a t e r than .80 and two had a d i f f i c u l t y i n dex l e s s than .20; i n p i l o t t e s t B, seven items had a d i f f i c u l t y index g r e a t e r than .80; one and two i t e m s , i n p i l o t t e s t s A and B, r e s p e c t i v e l y , had n e g a t i v e p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t s ) . In each i n s t a n c e , the items were m o d i f i e d t o make them more e f f e c t i v e and i n -c l u d e d i n t h e f i n a l v e r s i o n o f t h e t e s t as i t was f e l t t hey measured i m p o r t -ant g o a l s and o b j e c t i v e s which wou 1 d<„otherwi se not be measured. - 10 -T a b l e 1 S e l e c t i o n o f I terns from t h e 1977 Grade k S c i e n c e P i l o t T e s t s Goal P i l o t T e s t Commun i c a t i ng b U n d e r s t a n d i n g Concepts 13 (2) 13 (2) S k i l l i n the P r o c e s s e s o f S c i e n c e C l a s s i f y i n g 5 (1) O b s e r v i n g -- 5 a c M e a s u r i n g " 5 (2) 1 Combined S k i l l s -- 5 0) App1i c a t i on 5 S a f e t y -- 5 S c i e n t i f i c L i t e r a c y 3 1 ° T o t a l d . 32 (5) 35 (3) Goal c l u s t e r s h a v i n g s m a l l numbers o f items were embedded i n one p i l o t t e s t o n l y . k An a d d i t i o n a l i t e m , not run i n the p i l o t t e s t i n g was added t o t h i s c l u s t e r f o r the f i n a l v e r s i o n o f t h e t e s t . c A f t e r p i l o t t e s t i n g , some items were moved from one c l u s t e r t o a n o t h e r f o r the f i n a l v e r s i o n o f the t e s t . ^ The p i l o t t e s t i n g p r o v i d e d 67 items f o r the f i n a l v e r s i o n o f the t e s t . See note b, above. O r g a n i z a t i o n o f the P r e s e n t Study Chapter II d i s c u s s e s the l i t e r a t u r e r e l e v a n t t o the a p p l i c a t i o n of the c o n v e n t i o n a l and Rasch methods of item a n a l y s i s . Chapter 111 d e s c r i b e s the item pool used t o c o n s t r u c t t h e c o n v e n t i o n a l and Rasch t e s t s and the item a n a l y s i s c r i t e r i a a p p l i e d i n c o n s t r u c t i n g the d i f f e r e n t t e s t s . C h a p t e r III a l s o d e s c r i b e s the comparisons made between the t e s t s c o n s t r u c t e d . C h a p t e r - 11 -IV d i s c u s s e s the r e s u l t s o f t h e comparisons between t h e c o n v e n t i o n a l and Rasch t e s t s which were c o n s t r u c t e d . C h a p t e r V p r o v i d e s a summary o f the r e s e a r c h and c o n c l u s i o n s based upon the f i n d i n g s . - 12 -CHAPTER I I PRESENTATION OF BASIC PROCEDURES AND REVIEW OF RELEVANT LITERATURE The a p p l i c a t i o n o f c o n v e n t i o n a l t e s t item a n a l y s i s and the Rasch model i n the c o n s t r u c t i o n o f t e s t s i s reviewed i n t h i s c h a p t e r . P r o c e d u r e s and r e s e a r c h r e l a t i n g t o each method a r e o u t l i n e d . The c h a p t e r c o n c l u d e s w i t h a revi e w o f the r e s e a r c h i n which t h e two methods o f t e s t item a n a l y s i s and t h e i r p r o d u c t s a r e compared. C o n v e n t i o n a l I tern A n a l y s i s C h a r a c t e r i s t i c s o f i n d i v i d u a l t e s t items have a d i r e c t e f f e c t on the c h a r a c t e r i s t i c s o f the t o t a l t e s t ; a l l p r o p e r t i e s o f the t o t a l t e s t can be d i r e c t l y d e r i v e d from the p r o p e r t i e s o f the i n d i v i d u a l items w h i c h c o m p r i s e the t e s t (:Magnusson, 1967)- The purpose o f c o n v e n t i o n a l item a n a l y s i s i n t e s t b u i l d i n g i s t o o b t a i n item p a r a m e t e r s , such as item d i f f i c u l t y i n d i c e s and c o r r e l a t i o n s o f items w i t h s u b t e s t s and t o t a l t e s t s , w h i c h w i l l be use-f u l i n c o n s t r u c t i n g t e s t s h a v i n g p a r t i c u l a r d e s i r e d c h a r a c t e r i s t i c s . In the case o f m u l t i p l e - c h o i c e i t e m s , c o n v e n t i o n a l a n a l y s i s a l s o p r o v i d e s i n f o r m a -t i o n r e l a t i n g t o the e f f e c t i v e n e s s o f the f o i l s . I tern a n a l y s i s i s performed i n o r d e r t o e x e r c i s e some c o n t r o l o v e r the e v e n t u a l means, v a r i a n c e s , r e l i a -b i l i t i e s , and v a l i d i t i e s o f measurement i n s t r u m e n t s b e i n g b u i l t (Rentz & Rentz , 1978). In the f o l l o w i n g s e c t i o n s , the n a t u r e o f c o n v e n t i o n a l t e s t i t e m p arameters and t h e i r a p p l i c a t i o n i n t e s t c o n s t r u c t i o n w i l l be e l u c i -d a ted . I tern D i f f i c u l t y Index In c o n v e n t i o n a l t e s t item a n a l y s i s , the b a s i c index o f d i f f i c u l t y , o r d i f f i c u l t y v a l u e , f o r a di c h o t o m o u s 1 y s c o r e d item i s the p e r c e n t a g e , o r - 1 3 -p r o p o r t i o n , o f s u b j e c t s p a s s i n g the item. The p r o p o r t i o n , p_, i s the a r i t h -m e t i c mean o f the item. The mean t e s t s c o r e f o r a group o f examinees i s equal t o the sum ov e r items o f the i t e m a r i t h m e t i c means. C o n s e q u e n t l y , i f one w i s h e s t o c o n s t r u c t a t e s t such t h a t a group o f examinees s i m i l a r t o t h e p i l o t t e s t group w i l l have a p r e d e t e r m i n e d mean s c o r e , items h a v i n g a mean p i l o t t e s t sample d i f f i c u l t y e q u a l t o the d e s i r e d mean s c o r e d i v i d e d by t h e number o f items t o be i n c l u d e d i n the t e s t s h o u l d be s e l e c t e d from the item p o o l ( L o r d £ N o v i c k , 1 9 6 8 ) . The p r e d e t e r m i n e d mean s c o r e would be a p p r o x i -mate o n l y because the item d i f f i c u l t y index o f an item i s always a r e f l e c t i o n o f the a b i l i t y o f t h e group t r y i n g the item (Ebel , 1 9 7 2 a ) . F^-values o f i n d i v i d u a l items a r e a l s o i m p o r t a n t i n t e s t c o n s t r u c t i o n i n t h a t they i n f l u e n c e the c h a r a c t e r i s t i c s o f t e s t s c o r e d i s t r i b u t i o n s . The p_-values, t o g e t h e r w i t h the number of items i n the t e s t and the i n t e r c o r r e -l a t i o n s among t h e s e i t e m s , i n f l u e n c e the v a r i a n c e and shape o f the d i s t r i b u -t i o n o f t o t a l t e s t s c o r e s . T o t a l t e s t v a r i a n c e i s the sum o f a l l item v a r i -ances and item c o v a r i a n c e s i n the t e s t as a whole (Magnusson, 1 9 6 7 , pp. 5 5 ~ 5 6 ) • 2 Because item v a r i a n c e , s_. , i s a f u n c t i o n o f the f r e q u e n c y o f c o r r e c t — 2 response on the i n d i v i d u a l i t e m s , t h a t i s, s_. = P_jS_j . where q_. - 1 - p_. , t e s t v a r i a n c e w i l l be maximized, o t h e r t h i n g s b e i n g e q u a l , when the v a r i a n c e o f the i n d i v i d u a l items i s maximized. T h i s w i l l o c c u r when p. = q. = . 5 , and _ JL 2 s_.' = . 2 5 as a consequence (Magnusson, 1 9 6 7 , P- 2 6 ) . Thus, the c o n t r i b u t i o n o f item v a r i a n c e t o t o t a l t e s t v a r i a n c e i s g r e a t e s t when an item has a p_-v a l u e o f . 5 and approaches z e r o as the £-value o f an i t e m approaches z e r o o r one. £-values o f i n d i v i d u a l items a f f e c t t h e shape o f t h e d i s t r i b u t i o n o f t o t a l t e s t s c o r e s . I f a t e s t c o n s i s t s o f many easy i t e m s , the shape o f the d i s t r i b u t i o n o f t o t a l t e s t s c o r e s f o r a group o f normal a b i l i t y i s l i k e l y t o be n e g a t i v e l y skewed w i t h most examinees b e i n g p l a c e d i n the upper - 14 -p o r t i o n o f the d i s t r i b u t i o n o f t o t a l t e s t s c o r e s . I f a t e s t c o n t a i n s a d i s -p r o p o r t i o n a t e number o f d i f f i c u l t i t e m s , the shape o f the d i s t r i b u t i o n o f t o t a l t e s t s c o r e s f o r a group o f normal a b i l i t y w i l l p r o b a b l y be p o s i t i v e l y skewed w i t h most examinees b e i n g p l a c e d i n the lower p o r t i o n o f the d i s t r i -b u t i o n o f t o t a l t e s t s c o r e s . A t e s t d e v e l o p e r can c o r r e c t t h e s e extremes by i n s e r t i n g a d d i t i o n a l d i f f i c u l t items i n t o the former t e s t and e a s i e r items i n t o the l a t t e r t e s t i n o r d e r t o o b t a i n a more normal d i s t r i b u t i o n o f t o t a l t e s t s c o r e s . There i s no hard and f a s t r u l e c o n c e r n i n g optimum £-values f o r items to be i n c l u d e d i n t e s t s . Some r e s e a r c h e r s a d v o c a t e the i n c l u s i o n o f items o f m i d d l e d i f f i c u l t y i n t e s t c o n s t r u c t i o n as a means t o i n c r e a s e the s p r e a d o f t e s t s c o r e s and t o improve t e s t re 1 i ab i-'l i t y . ' In one-s tudy (Ebel , 1972a ) , t h r e e t e s t s each c o n s i s t i n g o f 16 items s e l e c t e d from an e x i s t i n g 6 l - i t e m pool were c o n s t r u c t e d t o have the f o l l o w i n g c h a r a c t e r i s t i c s : 1) a t e s t i n w h i c h the d i f f i c u l t y v a l u e s o f the items were c o n c e n t r a t e d as c l o s e t o .5 as p o s s i b l e ; 2) a t e s t i n w h i c h the d i f f i c u l t y v a l u e s o f the items were d i s t r i -buted i n a u n i f o r m f a s h i o n o v e r t h e e n t i r e range o f a v a i l a b l e d i f f i c u l t y v a l u e s , and 3) a t e s t c o n s i s t i n g o f extreme d i f f i c u l t y v a l u e s a c h i e v e d by i n c l u d i n g the e i g h t e a s i e s t and e i g h t most d i f f i c u l t i t e ms. P r e d i c t a b l y , the f i r s t t e s t was found t o have the w i d e s t d i s p e r s i o n o f t e s t s c o r e s and h i g h e s t r e l i a b i l i t y o f the t h r e e t e s t s . C o n s t r u c t i o n o f t e s t s by i n c l u d i n g o n l y items h a v i n g a d i f f i c u l t y v a l u e n ear .5 does n o t , however, g u a r a n t e e a s u i t a b l e measurement i n s t r u m e n t . Item d i f f i c u l t y i s o n l y one o f t h e i t em s t a t i s t i c s t o be c o n s i d e r e d when s e l e c t i n g items f o r a t e s t . T o t a l t e s t and s u b t e s t homogeneity must a l s o be c o n s i d e r e d i n s e l e c t i n g i t e m s . Maximum t e s t homogeneity i s a c h i e v e d when items w i t h h i g h i n t e r c o r r e l a t i o n s a r e i n c l u d e d i n a t e s t (Green, L i s s i t z S M u l a i k , 1977)-An i n d i c a n t o f h i g h i t e r n - i n t e r c o r r e 1 a t i o n i s a h i g h i t e m - t o t a l t e s t o r i t em-- 15 -s u b t e s t c o r r e l a t i o n f o r each item i n the t o t a l t e s t o r s u b t e s t . W h i l e £-val-ues o f .5 do s e t an upper l i m i t on c o r r e l a t i o n s o f i n d i v i d u a l items w i t h t o t a l t e s t o r s u b t e s t s c o r e s , they do not g u a r a n t e e h i g h i n t e r c o r r e l a t i o n s between i n d i v i d u a l items and t o t a l t e s t o r s u b t e s t s c o r e s . C o n s e q u e n t l y j3-values and i t e m - t o t a l t e s t o r i t e m - s u b t e s t c o r r e l a t i o n c o e f f i c i e n t s must be c o n s i d e r e d i n tandem when items a r e b e i n g s e l e c t e d f o r a t e s t ( N u n n a l l y , 1967). S i n c e t h e r e i s l i t t l e d i f f e r e n c e i n d i s c r i m i n a t i n g power between items h a v i n g c l o s e d i f f i c u l t y v a l u e s , i t i s suggested t h a t items w i t h £-values r a n g i n g from .5 t o .7 (Brown, 1976) or .8 ( S t a n l e y & H o p k i n s , 1972) be i n c l u d -ed i n t e s t s . In u s u a l p r a c t i c e , items w i t h £-values r a n g i n g from .2 t o .8 are i n c l u d e d i n t e s t s aimed at groups of examinees i n wh i c h t h e r e i s a normal d i s t r i b u t i o n o f a b i l i t y . T h i s was the g e n e r a l p r o c e d u r e f o l l o w e d f o r the development o f achievement t e s t s f o r the B r i t i s h C olumbia S c i e n c e Assessment (Rogers, 1979)-I tern D i s c r i m i n a t i o n I n d i c e s I tern d i s c r i m i n a t i o n i n d i c e s p r o v i d e a measure o f how w e l l a t e s t i t e m d i f f e r e n t i a t e s between examinees h a v i n g g r e a t e r and l e s s e r amounts o f a b i l i t y as r e g a r d s the t r a i t measured by t h e item. The c r i t e r i o n used may be t h e s u b t e s t s c o r e o r t h e t o t a l t e s t s c o r e , w i t h t h e d i s c r i m i n a t i o n i n d e x b e i n g an i t e m - s u b t e s t o r i t e m - t o t a l t e s t c o r r e l a t i o n c o e f f i c i e n t . A h i g h p o s i t i v e c o r r e l a t i o n i s e x p e c t e d between the c o r r e c t o p t i o n i n a m u l t i p l e - c h o i c e ques-t i o n and the c r i t e r i o n s c o r e . N e g a t i v e c o r r e l a t i o n s a r e e x p e c t e d f o r t h e d i s t r a c t o r s . T h i s i s an i n d i c a t i o n t h a t s u b j e c t s g e t t i n g the c o r r e c t answer f o r an item have h i g h s c o r e s on t h e c r i t e r i o n and s u b j e c t s c h o o s i n g one o f the d i s t r a c t o r s have low s c o r e s on the c r i t e r i o n : s t u d e n t s w i t h h i g h a b i l i t y on the t r a i t b e i n g measured g e t items measuring the t r a i t c o r r e c t w h i l e s t u -- 16 -d e n t s w i t h low a b i l i t y on the t r a i t get items measuring the t r a i t wrong. The p o i n t - b i s e r i a l and b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t s a r e d i s c r i m i n -a t i o n i n d i c e s which can be used w i t h dichotomous 1y s c o r e d i t e m s . They d i f f e r i n t h e i r a s s u m ptions r e g a r d i n g the n a t u r e of the t r a i t u n d e r l y i n g the d i c h o t -omy . P o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t . The p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t measures the r e l a t i o n s h i p between a dichotomous v a r i a b l e f o r which t h e r e a r e o n l y two c a t e g o r i e s o f s c o r e s and a c o n t i n u o u s v a r i a b l e . When t h e p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t i s used, the u n d e r l y i n g t r a i t o r a b i l i t y i s t r e a t e d as a t r u e dichotomy: t h o s e who pass the item and t h o s e who f a i l t he item form two m u t u a l l y e x c l u s i v e homogeneous groups. The c o r r e -l a t i o n c o e f f i c i e n t , r , i s a s p e c i a l case o f the more g e n e r a l Pearson p r o -duct-moment c o r r e l a t i o n c o e f f i c i e n t . It can be a r r i v e d a t by c a l c u l a t i n g the P earson product-moment c o r r e l a t i o n c o e f f i c i e n t on t h e d a t a as i t e x i s t s , o r use can be made o f s e v e r a l f o r m u l a s w h i c h have been d e v i s e d t o s i m p l i f y c a l c u l a t i o n (Magnusson, 1967, pp. 198-202). The magnitude o f t h e p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t f o r an item w i t h a c r i t e r i o n depends on the p r o p o r t i o n o f s u b j e c t s p a s s i n g t h e i t e m , £, and the p r o p o r t i o n o f s u b j e c t s f a i l i n g t he i t e m , q_ = 1 - p_. The t h e o r e t i c a l maximum v a l u e o f r ^ wi11 tend t o d i m i n i s h as the i t e m - d i f f i c u l t y v a l u e s d i -v e r g e from .5- Magnusson p o i n t s out t h a t , t h e o r e t i c a l l y , t h e p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t has a magnitude l i m i t a t i o n o f a p p r o x i m a t e l y .8 pro-v i d e d t h a t t h e c o n t i n u o u s c r i t e r i o n v a r i a b l e i s n o r m a l l y d i s t r i b u t e d and t h e dichotomous v a r i a b l e forms a t r u e d i c h o t o m y , s p l i t i n such a way t h a t p_ = q_ = -5. When the assumptions t h a t t he u n d e r l y i n g c o n t i n u o u s v a r i a b l e i s n o r m a l l y d i s t r i b u t e d and the dichotomous v a r i a b l e s form a t r u e dichotomy a r e v i o l a t e d , - 17 -the p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t has been found t o have v a l u e s g r e a t e r than .8. In an i n v e s t i g a t i o n i n v o l v i n g f i v e d i f f e r e n t d i s t r i b u t i o n shapes f o r the c o n t i n u o u s v a r i a b l e ( K a r a b i n u s , 1975), the p o i n t - b i s e r a i c o r r e -l a t i o n c o e f f i c i e n t was found t o have v a l u e s g r e a t e r than .8 w i t h r e c t a n g u l a r d i s t r i b u t i o n s o f the c o n t i n u o u s v a r i a b l e h a v i n g minimal o r no o v e r l a p on each o f the d i c h o t o m i z e d v a l u e s . V a l u e s o f r , g r e a t e r than .9 were found w i t h peaked bimodal d i s t r i b u t i o n s h a v i n g minimal o r no o v e r l a p w i t h the d i c h o t o -mized v a l u e s . Thus, p o i n t - b i s e r i a 1 c o e f f i c i e n t v a l u e s g r e a t e r than .8 can be found when the shape o f the c o n t i n u o u s v a r i a b l e d i s t r i b u t i o n more c l o s e l y resembles t h a t o f the d i c h o t o m i z e d v a r i a b l e . B i s e r i a l c o r r e l a t i o n c o e f f i c i e n t . B i s e r i a l c o r r e l a t i o n c o e f f i c i e n t s a r e a l s o used w i t h dichotomous 1y s c o r e d i t e m s , but use o f t h i s c o e f f i c i e n t r e -q u i r e s the a s s u m p t i o n t h a t the dichotomy i s an a r t i f i c i a l i m p o s i t i o n on an u n d e r l y i n g a b i l i t y w h i c h i n a c t u a l i t y i s n o r m a l l y d i s t r i b u t e d and c o n t i n u o u s i n n a t u r e . There i s no j u s t i f i c a t i o n f o r use o f the b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t , _ r ^ . , w i t h d a t a f o r w h i c h t h i s a s s u m p t i o n does not seem r e a s o n -a b l e . A b a s i c a s s u m p t i o n i n measurement t h e o r y i s t h a t t r u e s c o r e s f o r i n d i -v i d u a l s f o r a t r a i t have a normal d i s t r i b u t i o n . The s p l i t t i n g o f i n d i v i d u a l s w i t h r e g a r d t o p a s s i n g o r f a i l i n g a t e s t item i s c o n s i d e r e d t o be an a r b i t -r a r y p r o c e s s w i t h the l o c u s o f t h e s p l i t dependent upon the d i f f i c u l t y o f the item. Thus, the b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t might be more p l a u s i b l e f o r use w i t h p s y c h o m e t r i c d a t a than the use o f the p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t ( H e r i r y s s e n , 1971)-The b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t i s an e s t i m a t e o f the Pearson product-moment c o r r e l a t i o n c o e f f i c i e n t . S e v e r a l f o r m u l a s f o r c a l c u l a t i n g r _ k j s from raw s c o r e d a t a e x i s t (Magnusson, 1967, PP- 202-206). The b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t can t a k e on a range o f v a l u e s l e s s than -1 and g r e a t -- 18 -e r than +1. T h i s can o c c u r i f the a s s u m p t i o n o f u n d e r l y i n g n o r m a l i t y i s a c t u a l l y i n c o r r e c t , o r when a s m a l l number o f s u b j e c t s i s used i n t e s t i n g , p r o d u c i n g a p l a t y k u r t i c d i s t r i b u t i o n o f the c o n t i n u o u s l y measured v a r i a b l e ( G l a s s S S t a n l e y , 1970). A l s o , the range o r _ f _ u j s w i l l be l e s s than -1 o r g r e a t e r than +1 i f the d i s t r i b u t i o n o f the c o n t i n u o u s l y measured v a r i a b l e i s l e p t o k u r t i c . Because p o i n t - b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t s depend upon the p and q_ v a l u e s o f an i t e m , they tend t o d i m i n i s h as the i t e m - d i f f i c u l t y v a l u e s d i -v e r g e from .5. C o n s e q u e n t l y , the p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t f o r an item w i l l always be s m a l l e r than the c o r r e s p o n d i n g b i s e r i a l v a l u e . The r e l a t i o n between the two t y p e s o f c o e f f i c i e n t s i s shown i n the f o l l o w i n g e q u a t i o n (Magnusson, 19&7, P- 205): -^ b i s / pq r u y -pb where y_ i s the o r d i n a t e o f the d i v i d i n g l i n e between the p r o p o r t i o n s p_ and q_ i n a u n i t normal d i s t r i b i t i on. Even unde r opt i mum cond i t i ons f o r _ r ^ , when p_ = q_= .5, the b i s e r i a l c o e f f i -c i e n t v a l u e w i l l exceed the p o i n t - b i s e r i a 1 v a l u e by 25% when a p p l i e d t o the same d a t a . As item d i f f i c u l t y v a l u e s d i v e r g e from .5, the d i s c r e p a n c y be-tween b i s e r i a l and p o i n t - b i s e r i a l v a l u e s a p p l i e d t o the same i t e m w i l l become even l a r g e r . D i s t r a c t o r adequacy. A l l d i s t r a c t o r s , o r f o i l s , i n a m u l t i p 1 e - c h o i c e t e s t i t em s h o u l d a t t r a c t r e s p o n s e s . The i d e a l s i t u a t i o n would be t h a t a l l i n c o r r e c t responses were e q u a l l y d i s t r i b u t e d among the i n c o r r e c t o p t i o n s o f an item. D i s t r a c t o r s a t t r a c t i n g no responses a r e i n e f f i c i e n t deadwood, mere-l y s e r v i n g t o i n c r e a s e the r e a d i n g time o f a t e s t . A l l d i s t r a c t o r s s h o u l d have a n e g a t i v e p o i n t - b i s e r i a 1 o r b i s e r i a l c o r r e l a t i o n w i t h the s u b t e s t o r t o t a l t e s t c r i t e r i o n s c o r e u n l e s s t h e r e i s an adequate r a t i o n a l e t o s u p p o r t - 19 -t h e o p p o s i t e s i t u a t i o n . Thus, e f f e c t i v e d i s t r a c t o r s a r e chosen by examinees who have a low c r i t e r i o n s c o r e . The o p p o s i t e s i t u a t i o n i n d i c a t e s t h a t t h e item might be miskeyed o r a m b i g u o u s l y worded, o r t h a t the d i s t r a c t o r s might be so c l o s e i n meaning t o the c o r r e c t answer t h a t even examinees w i t h h i g h c r i t e r i o n s c o r e s a r e u n a b l e t o d i s c r i m i n a t e among them. It i s i m p o r t a n t t h a t t e s t item a n a l y s i s be employed f o r r e v i s i o n o f items as w e l l as f o r s e l e c t i o n o f good i t e m s . R e v i s i o n o f d i s t r a c t o r s i n a p o o r l y w r i t t e n i t em can do much t o enhance i t s p s y c h o m e t r i c p r o p e r t i e s . In a s t u d y i n v o l v i n g r e v i s e d items and newly w r i t t e n items embedded i n a t e s t , i t was found t h a t a l l r e v i s e d items improved i n d i s c r i m i n a t i o n power (Lange, Lehmann, S Mehrens, 1967)• The d i s c r i m i n a t i o n i ndex used i n the s t u d y was t h e D-index i n w h i c h the d i f f e r e n c e between the number o f c o r r e c t and i n c o r -r e c t d i s c r i m i n a t i o n s i s e x p r e s s e d as a p e r c e n t a g e o f t h e maximum p o s s i b l e c o r r e c t d i s c r i m i n a t i o n s ( F i n d l e y , 1956)- The r e v i s e d items d i s p l a y e d a h i g h e r a verage d i s c r i m i n a t i o n than the new items w r i t t e n f o r the t e s t . Lange e t a l , c o n c l u d e d t h a t r e v i s i o n o f poor items f o r subsequent use was more ec o n o m i c a l than c o n s t r u c t i n g e n t i r e l y new items f o r each t e s t a d m i n i s t e r e d . They found t h a t i t t o o k a p p r o x i m a t e l y f i v e t i m e s l o n g e r t o d e v e l o p a new i t em than i t d i d t o r e v i s e a poor item. C h o i c e of i t em d i s c r i m i n a t i o n i n dex . C o n t r o v e r s y r e g a r d i n g the c h o i c e o f e i t h e r p o i n t - b i s e r i a l o r b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t s i n i t em a n a l y -s i s c e n t e r s around the f a c t t h a t t h e p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t i s a f u n c t i o n o f item d i f f i c u l t y w h i l e the b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t i s n o t . For t h i i s r e a s o n , i t i s f e l t t h a t b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t s w i l l d e m o n s t r a t e l e s s v a r i a n c e between groups of examinees d i f f e r i n g i n a b i l -i t y . T h i s w i l l not be t r u e o f the p o i n t - b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t which n e c e s s a r i l y must be low f o r items found t o be e i t h e r t o o d i f f i c u l t o r - 20 -t o o easy f o r a g i v e n group o f examinees. The f a c t t h a t use o f the p o i n t -b i s e r i a l c o e f f i c i e n t f a v o u r s t e s t items o f medium d i f f i c u l t y may be advan-t a g e o u s , however, i n t h a t e x t r e m e l y d i f f i c u l t and e x t r e m e l y easy items con-t r i b u t e v e r y l i t t l e t o the d i s c r i m i n a t i n g power of a measurement i n s t r u m e n t ( L o r d & N o v i c k , 1968). In one s t u d y , n i n e d i f f e r e n t t y p e s o f d i s c r i m i n a t i o n i n d e x were comput-ed f o r each o f t h e 120 items e q u a l l y d i s t r i b u t e d between two p a r a l l e l forms o f a t e s t ( E n g e l h a r t , 1965) - These d i s c r i m i n a t i o n i n d i c e s i n c l u d e d r , and —pb _ r ^ . , both e s t i m a t e d from a c o m p u t a t i o n a l c h a r t , w h i c h r e q u i r e d o n l y t he up-per and lower p o r t i o n s o f the s c o r e d i s t r i b u t i o n s . I n t e r c o r r e l a t i o n s among the v a r i o u s d i s c r i m i n a t i o n i n d i c e s used were a l l h i g h , and f o r the two forms of the t e s t , c o r r e l a t i o n s between r . and r. . were .982 and .987 r e s p e c t i v e -—pb — b i s l y . C o r r e l a t i o n s between each index and the f i r s t c e n t r o i d f a c t o r o f each c o r r e l a t i o n m a t r i x were h i q h . The c o r r e l a t i o n s between r , and t h e f i r s t -pb f a c t o r f o r each form o f the. t e s t were .975 and .978 r e s p e c t i v e l y . The c o r r -e s p o n d i n g c o r r e l a t i o n s between _ r ^ . and the f i r s t f a c t o r were .952 and .960. A p p l y i n g what were c o n s i d e r e d t o be o b j e c t i v e s t a n d a r d s f o r the c r i t i c a l v a l -ues o f the d i f f e r e n t d i s c r i m i n a t i o n i n d i c e s , i t was found t h a t a c o r e o f 92 items from the 120-item pool were c l a s s i f i e d as e i t h e r s a t i s f a c t o r y o r unsa t -i s f a c t o r y by a l l i n d i c e s . In a n o t h e r s t u d y , f o u r language t e s t s , each c o n s i s t i n g o f two l e v e l s f o r both r e a d i n g and l i s t e n i n g , were a d m i n i s t e r e d t o 4,300 examinees (Aleamoni S S p e n c e r , 1969)- A computer item a n a l y s i s program was d e s i g n e d t o c a l c u l a t e the item d i f f i c u l t y , r , , and r, . s t a t i s t i c s f o r each item. It was found —pb —b i s t h a t the c o r r e l a t i o n s between the r , and r. . item s t a t i s t i c s were h i q h , —pb — b i s w i t h a range o f .92 t o .99 and a mean c o r r e l a t i o n o f .97- The c o r r e l a t i o n s between each i n d e x o f d i s c r i m i n a t i o n and the d i f f i c u l t y index f o r an ite m - 21 -were found t o be s i m i l a r w i t h r e s p e c t t o the magni t u d e , s i g n , and s i g n i f i -cance l e v e l o f the c o e f f i c i e n t s . The h i g h l i n e a r r e l a t i o n s h i p between the two c o r r e l a t i o n c o e f f i c i e n t s was ta k e n as an i n d i c a t i o n t h a t r , and r{ . —pb — b i s w ould g e n e r a l l y l e a d t o the same r a n k - o r d e r i n g o f items i n t h e t e s t s , and t h e h i g h l i n e a r r e l a t i o n s h i p between each o f the d i s c r i m i n a t i o n i n d i c e s and t h e d i f f i c u l t y index d i d not s u p p o r t the c o n t e n t i o n t h a t i s more h i g h l y r e -l a t e d t o the d i f f i c u l t y l e v e l o f an item than r. . . —b i s In a more r e c e n t s t u d y , a 5 0-item v e r b a l a n a l o g y t e s t was a d m i n i s t e r e d t o 2,311 examinees, from w h i c h a random sample o f 1,000 s u b j e c t s was drawn t o d e t e r m i n e a t o t a l o f 19 d i f f e r e n t item d i s c r i m i n a t i o n i n d i c e s , i n c l u d i n g and r. . ( O o s t e r h o f , 1976) . The 50 items were ranked s e p a r a t e l y f o r each o f —b i s ' the i n d i c e s and the r a n k - o r d e r c o r r e l a t i o n s were computed f o r each o f the p o s s i b l e c o m b i n a t i o n s . The m a j o r i t y o f t h e c o r r e l a t i o n s between i n d i c e s were g r e a t e r than . 9 , w i t h t he c o r r e l a t i o n between the r , and r, . i n d i c e s b e i n g —pb — b i s .934. Two f a c t o r s a c c o u n t i n g f o r 9 5 - 1 % o f the v a r i a n c e were e x t r a c t e d i n a p r i n c i p a l f a c t o r s o l u t i o n o f the i n t e r c o r r e 1 a t i o n m a t r i x . L o a d i n g s on the f i r s t common f a c t o r were c o n s i d e r e d t o be a guage of the degree t o wh i c h each type of d i s c r i m i n a t i o n index r e l a t e d t o the f a c t o r common t o a l l o f the d i s -c r i m i n a t i o n i n d i c e s . The f i r s t f a c t o r l o a d i n g s f o r r . and r. . were .979 a —pb — b i s and .981 r e s p e c t i v e l y . The f i r s t f a c t o r a c c o u n t e d f o r 86.4% o f t h e t o t a l v a r i a n c e . A second f a c t o r , a c c o u n t i n g f o r l e s s than 10% o f the v a r i a n c e was i n t e r p r e t e d as b e i n g a s s o c i a t e d w i t h item d i f f i c u l t y . The l o a d i n g s f o r r , -pb and r , . on t h i s second f a c t o r were - . 2 0 7 and .124 r e s p e c t i v e l y . O o s t e r h o f — b i s c o n c l u d e d t h a t , s i n c e a l l d i s c r i m i n a t i o n i n d i c e s loaded h e a v i l y on one f a c t o r , they s h a r e d a g r e a t de a l o f s i m i l a r i t y . C h o i c e o f one index o v e r a n o t h e r , i n the l i g h t o f t h i s r e s e a r c h , would m i n i m a l l y a f f e c t the p r o d u c t s r e s u l t i n g from item a n a l y s i s . O o s t e r h o f recommended t h a t the t e s t d e v e l o p e r s e l e c t - l i -the, index t o be used on t h e b a s i s o f c o m p u t a t i o n a l f a c i l i t i e s a v a i l a b l e and i n t u i t i v e p r e f e r e n c e . C o r r e c t i o n o f i t e m - t o t a l c o r r e l a t i o n s f o r s p u r i o u s n e s s . The p o i n t - b i s -e r i a l o r b i s e r i a l c o r r e l a t i o n of an i t em w i t h a s u b t e s t s c o r e o r t o t a l t e s t s c o r e used as a c r i t e r i o n w i l l be s p u r i o u s l y h i g h b e c a u s e , i n a d d i t i o n t o the item's common f a c t o r v a r i a n c e w i t h t h e whole t e s t , i t s s p e c i f i c v a r i a n c e c o n t r i b u t e s t o t h e c o r r e l a t i o n as w e l l . The magnitude o f t h i s a r t i f a c t o f i t e m - t e s t c o r r e l a t i o n s i s i n v e r s e l y p r o p o r t i o n a l t o the number o f items mak-ing up t h e c r i t e r i o n and t h e s i z e o f the i n t e r c o r r e 1 a t i o n s between the i t e m s . C o r r e c t i o n f o r m u l a s t h a t g i v e t h e ' c o r r e l a t i o n between an i t e m and t h e t o t a l s c o r e based on the r e m a i n i n g n_ - 1 items i n t h e c r i t e r i o n have been d e v e l o p e d f o r both p o i n t - b i s e r i a 1 and b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t s . G u i l f o r d (1953) p r o v i d e d a f o r m u l a f o r use i n a d j u s t i n g r_. , the p o i n t -b i s e r i a l c o r r e l a t i o n o b t a i n e d between i t e m i and the t o t a l s c o r e w i t h item _ i _ i n c l u d e d , t o o b t a i n r. , the c o r r e c t e d p o i n t - b i s e r i a 1 c o r r e l a t i o n o f i t e m i — i r — w i t h the r e m a i n d e r o f t h e t o t a l s c o r e a f t e r removal o f i t e m i . G u i l f o r d p o i n t e d out t h a t t h e amount o f c o r r e c t i o n would be g r e a t e r when r_ was s m a l l than when i t was. l a r g e . The s m a l l e r i t e m - t o t a l c o r r e l a t i o n s a r e t h e most p r o b l e m a t i c when a t e s t b u i l d e r i s d e c i d i n g what items t o i n c l u d e i n a t e s t . Whi1e Gui1 f o r d c o n t e n d e d t h a t the ' a p p r o p r i a t e ' c o r r e l a t i o n c o e f f i c i e n t f o r r. was t h e p o i n t - b i s e r i a 1 c o e f f i c i e n t , he s t a t e d t h a t t h e f o r m u l a c o u l d —11 be a p p l i e d i n c o r r e c t i n g b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t s once t h e y had been changed i n t o p o i n t - b i s e r i a l e s t i m a t e s by a p p 1 i c a t i o n o f a t r a n s f o r m a t i o n formu1 a. H e n r y s s e n (1 9 6 3 ) was c o n c e r n e d w i t h t h e f a c t t h a t G u i l f o r d ' s f o r m u l a d i d not t a k e i n t o c o n s i d e r a t i o n t h a t d i f f e r e n t items would be c o r r e l a t e d w i t h somewhat d i f f e r e n t sums o f the r e m a i n i n g i t e m s . These d i f f e r e n t r e m a i n d e r s o f items w o u l d p r o d u c e t o t a l s c o r e s t h a t v a r i e d i n c o n t e n t , c o n s e q u e n t l y - 2 3 -measuring somewhat d i f f e r e n t d i m e n s i o n s . In o r d e r t o c o r r e l a t e t h e items w i t h t h e same g e n e r a l f a c t o r measured by the t o t a l t e s t , Henryssen d e v e l o p e d c o r r e c t i o n f o r m u l a s f o r r , and r, . i n w h i c h the unique p a r t s o f the item —pb —b i s v a r i a n c e s do not i n f l u e n c e the c o r r e l a t i o n . T h i s was a c c o m p l i s h e d by s u b s t i -t u t i n g a verage c o v a r i a n c e s f o r the item v a r i a n c e s . Henryssen p r o v i d e d an e m p i r i c a l d e m o n s t r a t i o n i n w h i c h the b i s e r i a l i t e m -t e s t c o r r e l a t i o n c o e f f i c i e n t s were c a l c u l a t e d f o r each item w i t h the remain-ing n_ - 1 items i n a t e s t of r^ items. A l s o , c o r r e c t e d b i s e r i a l i t e m - t e s t c o r r e l a t i o n c o e f f i c i e n t s were computed u s i n g G u i l f o r d ' s f o r m u l a and Henrys-son's new f o r m u l a . I t was shown t h a t the c o r r e c t e d b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t s o b t a i n e d u s i n g H enryssen's f o r m u l a were i d e n t i c a l t o t h e c o r r e -l a t i o n c o e f f i c i e n t s a r r i v e d a t by d i r e c t l y computing each item's £_^-s w i t h the r e m a i n i n g n_ - 1 items i n the t e s t . C o r r e c t e d c o r r e l a t i o n s o b t a i n e d by a p p l y i n g G u i l f o r d ' s f o r m u l a d e v i a t e d from the d i r e c t l y c a l c u l a t e d v a l u e s . C u r e t o n (1966) p r o v i d e d a d i f f e r e n t c o n c e p t u a l i z a t i o n o f the problems i n h e r e n t i n c o r r e c t i n g i t e m - t o t a l c o r r e l a t i o n s f o r s p u r i o u s n e s s . H i s c o n c e r n was t h a t the r e l i a b i l i t y o f t h e remainder o f the t e s t would v a r y i n v e r s e l y w i t h the r e l i a b i l i t y o f t h e item o m i t t e d . To account f o r t h i s , C u r e t o n d e r i v -ed a f o r m u l a i n w h i c h t h e c o r r e l a t i o n between an item and a t e s t was a r r i v e d at by r e p l a c i n g the removed item w i t h a ' r a t i o n a l l y e q u i v a l e n t 1 i t e m . The c o r r e c t e d i t e m - t o t a l c o r r e l a t i o n i s i n v a r i a n t w i t h t e s t v a r i a n c e and r e l i a b i l -i t y . C u r e t o n p r o v i d e s f o r m u l a s f o r c o r r e c t i n g both and J_^js« The formu-l a s s h o u l d o n l y be a p p l i e d w i t h f a c t o r i a l l y homogeneous t e s t s i n which the same t r a i t o r c o m b i n a t i o n o f t r a i t s i s measured by e v e r y item. In one e m p i r i c a l s t u d y (Wolf, 1967) , the r e s u l t s o f a p p l y i n g G u i l f o r d ' s and Henryssen's f o r m u l a s i n c o r r e c t i n g i t e m - t o t a l b i s e r i a l c o r r e l a t i o n s were i n v e s t i g a t e d by u s i n g a 69-item s c i e n c e t e s t a d m i n i s t e r e d t o 171 examinees. U n c o r r e c t e d i t e m - t o t a l c o r r e l a t i o n s , c o r r e c t e d item t o t a l c o r r e l a t i o n s o b t a i n -- 2k -ed by c o r r e l a t i n g each item w i t h t h e r e m a i n i n g n_ - 1 items i n the t e s t , and c o r r e l a t i o n s o b t a i n e d by a p p l y i n g the G u i l f o r d and Henryssen f o r m u l a s were computed. The t e s t had been randomly assembled and the i n v e s t i g a t o r was thus a b l e t o examine the e f f e c t o f v a r y i n g the number o f t e s t items by e s t a b l i s h -ing samples o f 10, 20, 30, kO, 50 and 60 i t e m s . C o r r e l a t i o n s o b t a i n e d w i t h t h e Henryssen f o r m u l a agreed e x a c t l y w i t h the c o r r e l a t i o n s o b t a i n e d by c o r r e -l a t i n g each item w i t h t h e r e m a i n i n g n_ - 1 items i n the t e s t . Use o f the G u i l -f o r d f o r m u l a produced some d i s c r e p a n c i e s but t h e s e were not as g r e a t as the d i f f e r e n c e s between the u n c o r r e c t e d i t e m - t o t a l c o r r e l a t i o n s and the c o r r e l a -t i o n s computed between each item and the r e m a i n i n g rv - 1 items i n the t e s t . As the t e s t was l e n g t h e n e d , Wolf found t h a t t h e d i s c r e p a n c i e s between -the u n c o r r e c t e d i t e m - t o t a l c o r r e l a t i o n s and the G u i l f o r d c o r r e c t e d c o r r e l a -t i o n s w i t h the c o r r e l a t i o n s computed between each item and the r e m a i n i n g n - 1 items d e c r e a s e d . By i n c l u d i n g an i d e n t i c a l c l u s t e r o f ten items i n each of the seven t e s t s , the i n v e s t i g a t o r was a b l e t o examine the e f f e c t o f t e s t l e n g t h on u n c o r r e c t e d i t e m - t o t a l b i s e r i a l c o r r e l a t i o n s as compared t o the c o r r e l a t i o n s between each item and the r e m a i n i n g - 1 items i n t h e t e s t . As t e s t l e n g t h i n c r e a s e d i t was found t h a t the d i f f e r e n c e s between the u n c o r r e c t -ed b i s e r i a l c o r r e l a t i o n s and the c o r r e l a t i o n s o f each item w i t h the r e m a i n i n g items i n the t e s t d i m i n i s h e d f o r the t e n t r a c e r i t e m s . It was c o n c l u d e d t h a t c o r r e c t i o n s f o r i t e m - t o t a l c o r r e l a t i o n s would not be n e c e s s a r y when c o n s t r u c t -i n g t e s t s from a kO-itern o r l a r g e r item p o o l . I f c o r r e c t e d c o r r e l a t i o n s were , d e s i r e d , Wolf s u g g e s t e d t h a t the Henryssen f o r m u l a was the c o r r e c t one t o a p p l y t o i t e m - t o t a l b i s e r i a l c o r r e l a t i o n s . In a n o t h e r study ( B e r k , 1978), the G u i l f o r d , Henryssen and C u r eton c o r -r e c t i o n f o r m u l a s were a p p l i e d t o p o i n t - b i s e r i a 1 i t e m - t o t a l c o r r e l a t i o n s ob-t a i n e d i n a t e s t o f 20 E n g l i s h mechanics items a d m i n i s t e r e d t o 124 examinees. E i g h t a d d i t i o n a l random samples of k, 6, 8, 10, 12, 14, 16 and 18 items were - 25 -e s t a b l i s h e d t o s t u d y the e f f e c t s o f t e s t l e n g t h on p o i n t - b i s e r i a 1 i t e m - t o t a l t e s t c o r r e l a t i o n s . It was found t h a t the G u i l f o r d c o r r e c t e d p o i n t - b i s e r i a 1 c o r r e l a t i o n s were i d e n t i c a l t o t h e c o r r e l a t i o n s computed f o r each item and t h e r e m a i n i n g n_ - 1 items i n the t e s t f o r a l l t e s t s . The Henryssen and C u r e t o n c o r r e c t i o n f o r m u l a s y i e l d e d c o r r e c t e d p o i n t - b i s e r i a 1 c o r r e l a t i o n s w h i c h were g r e a t e r than the c o r r e l a t i o n s f o r each item w i t h the r e m a i n i n g items i n the t e s t but l e s s than t h e u n c o r r e c t e d i t e m - t o t a l c o r r e l a t i o n c o e f f i c i e n t s . In g e n e r a l , the u n c o r r e c t e d , Henryssen c o r r e c t e d , and C u r eton c o r r e c t e d c o r r e l a t i o n s d e v i -a t e d l e s s from the c o r r e l a t i o n s f o r each item w i t h the r e m a i n i n g items i n t h e t e s t as t e s t l e n g t h i n c r e a s e d . There was r a p i d c onvergence o f C u r e t o n c o r -r e c t e d c o r r e l a t i o n s w i t h c o r r e l a t i o n s f o r each item and the r e m a i n i n g items i n a t e s t f o r t e s t l e n g t h s above 10 i t e m s . For a l l t e s t l e n g t h s , and f o r each o f the t h r e e t y p e s o f c o r r e c t e d c o r r e l a t i o n s as w e l l as the u n c o r r e c t e d i t e m - t o t a l c o r r e l a t i o n s , Spearman r a n k - o r d e r c o r r e l a t i o n s of item ranks w i t h c o r r e l a t i o n s f o r each i t e m and t h e r e m a i n i n g items i n t h e t e s t were computed. Use o f G u i l f o r d ' s o r C u r e t o n ' s c o r r e c t i o n f o r m u l a s y i e l d e d i t em ranks i d e n -t i c a l t o t h o s e ranks o f c o r r e l a t i o n s between items and the r e m a i n i n g items i n a t e s t f o r t e s t s o f a l l l e n g t h s . Use o f Henryssen's f o r m u l a produced d i f f e r -ent r a n k - o r d e r i n g of items i n t e s t s o f l e s s than 10 items. Berk c o n c l u d e d t h a t c o r r e c t i o n o f i t e m - t o t a l p o i n t - b i s e r i a 1 c o e f f i c i e n t s o f .4 o r h i g h e r was u n n e c e s s a r y i n t e s t s w i t h as few as 18 o r 20 i t e m s , but t h a t the G u i l f o r d o r C u r e ton c o r r e c t i o n f o r m u l a s h o u l d be a p p l i e d t o p o i n t - b i s e r i a 1 c o r r e l a t i o n s f o r items w i t h d i s c r i m i n a t i o n i n d i c e s below t h a t p o i n t . T e s t E v a l u a t i o n I n t e r n a l C o n s i s t e n c y I n t e r n a l c o n s i s t e n c y c o e f f i c i e n t s such as Hoyt's Anova and Cronbach's c o e f f i c i e n t a measure the degree t o which items i n a s e t a r e i n t e r r e l a t e d . - 26 -In e s t i m a t i n g i n t e r n a l c o n s i s t e n c y c o e f f i c i e n t s i t i s assumed t h a t the items c o m p r i s i n g a t e s t a r e a random sample o f items from an item p o p u l a t i o n i n whi c h the items measure a s i n g l e , o r u n i d i m e n s i o n a 1 , t r a i t . High i n t e r i t e m c o r r e l a t i o n s c a r r y w i t h them the i m p l i c a t i o n t h a t a t e s t c o n s i s t s o f items w h i c h measure a s i n g l e , o r u n i d i m e n s i o n a l t r a i t . Many r e s e a r c h e r s f e e l t h a t i n t e r n a l c o n s i s t e n c y i s b e s t a c h i e v e d by i n -c l u d i n g items i n a t e s t which c o r r e l a t e h i g h l y w i t h the s u b t e s t o r t o t a l t e s t c r i t e r i o n . I terns can be c o r r e l a t e d w i t h the t o t a l s c o r e f o r t h e s e t o f i t e m s , and those w i t h low i t e m - s u b t e s t o r i t e m - t o t a l t e s t c o r r e l a t i o n s a r e c u l l e d o u t . T h i s p r o c e d u r e can be rep e a t e d u n t i l a h i g h l y homogeneous, o r i n t e r n a l -l y c o n s i s t e n t t e s t i s o b t a i n e d (DuBois, 1970)- An a l t e r n a t e approach i s t o beg i n w i t h a c o r e o f items from an item pool which c o r r e l a t e most h i g h l y w i t h t h e i n t e r n a l c r i t e r i o n and, i n s u c c e s s i v e s t a g e s , add o t h e r items t h a t c o r r e -l a t e h i g h l y w i t h the s u b t e s t o r t o t a l t e s t u n t i l t he d e s i r e d l e v e l o f homogen-e i t y i s a c h i e v e d ( N u n n a l l y , 1967)• To de t e r m i n e whether the t a s k o f c o n s t r u c t -in g a homogeneous t e s t has been a c c o m p l i s h e d , measures o f i n t e r n a l c o n s i s t e n c y such as Hoyt's Anova and Cronbach's c o e f f i c i e n t a a r e then a p p l i e d . Hoyt's Anova. Hoyt (19^1) o u t l i n e d a method o f o b t a i n i n g a unique i n -t e r n a l c o n s i s t e n c y r e l i a b i l i t y c o e f f i c i e n t f o r a t e s t t h r o u g h a p p l i c a t i o n o f a n a l y s i s o f v a r i a n c e . Any g i v e n s c o r e on a t e s t , _Y_. , i s c o n s i d e r e d t o be a l i n e a r c o m b i n a t i o n o f the f o l l o w i n g components: p, a component common t o a l l persons and t o a l l v a r i a b l e s ( i t e m s ) ; I T . , a component a s s o c i a t e d w i t h the p e r s o n ; , a component a s s o c i a t e d w i t h t h e ite m , and e.^, an e r r o r component independent o f u ; I T . , and a., and w h i c h i s n o r m a l l y d i s t r i b u t e d w i t h a mean — ~2 o f z e r o and a v a r i a n c e o f a . The v e c t o r , Y.., i s thus composed o f f o u r i n -e In-dependent f a c t o r s o r components: Y. . = u + TT. + a. + e. . ~11 1 1 11 - 27 -Assuming t h a t items and examinees a r e random samples drawn from t h e i r r e s p e c t i v e p o p u l a t i o n s , t h e two-way random e f f e c t s model i s a p p l i e d t o o b t a i n an ANOVA t a b l e w i t h MS^ ^ . MS.„ , MS , and MS terms. MS — t o t a l , — i terns —^-persons — e r r o r — e r r o r i s the mean square a s s o c i a t e d w i t h the p e r s o n s - b y - i t e r n s i n t e r a c t i o n , MS x . The r e l i a b i l i t y c o e f f i c i e n t i s o b t a i n e d by d i v i d i n q the — p e r s o n s items 3 d i f f e r e n c e between MS and MS- by MS (Stan l e y 1 9 7 1 , P- 399). — p e r s o n s — e r r o r — p e r s o n s Hoyt's f o r m u l a f o r i n t e r n a l c o n s i s t e n c y r e l i a b i l i t y g i v e s t h e r a t i o o f t r u e v a r i a n c e f o r p e r s o n s , t h a t i s , v a r i a n c e not due t o the u n r e l i a b i l i t y o f the measurement i n s t r u m e n t , t o t o t a l t e s t v a r i a n c e . The s t a n d a r d e r r o r o f measurement i s o b t a i n e d by t a k i n g the square r o o t o f t h e q u o t i e n t o f SS d i v i d e d by jc - 1, where k_ i s the number o f persons t a k i n g the t e s t . Hoyt p o i n t e d out t h a t h i s method o f e s t i m a t i n g t e s t r e l i a b i l i t y y i e l d e d i d e n t i c a l r e s u l t s t o t h o s e a r r i v e d a t w i t h a p p l i c a t i o n o f t h e K u d e r - R i c h a r d s o n f o r m u l a 20, KR 20, (Kuder & R i c h a r d s o n , 1937) i f items were s c o r e d dichotomous1y--1 i f c o r r e c t and 0 i f I n c o r r e c t . Cronbach's c o e f f i c i e n t a l p h a . C o e f f i c i e n t a l p h a i s a l s o a two-way a n a l y -s i s o f v a r i a n c e approach t o o b t a i n i n g a c o e f f i c i e n t of i n t e r n a l c o n s i s t e n c y f o r a t e s t (Cronbach, 1970). When a p p l i e d t o the same dichotomous 1y s c o r e d t e s t t a k e n as a who l e , i n t e r n a l c o n s i s t e n c y c o e f f i c i e n t s o b t a i n e d u s i n g KR 20, Hoyt's Anova, and Cronbach's c o e f f i c i e n t a l p h a y i e l d e q u i v a l e n t r e s u l t s . Cron-bach's a l p h a i s a more g e n e r a l f o r m u l a , however, i n t h a t i t can be a p p l i e d t o e s t i m a t e an i n t e r n a l c o n s i s t e n c y c o e f f i c i e n t f o r a t e s t w h i c h i s a co m p o s i t e o f s u b t e s t s (Cronbach, 1951). T e s t s d i v i s i b l e i n t o d i s t i n c t s u b t e s t s a r e broken up i n t o t h e s e b e f o r e a p p l i c a t i o n o f the f o r m u l a . To o b t a i n i n t e r p r e t -a b l e s u b t e s t s c o r e s , the b e s t s e t o f items i s one h a v i n g a l a r g e c o e f f i c i e n t a l p h a and i s not f u r t h e r d i v i s i b l e i n t o s m a l l e r c l u s t e r s o f i t e m s . The a l p h a c o e f f i c i e n t i s an i n d i c a t i o n o f how w e l l t h e . s u b t e s t s i n a t o t a l t e s t tend t o measure the same f a c t o r . - 2 8 -The Hoyt index i s computed o v e r s u b t e s t s , w h i l e the a l p h a c o e f f i c i e n t i s an i n d e x o f i n t e r n a l c o n s i s t e n c y f o r s u b t e s t s . The two i n d i c e s w i l l o n l y be e q u i v a l e n t when the c o v a r i a n c e s o f s u b t e s t i t e m s , computed w i t h i n each sub-t e s t , a l l equal z e r o . S u b t e s t items c o r r e l a t i n g p o s i t i v e l y w i t h each o t h e r in each s u b t e s t . w i l l y i e l d a Hoyt v a l u e g r e a t e r t h a n a l p h a . A l p h a w i l l be g r e a t e r than the Hoyt index i f s u b t e s t items c o r r e l a t e n e g a t i v e l y w i t h i n each s u b t e s t . S i n c e , i n d e s i g n a t i n g s u b t e s t s , one w i s h e s t o i n c l u d e items h a v i n g h i g h i n t e r c o r r e l a t i o n s , the former s i t u a t i o n w i l l n o r m a l l y a r i s e . C o e f f i c -i e n t a l p h a r e p r e s e n t s a lower bound as an i n d e x o f common-factor c o n c e n t r a t i o n i n a t e s t , and an upper bound as an i n d e x o f f i r s t - f a c t o r c o n c e n t r a t i o n among the items (Cronbach, 1951)-S t a n d a r d E r r o r o f Measurement In c o n v e n t i o n a l t e s t t h e o r y i t i s hoped t o a p p r o x i m a t e a p e r s o n ' s t r u e s c o r e on the t r a i t b e i n g measured by a t e s t . Because a person's t e s t s c o r e i s a sample s c o r e , i t w i l l d e v i a t e from h i s / h e r t r u e s c o r e because o f d i f f e r -ences i n the items used i n the p a r a l l e l forms o f t e s t s and because o f such measurement e r r o r s caused by d i f f e r e n c e s i n examiner/examinee i n t e r a c t i o n s , changes i n examinee h e a l t h and m o t i v a t i o n , c l a r i t y o f i n s t r u c t i o n s , o u t s i d e d i s t u r b a n c e s , g u e s s i n g , and s u b j e c t i v i t y i n s c o r i n g ( S t a n l e y , 1971, P- 364). The measure o f t h i s d e v i a t i o n from a p e r s o n ' s t r u e s c o r e i s c a l l e d t h e s t a n -d a r d e r r o r of measurement (SEM) o f the t e s t (Magnusson, 1967, P- 79). In c o n v e n t i o n a l , o r c l a s s i c a l , t e s t t h e o r y the SEM i s an e s t i m a t e o f t h e a v e r a g e s t a n d a r d d e v i a t i o n o f t h e d i s t r i b u t i o n o f measurement e r r o r s made when we a l l o w a g i v e n i n d i v i d u a l ' s o b t a i n e d s c o r e on each one o f a t h e o r e t i c -a l l y i n f i n i t e number of p a r a l l e l t e s t s t o r e p r e s e n t h i s / h e r t r u e s c o r e . Measurement e r r o r s a r e assumed t o be n o r m a l l y d i s t r i b u t e d , m u t u a l l y independ-e n t , and independent o f the t r u e s c o r e w h i c h they r e p r e s e n t . The l a s t assump-- 2 9 -t i o n i s p a r t i c u l a r l y d e b a t a b l e and has l e t t o the development o f d i f f e r e n t e q u a t i o n s f o r the SEM i n w h i c h the assum p t i o n o f equal v a r i a n c e o f e r r o r d i s t r i b u t i o n s f o r v a r y i n g t r u e s c o r e s i s not made (Magnusson, 1 9 6 7 , pp. 8 2 - 8 6 ) . The s t a n d a r d e r r o r o f measurement f o r a t e s t , as d e f i n e d above, i s d i r e c t -l y p r o p o r t i o n a l t o the square r o o t o f the number o f items on the t e s t . T h i s r e l a t i o n s h i p has been proved m a t h e m a t i c a l l y (Gardner, 1 9 7 0 ) and has been sub-s t a n t i a t e d t h r o u g h e m p i r i c a l i n v e s t i g a t i o n ( L o r d , 1 9 5 9 ) . L o r d i n v e s t i g a t e d the r e l a t i o n s h i p s between the s t a n d a r d e r r o r o f measurements f o r 5 0 t e s t s o f moderate d i f f i c u i t y . and t e s t means, s t a n d a r d d e v i a t i o n s , item d i f f i c u l t i e s , and number o f i t e m s . . He found t h a t t h e number o f items i n a t e s t was the o n l y s i g n i f i c a n t f a c t o r r e l a t i n g t o the t e s t s t a n d a r d e r r o r s o f measurement. The c o r r e l a t i o n s between t e s t s t a n d a r d e r r o r s o f measurement, when r e l i a b i l -i t y o f the t e s t s was d e t e r m i n e d u s i n g t h e KR 2 0 f o r m u l a , and t h e sq u a r e r o o t of the number o f items on the t e s t s was . 9 9 6 . The best f i t t i n g r e g r e s s i o n l i n e y i e l d e d t h e r e l a t i o n s h i p SEM = . 4 3 2 / n. , where £ i s the number o f items on a t e s t . In an e m p i r i c a l s t u d y t o t e s t Lord's f i n d i n g s , Guretoh,..Cook, F i s c h e r , L a s e r , R o c k w e l l , & Simmons ( 1 9 7 3 ) i n v e s t i g a t e d the r e l a t i o n s h i p between the SEM and the square r o o t o f the number o f items i n a t e s t f o r s i x d i f f e r e n t t e s t s . These r e s e a r c h e r s found t h a t the r e g r e s s i o n c o e f f i c i e n t , when Lord's e a r l i e r d a t a was merged w i t h t h e i r d a t a , was . 4 1 7 , a somewhat lower v a l u e than Lord's c o n s t a n t o f p r o p o r t i o n a l i t y . Ebel ( 1 9 7 2 b ) p r o v i d e s a r a t i o n a l e x p l a n a t i o n o f why i n c r e a s i n g the num-ber o f items i n a t e s t s h o u l d improve t e s t r e l i a b i l i t y : • 1. The t r u e component of a s c o r e i s p r o p o r t i o n a l t o the number of e q u i v a l e n t items t h a t c o n t r i b u t e t o i t . 2 . The e r r o r component o f a s c o r e i s p r o p o r t i o n a l t o the square r o o t o f the number o f e q u i v a l e n t elements t h a t c o n t r i b u t e t o i t . - 30 -Magnusson (1967) d e s c r i b e s the same p r i n c i p l e i n somewhat d i f f e r e n t terms: W h i l e the t r u e v a r i a n c e i n c r e a s e s as the s q u a r e o f the number o f t i m e s the t e s t i s i n c r e a s e d i n l e n g t h , the i n c r e a s e i n e r r o r v a r -i a n c e i s d i r e c t l y p r o p o r t i o n a l t o the i n c r e a s e i n t e s t l e n g t h . T h i s means t h a t , when the t e s t i n c r e a s e s i n l e n g t h , the t r u e v a r -i a n c e r e p r e s e n t s a g r e a t e r s hare o f the t o t a l v a r i a n c e . T h i s i n t u r n means t h a t the t e s t w i l l become more r e l i a b l e , (p. 72) The Rasch L o g i s t i c Model General L a t e n t T r a i t Theory L a t e n t t r a i t t h e o r y , a l s o r e f e r r e d t o as item response t h e o r y , o r item c h a r a c t e r i s t i c c u r v e (ICC) t h e o r y , assumes t h a t examinee performance i n t e s t -i n g s i t u a t i o n s i s p r e d i c t a b l e o r e x p l a i n a b l e by d e f i n i n g examinee c h a r a c t e r -i s t i c s , termed t r a i t s , and u s i n g t e x t s c o r e s t o p r e d i c t o r e x p l a i n p e r f o r m -ance on t e s t s . The t r a i t s o r a b i l i t i e s a r e not d i r e c t l y m e asurable but a r e i n f e r r e d from t e s t performance (Hambleton, Swaminathan, Cook, E i g n o r S G i f -f o r d , 1978). L a t e n t t r a i t models a r e m a t h e m a t i c a l models w h i c h attempt t o e x p l a i n , as a c o n c i s e m a t h e m a t i c a l f u n c t i o n , the r e l a t i o n s h i p between exami-nee performance on a t e s t and the u n o b s e r v a b l e l a t e n t t r a i t s o r a b i l i t i e s w h i c h u n d e r l i e t h i s p erformance (Hambleton £ Cook, 1977). When a s i n g l e l a t e n t t r a i t i s assumed t o u n d e r l i e performance on a t e s t , t h e m a t h e m a t i c a l f u n c t i o n t h a t r e l a t e s t h e p r o b a b i l i t y o f s u c c e s s on an item t o the u n d e r l y i n g t r a i t measured by the i t e m s e t o r t e s t c o n t a i n i n g the item i s c a l l e d an item c h a r a c t e r i s t i c c u r v e , o r t r a c e l i n e . The item c h a r a c t e r i s -t i c c u r v e i s the n o n l i n e a r r e g r e s s i o n f u n c t i o n of item s c o r e on the l a t e n t t r a i t u n d e r l y i n g the t e s t (Hambleton & Cook, 1977)- The ICC g r a p h i c a l l y d e p i c t s the r e l a t i o n s h i p between an examinee's p o s i t i o n on the l a t e n t t r a i t measured by a t e s t and the p r o b a b i l i t y o f p a s s i n g an item i n t h a t t e s t . L a t e n t T r a i t Models Three l a t e n t t r a i t models have been d e v e l o p e d . Each model i s an attempt t o m a t h e m a t i c a l l y s p e c i f y , as c o n c i s e l y as p o s s i b l e , what o c c u r s when a per -- 31 -son p o s s e s s i n g a g i v e n amount o f a b i l i t y r e l a t e d t o a p a r t i c u l a r t r a i t a t -tempts a t e s t item b e l i e v e d t o measure t h a t t r a i t . The models d i f f e r i n the number o f v a r i a b l e s c o n s i d e r e d n e c e s s a r y t o p r o v i d e an a c c u r a t e d e s c r i p t i o n o f the outcome o f the e n c o u n t e r between examinee and t e s t item. A l l t h r e e models employ o n l y one person p a r a m e t e r , o r a b i l i t y p a r a m e t e r , i n t h e i r form-u l a t i o n o f the ICC f u n c t i o n . They d i f f e r i n t h e i r number o f item p a r a m e t e r s c o n s i d e r e d t o be n e c e s s a r y i n d e f i n i n g the f u n c t i o n . The most complex model i n c l u d e s t h r e e item p a r a m e t e r s - - i t e r n d i f f i c u l t y , i t e m d i s c r i m i n a t i o n , and a parameter f o r g u e s s i n g . A l e s s g e n e r a l two-parameter model o m i t s the g u e s s i n g parameter i n p r e d i c t i n g an examinee's s u c c e s s on an item. The Rasch model i s the s i m p l e s t o f t h e t h r e e c o n c e p t u a l i z a t i o n s , e m p l o y i n g o n l y the i t e m d i f f i -c u l t y parameter i n i t s m a t h e m a t i c a l d e s c r i p t i o n o f the ICC. U n t i l v e r y r e c e n t l y , the Rasch model has been the most h e a v i l y i n v e s t i -g a t e d o f the t h r e e models. The o t h e r two models r e q u i r e complex s c o r i n g r u l e s and m a t h e m a t i c a l l y u n w i e l d y s o l u t i o n s a r e n e c e s s a r y t o o b t a i n v a l u e s f o r the a d d i t i o n a l item p a r a m e t e r s . C u r r e n t l y , however, r a p i d and e f f i c i e n t computer programs have been d e v e l o p e d f o r the two- and t h r e e - p a r a m e t e r models as wel 1 . The Rasch S i m p l e L o g i s t i c Model The Rasch model was d e v e l o p e d i n d e p e n d e n t l y of the o t h e r l a t e n t t r a i t models by the D a n i s h m a t h e m a t i c i a n Georg Rasch i n t h e e a r l y 1960's. In f o r m u l a t i n g h i s model, Rasch made t h r e e a s s u m p t i o n s (Rasch, 1966): (a) To each s i t u a t i o n i n w h i c h a s u b j e c t (s_ = 1, 2, ... n_) has t o answer an i t e m (_[_ = 1, 2, ... m) t h e r e i s a c o r r e s p o n d i n g p r o b a b i l i t y of a c o r r e c t answer (X . = 1) w h i c h we s h a l l w r i t e i n the form P r ( X . = 1) = ^ s i (R . > 0). - L L " 1 + R . - 5 1 — s 1 - 3 2 -(b) The s i t u a t i o n p a r a m e t e r , R^., i s the p r o d u c t o f two f a c t o r s , R . = A W., —s i — s — _ i _ where A p e r t a i n s t o the s u b j e c t and W. t o the item. — s J — j _ (c) Given the v a l u e s o f the p a r a m e t e r s , a l l answers a r e s t o c h a s t i c -a l l y independent, (p. 5 0 ) A^, t h e s u b j e c t p a r a m e t e r , may tak e on any v a l u e g r e a t e r than z e r o and i s a measure o f the a b i l i t y o f the s u b j e c t w i t h r e s p e c t t o the k i n d s o f items b e i n g answered. H i g h e r A^ v a l u e s i n d i c a t e g r e a t e r a b i l i t y . W., the i t e m p a rameter, may a l s o t a k e any v a l u e g r e a t e r than z e r o and measures the s i m p l i c -i t y , o r e a s i n e s s , o f an item. I t s r e c i p r o c a l , JD. = 1/W., measures the d i f f i -c u l t y of an i t e m , w i t h h i g h e r v a l u e s i n d i c a t i n g items o f g r e a t e r d i f f i c u l t y . I f JJ. i s used i n s t e a d o f VJ. , the Rasch e q u a t i o n becomes: Pr( X . = 1) = — s i = — s — i — s I 1 + R . 1 + A W. —s i — s _ — j _ A / D. A —s — i — s 1 + A /D. A + D. —_s_ — \ _ —s_ — j _ A^ and JJ. are measureable on an u n d e r l y i n g r a t i o s c a l e r a n g i n g from 0 t o +°°". The s i m p l e l o g i s t i c form o f the Rasch e q u a t i o n may a l s o be d e r i v e d from the above r e l a t i o n s h i p s (O'Shea, 1979). T h i s e q u a t i o n i s as f o l l o w s : a - d. Pr( X . = 1) = e -—s i - i a - d. 1 + e 1 ^ where i s the n a t u r a l l o g a r i t h m o f t h e p e r s o n , o r s u b j e c t , a b i l i t y p a r a meter, and d_. i s t h e n a t u r a l l o g a r i t h m o f the item d i f f i c u l t y parameter. The above f o r m u l a t i o n o f the Rasch s i m p l e l o g i s t i c model i s common i n the c u r r e n t l i t e r a t u r e (Hamb1eton,et a l , 1978). I t i s t h e d i f f e r e n c e between t h e person a b i l i t y p a r ameter, a_^, and the item d i f f i c u l t y p a r a m e t e r , c[. , which a f f e c t s the p r o b a b i l i t y o f a s u c c e s s f u l outcome when a per s o n o f - 3 3 -g i v e n a b i l i t y a t t e m p t s an item o f g i v e n d i f f i c u l t y . The p a r a m e t e r s , a^ and d_. , can t h e o r e t i c a l l y v a r y from _ c° t o +°° and c o n s e q u e n t l y so can t h e i r d i f f e r -ence ( W r i g h t , 1 9 7 7 a ) - The u n i t o f measurement f o r both parameters i s the 1 l o g i t ' , w h i c h i n p r a c t i c a l a p p l i c a t i o n s u s u a l l y ranges from - 3 t o + 3 ( R e n t z , 1 9 7 9 ) - N e g a t i v e v a l u e s o f a_^  and cL i n d i c a t e p e r s o n s o f lower a b i l i t y and e a s i e r i t e m s , r e s p e c t i v e l y . P r o b a b i l i t y o f a c o r r e c t answer t o an i t e m , as d e t e r m i n e d by the l o g i s t i c e q u a t i o n , remains i n i t s r e s t r i c t e d range between z e r o and one, because the d i f f e r e n c e between the parameters i s a p p l i e d as the exponent t o e, the base o f the n a t u r a l l o g a r i t h m s . The f o l l o w i n g examples i l l u s t r a t e how the i n t e r p l a y between a^ and cL v a l u e s a f f e c t the p r o b a b i l i t y o f a c o r r e c t r e s p o n s e t o an i t e m ( W r i g h t , 1 9 7 7 a ) . Suppose a person o f h i g h a b i l i t y , = 3 , a t t e m p t s an item o f average d i f f i -c u l t y , d_. = 0 . The l o g odds f o r s u c c e s s on t h i s item i s equal t o - d. = 3 - 0 = 3 , and the v a l u e o f the p r o b a b i l i t y e x p r e s s i o n , P r ( X ^ . = 1 ) i s . 9 5 -Now suppose a person o f average a b i l i t y , a_^  = 0 , a t t e m p t s a v e r y d i f f i c u l t i t e m , d_. = 3. The l o g odds f o r s u c c e s s on the item f o r t h a t p e r s o n w i l l be 0 - 3 = _ 3 , and the p r o b a b i l i t y i s . 0 5 . I f the a b i l i t y o f the pe r s o n answer-ing the item matches t h e d i f f i c u l t y o f the item e x a c t l y , the l o g odds f o r suc-cess on the item i s equal t o 0 and the p r o b a b i l i t y f o r s u c c e s s on the item i s • 5 . The p r o b a b i l i t y e x p r e s s i o n i l l u s t r a t e s t h a t the g r e a t e r the p o s i t i v e d i f f e r e n c e s between a^ and d_. becomes, t h e g r e a t e r a person's chance f o r suc-c e s s on the it e m . The g r e a t e r the n e g a t i v e d i f f e r e n c e becomes, the lower i s the p e r s o n ' s chance f o r s u c c e s s . When person a b i l i t y and item d i f f i c u l t y a r e matched p e r f e c t l y , t h e r e i s a 5 0 % chance f o r s u c c e s s on the it e m . It can be demonstrated t h a t Rasch person a b i l i t y p arameters and item d i f f i c u l t y p a r a m e t e r s a r e measured on a r a t i o s c a l e . Suppose t h r e e p e r s o n s , /\, B_ and C_, w i t h a b i l i t i e s ! , 2 and 3 r e s p e c t i ve 1 y, attempt the same i tern - 34 -w h i c h has a d i f f i c u l t y o f . 5 - The odds on s u c c e s s f o r the same item would be e 0 ' 5 , e 1 ; 5 and e 2 * 5 f o r p e r s o n s A_, B_ and r e s p e c t i v e l y . The v a l u e s a r e 1.65, 4 .48 and 12.18. In each c a s e , the odds on s u c c e s s f o r a g i v e n person on the item compared t o the odds on s u c c e s s f o r the person o f next h i g h e s t a b i l i t y r e s u l t s i n the r a t i o 4 .48 / 1 .65 = 12.18/4.48 = 2.72. Thus, persons d i f f e r i n g by equal i n t e r v a l s on th e Rasch a b i l i t y s c a l e d i s p l a y the same r a t i o s f o r odds on s u c c e s s w i t h an item o f g i v e n Rasch d i f f i c u l t y . S i m i l a r examples, i n w h i c h the a b i l i t y o f a g i v e n person i s h e l d c o n s t a n t and t h a t p e r s o n ' s odds on su c c e s s w i t h items h a v i n g Rasch d i f f i c u l t y e s t i m a t e s d i f f e r -ing by equal i n t e r v a l s c o u l d be c a l c u l a t e d . The Rasch model i s the o n l y l a t e n t t r a i t model w h i c h p e r m i t s t h i s s o r t o f i n t e r p r e t a t i o n o f p e r s o n a b i l i -t y and i t e m d i f f i c u l t y p arameters (Hambleton e t a l , 1978). Item C h a r a c t e r i s t i c Curves f o r t h e Rasch Model F i g u r e 1 d e p i c t s t h r e e item c h a r a c t e r i s t i c c u r v e s which a r e c o m p a t i b l e w i t h the Rasch model. The v a l u e s o f the a b i l i t y p a r a m e t e r , a , a r e p l o t t e d — s a l o n g t h e a b s c i s s a o f the gr a p h i n the p r a c t i c a l range, -3 t o +3. The o r d i -n a te shows the p r o b a b i l i t y o f p a s s i n g an item o f g i v e n d i f f i c u l t y as i t r e -l a t e s t o i n c r e a s i n g a b i l i t y . The ICC ' s a r e smooth and monotonica1 1y i n c r e a s e as a b i l i t y i n c r e a s e s . They have e q u a l s l o p e s because the Rasch model assumes equal d i s c r i m i n a t i n g power f o r a l l items. In ICC t h e o r y , the item d i f f i c u l t y index i s d e f i n e d as the a b i l i t y v a l u e r e q u i r e d t o have a 50% chance o f ob-t a i n i n g the p a r t i c u l a r item i n q u e s t i o n c o r r e c t . The d i f f i c u l t y i n d e x o f the t h r e e items shown i s -1, 0 and +1, r e s p e c t i v e l y . Assumptions and Robustness o f t h e Rasch L o g i s t i c Model Assumptions a r e the n e c e s s a r y c o n d i t i o n s f o r the consequences o f a p r o -posed model t o be t r u e ( R e n t z , 1979). Rasch's t h r e e o r i g i n a l a s s u m p t i o n s - -a) a d o p t i o n o f a p r o b a b i l i t y model; b) d e f i n i t i o n o f the s i t u a t i o n p a r ameter, - 3 5 -3 - 2 - 1 0 1 2 3 F i g u r e 1 . Three ICC 1s t h e o r e t i c a l l y o b t a i n a b l e u s i n g the Rasch model. - 3 6 -— s i , as b e i n g the p r o d u c t o f two f a c t o r s o n l y , person a b i l i t y , , and i t e m d i f f i c u l t y , W^. , and c) s t o c h a s t i c independence o f answers, were the o n l y a s s u mptions made by him i n d e v e l o p i n g the model (Rasch, 1 9 6 6 ) . In a d d i t i o n t o the o r i g i n a l a s s u m p t i o n s made by Rasch, the a s s u m p t i o n s o f u n i d i m e n s i o n a 1 -i t y o f the t r a i t b e i n g measured, l a c k o f g u e s s i n g on m u l t i p l e - c h o i c e i t e m s , and equal i t e m - d i s c r i m i n a t i o n s have a r i s e n from R a s c h 1 s second a s s u m p t i o n . The l a t t e r t r i a d o f a s sumptions has been i n v o l v e d i n most o f the e m p i r i c a l i n v e s t i g a t i o n s c o n c e r n i n g the r o b u s t n e s s o f t h e Rasch model. The r o b u s t n e s s o f a model i s the c o n d i t i o n t h a t e x i s t s when the consequences o f a model h o l d a l t h o u g h a l l o f the assumptions o f t h e model may not have been s t r i c t l y met ( R e n t z , 1 9 7 9 ) . A d o p t i o n o f a p r o b a b i l i t y model. In Rasch's f i r s t a s s u m p t i o n , a proba-b i l i t y , o r s t o c h a s t i c , model i s adopted as a c o n v e n i e n t d e v i c e t o d e s c r i b e the i n t e r a c t i o n between an examinee and a t e s t item. Rasch p r e s e n t e d a model w h i c h d e s c r i b e d t h e p r o b a b i l i t y o f o b t a i n i n g a t e s t i t e m c o r r e c t as a f u n c t i o n o f the item's d i f f i c u l t y and the examinee's a b i l i t y w i t h r e s p e c t t o the t r a i t b e i n g measured by the item. The model appears t o be c o m p a t i b l e w i t h r e a l i t y . Rasch, however, does not t a k e a p o s i t i o n on the q u e s t i o n of whether or not t e s t i t e m responses can be u l t i m a t e l y e x p l a i n e d i n c a u s a l terms (Rasch, 1 9 6 6 ) . S t o c h a s t i c independence. In a p p l i c a t i o n s of the Rasch model t o d a t a , a l l answers must be s t o c h a s t i c a l l y i n d e p e n d e n t , o r e x h i b i t ' l o c a l independence'. T h i s means t h a t a l l items must meet a c r i t e r i o n o f i n t e r - i t e m independence i n t h a t the response g i v e n by a person t o one item does not a f f e c t the response g i v e n by t h a t person on o t h e r items i n the t e s t ( W h i t e l y £ Dawis, 197*0. Thus, m u l t i - p a r t , o r c h a i n e d t e s t items a r e not c o m p a t i b l e w i t h the a p p l i c a -t i o n o f the Rasch model. A l s o , t e s t s a n a l y z e d u s i n g the Rasch p r o c e d u r e must e s s e n t i a l l y be. power t e s t s u n i n f l u e n c e d by speededness. Rasch ( 1 9 6 6 ) - 37 -e n c o u n t e r e d the problem o f speededness when h i s model f a i l e d t o f i t the d a t a from t e s t s i n which examinees were w o r k i n g under a time s t r e s s . When he s t r a t i f i e d t h e examinees a c c o r d i n g t o o v e r a l l w o r k i n g speed, the d a t a f i t the model. Thus, item s c o r e s must be s t a t i s t i c a l l y r e l a t e d t o each o t h e r o n l y t h r o u g h the u n d e r l y i n g t r a i t measured by a t e s t . When items a re not a t t e m p t -t e d on a speed t e s t , p erformance becomes a f u n c t i o n o f an e x t r a n e o u s f a c t o r o t h e r than the l a t e n t t r a i t b e i n g measured. A n o t h e r f a c e t o f s t o c h a s t i c independence i s i n t e r - p e r s o n independence (O'Shea, 1979) w h i c h i m p l i e s t h a t t h e r e must be no c o p y i n g o r c h e a t i n g e x i s t -i n g under group t e s t i n g c o n d i t i o n s . The p r o b a b i l i t y t h a t two examinees get a p a r t i c u l a r t e s t i t em c o r r e c t must r e l a t e o n l y t o t h e i r a b i l i t i e s on the l a -t e n t t r a i t b e i n g measured and cannot be i n f l u e n c e d by any o t h e r f a c t o r s . The u n i d i m e n s i o n a 1 i t y a s s u m p t i o n . In the a p p l i c a t i o n o f l a t e n t t r a i t models, i t i s t y p i c a l t o assume t h a t the l a t e n t space i s u n i d i m e n s i o n a 1 - -items i n a t e s t a r e homogeneous, measuring o n l y a s i n g l e a b i l i t y o r t r a i t (Hambleton & Cook, 1977). Rasch (1966) d i d not e x p l i c i t l y s t a t e t h i s assump-t i o n but i t i s a n e c e s s a r y a s s u m p t i o n i n t h e a p p l i c a t i o n o f l a t e n t t r a i t mod-e l s i n g e n e r a l ( L o r d & N o v i c k , 1968). W h i t e l y and Dawis (1974) p r e s e n t a d e f i n i t i o n o f u n i d i m e n s i o n a l i t y as they b e l i e v e i t a p p l i e s t o t h e Rasch model. Cons i d e r : ... an item by s c o r e group m a t r i x i n wh i c h k. items a r e o r d e r e d by t h e i r d i f f i c u l t y l e v e l and k - 1 s c o r e groups by o b t a i n e d raw s c o r e s . The s c o r e groups f o r w h i c h a l l items a r e e i t h e r passed o r f a i l e d a r e e x c l u d e d from the m a t r i x , s i n c e t h e s e extreme s c o r e groups p r o v i d e no d i f f e r e n t i a l i n f o r m a t i o n about the it e m s . The c e l l e n t r i e s r e p r e s e n t the p r o b a b i l i t y , t n a t item J_ w i l l be passed by th e s c o r e group j _ . ... un i d i mens i ona 1 i t y i m p l i e s t h a t f o r each i t e m , P.. i s l e s s than P . „ and P.„ i s l e s s than P.. and —j_1 j_2 —j_2 —\_} so on t o P. , so t h a t the p r o b a b i l i t y o f p a s s i n g the item 1 , K — I i n c r e a s e s r e g u l a r l y w i t h the t o t a l s c o r e . Each i t e m , then o r d e r s the s u b j e c t s i n the same way. A n o t h e r i m p l i c a t i o n r e q u i r e d f o r c o n j o i n t measurement o f s u b j e c t s and i t e m s , i s t h a t items a r e o r d e r e d the same way w i t h i n each s c o r e group. On the item by s c o r e group m a t r i x , t h i s im-- 3 8 -p l i e s t h a t P , . i s l e s s than P.. and P.. i s l e s s than P n., e t c . t o - 1 j _ - 2 j - 2 j _ - 3 j _ P, . w i t h i n each s c o r e group. Tpp- 1 6 4 - 1 6 6 ) -kj_ The above c o n c e p t u a l i z a t i o n o f u n i d i m e n s i o n a 1 i t y can be d e p i c t e d d i a g r a m -a t i c a l l y as shown i n F i g u r e 2 (O'Shea, 1 9 7 9 ) . The d i a g r a m shows t h e m a t r i x o f p r o b a b i l i t i e s f o r t h e c e l l s o f a t e s t o f k_ i t e m s , the i t e m s h a v i n g been a r r a n g e d i n o r d e r o f i n c r e a s i n g d i f f i c u l t y . For t h e c o n d i t i o n o f u n i d i m e n -s i o n a l i t y t o be s a t i s f i e d , the items must be o r d e r e d the same way w i t h i n each s c o r e group as f o l l o w s , P... > P.„ > P.. > ... > P., . -^_1 — j _ 2 — j _ 3 — i k A l s o , each i t e m must o r d e r s u b j e c t s by membership i n a s c o r e g r o u p i n the same way as f o l l o w s , P. . < P_. < P.. < ... < P,. .... - 1 j_ - 2 j _ - 3 j _ -(k - 1 ) j _ Raw S c o r e s 1 terns 1 2 3 k 1 ^ 1 1 p-u P-n - i J i 2 ^ 2 1 P-22 P-21 " 2 k 3 ^ 3 1 P-32 ?-}} ^ 3 k i i " 1 -(k - l ) 2 -(k - 1 )3 ' .... -(k-Dk F i g u r e 2 . Raw s c o r e by item m a t r i x i l l u s t r a t i n g u n i d i m e n s i o n -a l i t y . F a c t o r a n a l y s i s i s the method commonly s u g g e s t e d f o r c o n s t r u c t i n g u n i -d i m e n s i o n a l t e s t s (Lumsden, 1 9 6 l ) o r t o d e t e r m i n e whether e x i s t i n g t e s t s a r e u n i d i m e n s i o n a 1 . Hambleton et a l , ( 1 9 7 8 ) s u g g e s t t h a t t h e f a c t o r a n a l y t i c - 3 9 -approach t o t e s t c o n s t r u c t i o n o u t l i n e d by Lumsden be a p p l i e d i n c o n s t r u c t i n g t e s t s w hich a r e t o be a n a l y z e d by l a t e n t t r a i t methods. Hambleton and Traub ( 1 9 7 3 ) performed a p r i n c i p a l components a n a l y s i s on the m a t r i c e s o f t e t r a c h o r i c item i n t e r c o r r e l a t ions t o d e t e r m i n e whether t h r e e t e s t s were u n i d i m e n s i o n a 1 b e f o r e a p p l y i n g the Rasch model i n t e s t a n a l y s i s . In a l l t h r e e c a s e s , the f i r s t f a c t o r a c c o u n t e d f o r 2 0 . 5 % t o 3 1 - 7 % o f t h e t o t a l t e s t v a r i a n c e . The r e s e a r c h e r s proceeded w i t h the a n a l y s e s as i f the t e s t s were u n i f a c t o r i a l p o i n t i n g o u t , however, t h a t from two t o t h r e e f a c t o r s would p r o b a b l y be r e q u i r e d t o a r r i v e a t an a c c e p t a b l e s o l u t i o n f o r t h e t e s t s . The b e s t g o o d n e s s - o f - f i t o f items r e s u l t s o b t a i n e d , a p p l y i n g the Rasch model and Birnbaum's two-parameter model, were f o r the most homogeneous t e s t . In a l a t e r p a p e r , Hambleton and Cook ( 1 9 7 7 ) s t a t e d t h a t the u n i d i m e n s i o n a 1 i t y a ssumption does not h o l d f o r most t e s t s , but t h a t a f a c t o r a n a l y s i s c o u l d be used t o c l u s t e r items i n a heterogeneous t e s t . L a t e n t t r a i t model i t e m a n a l -y s i s c o u l d then be a p p l i e d t o each homogeneous c l u s t e r o f items. The f a c t o r a n a l y t i c approach t o d e t e r m i n i n g the u n i d i m e n s i o n a 1 i t y o f a t e s t has been c r i t i c i z e d as u n n e c e s s a r y o r m i s l e a d i n g by o t h e r r e s e a r c h e r s . F a c t o r a n a l y s i s i s i t s e l f a s e r i e s o f t h e o r e t i c a l models w i t h d i f f e r i n g and sometimes c o n f l i c t i n g c o n c e p t u a l i z a t i o n s o f d i m e n s i o n a l i t y (Rentz & R e n t z , 1 9 7 8 ) . The d i m e n s i o n a l i t y o f a s e t o f t e s t items i s t o o dependent upon t h e types o f i n t e r - i t e m c o r r e l a t i o n c o e f f i c i e n t s used i n t h e f a c t o r a n a l y s i s . For example, a n a l y s i s o f phi c o e f f i c i e n t s g e n e r a l l y y i e l d s more f a c t o r s than a n a l y s i s o f t e t r a c h o r i c c o r r e l a t i o n c o e f f i c i e n t s (Hambleton £ T.rraub, 1 9 7 3 ) -W r i g h t ( 1 9 7 7 a ) c a u t i o n s a g a i n s t the use o f the f a c t o r a n a l y t i c a p p r o a c h , c i t i n g t h e extreme i n s t a b i l i t y o f f a c t o r s o v e r d i f f e r e n t samples o f examinees as a n e g a t i v e f e a t u r e o f the method. I t has been p o i n t e d out t h a t the best t e s t o f u n i d i m e n s i o n a 1 i t y may be the d i r e c t a p p l i c a t i o n of t h e Rasch model t o a group o f items (Rentz £ Re n t z , 1 9 7 8 ) . In many p r a c t i c a l a p p l i c a t i o n s o f - ko -the Rasch model, items have been s u c c e s s f u l l y a n a l y z e d d e s p i t e t h e f a c t t h a t t e s t s were a se e m i n g l y h e t e r o g e n e o u s , m u l t i d i m e n s i o n a l s e t o f i t e m s . Thus, the Rasch model appears to d i s p l a y r o b u s t n e s s w i t h r e s p e c t t o the v i o l a t i o n o f t h e u n i d i m e n s i o n a 1 i t y a s s u m p t i o n i n h e r e n t i n l a t e n t t r a i t models. The assumptions o f equal item d i s c r i m i n a t i o n and l a c k o f g u e s s i n g . I t has been s t a t e d t h a t t h e Rasch model may not be r o b u s t f o r some d a t a i n t h a t the o n l y item c h a r a c t e r i s t i c i t t a k e s i n t o a ccount i s item d i f f i c u l t y , w h i l e n e g l e c t i n g t o i n c o r p o r a t e o t h e r p o s s i b l e i t e m c h a r a c t e r i s t i c s such as item d i s c r i m i n a t i o n and g u e s s i n g (Wright S Panchapakesan, 1969)- Thus, an ite m may a c t u a l l y measure the same t r a i t as the o t h e r items w i t h which i t i s group-ed, but i t may be r e j e c t e d i n Rasch a n a l y s i s o f the t e s t d a t a because o f t h e model's i m p l i e d r e s t r i c t i v e a s s u m p t i o n s t h a t a l l item d i s c r i m i n a t i o n c h a r a c -t e r i s t i c s a r e equal and t h a t t h e r e i s minimal g u e s s i n g i n v o l v e d i n a n s w e r i n g m u l t i p l e - c h o i c e i t e m s . Two-parameter and t h r e e - p a r a m e t e r models i n c o r p o r a t i n g item c h a r a c t e r i s -t i c s f o r d i s c r i m i n a t i o n and g u e s s i n g e x i s t i n the f i e l d o f l a t e n t t r a i t t h e o r y . One l i n e o f r e s e a r c h has been t o compare t h e Rasch l o g i s t i c model t o t h e s e o t h e r models i n s i t u a t i o n s where the a s s u m p t i o n s o f equal item d i s c r i m i n a t i o n and no g u e s s i n g have been known t o be v i o l a t e d . The forms o f item c h a r a c t e r i s t i c c u r v e s v a r y depending upon the number o f parameters used i n d e f i n i n g them (Hambleton S Cook, 1977)- For t h e Rasch l o g i s t i c model, the ICC's a r e n o n - i n t e r s e c t i n g , d i f f e r i n g o n l y by a t r a n s l a -t i o n a l o n g the a b i l i t y s c a l e . These c u r v e s r e f l e c t the n o t i o n t h a t items a r e h e l d t o v a r y o n l y w i t h r e s p e c t t o t h e i r d i f f i c u l t y . I tern c h a r a c t e r i s t i c c u r v e s f o r the two-parameter model v a r y i n both s l o p e and t r a n s l a t i o n a l o n g the a b i l i t y s c a l e . The v a r i a t i o n i n s l o p e i s a r e f l e c t i o n o f t h e f a c t t h a t some i t e m s , w i t h a s l o p e g r e a t e r than u n i t y , a r e more d i s c r i m i n a t i n g than - 41 -o t h e r items w h i c h have a s l o p e l e s s than one. I tern c h a r a c t e r i s t i c c u r v e s f o r t h e t h r e e - p a r a m e t e r model d i f f e r i n s l o p e , t r a n s l a t i o n a l o n g t h e a b i l i t y s c a l e , and i n lower asymptote. The lower asymptote i s g r e a t e r than z e r o and approaches a h y p o t h e t i c a l lower l i m i t w h i c h i s a f a c t o r o f the number of a l t e r n a t i v e s i n a m u l t i p l e - c h o i c e q u e s t i o n . In a computer s i m u l a t i o n s t u d y (Hambleton S Traub, 1971), the one-para-meter Rasch model was compared w i t h r e s p e c t t o i t s e f f e c t i v e n e s s i n d e t e r m i n -ing p e r s o n a b i l i t y e s t i m a t e s t o Birnbaum's two-parameter and t h r e e - p a r a m e t e r l a t e n t t r a i t models. Four ranges o f item d i s c r i m i n a t i n g l e v e l (0.0, .2, .4, .8) and t h r e e mean l e v e l s o f g u e s s i n g (0.0, .1 , .2) were g e n e r a t e d by means o f a computer program. The i n t e r a c t i o n o f v a r y i n g item d i s c r i m i n a t i o n and g u e s s i n g l e v e l s was compared f o r each model as regard e d the model's e f f i -c i e n c y i n d e t e r m i n i n g person a b i l i t y l e v e l s r a n g i n g from -3 t o 3. I t was c o n c l u d e d t h a t t h e Rasch model y i e l d s comparably e f f i c i e n t e s t i m a t e s o f a b i l -i t y when t h e r e was no g u e s s i n g and the range of item d i s c r i m i n a t i o n s was s m a l l . It was found t h a t t h e Rasch model y i e l d e d t h e l e a s t e f f i c i e n t e s t i -mates o f p e r s o n a b i l i t y when low a b i l i t y 'examinees' responded t o a t e s t i n whi c h g u e s s i n g was a f a c t o r . In a subsequent s t u d y , Hambleton and Traub (1973) compared the Rasch model and the two-parameter model i n an e m p i r i c a l a n a l y s i s o f t h r e e d i f f e r e n t t e s t s o f a b i l i t y . They c o n c l u d e d t h a t both models e s t i m a t e d examinee a b i l i t y i n an e q u a l l y e f f i c i e n t f a s h i o n p r o v i d e d t h a t the number o f items i n t h e t e s t was l a r g e . The Rasch model was l e a s t e f f i c i e n t i n e s t i m a t i n g person a b i l i t y l e v e l s when t h e t e s t was s h o r t and t h e r e was s u b s t a n t i a l v a r i a n c e i n the item d i s c r i m i n a t i o n p a r a m e t e r s . The numbers o f items i n the t e s t s a n a l y z e d were 20, 45 and 80. In a n o t h e r s i m u l a t e d s t u d y ( D i n e r o & H a e r t e l , 1977), d a t a from t h e r e -sponses o f 75 ' s u b j e c t s ' i n t e s t s o f 3 0 a r t i f i c i a l l y g e n e r a t e d ' i t e m s ' were - kl -f i t t e d t o the Rasch model. F i v e d i f f e r e n t v a r i a n c e s f o r item d i s c r i m i n a t i o n v a l u e s were g e n e r a t e d (.05, .10, -15, .20, .25). For each v a r i a n c e l e v e l , t h r e e d i f f e r e n t t y p e s o f item d i s t r i b u t i o n ( n o r m a l , u n i f o r m , p o s i t i v e l y skew-ed) were a l s o s i m u l a t e d . I t was c o n c l u d e d t h a t the u n i f o r m d i s t r i b u t i o n y i e l d e d the p o o r e s t f i t o f items t o t h e model o v e r a l l l e v e l s o f item d i s c r i m -i n a t i o n v a r i a n c e and t h a t t h e r e g e n e r a l l y was a s l i g h t i n c r e a s e i n l a c k o f f i t o f items as the d i s c r i m i n a t i o n l e v e l v a r i a n c e s i n c r e a s e d . The r e s e a r c h -e r s c o n c l u d e d , however, t h a t the l a c k o f an item d i s c r i m i n a t i o n c h a r a c t e r i s -t i c i n the Rasch model does not g e n e r a l l y r e s u l t i n poor f i t o f items w i t h v a r y i n g d i s c r i m i n a t i o n s . They p o i n t e d out t h a t the f i t o f an i t e m t o t h e model p r o b a b l y a f f e c t s e s t i m a t e s o f d i s c r i m i n a t i o n , w i t h poor f i t t i n g items h a v i n g l a r g e r e r r o r s o f e s t i m a t e s o f t h e i r d i s c r i m i n a t i o n s . O'Shea (1979) c i t e d a s i m u l a t e d s t u d y by C a r t l e d g e (1976) i n w h i c h the r e s e a r c h e r found t h a t items w i t h d i s c r i m i n a t i o n v a l u e s r a n g i n g from .9 t o 1.1 were found t o f i t the Rasch model, and t h a t the model c o n t i n u e d t o d i s p l a y r o b u s t n e s s when the item d i s c r i m i n a t i o n s v a r i e d from .8 t o 1.2. In a s i m u l a t e d s t u d y , Panchapakesan (1969) found t h a t the items w i t h e x t r e m e l y d e v i a n t d i s c r i m i n a t i o n s d i d not f i t the model, but t h a t items w i t h d i s c r i m i n a t i o n v a l u e s r a n g i n g from .8 t o 1.25 d i s p l a y e d f i t t o the model. Using a sample s i z e o f 500 f o r a 2 0 -itern s i m u l a t e d t e s t c o n t a i n i n g items w i t h d i s c r i m i n a t i o n v a l u e s r a n g i n g from .8 t o 1.2, Panchapakesan found t h a t the mean square f o r o v e r a l l f i t o f items was not l a r g e enough t o be s i g n i f i c a n t . She c o n c l u d e d t h a t t h e Rasch model d i s p l a y e d r o b u s t n e s s when t h e a s s u m p t i o n o f items w i t h equal d i s c r i m i n a t i o n s was not met. There i s e v i d e n c e from s i m u l a t e d s t u d i e s e m p l o y i n g e x a c t l y equal i t e m d i s c r i m i n a t i o n v a l u e s , t h a t the s t a n d a r d d e v i a t i o n f o r e s t i m a t e s o f d i s c r i m -i n a t i o n v a l u e s a r e as l a r g e as .2 (Wright & Mead, 1978). The i n f e r e n c e w h i c h might p o s s i b l y be drawn from t h i s and t h e p r e v i o u s s t u d i e s c i t e d , i s t h a t a - 4 3 -range i n d i s c r i m i n a t i o n v a l u e s f o r items from . 6 t o 1 . 4 might be a c c e p t a b l e (O'Shea, 1 9 7 9 ) . S i n c e g u e s s i n g i s a pro b l e m i n h e r e n t i n t h e use o f m u l t i p l e - c h o i c e t e s t i t e m s , i t might be w e l l t o s e t some lower l i m i t t o s c o r e s w h i c h a r e a l l o w e d t o e n t e r i n t o d a t a a n a l y s i s u s i n g t h e Rasch model. T h i s w o u l d compensate f o r the absence o f a g u e s s i n g c h a r a c t e r i s t i c i n the model. One a p p r o a c h s u g g e s t -ed by W r i g h t and Mead ( 1 9 7 8 ) i s t o s e t the lower l i m i t somewhat above t h e random g u e s s i n g l e v e l . T h i s approach i s o f t e n the one f o l l o w e d t o c o n t r o l f o r g u e s s i n g i n c o n v e n t i o n a l item a n a l y s i s . Panchapakesan ( 1 9 & 9 ) ' n s i m u l a t -ed s t u d i e s i n v o l v i n g a 2 0 - i t e r n t e s t composed o f f i v e - a 1 t e r n a t i v e m u l t i p l e -c h o i c e i t e m s , c o n c l u d e d t h a t the e f f e c t o f g u e s s i n g was n e g l i g i b l e p r o v i d e d t h a t the "examinees' were o f h i g h a b i l i t y . For s i t u a t i o n s i n w h i c h t h i s i s not the c a s e , she p r o p o s e d the f o l l o w i n g f o r m u l a f o r e l i m i n a t i n g examinees from the c a l i b r a t i o n sample: where J< = t h e number o f items i n t h e t e s t , m^  = t h e number o f a l t e r n a t i v e s per i t e m , and, _r = t h e s c o r e below w h i c h examinees a r e e l i m i n a t e d from the c a l i b r a t i o n sample. In t h e f o r m u l a above, k/m i s t h e e x p e c t e d s c o r e based on random g u e s s i n g 2 0.5 and (k_(m_ - l ) / m ) " i s t h e s t a n d a r d d e v i a t i o n o f t h a t s c o r e . E s t i m a t i o n o f I tern P a r a m e t e r s f o r t h e Rasch L o g i s t i c Model Once t h e fo r m o f t h e i t e m c h a r a c t e r i s t i c c u r v e f o r t h e Rasch model has been s p e c i f i e d and the n e c e s s a r y a s s u m p t i o n s o f t h e model have been met, i t i s p o s s i b l e t o a r r i v e a t e s t i m a t e s o f the p a r a m e t e r s a_^  and _d . E s t i m a t o r s f o r t h e Rasch model have been d e v e l o p e d w h i c h have been shown t o be s u f f i -c i e n t , c o n s i s t e n t , e f f i c i e n t , and u n b i a s e d ( W r i g h t , 1 9 7 7 a ) . S e v e r a l computer - hk -programs e x i s t f o r the c o m p u t a t i o n o f the p a r a m e t e r s , as w e l l as a method by Cohen ( 1 9 7 6 ) w h i c h can be used t o a r r i v e a t a p p r o x i m a t e e s t i m a t e s by hand. The d a t a n e c e s s a r y f o r the e s t i m a t i o n o f t h e J< item p a r a m e t e r s and the N_ p e r s o n parameters f o r a t e s t c o n s i s t s o f the m a t r i x o f _K items on t h e t e s t (1 , 2 , . . . , K_) a r r a n g e d i n t h e columns and a l l p o s s i b l e s c o r e s o t h e r than z e r o and K ( 1 , 2 , . . . , K - 1 ) a r r a n g e d i n t h e rows. P e r f e c t and z e r o s c o r e s ar e removed from the d a t a base because such s c o r e s do not y i e l d f i n i t e e s t i -mates o f a b i l i t y f o r persons o b t a i n i n g them (Wright & D o u g l a s , 1 9 7 7 a ) . S i m i -l a r l y , items w h i c h a l l examinees g e t c o r r e c t p r o v i d e no d i f f e r e n t i a l i n f o r m -a t i o n about a c t u a l d i f f e r e n c e s i n d i f f i c u l t y o f these items. Rasch ( 1 9 6 6 ) showed t h a t the column and row t o t a l s f o r items and persons in a t e s t s c o r e m a t r i x were j o i n t l y s u f f i c i e n t s t a t i s t i c s f o r the s i m u l t a n e -ous e s t i m a t i o n o f the person a b i l i t y p a r ameter, a^, and t h e item d i f f i c u l t y p a r a m e t e r , d_. . He a l s o deduced from the model t h a t the column and row t o t a l s o f t he m a t r i x c o u l d be used as s e p a r a t e and independent e s t i m a t e s o f the parameters d_. and a_^  r e s p e c t i v e l y . Two computer programs f o r o b t a i n i n g parameter e s t i m a t e s , LOG, which uses an unweighted l e a s t s q u a r e s a p p r o a c h , and MAX, whi c h uses a maximum l i k e l i h o o d p r o c e d u r e , were o u t l i n e d by Wri g h t and Panchapakesan ( 1 9 6 9 ) - They f a v o u r e d the maximum l i k e l i h o o d p r o c e d u r e as i t y i e l d e d b e t t e r e s t i m a t e s o f a and d. —s_ — _ i _ as w e l l as b e t t e r a p p r o x i m a t i o n s o f the s t a n d a r d e r r o r s o f the e s t i m a t e s . They p o i n t e d out t h a t when N_ was l a r g e , and the a b i l i t y range o f the exami-nees was w i d e r than the d i f f i c u l t y range o f the item p a r a m e t e r s , the item e s t i m a t e s o b t a i n e d by LOG and MAX were e q u i v a l e n t . Andersen ( 1 9 7 3 ) c r i t i c i z e d the W r i g h t and Panchapakesan approach be-cause i t y i e l d e d u n c o n d i t i o n a l e s t i m a t e s o f a and d.. He showed t h a t , as N —s —1_ — i n c r e a s e s , the number o f a b i l i t y parameters i n c r e a s e s w i t h o u t bound l e a d i n g t o b i a s e d e s t i m a t e s o f the item p a r a m e t e r s . - 4 5 -W r i g h t and Douglas ( 1 9 7 7 b ) o u t l i n e d two computer prognams, FCON, the f u l l c o n d i t i o n a l p r o c e d u r e based on Andersen's ( 1 9 7 3 . ) s o l u t i o n o f the c o n d i -t i o n a l l i k e l i h o o d e q u a t i o n s , and ICON, an i n c o m p l e t e c o n d i t i o n a l p r o c e d u r e f o r e s t i m a t i n g the a_s and d_. p a r a m e t e r s . They found the former p r o c e d u r e i m p r a c t i c a l f o r use w i t h more than 1 0 o r 1 5 items because i t d i d not converge r a p i d l y w i t h numbers o f items g r e a t e r than t h i s . They found t h a t the ICON p r o c e d u r e c o n v e r g e d more r a p i d l y , but t h a t b i a s e d e s t i m a t e s o f t h e i t e m p a r a -meters r e s u l t e d when t h e r e were more than 2 0 o r 3 0 items i n a t e s t . S i n c e ICON r e q u i r e d a c o r r e c t i o n f a c t o r f o r b i a s , the r e s e a r c h e r s c o n c l u d e d t h a t the most f r u i t f u l approach would be t o u t i l i z e the o r i g i n a l u n c o n d i t i o n a l p r o c e d u r e , UCON, (Wright S Panchapakesan, 1 9 6 9 ) w i t h a c o r r e c t i o n f o r b i a s s i n c e t h i s was the most p r a c t i c a l program f o r i m p l e m e n t a t i o n . They d e t e r m i n -ed t h a t a c o r r e c t i o n f a c t o r o f (_K - 1)/K, when a p p l i e d t o the b i a s e d e s t i -mates o f the item p a r a m e t e r s , y i e l d e d v a l u e s v e r y c l o s e t o the c o r r e c t ones (Wright S D o u g l a s , 1 9 7 7 a ) . T h i s c o r r e c t i o n f a c t o r f o r item parameter b i a s i s i n c o r p o r a t e d i n t o t h e BICAL computer package p r e p a r e d by W r i g h t and Mead ( 1 9 7 8 ) . W r i g h t and Douglas ( 1 9 7 7 a ) a l s o p r e s e n t e d an a p p r o x i m a t e p r o c e d u r e , PROX, which r e q u i r e s the a s s u m p t i o n s t h a t p e r s on a b i l i t i e s and item d i f f i c u l -t i e s a re n o r m a l l y d i s t r i b u t e d . They found t h a t the c a l c u l a t i o n time u s i n g PROX f o r 1arge-numbers o f items was a p p r o x i m a t e l y hO t i m e s f a s t e r than the time i t took f o r UCON t o converge. Hambleton e t a l , ( 1 9 7 8 ) c a u t i o n e d t h a t PROX r e q u i r e s f u r t h e r i n v e s t i g a t i o n as i t i s an a p p r o x i m a t e p r o c e d u r e and the e x t e n t t o w h i c h d e p a r t u r e s from the a s s u m p t i o n t h a t i t em d i f f i c u l t i e s and p e r s o n a b i l i t i e s a r e n o r m a l l y d i s t r i b u t e d a f f e c t t h e e s t i m a t e s i s not w e l l u n d e r s t o o d . PROX i s an o p t i o n a v a i l a b l e i n the BICAL computer package. G u s t a f s s o n ( 1 9 8 0 ) d e s c r i b e d a p r o c e d u r e f o r c o n d i t i o n a l e s t i m a t i o n o f item parameters f o r the Rasch model which p r o v i d e s r a p i d c o m p u t a t i o n o f nu-- 46 -m e r i c a l l y s t a b l e e s t i m a t e s even f o r long t e s t s c o n s i s t i n g o f 80 t o 100 items. He c o n c l u d e d t h a t f o r t e s t s o f t h i s l e n g t h a c o n d i t i o n a l approach t o item parameter e s t i m a t i o n i s s u p e r i o r t o an u n c o n d i t i o n a l approach i n t h a t power-f u l t e s t s o f g o o d n e s s - o f - f i t o f d a t a t o the Rasch model, w i t h known s t a t i s -t i c a l p r o p e r t i e s , can be d e v i s e d . S t a n d a r d e r r o r of e s t i m a t e f o r item d i f f i c u l t i e s . The Rasch method o f t e s t a n a l y s i s p r o v i d e s a measure of the s t a n d a r d e r r o r o f e s t i m a t e a s s o c i a t e d w i t h the i t e m d i f f i c u l t y p a r a m e t e r s . The s t a n d a r d e r r o r o f e s t i m a t e f o r item d i f f i c u l t y i n the Rasch model i s m i n i m i z e d when t h e r e i s a h i g h c o r r e s p o n d e n c e between item d i f f i c u l t i e s and the mean a b i l i t y o f p e r s o n s in;. the•Sample. The b e s t e s t i m a t e s o f item d i f f i c u l t y a re o b t a i n e d f o r items on w h i c h the proba-b i l i t y o f p a s s i n g the item o r f a i l i n g the i t e m i s e q u a l o r n e a r l y equal f o r the maximum number o f persons i n the sample ( W h i t e l y & Dawis, 1974) . The c o n v e r s e i s t r u e i n the u s u a l c o n c e p t u a l i z a t i o n o f item v a r i a n c e 2 employed i n c l a s s i c a l t e s t t h e o r y . I tern v a r i a n c e , s_ = P_jS_j > i s the h i g h e s t when t h e r e i s a 50% chance o f p a s s i n g o r f a i l i n g an i t e m , i . e . , when _p. = q_. = . 5 , and i s m i n i m i z e d as items become e a s i e r o r more d i f f i c u l t . T h i s s i t u -a t i o n seems p a r a d o x i c a l , e s p e c i a l l y i n the case o f v e r y d i f f i c u l t m u l t i p l e -c h o i c e items where g u e s s i n g might be most p r e v a l e n t . W h i l e some i n v e s t i g a t o r s m a i n t a i n t h a t the Rasch model y i e l d s i t e m - d i f -f i c u l t y e s t i m a t e s w h i c h a r e i n v a r i a n t o v e r samples d i f f e r i n g i n a b i l i t y , t h i s i s not the case f o r s t a n d a r d e r r o r s o f e s t i m a t e f o r item d i f f i c u l t i e s (White-l y £ Dawis, 197*0- The s t a n d a r d e r r o r o f item d i f f i c u l t y i s s a m p l e - s p e c i f i c , b e i n g s t r o n g l y i n f l u e n c e d by the r e l a t i o n s h i p between item d i f f i c u l t y and the d i s t r i b u t i o n o f a b i l i t y i n the sample ( W r i g h t , 1977a). ' S t a n d a r d e r r o r o f e s t i m a t e f o r person a b i l i t i e s ; Use of the Rasch model a l s o p r o v i d e s s e p a r a t e e r r o r s o f measurement f o r each of the p e r s o n a b i l i t y p a r a m e t e r s . The most p r e c i s e measurements o f p e r s o n a b i l i t y a r e o b t a i n a b l e - hi -from c e n t r a l s c o r e s on a t e s t and e r r o r i n e s t i m a t i n g person a b i l i t y i s max-i m i z e d as s c o r e s become more extreme i n e i t h e r d i r e c t i o n . Thus the most p r e -c i s e e s t i m a t e s o f person a b i l i t y w i l l be o b t a i n e d i f t e s t s a r e composed o f items c l o s e l y complementing the a b i l i t i e s o f t h e examinees ( W h i t e l y & Dawis, 1974). The s t a n d a r d e r r o r s o f e s t i m a t e s f o r person a b i l i t i e s o b t a i n e d u s i n g the Rasch model behave i n the o p p o s i t e manner from the c o n v e n t i o n a l s t a n d a r d e r -r o r s o f measurement f o r randomly p a r a l l e l t e s t s c o n s i s t i n g o f items randomly sampled from a g i v e n domain. In the c o n v e n t i o n a l c a s e , the s t a n d a r d e r r o r s o f measurement f o r s c o r e s become minimal as the s c o r e s d e p a r t i n v a l u e from the c e n t r a l c a s e s . They i l l o g i c a l l y become z e r o a t s c o r e s o f z e r o o r 1 0 0 % , where the e s t i m a t i o n o f person a b i l i t y i s a c t u a l l y e x t r e m e l y i m p r e c i s e ( W r i g h t , 1 9 7 7 a ) . A g a i n , c l a i m s have been made t h a t the Rasch model p r o v i d e s i n v a r i a n t p e r s o n a b i l i t y e s t i m a t e s which do not depend upon the p a r t i c u l a r s e t o f items chosen t o measure a b i l i t y ( W r i g h t , 1 9 7 7 a ) . However, the p r e c i s i o n o f the pe r -son a b i l i t y e s t i m a t e s w i l l v a r y d i r e c t l y w i t h t h e e x t e n t t o w h i c h t h e d i f f i -c u l t y o f the items chosen t o measure t h a t a b i l i t y complements the a c t u a l a b i l -i t y o f the per s o n s b e i n g measured. T e s t i n g item F i t t o the Rasch Model S e v e r a l approaches have been used i n e v a l u a t i n g the f i t o f t e s t items t o the Rasch model. Rasch ( i 9 6 0 ) o r i g i n a l l y employed a s i m p l e g r a p h i c a l approach t o a p p r o x i m a t e l y d e t e r m i n e how w e l l h i s model a p p l i e d t o t e s t d a t a (O'Shea, 1 9 7 9 ) . Rasch grouped examinees i n t o f i v e a b i l i t y groups r a n g i n g from h i g h t o low on the b a s i s o f raw s c o r e s o b t a i n e d i n a t e s t . H i s approach t o t e s t i n g the f i t o f the d a t a t o h i s model was t o p l o t two-way graphs o f item d i f f i c u l t y f o r a l l p a i r s o f a d j a c e n t s c o r e groups. Rasch then made a v i s u a l c o n f i r m a -- 48 -t i o n o f the f i t o f h i s d a t a by n o t i n g whether the f o u r l i n e s o f b e s t f i t were a p p r o x i m a t e l y p a r a l l e l and had a s l o p e a p p r o a c h i n g u n i t y . I f s c o r e groups a r e o f adequate s i z e , s l o p e s o f items can be a d e q u a t e l y e s t i m a t e d w i t h u n i t s l o p e b e i n g the c r i t e r i o n f o r f i t . P r o c e d u r e s u s i n g a l l s c o r e groups prob-a b l y do not y i e l d good e s t i m a t e s f o r extreme s c o r e groups. The BICAL p r o -cedure c o l l a p s e s s c o r e groups i n t o l a r g e r e n t i t i e s . C o n s e q u e n t l y persons hav-ing extreme s c o r e s a r e a s s i m i l a t e d i n t o l a r g e r groups which p r o b a b l y p r o v i d e more s t a b l e e s t i m a t e s (Rentz & R e n t z , 1978) . A more p r e c i s e p r o c e d u r e was t o use a s t a t i s t i c a l approach t o t e s t the s i g n i f i c a n c e o f the d e p a r t u r e o f r e g r e s s i o n s from a u n i t s l o p e (Anderson, Kearney & E v e r e t t , 1968). A group o f pe r s o n s t a k i n g a t e s t was d i v i d e d i n t o s i x a b i l i t y groups. For each i t e m , a d i f f i c u l t y e s t i m a t e s p e c i f i c t o each s c o r e group was computed. A mean v a l u e f o r a l l item d i f f i c u l t i e s was o b t a i n -ed f o r each a b i l i t y group as w e l l . T h e . s i x d i f f i c u l t y e s t i m a t e s f o r each item were then r e g r e s s e d on the s i x means o f a l l d i f f i c u l t y e s t i m a t e s f o r each s c o r e group. A t - v a l u e was o b t a i n e d f o r each item t o d e t e r m i n e whether the r e g r e s s i o n o f the s i x item d i f f i c u l t y e s t i m a t e s on the s i x o v e r a l l d i f f i -c u l t y means d i f f e r e d s t a t i s t i c a l l y from a u n i t s l o p e . Two p r o b a b i l i t y l e v e l s were used f o r the t - t e s t s . The t e s t u s i n g a p r o b a b i l i t y o f .05 was the more s t r i n g e n t t e s t o f f i t t o the model as more items were r e j e c t e d u s i n g t h i s c r i t e r i o n than were r e j e c t e d when a p r o b a b i l i t y v a l u e o f .01 was used. T e s t o f f i t o f items t o t h e Rasch model by d e t e r m i n i n g a c h i - s q u a r e s t a t i s t i c , o r some v a r i a t i o n t h e r e o f , has become the common method (Rentz & Re n t z , 1978) . The g e n e r a l p r o c e d u r e was o u t l i n e d by W r i g h t and Panchapakesan (1969) and i n v o l v e s the comparison of d a t a o b t a i n e d w i t h what i s e x p e c t e d g i v e n the model. The s t a r t i n g p o i n t i s a raw s c o r e m a t r i x f o r a t e s t w i t h k_ items and m s c o r e groups w i t h no empty c e l l s , o r a m a t r i x . I n w h i c h s c o r e groups, have been c o l l a p s e d i n t o a s m a l l e r number o f g r o u p s , j _ . For each c e l l , - k3 -the s t a n d a r d d e v i a t e i s the d i f f e r e n c e between the o b s e r v e d v a l u e and the e s t i m a t e d e x p e c t e d v a l u e o f the p r o b a b i l i t y o f s u c c e s s on the item. Given t h e s e v a l u e s , a s u i t a b l e s t a t i s t i c f o r t e s t i n g t h e o v e r a l l f i t o f the e n t i r e group o f items t o the model i s the c h i - s q u a r e s t a t i s t i c - - t h e sum o f a l 1 squared s t a n d a r d d e v i a t e s w i t h degrees o f freedom (k. - 1) (rn - 1 ) , o r i n the case o f grouped s c o r e s (k_ - 1 ) (j_ - 1 ) . The f i t o f each i n d i v i d u a l item t o t h e model can a l s o be e v a l u a t e d by summing t h e squ a r e d s t a n d a r d d e v i a t e s o v e r t h e s c o r e groups w i t h (rn - 1 ) o r (j_ - l ) degrees o f freedom. T h i s c h i - s q u a r e s t a t i s t i c was used by Fryman ( 1 9 7 6 ) who e l i m i n a t e d items w h i c h had a p r o b a b i -l i t y o f f i t t o the model by chance of l e s s than . 0 5 . The q u o t i e n t o f the c h i - s q u a r e d i v i d e d by the a p p r o p r i a t e degrees o f freedom y i e l d s the mean square f i t s t a t i s t i c . Care must be ta k e n i n the i n t e r p r e t a t i o n o f the MS f i t s t a t i s t i c s i n c e , as w i t h a l l s t a t i s t i c a l t e s t s o f s i g n i f i c a n c e , s u f f i c i e n t power i s i n v a r i a b l y g a i n e d t o r e j e c t the hypo-t h e s i s — i n t h i s case a q u e s t i o n o f whether an item f i t s a p a r t i c u l a r model--i f a l a r g e enough sample i s used (Rentz & Re n t z , 1 9 7 8 ) . The MS f i t s t a t i s t i c w i l l l e a d t o the e l i m i n a t i o n o f items i n a l l i n s t a n c e s i f v e r y l a r g e samples a r e used. C o n s e q u e n t l y , MS f i t s t a t i s t i c s s h o u l d be c o n s i d e r e d as a group i n r e l a t i o n t o each o t h e r . O u t l i e r s w h i c h do not f i t the model s h o u l d be s p o t t e d i n s t e a d o f s e t t i n g a b s o l u t e v a l u e s o f t h e MS f i t s t a t i s t i c f o r s e l e c -t i o n / r e j e c t i o n o f it e m s . If t h e sample i s a v e r y l a r g e one, items w i t h MS f i t v a l u e s as h i g h as 1 0 may s t i l l f i t the model i f t h i s v a l u e does not d i f -f e r g r e a t l y from the MS f i t v a l u e s shown by the items i n the whole group (Rentz £ R e n t z , 1 9 7 8 ) . Hashway ( 1 9 7 7 ) used the MS f i t s t a t i s t i c t o g e t h e r w i t h the item d i s c r i m -i n a t i o n i n d i c e s p r o v i d e d by t h e Rasch model t o r e j e c t items w h i c h had a MS f i t g r e a t e r than 2 . 2 and a Rasch d i s c r i m i n a t i o n index o u t s i d e the range . 8 - 50 -t o 1.2. O'Shea (1979) e s t a b l i s h e d s i m i l a r c r i t e r i a f o r r e j e c t i n g n o n f i t t i n g i t e m s , e l i m i n a t i n g items f o r which the MS f i t exceeded u n i t y by f o u r o r more s t a n d a r d e r r o r s o f the MS f i t , and f o r wh i c h the Rasch d i s c r i m i n a t i o n index was l e s s than .7- The Rasch d i s c r i m i n a t i o n index p r o v i d e d by computer p r o -grams such as BICAL i s r e l a t e d t o the c o n v e n t i o n a l p o i n t - b i s e r i a l c o e f f i c i e n t (O'Shea, 1979). Indeed, h y b r i d approaches t o item a n a l y s i s , u s i n g the Rasch MS f i t s t a t i s t i c t o g e t h e r w i t h the c o n v e n t i o n a l p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t have been employed. Forbes (1976) e l i m i n a t e d items h a v i n g a p o i n t - b i s e r i a l i t e m - t o t a l s c o r e c o r r e l a t i o n l e s s than .25 and a mean sq u a r e f i t e x c e e d i n g 2.5- These l a t t e r s t u d i e s i n d i c a t e t h a t r e s e a r c h e r s have used f i t c r i t e r i a o t h e r than c h i - s q u a r e r e l a t e d s t a t i s t i c s i n d e t e r m i n i n g the f i t o f t e s t items t o the Rasch model. The q u e s t i o n o f sample s i z e i n a p p l i c a t i o n o f the Rasch model. There i s some q u e s t i o n as r e g a r d s t h e s i z e o f t h e sampde r e q u i r e d t o t e s t the f i t o f items t o the Rasch model. In an e a r l y s t u d y u s i n g the Rasch p r o c e d u r e , sam-p l e s o f 4 4 6 , 129, and 279 s u b j e c t s were c o n s i d e r e d too s m a l l f o r a p p l i c a t i o n , o f t h e t e c h n i q u e (Anderson e t a l , 1968). In a n o t h e r s t u d y , the i n v e s t i -g a t o r s p o i n t e d out t h a t c o r r e l a t i o n s between item d i f f i c u l t y e s t i m a t e s ob-t a i n e d from 89- s u b j e c t samples may have been low because o f the s m a l l sample s i z e ( T i n s l e y 6 Dawis, 1975). They p o i n t e d o u t , however, t h a t e s t i m a t e s from a l a r g e r sample o f 269 s u b j e c t s a l s o d i d not c o r r e l a t e h i g h l y . W h i t e l y (1977) has ta k e n up th e i s s u e o f sample s i z e s used i n t e s t i n g f i t o f items t o the Rasch model. She p o i n t s out t h a t the t e s t o f f i t i s a s t a t i s t i c a l t e s t and l i k e a l l s t a t i s t i c a l p r o c e d u r e s r e q u i r e s a sample o f a re a s o n a b l e s i z e t o d i s c r i m i n a t e between the s m a l l d i f f e r e n c e s t h a t may e x i s t between good and bad items. W h i t e l y s t a t e d t h a t an N_ as l a r g e as 800 may f a i l t o d e t e c t s i z a b l e d i f f e r e n c e s between the q u a l i t y o f f i t o f t e s t i tems. - 51 -In r e p l y t o W h i t e l y , Wright ( 1 9 7 7 b ) argued t h a t the sample s i z e r e q u i r e d t o t e s t f i t o f items t o the Rasch model i s s i m p l y d i c t a t e d by the p r e c i s i o n o f t he c a l i b r a t i o n o f items w h i c h the t e s t b u i l d e r w i s h e s t o a c h i e v e . W r i g h t f e l t t h a t a sample s i z e as s m a l l as 1 0 0 may p r o v i d e u s e f u l i n f o r m a t i o n i n c e r t a i n t e s t b u i l d i n g s i t u a t i o n s . In an e m p i r i c a l i n v e s t i g a t i o n o f the m a t t e r o f sample s i z e , i t was found t h a t an N_ between 2 0 0 and 3 0 0 was s u f f i c i e n t f o r f i e l d t e s t i n g items t o be s u b j e c t e d t o a n a l y s i s u s i n g the Rasch model, w i t h a sample s i z e o f 2 0 0 p r o -v i d i n g i n f o r m a t i o n n e a r l y as a c c u r a t e as a sample s i z e o f 3 0 0 ( F o r s t e r et a l , un d a t e d ) . T e s t A n a l y s i s and Development U s i n g the Rasch Model Once item parameters have been e s t i m a t e d and f i t o f the items t o the Rasch model has been e v a l u a t e d , t e s t s may be c o n s t r u c t e d and c a l i b r a t e d t o o b t a i n an a b i l i t y - s c a l e v a l u e c o r r e s p o n d i n g t o each raw s c o r e on the t e s t . The Rasch model has been a p p l i e d i n t e s t c o n s t r u c t i o n and a n a l y s i s i n the s c h o o l c o n t e n t a r e a s o f r e a d i n g ( F o r s t e r et a l , undated; Rentz & Bashaw, 1 9 7 7 ) and mathematics ( F o r s t e r e t a l , undated; Hashway, 1 9 7 7 ; 0'Shea, 1 9 7 9 ) a t t he e l e m e n t a r y l e v e l . School achievement t e s t s i n v e s t i g a t e d have t y p i c a l -l y been m u l t i p l e - c h o i c e , but i t i s p r o b a b l e t h a t t h e model w i l l accommodate items w i t h o t h e r modes o f response p r o v i d e d the t a s k s can be d i c h o t o m i z e d i n t o pass o r f a i l s c o r e s (Rentz & R e n t z , 1 9 7 8 ) . There i s e v i d e n c e t h a t t h e Rasch p r o c e d u r e i s somewhat h a r s h e r i n i d e n -t i f y i n g u n s u i t a b l e items than i s t h e c o n v e n t i o n a l p r o c e d u r e o f i t e m a n a l y s i s . Between 5 % t o 1 0 % of the items i n some t e s t s have been r e j e c t e d by Rasch c r i t e r i a a f t e r c o n v e n t i o n a l a n a l y s i s has been a p p l i e d (Rentz & Re n t z , 1 9 7 8 ) . However, t h i s would n a t u r a l l y depend upon the v a l u e s o f the r e j e c t i o n c r i t e -r i a chosen. When the range o f item d i s c r i m i n a t i o n s i s w i d e , Rasch p r o c e d u r e s and c o n v e n t i o n a l p r o c e d u r e s w i l l tend t o d i f f e r i n the r e j e c t i o n o f i t e m s . - 52 -Rasch p r o c e d u r e s e l i m i n a t e items a t both extremes o f the d i s c r i m i n a t i o n v a l u e spectrum w h i l e c o n v e n t i o n a l p r o c e d u r e s e l i m i n a t e o n l y low d i s c r i m i n a t i n g i t e m s . However, s t u d i e s comparing t h e c h a r a c t e r i s t i c s o f items s e l e c t e d f o r t e s t s u s i n g c o n v e n t i o n a l and Rasch p r o c e d u r e s have f a i l e d t o p i n p o i n t sub-s t a n t i a l d i f f e r e n c e s between items s e l e c t e d / r e j e c t e d by the two methods (Hamb-l e t o n e t a l , 1978). C o n v e n t i o n a l item a n a l y s i s and t h e Rasch p r o c e d u r e were used to d e v e l o p two homogeneous s u b s c a l e s from the k3~itern MacAndrew a l c o h o l i s m s c a l e admin-i s t e r e d t o 373 u n i v e r s i t y s t u d e n t s ( D o u g l a s s , K h a v a r i & F a r b e r , 1979). Con-v e n t i o n a l item a n a l y s i s y i e l d e d a s u b s c a l e o f 29 items from t h e o r i g i n a l s c a l e each h a v i n g i t e m - t o t a l t e s t c o r r e l a t i o n s o f .2 o r g r e a t e r . The s u b s c a l e c o n s t r u c t e d u s i n g t h e Rasch method c o n s i s t e d o f 36 items r e m a i n i n g a f t e r items h a v i n g a c h i - s q u a r e v a l u e f a i l i n g t o f i t the model w i t h a p r o b a b i l i t y l e s s than o r equal t o .001 were d i s c a r d e d . The i n v e s t i g a t o r s found s u b s t a n t i a l d i f f e r e n c e s i n the t o t a l number o f items chosen by each method, s p e c i f i c items chosen by each method, and the r e l a t i v e f i t o f items depending upon w h i c h p r o c e d u r e was used. They were a l s o a b l e t o a r r i v e at d i f f e r e n t d e f i n i t i o n s o f the t r a i t s measured by the c o n v e n t i o n a l s u b s c a l e and t h e Rasch s u b s c a l e . The Rasch raw s c o r e s c a l e s and the Rasch a b i l i t y e s t i m a t e s c a l e s c o r r e -l a t e d more h i g h l y w i t h the o r i g i n a l 49-item t e s t than d i d the c o n v e n t i o n a l raw s c o r e s c a l e s . However, e v a l u a t i o n o f the c o r r e l a t i o n s w i t h a t - t e s t f o r dependent c o r r e l a t i o n s y i e l d e d no s i g n i f i c a n t d i f f e r e n c e s . The c o n v e n t i o n a l s u b s c a l e , the Rasch s u b s c a l e , and the o r i g i n a l k3~itern f u l l s c a l e a l l had e q u a l l y low c o r r e l a t i o n s w i t h an e x t e r n a l c r i t e r i o n , the K h a v a r i A l c o h o l t e s t f o r a s s e s s i n g a l c o h o l consumption. The r e s e a r c h e r s c o n c l u d e d t h a t n e i t h e r the c o n v e n t i o n a l s u b s c a l e nor the Rasch s u b s c a l e had been e f f e c t i v e as a p u r i f i e d s u b s c a l e d e r i v e d from an a l -- 53 -ready w e l l - c o n s t r u c t e d homogeneous s c a l e . They f e l t t h a t the c o n v e n t i o n a l method s h o u l d remain the method o f c h o i c e f o r t e s t c o n s t r u c t i o n s i n c e i t was more e a s i l y u n d e r s t o o d and was l e s s c o s t l y as r e g a r d s computer t i m e . Ryan and Hamm (1976) used the Rasch p r o c e d u r e i n an attempt t o i n c r e a s e i n t e r n a l c o n s i s t e n c y , as measured by KR 20, and t o c l a r i f y the f a c t o r s t r u c -t u r e o f t h e items on e i g h t teacher-made t e s t s i n e d u c a t i o n a l r e s e a r c h . The number o f items i n t h e t e s t s v a r i e d from 15 t o 50. A l l items were f o u r - o p t i o n m u l t i p l e c h o i c e q u e s t i o n s . Two s u b s c a l e s were c o n s t r u c t e d from each o f the o r i g i n a l t e s t s , a s u b s c a l e o f items r e m a i n i n g a f t e r d e l e t i o n a c c o r d i n g t o Rasch c r i t e r i a , and a s u b s c a l e c o n s i s t i n g o f items l o a d i n g on the f i r s t f a c -t o r i n a p r i n c i p a l components a n a l y s i s of the o r i g i n a l t e s t . A KR 20 r e l i -a b i l i t y c o e f f i c i e n t was c a l c u l a t e d f o r each o f t h e e i g h t o r i g i n a l t e s t s as w e l l as f o r t h e two t e s t s d e r i v e d from each o r i g i n a l t e s t i n o r d e r t o d e t e r -mine the e f f e c t which item s e l e c t i o n approach had on t e s t i n t e r n a l c o n s i s t -ency. S i n c e t h e two s u b s c a l e s d e v e l o p e d from each t e s t c o n s i s t e d of fewer items than the o r i g i n a l t e s t , t h e r e s e a r c h e r s a p p l i e d the Spearman-Brown f o r m u l a t o the KR 20 v a l u e f o r each s u b s c a l e so t h a t the v a l u e s would be comparable f o r a t e s t as long as the o r i g i n a l t e s t . With the Spearman-Brown f o r m u l a a p p l i e d t o the KR 20 v a l u e s i t was found t h a t a l l s u b s c a l e s c o n s t r u c t e d u s i n g t h e Rasch a p p r o a c h , and seven o f the s u b s c a l e s c o n s t r u c t e d u s i n g f a c t o r a n a l y s i s , y i e l d e d a KR 20 c o e f f i c i e n t s u p e r i o r t o t h a t o f each r e l a t e d o r i g i n a l t e s t . O m i t t i n g t h e Spearman-Brown c o r r e c t i o n f o r m u l a , seven o f the Rasch s u b s c a l e s and two o f t h e f a c t o r a n a l -yses s u b s c a l e s s t i l l had KR 20 r e l i a b i l i t i e s s u p e r i o r t o t h a t o f t h e o r i g i n a l t e s t s . I t s h o u l d be n o t e d t h a t the Rasch approach r e t a i n e d more items from t h e o r i g i n a l t e s t s than d i d the f a c t o r a n a l y t i c a p p roach. The e f f e c t s on t h e f a c t o r s t r u c t u r e o f t h e t e s t s by d e l e t i n g items were compared by p e r f o r m i n g a f a c t o r a n a l y s i s on the o r i g i n a l s c a l e , the Rasch - 5k -s u b s c a l e , and the f a c t o r a n a l y s i s s u b s c a l e o f each t e s t . The p r o p o r t i o n o f v a r i a n c e a c c o u n t e d f o r by t h e ' f i r s t f a c t o r was h i g h e s t f o r t e s t s c o n s t r u c t e d u s i n g f i r s t f a c t o r i t e m s . The f i r s t f a c t o r o f the Rasch s u b s c a l e s a c c o u n t e d f o r o n l y a s l i g h t l y h i g h e r p r o p o r t i o n o f t h e v a r i a n c e than d i d the f i r s t f a c t o r on the o r i g i n a l t e s t s . - 5 5 -CHAPTER I I I DESIGN OF THE STUDY The purpose o f the s t u d y was t o o b j e c t i v e l y a p p l y c o n v e n t i o n a l and Rasch item a n a l y s i s p r o c e d u r e s t o a pool o f t e s t items and t o compare the s p e c i f i c n a t u r e o f the items s e 1 e c t e d / r e j e c t e d as w e l l as the o v e r a l l c h a r a c t e r i s t i c s o f t e s t s r e s u l t i n g from the a p p l i c a t i o n o f the two approaches. The stu d y concerned i t s e l f w i t h the p r a c t i c a l a p p l i c a t i o n o f two d i f f e r e n t s t a t i s t i c a l item a n a l y s i s p r o c e d u r e s . I t was not con c e r n e d p e r se w i t h e n s u r i n g t h a t the r e s u l t i n g t e s t s p o s s e s s e d c o n t e n t v a l i d i t y w hich would have i n v o l v e d s u b j e c -t i v e judgements r e g a r d i n g the items t o be i n c l u d e d . I tern Pool The item pool used i n t h i s s t u d y was a m o d i f i e d v e r s i o n o f the 7 5 " i t e m pool used f o r the development o f the Grade 4 S c i e n c e Assessment p i l o t t e s t s . F i r s t , the item pool used i n t h i s s t u d y c o n t a i n e d an a d d i t i o n a l i t e m w h i c h had been added t o the f i n a l v e r s i o n o f the t e s t used i n the assessment. Second, t h r e e items w h i c h a l l r e f e r r e d t o the same g r a p h i c a l m a t e r i a l and appeared i n c o n s e c u t i v e o r d e r were s e p a r a t e d . A l t h o u g h t h e s e items were not c h a i n e d , i t was f e l t t h a t k e e p i n g the items c l u s t e r e d t o g e t h e r might v i o l a t e the assumption o f l o c a l independence f o r the Rasch model. T h i r d , item 2 2 from p i l o t t e s t b o o k l e t A and item 3 0 from p i l o t t e s t b o o k l e t B were r e w r i t -t e n . The former item d i d not have a stem c o m p a t i b l e w i t h i t s o p t i o n s and the l a t t e r item had two c o r r e c t o p t i o n s . F o u r t h , the f i n a l v e r s i o n o f the 6 8 items used i n the a c t u a l assessment, was adopted. And f i f t h , i n s t e a d o f u s i n g two s e p a r a t e b o o k l e t s , a l l 7 6 items were i n c l u d e d i n one t e s t . The goa l areas and the items a s s i g n e d t o them i n t h e 7 6 - i t e m v e r s i o n o f the item pool - 56 -used i n the p r e s e n t s t u d y were the same as t h o s e used f o r the o r i g i n a l item pool and a r e shown i n T a b l e 2. A copy o f t h e t e s t b o o k l e t used i s p r o v i d e d i n Appendix A. The Sample Sample S i z e The i n i t i a l minimum sample s i z e was s e t a t n_ = 500. An r^ o f 500 would s a t i s f y the a p p a r e n t minimum r e q u i r e m e n t o f a t l e a s t 200 t o 300 p e r s o n s need-ed f o r Rasch item a n a l y s i s as d i s c u s s e d i n Chapter I I . A sample o f 500 would T a b l e 2 Assignment o f I terns t o Goals Goal 1--Understanding Concepts (30 items) 1, 3, 4, 5, 6, 7, 10, 12, 13, 19, 20, 21, 2 4 , 27, 28, 30, 34 38, 4 0 , 4 2 , 45, 4 6 , 51, 52, 53, 54, 55, 58, 67, 72 Goal 2— — Ski 11 i n the P r o c e s s e s o f S c i e n c e (32 items) C l a s s i f y i n g : 9, 11, 44, 49, 65, 76 O b s e r v i n g : 16, 17, 23, 35, 60 Communicating: 1 4 , 31, 32, 59, 70, 71 M e a s u r i n g : 8, 18, 26, 29, 37, 4 1 , 4 8 , 63, 73 Combined S k i l l s : 22, 33, 36, 56, 62, 74 Goal 3 - - A p p l i c a t ion (5 items) 39, 43, 50, 66, 69 Goal 4--Safety (5 items) 2, 25, 61, 68, 75 Goal 5 ~ ~ S c i e n t i f i c L i t e r a c y (4 items) 15, 47, 57, 6 4 a l s o s a t i s f y the r e q u i r e m e n t o f 300 t o 500 s u b j e c t s s u g g e s t e d by Henryssen (1971) f o r a m e a n i n g f u l c o n v e n t i o n a l item a n a l y s i s . F u r t h e r m o r e , f a c t o r - 57 -a n a l y s i s was employed i n the p r e s e n t s t u d y as a method o f d e t e r m i n i n g t e s t homogeneity. Where f a c t o r a n a l y s i s o f t e s t s i s i n v o l v e d , i t i s s u g g e s t e d t h a t t h e r e be a sample c o n s i s t i n g o f a t l e a s t 10 s u b j e c t s per item ( N u n n a l l y , 1967). The two l a r g e s t s u b t e s t s used i n t h i s study c o n s i s t e d o f 30 items (Concepts) and 32 items ( P r o c e s s e s ) . For f a c t o r a n a l y s i s o f a 32-item sub-t e s t a sample s i z e o f 500 s u b j e c t s exceeds t h i s g u i d e l i n e by 180 s u b j e c t s . O b t a i n i n g the Sample In o r d e r t o e x e r c i s e g r e a t e r c o n t r o l o v e r t h e d i s t r i b u t i o n and c o l l e c -t i o n o f t e s t b o o k l e t s , and s i n c e a v a i l a b l e time and funds were l i m i t e d , the sample s e l e c t e d was r e s t r i c t e d t o the Okanagan V a l l e y o f B r i t i s h Columbia. A c c o r d i n g l y , p e r m i s s i o n was sought from the s u p e r i n t e n d e n t s o f t h e s i x s c h o o l d i s t r i c t s i n t h i s a r e a t o a d m i n i s t e r the t e s t s t o the grade f o u r s t u d e n t s i n t h e i r di s t r i c t s . S h o r t l y a f t e r m i d - O c t o b e r , 1979, the s u p e r i n t e n d e n t s were s e n t a l e t t e r (see A ppendix B) i n w h i c h the purpose o f the study as w e l l as the n a t u r e o f the t e s t t o be used was d e s c r i b e d . I t was s t r e s s e d t h a t the purpose o f the s t u d y was t o compare methods o f item a n a l y s i s and not t o compare i n d i v i d u a l -s t u d e n t , s c h o o l , o r d i s t r i c t achievement. The s t r i c t e s t c o n f i d e n t i a l i t y c o n c e r n i n g such d a t a was a s s u r e d . A stamped, s e l f - a d d r e s s e d r e t u r n e n v e l o p e was i n c l u d e d w i t h each l e t t e r . F i v e s u p e r i n t e n d e n t s g r a n t e d p e r m i s s i o n t o c o n t a c t the e l e m e n t a r y s c h o o l p r i n c i p a l s i n t h e i r d i s t r i c t s ^ . A c c o r d i n g l y , i n November 1979, 59 e l e m e n t a r y s c h o o l p r i n c i p a l s and head t e a c h e r s were c o n t a c t e d by l e t t e r (see Appendix B). The l e t t e r s c o n t a i n e d i n f o r m a t i o n s i m i l a r t o t h a t c o v e r e d i n the 1. The s i x t h s u p e r i n t e n d e n t answered a f f i r m a t i v e l y , but h i s l e t t e r was not r e c e i v e d u n t i l a f t e r the l e t t e r s t o p r i n c i p a l s had been s e n t and j u s t s h o r t l y b e f o r e the t e s t i n g p e r i o d was t o commence. C o n s e q u e n t l y , s t u -dents from t h i s d i s t r i c t were not t e s t e d . - 58 -l e t t e r s t o the s u p e r i n t e n d e n t s . A t e s t i n g p e r i o d d u r i n g t h e week o f December 3 - 7 , 1979, was s u g g e s t e d . I t was emphasized t h a t the t e s t s would be com-p u t e r s c o r e d . The l e t t e r i n c l u d e d a t e a r o f f sheet on w h i c h p r i n c i p a l s c o u l d i n d i c a t e t h e i r d e c i s i o n t o a l l o w grade f o u r c l a s s e s i n t h e i r s c h o o l s t o take p a r t i n the s t u d y o r not. The t e a r o f f s h eet a l s o i n c l u d e d a space f o r t h e number o f c o p i e s o f t h e t e s t r e q u i r e d f o r each s c h o o l i f the d e c i s i o n t o t a k e p a r t i n the s t u d y was p o s i t i v e . Stamped, s e l f - a d d r e s s e d r e t u r n e n v e l o p e s were a g a i n p r o v i d e d . Of t h e 59 p r i n c i p a l s a p p r oached, 31 r e p l i e d . N i n e t e e n p r i n c i p a l s g r a n t e d p e r m i s s i o n t o conduct the t e s t i n g i n t h e i r s c h o o l s . Of the 12 p r i n c i p a l s who r e p l i e d n e g a t i v e l y , s e v e r a l p r o v i d e d e x p l a n a t i o n s f o r t h e i r d e c i s i o n s . In one c a s e , the s c h o o l was a p r i m a r y s c h o o l w i t h K - 3 c l a s s e s o n l y . A n o t h e r s m a l l s c h o o l d i d not w i s h t o take p a r t i n the s t u d y because they had o n l y 11 grade f o u r s t u d e n t s d i s t r i b u t e d i n s p l i t grade t h r e e / f o u r and grade f o u r / f i v e c l a s s e s . Two p r i n c i p a l s mentioned t h a t t h e i r s c h o o l s would be h e a v i l y i n -v o l v e d i n s e a s o n a l p r o j e c t s a t the time o f t e s t i n g . In one s c h o o l the p r i n -c i p a l was on m e d i c a l l e a v e o f absence and t h e p e r s o n i n c h a r g e w i s h e d as few c o m p l i c a t i o n s as p o s s i b l e d u r i n g the r e g u l a r a d m i n i s t r a t o r ' s absence. A l s o , s c h o o l s i n the t a r g e t group were i n v o l v e d i n a s c i e n c e p i l o t t e s t p r o j e c t a t the grade f o u r t o seven l e v e l b e i n g c onducted by the E d u c a t i o n a l Research I n s t i t u t e o f B r i t i s h Columbia ( E d u c a t i o n a l Research I n s t i t u t e o f B r i t i s h C o l u m b i a , 1980). S t u d e n t s were t e s t e d i n 15 o f the 19 s c h o o l s i n w h i c h p e r m i s s i o n t o t e s t had been g r a n t e d . These s c h o o l s e n r o l l e d a p p r o x i m a t e l y 550 grade f o u r s t u -dents and were a l l l o c a t e d i n a compact g e o g r a p h i c a l a r e a i n t h e s o u t h e r n p a r t o f the Okanagan V a l l e y , t h e r e b y f a c i l i t a t i n g the d a t a c o l l e c t i o n . Three s c h o o l s had a grade f o u r enrolment o f 20 o r l e s s ; seven had a grade f o u r en-rolment o f between 21 and hO p u p i l s , and f i v e had a grade f o u r e n r o l m e n t o f between k] and 60 p u p i l s . - 59 -Tes t i ng Te s t packages were d e l i v e r e d t o the 15 sample s c h o o l s d u r i n g the week p r i o r t o December 3 ~ 7, 1979- A c o v e r i n g l e t t e r t o the p r i n c i p a l o f each s c h o o l gave i n s t r u c t i o n s as t o when the t e s t s were t o be a d m i n i s t e r e d and s t a t e d t h a t completed t e s t s would be c o l l e c t e d t h e week a f t e r t e s t i n g (see Appendi x A ) . The t e s t packages i n c l u d e d s e t s o f d i r e c t i o n s f o r t h e t e a c h e r s who were t o a d m i n i s t e r the t e s t s (see Appendix A ) . The d i r e c t i o n s i n c l u d e d an example q u e s t i o n w h i c h was t o be put on the b l a c k b o a r d f o r a l l s t u d e n t s t o see. The t e a c h e r s were asked t o en s u r e t h a t a l l s t u d e n t s had a c e n t i m e t e r r u l e r w h i c h was r e q u i r e d f o r the c o m p l e t i o n o f q u e s t i o n 37- I t was s t r e s s e d t h a t s t u -dents r e t a i n t h e i r t e s t b o o k l e t s i f t e s t i n g were i n t e r r u p t e d by a r e s t p e r i -od. An i d e n t i f i c a t i o n p r o c e d u r e t o en s u r e t h i s was s u g g e s t e d . The p r o c e d u r e p r e s e r v e d the c h i l d ' s anonymity. Teachers were f r e e t o d e v i s e t h e i r own i d e n -t i f i c a t i o n p r o c e d u r e s i f they w i s h e d . Teachers were asked t o encourage s t u -d ents t o attempt a l l q u e s t i o n s and t o use the 'I don't know' o p t i o n o n l y i f . they d i d not know the answer. To a v o i d problems a s s o c i a t e d w i t h speeded t e s t s i n c o n v e n t i o n a l and Rasch item a n a l y s i s , t e a c h e r s were asked t o a l l o w s u f f i c i e n t time f o r at l e a s t 95% o f the s t u d e n t s i n t h e i r c l a s s e s t o complete a l l i t e m s . The t e a c h e r s a d m i n i s t e r i n g the t e s t s were asked t o p r o v i d e p e r t i -nent w r i t t e n comments about the t e s t i f they w i s h e d . Data P r e p a r a t i o n A t o t a l o f 527 completed t e s t b o o k l e t s was c o l l e c t e d . P r e l i m i n a r y i n -s p e c t i o n o f the t e s t b o o k l e t s i n d i c a t e d t h a t s t u d e n t s g e n e r a l l y had s u f f i -c i e n t t i m e t o complete the t e s t . To a l l e v i a t e p o t e n t i a l e f f e c t s due t o speededness upon subsequent a n a l y s e s , b o o k l e t s w h i c h s a t i s f i e d t he f o l l o w i n g c r i t e r i a were e l i m i n a t e d : - 60 -1. I f a t e s t b o o k l e t had mo.re than f i v e unanswered items at the end o f e i t h e r o f the two t e s t s e c t i o n s the b o o k l e t was e l i m i n a t e d on the b a s i s t h a t the p u p i l a p p a r e n t l y d i d not have enough time t o complete one o r both s e c t i o n s o f the t e s t . 2. I f a t e s t b o o k l e t had more than 10 items w h i c h were unanswered o r f o r w h i c h more than one response had been i n d i c a t e d , the b o o k l e t was e l i m i n a t e d from the a n a l y s i s . The b a s i s f o r t h i s d e c i s i o n r u l e was t h a t a s t u d e n t who had more than ten such items i n h i s / h e r b o o k l e t may have been too h u r r i e d t o g i v e f u l l c o n s i d e r -a t i o n t o each i t e m , o r may have had a c a r e l e s s a t t i t u d e towards c o m p l e t i o n o f the t e s t . In c ases where s t u d e n t s had not answered i s o l a t e d items t h r o u g h o u t the t e s t , o r had p r o v i d e d more than one answer t o an i t e m , the 'I don't know 1 o p t i o n was keypunched as the response f o r such items t o a maximum o f ten items per b o o k l e t . I t was f e l t t h a t s t u d e n t s who d i d not answer a p a r t i c -u l a r item o r who were c o n f u s e d r e g a r d i n g the number o f p o s s i b l e answers t o an item l i k e l y d i d not know the answer t o t h a t item. O f , t h e 527 t e s t b o o k l e t s o b t a i n e d , 11 were e l i m i n a t e d on t h e b a s i s o f the c r i t e r i a above. Data f o r the r e m a i n i n g 516 s u b j e c t s were keypunched and v e r i f i e d by comparing the keypunched d a t a t o the a c t u a l r e s p o n s e s i n the s t u -dent b o o k l e t s . The o v e r a l l e r r o r r a t e was .16 p e r c e n t . A l l e r r o r s were c o r -r e c t e d . Data A n a l y s i s R e s t r i c t i o n o f A n a l y s e s t o Concepts and P r o c e s s e s S u b t e s t s P r e l i m i n a r y a n a l y s e s i n d i c a t e d t h a t t r e a t m e n t o f s m a l l s u b t e s t s c o n t a i n -ing f o u r o r f i v e items became p r o b l e m a t i c a l when u s i n g the BICAL computer program. N o r m a l l y BICAL employs s i x s c o r e groups i n computing item f i t t o the model. I f a s u b t e s t c o n s i s t s o f f i v e i t e m s , t h i s l e a v e s o n l y f o u r i n t e -- 61 -g r a l v a l u e s - - o n e , two, t h r e e , and f o u i — w h i c h can be used t o s t r i k e s c o r e groups f o r the a n a l y s i s . F i v e , r e p r e s e n t i n g a p e r f e c t s c o r e , i s not used i n the a n a l y s i s f o r reasons d i s c u s s e d i n Chap t e r I I . W h i l e the number o f s c o r e groups used i n BICAL can be v a r i e d , i t would seem t h a t Rasch i n d i c e s o b t a i n e d f o r a c l u s t e r o f f i v e items might be q u i t e u n r e l i a b l e because o f t h e r e s t r i c -t e d number o f s c o r e groups i n v o l v e d i n t h e i r c a l c u l a t i o n , and t h e v a s t l y d i s -p r o p o r t i o n a t e numbers of s u b j e c t s w h i c h might o c c u r i n each group. For t h e above r e a s o n , i t was d e c i d e d t o fo c u s a t t e n t i o n on the two l a r g -e r s u b t e s t s i n v o l v i n g s c i e n c e c o n c e p t s and the p r o c e s s e s o f s c i e n c e . As shown i n T a b l e 2, t h e s e s u b t e s t s c o n s i s t e d o f 30 items and 32 i t e m s , r e s p e c -t i v e l y . I t was f e l t t h a t t he most r e l i a b l e and u s e f u l comparisons c o u l d be made by a n a l y z i n g items from t h e s e l a r g e r s u b t e s t s . C o n s e q u e n t l y , the items i n t he A p p l i c a t i o n , S a f e t y , and S c i e n t i f i c L i t e r a c y s u b t e s t s were e x c l u d e d from the subsequent a n a l y s e s . T e s t s f o r U n i d i m e n s i o n a 1 i t y As mentioned i n Chapter I I , i t i s g e n e r a l l y f e l t t h a t l a t e n t t r a i t t e s t models such as the Rasch model a r e a p p l i c a b l e o n l y t o t e s t s w h i c h a r e u n i d i -m e n s i o n a l (Hambleton S Traub, 1973). A c c o r d i n g l y , the f o l l o w i n g s t e p s were taken t o d e t e r m i n e whether the Concepts and P r o c e s s e s s u b t e s t s were unidimen-s i o n a l : 1. Cronbach's c o e f f i c i e n t a l p h a . Cronbach's c o e f f i c i e n t a l p h a was .76 and .83 f o r t h e Concepts and P r o c e s s e s s u b t e s t s , r e s p e c t i v e l y . When the P r o c e s s e s s u b t e s t was s t r a t i f i e d i n t o f i v e groups o f i t e m s , each group r e l a t -ing t o one o f the o b j e c t i v e s - - C 1 a s s i f y i n g ; O b s e r v i n g , Communicating, Measur-i n g , Combined P r o c e s s e s - - s t r a t i f i e d c o e f f i c i e n t a l p h a was .79- These r e l a t -i v e l y h i g h v a l u e s s u g g e s t e d t h a t each o f the two s u b t e s t s was e s s e n t i a l l y un i d i mens iona 1 . - 6 2 -2 . F a c t o r a n a l y s i s . F a c t o r a n a l y s i s was performed on the Concepts and P r o c e s s e s s u b t e s t s . A s h o r t computer program was w r i t t e n t o render the o r i -g i n a l t e s t responses i n t o b i n a r y s c o r e d f o r m - - 0 f o r an i n c o r r e c t answer and 1 f o r a c o r r e c t answer. T e t r a c h o r i c c o r r e l a t i o n c o e f f i c i e n t s were then com-puted among the items f o r each s u b t e s t . The r e s u l t i n g c o r r e l a t i o n m a t r i c e s were s u b j e c t e d t o a p r i n c i p a l components a n a l y s i s u s i n g the AGFAP f a c t o r a n a l y s i s program ( H a k s t i a n & Bay, 1 9 7 3 ) - The number o f f a c t o r s r e t a i n e d f o r t r a n s f o r m a t i o n was d e t e r m i n e d by two c r i t e r i a - - a ) the number o f e i g e n v a l u e s g r e a t e r than u n i t y , t h a t i s , the K a i s e r - G u t t m a n c r i t e r i o n , and b) the s c r e e t e s t as d e s c r i b e d by Gorsuch ( 197*0. The r e s u l t s from the p r i n c i p a l components a n a l y s e s on each o f the two s u b t e s t s a r e p r e s e n t e d i n T a b l e 3 - As shown, the number of f a c t o r s t o be e x t r a c t e d was p r o b l e m a t i c a l . A p p l i c a t o n o f the K a i s e r - G u t t m a n c r i t e r i o n sug-g e s t e d a g r e a t e r number o f f a c t o r s than the s c r e e t e s t . To f u r t h e r c l a r i f y the number o f f a c t o r s , the s o l u t i o n s were t r a n s f o r m e d u s i n g t h e H a r r i s - K a i s e r method o f o b l i q u e t r a n s f o r m a t i o n w i t h c_ = 0 (independent c l u s t e r s o l u t i o n ) . P a t t e r n c o e f f i c i e n t s g r e a t e r than | . * t 0 | were c o n s i d e r e d t o be s a l i e n t . In the case o f both s u b t e s t s , f o r each number o f f a c t o r s r e t a i n e d , groups o f items w i t h s a l i e n t l o a d i n g s d i d not lend t hemselves t o m e a n i n g f u l i n t e r p r e t a -t i o n as b e i n g homogeneous c l u s t e r s o f items measuring an o b v i o u s l y d e f i n a b l e t r a i t . Because f a c t o r a n a l y s i s d i d not s e p a r a t e the items i n t o c l u s t e r s meas-u r i n g c l e a r l y d i f f e r e n t t r a i t s i t p r o v i d e d a d d i t i o n a l e v i d e n c e f o r the u n i d i -m e n s i o n a l n a t u r e o f the two s u b t e s t s used i n t h i s s t u d y . I t was f e l t t h a t the i d e n t i f i c a t i o n o f more than one f a c t o r a f t e r e x t r a c t i o n was a t t r i b u t a b l e t o problems a s s o c i a t e d w i t h dichotomous v a r i a b l e s i n p r o d u c i n g s p u r i o u s f a c t o r s ( G o r s u c h , 1 9 7 * * ) . In a t h i r d i n v e s t i g a t i o n a p r i n c i p a l components f a c t o r a n a l y s i s o f the m a t r i x o f Pearson product-moment c o r r e l a t i o n s among the f i v e item c l u s t e r s i n - 63 -the P r o c e s s e s s u b t e s t was performed. .As shown i n T a b l e 3, the r e s u l t s o f t h i s a n a l y s i s p r o v i d e d f u r t h e r e v i d e n c e f o r the u n i f a c t o r i a l n a t u r e o f t h e P r o c e s s e s s u b t e s t . T a b l e 3 F a c t o r A n a l y s e s o f S u b t e s t s S u b t e s t V a r i a b l e s Type o f Cor re 1 a t i on C o e f f i c i ent F a c t o r s E i g e n v a l u e s >1 Rule Ext r a c t e d S c r e e T e s t Concepts 30 items t e t r a c h o r i c 10 2, 6 P r o c e s s e s 32 items t e t r a c h o r i c 8 k P r o c e s s e s 5 i tern: c l u s t e r s P e arson p roduct-moment 1 I tern A n a l y s i s and T e s t C o n s t r u c t i o n The 30 Concepts items and t h e 32 P r o c e s s e s items were t r e a t e d as two s u b t e s t s . I terns were a n a l y z e d s e p a r a t e l y i n each s u b t e s t . U s i n g both the c o n v e n t i o n a l and Rasch p r o c e d u r e s , Concepts and P r o c e s s e s s u b t e s t s were b u i l t a c c o r d i n g t o the f o l l o w i n g seven s e t s o f c r i t e r i a : 1. C o n v e n t i o n a l Approach The items i n each o f t h e two s u b t e s t s were s u b j e c t e d t o a con-v e n t i o n a l item a n a l y s i s u s i n g the LERTAP ( N e l s o n , 197^ 0 computer program. I terns were d i s c a r d e d i f t h e y f a i l e d t o meet one o r more o f t h e f o l l o w i n g c r i t e r i a ; a. The d i f f i c u l t y i n d e x , p_, must l i e w i t h i n t h e range .20 < p < .80. b. The c o r r e c t e d i t e m - s u b t e s t p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t must be equal t o o r g r e a t e r than .20. c. A l l d i s t r a c t o r s must have an u n c o r r e c t e d i t e m - s u b t e s t - 6 4 -p o i n t - b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t l e s s than 0. d. A l l d i s t r a c t o r s must have a nonzero d i f f i c u l t y i n d ex. 2. Rasch Approach I (Panchapakesan) Examinees w i t h s c o r e s below the g u e s s i n g l e v e l s e t u s i n g P a n c h a p a k e s a n 1 s f o r m u l a were e x c l u d e d from the c a l i b r a t i o n sample f o r t h a t s u b t e s t . I terns were d i s c a r d e d i f they f a i l e d t o meet one o r more o f the f o l l o w i n g c r i t e r i a : a. The mean s q u a r e f i t must be l e s s than one s t a n d a r d d e v i a t i o n above u n i t y . b. The d i s c r i m i n a t i o n i ndex must be w i t h i n . one s t a n d -a r d d e v i a t i o n of the mean i tern 'di s c r i m i nat i o n . c. The d i f f i c u l t y i n dex must be w i t h i n one s t a n d a r d d e v i a t i o n o f the mean item d i f f i c u l t y . 3. Rasch Approach I I (Panchapakesan) P a n c h a p a k e s a n 1 s f o r m u l a was a g a i n a p p l i e d i n d e t e r m i n i n g the compo-s i t i o n o f t h e c a l i b r a t i o n sample. For t h i s a n a l y s i s , c r i t e r i o n c as a p p l i e d i n Rasch Approach I above was dropped. 4 . Rasch Approach I I I (Panchapakesan) A p p l i c a t i o n o f P a n c h a p a k e s a n 1 s f o r m u l a a g a i n d e t e r m i n e d the make-up o f t h e c a l i b r a t i o n sample. C r i t e r i a b and c as a p p l i e d i n Rasch Approach I above were dropped. 5. Rasch Approach IV (Random) T h i s a n a l y s i s used the same c r i t e r i a as Rasch Approach I above. In d e t e r m i n i n g the c a l i b r a t i o n sample, examinees w i t h s c o r e s below the random g u e s s i n g l e v e l f o r each s u b t e s t , s e t u s i n g the k/m_ f o r m u l a , were e x c l u d e d . 6 . Rasch Approach V (Random) T h i s a n a l y s i s used the same c r i t e r i a as Rasch Approach II above. - 65 -The c o m p o s i t i o n o f t h e c a l i b r a t i o n sample was d e t e r m i n e d by a p p l y -i n g the random g u e s s i n g l e v e l f o r m u l a . 7. Rasch Approach VI (Random) T h i s a n a l y s i s used the same c r i t e r i o n as Rasch Approach I I I above. A p p l i c a t i o n o f the random g u e s s i n g l e v e l f o r m u l a d e t e r m i n e d the make-up o f the c a l i b r a t i o n sample. C o r r e c t i o n o f i t e m - s u b t e s t p o i n t - b i s e r i a l c o r r e l a t i o n s . For s h o r t e r t e s t s i t i s a d v i s a b l e t o c o r r e c t i t e m - t e s t c o r r e l a t i o n c o e f f i c i e n t s f o r s p u r i o u s n e s s . An a p p r o p r i a t e c o r r e c t i o n f o r m u l a f o r use w i t h the p o i n t -b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t i s G u i l f o r d ' s (1953) f o r m u l a as d i s c u s s e d by Berk (1978) : r. = r . s - s. —1 r — 1 1 — t —1 - _ , where r. = the c o r r e l a t i o n o f an item w i t h the remainder o f the t o t a l —1 r s c o r e a f t e r removal o f the i t e m , r . ^ = the p o i n t - b i s e r i a 1 c o r r e l a t i o n o b t a i n e d between the item — i t and the t o t a l s c o r e w i t h the item i n c l u d e d , s_. = the s t a n d a r d d e v i a t i o n o f the i t e m , £j£j > p r o v i d e d the items a re dichotomous1y s c o r e d w i t h w e i g h t s o f one f o r c o r r e c t items and z e r o f o r i n c o r r e c t i t e m s , and s_ t = t h e s t a n d a r d d e v i a t i o n o f t h e t o t a l t e s t s c o r e . When the i t e m - t e s t c o r r e l a t i o n s a r e h i g h - - . k O o r g r e a t e i — a p p l i c a t i o n o f the c o r r e c t i o n f o r m u l a does not appear t o be n e c e s s a r y . However, the s t a t u s o f items h a v i n g p o i n t - b i s e r i a 1 c o r r e l a t i o n s c l o s e t o the c u t o f f p o i n t - - _ r ^ = .20 i n t h i s s t u d y - - i s o f t e n changed when the c o r r e c t i o n f o r m u l a i s a p p l i e d . For example, c o n s i d e r an i t e m h a v i n g an _ r ^ ^ o f .25 w i t h the t o t a l t e s t and a p-v a l u e o f .31. I f the t o t a l t e s t has a s t a n d a r d d e v i a t i o n o f 4.86, the - 66 -c o r r e c t e d r , i s .16. A p p l i c a t i o n o f the c o r r e c t i o n f o r m u l a would i n d i c a t e -Pb r e j e c t i o n o f t h e item u s i n g t h e c r i t e r i a o f t h i s s t u d y . E l i m i n a t i o n of s u b j e c t s from the Rasch a n a l y s e s . Two methods were used t o e l i m i n a t e s u b j e c t s from t h e c a l i b r a t i o n samples used i n the s i x Rasch approaches d e s c r i b e d above. For t h e f i r s t t h r e e (Approaches I, I I , and I I I ) the minimum s c o r e f o r i n c l u s i o n o f s u b j e c t s i n the c a l i b r a t i o n sample was d e t e r m i n e d by u s i n g P a nchapakesan 1s (1969) f o r m u l a : £ = k + 2 / k ( m - 1), m V rn 2 where Ik = the number o f i t e m s , m_ = the number o f a l t e r n a t i v e s per i t e m , and r_ = the s c o r e below w h i c h examinees a r e e l i m i n a t e d . P a n chapakesan 1s f o r m u l a i s based on the random g u e s s i n g s c o r e , k/rn, and t a k e s i n t o a c c o u n t t h e v a r i a n c e a t t r i b u t a b l e t o random g u e s s i n g , (k.Cm - D)/m 2. S i n c e the number o f o p t i o n s f o r items i n the t e s t s v a r i e d from f o u r t o s i x , the minimum s c o r e f o r i n c l u s i o n o f examinees i n the c a l i b r a t i o n sample was d e t e r m i n e d as a com p o s i t e . The minimum s c o r e s o b t a i n a b l e by g u e s s i n g on the f o u r - , f i v e - , and s i x - o p t i o n items were computed s e p a r a t e l y and then added t o o b t a i n the t o t a l minimum s c o r e w h i c h was rounded up t o t h e n e a r e s t whole number. For t h e l a s t t h r e e Rasch approaches o u t l i n e d above (Approaches IV, V, and V I ) , the minimum s c o r e f o r i n c l u s i o n o f s u b j e c t s i n the c a l i b r a t i o n sample was d e t e r m i n e d by a p p l i c a t i o n o f t h e l e s s c o n s e r v a t i v e f o r m u l a used t o f i n d t h e s c o r e o b t a i n a b l e by random g u e s s i n g on m u l t i p l e - c h o i c e q u e s t i o n s . T h i s approach has been s u g g e s t e d by Wri g h t and Mead (1978). The minimum s c o r e i s o b t a i n e d by c a l c u l a t i n g the e x p e c t e d s c o r e based on random g u e s s i n g , _k/m. A g a i n , the s c o r e f o r each s u b t e s t was d e t e r m i n e d as a com p o s i t e and t h e minimum s c o r e was then rounded up t o the n e a r e s t whole number. T a b l e k - 67 -p r o v i d e s a summary o f the number o f s u b j e c t s whose t e s t s were used i n each o f the i t e m a n a l y s e s approaches. As can be seen, t h e r e i s a l a r g e r e d u c t i o n i n the number o f s u b j e c t s e n t e r i n g the Rasch a n a l y s e s i n which P a n c h a p a k e s a n 1 s f o r m u l a was a p p l i e d . The numbers, however, a r e above the range o f 200 t o 300 s u b j e c t s s u g g e s t e d f o r m e a n i n g f u l a p p l i c a t i o n o f the Rasch method i n i t e m ana l y s i s . T a b l e 4 Number o f S u b j e c t s i n Each Item A n a l y s i s A n a l y s i s Concepts S u b t e s t Minimum Score Number o f S u b j e c t s 516 13 394 7 501 P r o c e s s e s S u b t e s t C o n v e n t i o n a l — 516 Rasch I, I I , I I I 13 462 Rasch IV, V, VI 7 508 S u b t e s t Comparisons The above item a n a l y s i s approaches y i e l d e d a c o n v e n t i o n a l Concepts sub-t e s t , a c o n v e n t i o n a l P r o c e s s e s s u b t e s t , s i x Rasch Concepts s u b t e s t s , and s i x Rasch P r o c e s s e s s u b t e s t s . The s u b t e s t s c o n s t r u c t e d were s u b j e c t e d t o a s e r i e s o f c o m p a r a t i v e a n a l y s e s as f o l l o w s : 1. P e r c e n t a g e o f items r e j e c t e d by each approach. The p e r c e n t a g e o f items d i s c a r d e d from the Concepts and P r o c e s s e s item p o o l s u s i n g t h e c r i t e r i a C onventiona1 Rasch I , I I , I I I Rasch IV, V, VI - 68 -i n each o f t h e approaches was computed. 2. P e r c e n t a g e o v e r l a p o f r e j e c t e d i t e m s . For each p o s s i b l e p a i r o f Concepts s u b t e s t s c o n s t r u c t e d u s i n g d i f f e r e n t a p p r o a c h e s , t h e r e was a common group o f items r e j e c t e d by both a p p r o a c h e s . T h i s was a l s o t r u e o f each pos-s i b l e p a i r o f P r o c e s s e s s u b t e s t s c o n s t r u c t e d . To d e t e r m i n e the p e r c e n t a g e o v e r l a p o f items commonly r e j e c t e d by two a p p r o a c h e s , the number of commonly r e j e c t e d items was d i v i d e d by the number o f commonly r e j e c t e d items p l u s the number o f items u n i q u e l y r e j e c t e d by each o f the a p p r o a c h e s . The q u o t i e n t was then m u l t i p l i e d by 100. For i n s t a n c e , the number of items commonly r e -j e c t e d by the C o n v e n t i o n a l Approach and Rasch Approach I f o r t h e Concepts s u b t e s t was 10. Two items were u n i q u e l y r e j e c t e d by the C o n v e n t i o n a l Approach and s i x items were u n i q u e l y r e j e c t e d by Rasch Approach I. Thus, the p e r c e n t -age o v e r l a p o f items r e j e c t e d by t h e two approaches f o r t h e Concepts s u b t e s t was computed as f o l l o w s - - ( 1 0 / ( 1 0 + 2 + 6)) x 100 = 56%. The p e r c e n t a g e o v e r l a p o f r e j e c t e d items was computed f o r each p o s s i b l e p a i r o f approaches f o r t h e Concepts s u b t e s t s and f o r t h e P r o c e s s e s s u b t e s t s . 3. Rasch e l i m i n a t i o n o f items not meeting c o n v e n t i o n a l c r i t e r i a . S i n c e the Rasch model i s a r e l a t i v e newcomer t o item a n a l y s i s , i t was d e c i d e d t o o b t a i n a measure o f the a b i l i t y o f Rasch i t e m a n a l y s i s t o r e j e c t items w h i c h would be c o n s i d e r e d u n s u i t a b l e because they d i d not meet c o n v e n t i o n a l c r i t e -r i a o f item s e l e c t i o n . A c c o r d i n g l y , t h e p e r c e n t a g e o f Rasch r e j e c t e d items i n each o f t h e Rasch Approaches I t o VI w h i c h a l s o d i s p l a y e d problems r e l a t e d t o c o n v e n t i o n a l c r i t e r i a was computed by d i v i d i n g the number o f t h e s e items by the number o f items r e j e c t e d i n the C o n v e n t i o n a l Approach and m u l t i p l y i n g t h i s q u o t i e n t by 100. For i n s t a n c e , a p p l i c a t i o n o f the p - v a l u e c r i t e r i o n t o t h e Concepts item pool i n the C o n v e n t i o n a l Approach l e d t o r e j e c t i o n o f f o u r i t e m s . Rasch Approach II was s u c c e s s f u l i n i d e n t i f y i n g o n l y one o f t h e s e i t e m s . T h e r e f o r e , the p e r c e n t a g e s u c c e s s f o r Rasch Approach II i n i d e n t i f y -- 69 -i n g Concepts items d i s p l a y i n g p - v a l u e problems was 1/4 x 100 = 25%. These p e r c e n t a g e s were computed f o r each t y p e o f c o n v e n t i o n a l item p r o b l e m , f o r each c o m b i n a t i o n o f c o n v e n t i o n a l item p r o b l e m s , and f o r t h e t o t a l number o f items i n the Concepts and P r o c e s s e s item p o o l s d i s p l a y i n g c o n v e n t i o n a l prob-lems. P e r c e n t a g e s were o b t a i n e d f o r each Concepts and P r o c e s s e s s u b t e s t b u i l t u s i n g a Rasch approach. 4. Rasch s p e c i f i c i t e m s . For each o f the Rasch s u b t e s t s c o n s t r u c t e d the p e r c e n t a g e o f items d i s c a r d e d s o l e l y on the b a s i s o f Rasch c r i t e r i a was computed. The t o t a l number o f items unique 1y d i s e a r d e d by t h e Rasch c r i t e r i a was d i v i d e d by the t o t a l number o f items d i s c a r d e d by the approach and t h e q u o t i e n t was m u l t i p l i e d by 100. Thus, the denominator i n t h e c o m p u t a t i o n was t h e sum o f the number o f items u n i q u e l y r e j e c t e d by Rasch c r i t e r i a and the number o f items r e j e c t e d by Rasch c r i t e r i a w hich a l s o d i s p l a y e d problems r e l a t e d t o c o n v e n t i o n a l c r i t e r i a . For i n s t a n c e , i n Rasch Approach IV, seven items were u n i q u e l y d i s c a r d e d by Rasch c r i t e r i a from the Concepts item p o o l . An a d d i t i o n a l n i n e i t e m s , which a l s o d i s p l a y e d problems r e l a t e d t o conven-t i o n a l c r i t e r i a were a l s o d i s c a r d e d by Rasch Approach IV. T h e r e f o r e , the p e r c e n t a g e o f R a s c h - s p e c i f i c items d i s c a r d e d from the Concepts item pool by Rasch Approach IV was (7/(7 + 9)) x 100 = 44%. 5. T e s t c h a r a c t e r i s t i c s . Each Concepts and P r o c e s s e s s u b t e s t r e s u l t i n g from the seven d i f f e r e n t i t e m a n a l y s i s p r o c e d u r e s was 1 r e a d m i n i s t e r e d 1 t o t h e o r i g i n a l 516 s u b j e c t s used i n the seven a p p r o a c h e s . Each s u b t e s t was s u b j e c t -ed t o a n a l y s i s u s i n g t h e LERTAP computer program. Data o b t a i n e d f o r each s u b t e s t i n c l u d e d t e s t means, s t a n d a r d d e v i a t i o n s , and range o f s c o r e s , as w e l l as Hoyt e s t i m a t e s o f r e l i a b i l i t y and s t a n d a r d e r r o r s o f measurement. 6. C o r r e l a t i o n o f s u b j e c t s ' s c o r e s on s u b t e s t s . In o r d e r t o d e t e r m i n e whether the seven Concepts s u b t e s t s were o r d e r i n g s u b j e c t s i n a s i m i l a r f a s h -i o n , c o r r e l a t i o n s were o b t a i n e d f o r the 516 s u b j e c t s ' s c o r e s on each p o s s i b l e - 70 -p a i r o f Concepts s u b t e s t s . S i m i l a r c o r r e l a t i o n s were computed f o r s u b j e c t s ' s c o r e s on each p o s s i b l e p a i r o f P r o c e s s e s s u b t e s t s . The s c o r e s used were the s u b j e c t s ' a r c s i n e t r a n s f o r m e d p r o p o r t i o n a l s c o r e s on each of the s u b t e s t s . Because the number o f items on each s u b t e s t v a r i e d i t was n e c e s s a r y t o con-v e r t the s u b t e s t s c o r e s o f i n d i v i d u a l s t o p r o p o r t i o n s i n o r d e r t o o b t a i n mean-i n g f u l c o m p a r i s o n s . S i n c e p r o p o r t i o n s form a b i n o m i a l r a t h e r than a normal d i s t r i b u t i o n , i t was n e c e s s a r y t o p e r f o r m an a r c s i n e t r a n s f o r m a t i o n on each p r o p o r t i o n a l s c o r e b e f o r e p r o c e e d i n g w i t h the c o r r e l a t i o n s . The c o r r e l a t i o n s were o b t a i n e d u s i n g the PEARSON CORR program i n the SPSS computer package ( N i e , H u l l , J e n k i n s , S t e i n b r e n n e r , & Bent, 1975). 7. Comparison o f s u b t e s t means. C o r r e l a t e d t - t e s t s were p e r f o r m e d on a l l p o s s i b l e p a i r s o f Concepts s u b t e s t means, and on a l l p o s s i b l e p a i r s o f P r o c e s s e s s u b t e s t means. A g a i n , a r c s i n e t r a n s f o r m a t i o n s of the 516 s u b j e c t s ' p r o p o r t i o n a l s c o r e s on each s u b t e s t were used i n computing the means o f each s u b t e s t . The T-TEST program o f the SPSS computer package was used t o p e r f o r m the a n a l y s e s . 8. D e t e r m i n a t i o n of f i t t o o r i g i n a l assessment t e s t . In e m p l o y i n g the C o n v e n t i o n a l and Rasch approaches t o item a n a l y s i s i n t h i s s t u d y o n l y s t a t i s -t i c a l i n f o r m a t i o n r e l a t i n g t o items was c o n s i d e r e d . A t e s t item was s e l e c t e d / r e j e c t e d from the s u b t e s t b e i n g b u i l t s o l e l y on the b a s i s of c o n v e n t i o n a l o r Rasch c r i t e r i a a p p l i e d t o i t s item i n d i c e s . A p u r e l y s t a t i s t i c a l approach t o t e s t c o n s t r u c t i o n i s not f o l l o w e d i n p r a c t i c e . I terns w i t h low i n d i c e s may s t i l l r e l a t e t o the c o n t e n t domain b e i n g measured by the t e s t (Cronbach, 1 9 7 0 -In a c t u a l p r a c t i c e , items w i t h i n a p p r o p r i a t e i n d i c e s a r e improved upon where p o s s i b l e and c a r r i e d i n t o t h e f i n a l v e r s i o n o f a t e s t i f i t i s f e l t t h a t they measure an i m p o r t a n t a r e a of the c o n t e n t domain. T h i s i s e s p e c i a l l y t r u e when the number o f items b e i n g p i l o t e d i s s m a l l . - 71 -Content v a l i d i t y o f the t e s t items was a p r i m a r y c o n s i d e r a t i o n i n the c o n s t r u c t i o n o f the Grade 4 S c i e n c e Assessment T e s t . Because the number o f items p i l o t e d was s m a l l , most items d i s p l a y i n g poor item i n d i c e s were r e f i n e d and c a r r i e d i n t o the f i n a l v e r s i o n o f the assessment t e s t . To d e t e r m i n e t h e f i t o f the s u b t e s t s c o n s t r u c t e d i n t h i s s t u d y t o the o r i g i n a l assessment t e s t the p e r c e n t a g e o f items on each s u b t e s t w h i c h a l s o were used i n the Concepts and P r o c e s s e s g o a l a r e a s o f the f i n a l assessment t e s t was computed. T a b l e 5 l i s t s t h e s e items. The numbers used f o r the items a r e the ones a s s i g n e d t o them i n the p r e s e n t s t u d y . T a b l e 5 I terns Used i n the F i n a l V e r s i o n of t h e Grade 4 Assessment Test U n d e r s t a n d i n g Concepts (26 items) 6, 7, 10, 12, 13, 19, 20, 21, 24 , 27, 28, 30, 3k, 38, kO, kl, k5, 46 , 51, 52, 53, 5k, 55, 58, 67, 72 S k i l l i n the P r o c e s s e s o f S c i e n c e (28 items) C l a s s i f y i n g (5 i t e m s ) : 9, 11, kk, 49, 65 O b s e r v i n g (5 i t e m s ) : 16, 17, 23, 35, 60 Communicating (6 i t e m s ) : 14, 31, 32, 59, 70, 71 M e a s u r i n g (7 i t e m s ) : 8, 18, 29, 37, 4 1 , 48 , 63 Combined S k i l l s (5 i t e m s ) : 22, 33, 36, 56, 62 - 72 -CHAPTER IV RESULTS The f o r m a t f o r C h a p t e r IV f o l l o w s t h e s e q u e n c e o f s t e p s o u t l i n e d i n C h a p t e r I I I . A d e s c r i p t i o n o f t h e f i n a l s a m p l e u s e d i n t h e s t u d y i s p r o v i d -e d . A b r i e f summary o f t h e s e v e n d i f f e r e n t i t e m a n a l y s i s p r o c e d u r e s c a r r i e d o u t on t h e i t e m p o o l f o l l o w s . The r e s u l t s o f c o m p a r i s o n s made b e t w e e n t h e s u b t e s t s r e s u l t i n g f r o m t h e s e v e n d i f f e r e n t a p p r o a c h e s a r e t h e n p r e s e n t e d . F i n a l S a m p l e In o r d e r t o e l i m i n a t e t h e p o s s i b l e e f f e c t s o f s p e e d e d n e s s i n t e r f e r i n g w i t h t h e R a s c h i t e m a n a l y s i s , t h e d e c i s i o n r u l e s f o r e l i m i n a t i o n o f t e s t b o o k l e t s d e s c r i b e d i n C h a p t e r I I I w e r e a p p l i e d . T h i s p r o c e d u r e l e d t o t h e e l i m i n a t i o n o f 11 b o o k l e t s . C o n s e q u e n t l y , t h e f i n a l s a m p l e s i z e was 5 1 6 , w i t h 246 g r a d e f o u r g i r l s a n d 270 g r a d e f o u r b o y s . The s u b j e c t s came f r o m t h r e e a d j a c e n t s c h o o l d i s t r i c t s l o c a t e d i n t h e S o u t h O k a n a g a n . T h e two d i s t r i c t s w i t h t h e s m a l l e s t s c h o o l p o p u l a t i o n s w e r e semi r u r a l w i t h s c h o o l s l o c a t e d i n s m a l l t o w n s . The d i s t r i c t w i t h t h e l a r g e s t s c h o o l p o p u l a t i o n was a s m a l l u r b a n c e n t e r . A g r i c u l t u r e i s an i m p o r t a n t f a c e t o f t h e economy i n a l l t h r e e d i s t r i c t s . In t h e s m a l l u r b a n c e n t e r , t h e s e r v i c e i n d u s t r i e s and t r a d e a r e i m p o r t a n t c o n t r i b u t o r s t o t h e economy. T h i s c i t y h a s become t h e l e a d i n g c e n t e r f o r m a n u f a c t u r e o f m o b i l e homes i n t h e P r o v i n c e . A l l t h r e e d i s t r i c t s a r e i m p o r -t a n t r e t i r e m e n t a n d t o u r i s t a r e a s . L o g g i n g a n d m i n i n g a r e s m a l l b u t s t i l l i m p o r t a n t c o n t r i b u t o r s t o t h e economy o f t h e r e g i o n . P o p u l a t i o n g r o w t h i n t h e t h r e e d i s t r i c t s r a n g e d f r o m 1 5 - 1 % t o 2 1 . 8 % o v e r t h e s p a n o f y e a r s 1971 t o 1976. E n g l i s h i s t h e m o t h e r t o n g u e s p o k e n by t h e m a j o r i t y o f t h e p e o p l e i n t h e a r e a . L e s s t h a n one p e r c e n t o f t h e p o p u l a t i o n l i s t s t h e i r m o t h e r t o n g u e as b e i n g N a t i v e I n d i a n , C h i n e s e / J a p a n e s e , o r I n d o -P a k i s t a n i . - 73 -I tern A n a l y s i s and T e s t C o n s t r u c t i o n For reasons o u t l i n e d i n Chapter I I I , i t was d e c i d e d t o r e s t r i c t the a n a l y s e s t o t h e Concepts and P r o c e s s e s item p o o l s . A c c o r d i n g l y , the u n i d i -m e n s i o n a l i t y o f t h e two item p o o l s was t e s t e d u s i n g Cronbach's c o e f f i c i e n t a l p h a and f a c t o r a n a l y s i s as d e t a i l e d i n t h e p r e v i o u s c h a p t e r . I t was d e c i d -ed t o p e r f o r m a s e r i e s o f seven item ana 1yses--one c o n v e n t i o n a l and s i x a p p l y -ing the Rasch a p p r o a c h — t o the Concepts and P r o c e s s e s item p o o l s . To f a c i l i -t a t e subsequent p r e s e n t a t i o n o f the r e s u l t s o b t a i n e d by a p p l y i n g the t e s t c o n s t r u c t i o n c r i t e r i a o u t l i n e d i n the p r e v i o u s c h a p t e r , the codes summarized i n T a b l e 6 wi11 be used. A p p l i c a t i o n o f the c r i t e r i a i n the seven d i f f e r e n t approaches i s summarized i n T a b l e 7- Raw d a t a from the LERTAP and BICAL com-p u t e r programs a r e p r e s e n t e d f o r each item i n Appendix C. T a b l e 6 C r i t e r i a Used i n t h e I tern A n a l y s e s Code C o n v e n t i o n a l C r i t e r i a a The d i f f i c u l t y i n d e x , p_, must l i e w i t h i n t he range .20 < £ < .80. b The c o r r e c t e d i t e m - s u b t e s t p o i n t - b i s e r i a 1 c o r r e l a -t i o n c o e f f i c i e n t must be eq u a l t o o r g r e a t e r than .20. c A l l d i s t r a c t o r s must have u n c o r r e c t e d i t e m - s u b t e s t p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t s l e s s than z e r o . d A l l d i s t r a c t o r s must have a nonzero d i f f i c u l t y index. Rasch Cr i t e r i a MS The mean square f i t must be l e s s than one s t a n d a r d d e v i a t i o n above u n i t y . Dl The d i s c r i m i n a t i o n index must be w i t h i n one s t a n d a r d d e v i a t i o n o f the mean item d i s c r i m i n a t i o n . DF The d i f f i c u l t y i n d e x must be w i t h i n one s t a n d a r d d e v i a t i o n o f the mean item d i f f i c u l t y . - Ik -S u b t e s t Comparisons 1. P e r c e n t a g e o f items r e j e c t e d by each approach. The p e r c e n t a g e o f items r e j e c t e d by each o f the item a n a l y s i s approaches i s summarized i n T a b l e 8. The p e r c e n t a g e i s g i v e n f o r each Concepts and P r o c e s s e s s u b t e s t . The number o f items i n each item p o o l , k, i s i n d i c a t e d . 2. P e r c e n t a g e o v e r l a p o f r e j e c t e d i t e m s . The p e r c e n t a g e o v e r l a p o f items commonly r e j e c t e d by two d i f f e r e n t approaches was computed f o r each p o s s i b l e p a i r o f Concepts s u b t e s t s and each p o s s i b l e p a i r o f P r o c e s s e s sub-t e s t s . These p e r c e n t a g e s a r e summarized i n T a b l e s 9 and 10. 3- Rasch e l i m i n a t i o n o f items not meeting c o n v e n t i o n a l c r i t e r i a . The sequence number o f items r e j e c t e d by a p p l i c a t i o n o f the C o n v e n t i o n a l Approach i s shown i n T a b l e 11. The c o n v e n t i o n a l c r i t e r i a w hich d e t e r m i n e d the r e j e c -t i o n o f the items a r e coded as i n T a b l e 6. By u s i n g the i n f o r m a t i o n i n T a b l e 11, i t i s p o s s i b l e t o d e t e r m i n e the p e r c e n t a g e o f t h e s e items w h i c h were a l s o T a b l e 7 Item A n a l y s i s C r i t e r i a Approach C o r r e c t i o n f o r Guess i ng C r i t e r i a Convent i ona1 Rasch I Rasch I I Rasch I I I Rasch IV Rasch V Rasch VI none Panchapakesan Panchapakesan Panchapakesan random random random a , b , c, d MS, Dl , DF MS, Dl MS MS, D l , DF MS, Dl MS - 75 -T a b l e 8 P e r c e n t a g e o f I terns R e j e c t e d Approach P e r c e n t a g e of Items R e j e c t e d Concepts (k = 30) P r o c e s s e s (k =32) Convent i ona 1 4o 28 Rasch 1 53 41 Rasch 11 30 22 Rasch 111 10 6 Rasch IV 53 44 Rasch V 37 16 Rasch VI 13 3 r e j e c t e d i n each o f t h e Rasch approaches. The p e r c e n t a g e e f f e c t i v e n e s s o f the Rasch approaches i n i d e n t i f y i n g t h e s e items i s summarized i n T a b l e s 12 and 13- P e r c e n t a g e Rasch e f f i c i e n c i e s i n r e j e c t i n g items h a v i n g s p e c i f i c conven-t i o n a l problems o r c o m b i n a t i o n s o f s e v e r a l problems i s shown as w e l l as per-centage e f f i c i e n c y i n r e j e c t i n g a l l items i n Concepts and P r o c e s s e s s u b t e s t s w h i c h d i s p l a y c o n v e n t i o n a l c r i t e r i a p roblems. The number o f items d i s p l a y i n g each c o n v e n t i o n a l problem i s shown i n b r a c k e t s . 4. Rasch s p e c i f i c i t ems. For each Rasch a p p r o a c h , the p e r c e n t a g e o f items r e j e c t e d s o l e l y on the b a s i s o f Rasch c r i t e r i a was d e t e r m i n e d . These p e r c e n t a g e s a r e shown f o r each Concepts and P r o c e s s e s s u b t e s t i n T a b l e 14. - 76 -T a b l e 9 P e r c e n t a g e O v e r l a p o f R e j e c t e d Items Concepts S u b t e s t Approach I II I I I IV V VI C o n v e n t i o n a l 56 40 25 47 35 33 Rasch I 56 19 68 42 25 Rasch II 33 32 43 30 Rasch I I I 19 27 75 Rasch IV 69 25 Rasch V 36 Rasch VI T a b l e 10 P e r c e n t a g e O v e r l a p o f R e j e c t e d I terns P r o c e s s e s S u b t e s t App roach 1 1 1 I I I IV V VI Convent i ona1 57 23 22 44 8 11 Rasch 1 54 15 69 20 8 Rasch 11 29 31 33 14 Rasch 111 7 17 50 Rasch IV 36 7 Rasch V 20 Rasch VI T a b l e 11 Items R e j e c t e d - - C o n v e n t i o n a l Approach C r i t e r i a Sequence Numbe rs o f I terns R e j e c t e d Concepts P r o c e s s e s a 6, 2 1 , 38 , 54 11, 14, 17, 2 2 , 35 , 49 , 71 b 46 , 58 , 67 73 c 10, 42 --b, c 30 --a, b, c 24 , 53 --a , c, d -- 16 T o t a l 12 items 9 i terns - 77 -T a b l e 12 Rasch Approaches — Concepts S u b t e s t P e r c e n t a g e o f Items R e j e c t e d A c c o r d i i ng t o C o n v e n t i o n a l C r i t e r i a Approach C r i t e r i a a (4.) b(3) c ( 2 ) b , c ( l ) a,b,c(2) T o t a l ( 1 2 ) Rasch I 100 67 50 1 00 100 83 Rasch I I 25 33 50 100 100 50 Rasch I I I 0 33 0 0 100 25 Rasch IV 100 100 0 0 100 75 Rasch V 25 100 0 0 100 50 Rasch VI 0 67 0 0 100 33 T a b l e 1 3 Rasch Approaches — P r o c e s s e s S u b t e s t P e r c e n t a g e o f Items R e j e c t e d A c c o r d i n g t o C o n v e n t i o n a l C r i t e r i a Approach C r i t e r i a a (7) b(D a, c, d(D T o t a l ( 9 ) 8 6 100 100 8 9 2 9 100 0 33 14 100 0 2 2 71 100 100 78 0 100 0 11 0 100 0 11 T a b l e 14 P e r c e n t a g e o f R a s c h - S p e c i f i c Items Approach P e r c e n t a g e of R a s c h - S p e c i f i c I terns Concepts P r o c e s s e s Rasch I 3 8 3 8 Rasch I I 3 3 5 7 Rasch I I I 0 0 Rasch IV 4 4 5 0 Rasch V 4 5 8 0 Rasch VI 0 0 Rasch I Rasch I I Rasch I I I Rasch IV Rasch V Rasch VI - 78 -5. S u b t e s t c h a r a c t e r i s t i c s . Comparisons o f the Concepts and P r o c e s s e s s u b t e s t s on the b a s i s o f number o f i t e m s , t e s t means, t e s t s t a n d a r d d e v i a -t i o n s , range o f s c o r e s , Hoyt r e l i a b i l i t y e s t i m a t e s , and s t a n d a r d e r r o r s o f measurement a r e summarized i n T a b l e s 15 and 16. 6. C o r r e l a t i o n s among s u b j e c t s ' s c o r e s on s u b t e s t s . The c o r r e l a t i o n s among s u b j e c t s ' s c o r e s on each p o s s i b l e p a i r o f Concepts s u b t e s t s and on each p o s s i b l e p a i r o f P r o c e s s e s s u b t e s t s were o b t a i n e d . The s c o r e s used were the s u b j e c t s ' a r c s i n e t r a n s f o r m e d p r o p o r t i o n a l s c o r e s on each s u b t e s t . The r e -s u l t s o f t h e s e a n a l y s e s a r e summarized i n T a b l e s 17 and 18. 7. Comparisons o f s u b t e s t means. C o r r e l a t e d t - t e s t s were performed t o de t e r m i n e whether s i g n i f i c a n t d i f f e r e n c e s e x i s t e d between t h e means o f a l l p o s s i b l e p a i r s o f Concepts s u b t e s t s and a l l p o s s i b l e p a i r s o f P r o c e s s e s sub-t e s t s . The l e v e l o f p r o b a b i l i t y f o r the r e j e c t i o n o f t h e n u l l h y p o t h e s i s o f no s i g n i f i c a n t d i f f e r e n c e s between means was s e t a t .05. The r e s u l t s o f t h e s e a n a l y s e s a r e summarized i n T a b l e s 19 and 20. I f a s i g n i f i c a n t d i f f e r -ence was found between means, the t a b l e s i n d i c a t e the approach which y i e l d e d the h i g h e r mean. An a s t e r i s k i n d i c a t e s no s i g n i f i c a n t d i f f e r e n c e found be-tween t h e two means compared. 8. D e t e r m i n a t i o n o f f i t t o o r i g i n a l assessment t e s t . The f a t e o f each i t e m i n each o f the seven approaches t o item a n a l y s i s i s summarized i n T a b l e s 21 and 22. R e j e c t e d items a r e i n d i c a t e d w i t h an 'X'. The sequence number o f the item i s the one used i n t h i s s t u d y . An item number w i t h an a s t e r i s k i n d i -c a t e s t h a t the item appeared i n the f i n a l v e r s i o n o f t h e Grade 4 S c i e n c e Assessment T e s t . In an attempt t o a s s e s s the f i t o f each s u b t e s t c o n s t r u c t e d i n t h i s s t u d y t o the o r i g i n a l assessment t e s t , the p e r c e n t a g e o f items s e l -e c t e d by each approach w h i c h a l s o were c o n t a i n e d i n the Concepts and P r o c e s s -es c a t e g o r i e s o f the f i n a l assessment t e s t was d e t e r m i n e d . The r e s u l t s o f - 79 -t h i s a n a l y s i s a r e summarized i n T a b l e 23-A d i s c u s s i o n o f the r e s u l t s p r e s e n t e d i n t h i s c h a p t e r and c o n c l u s i o n s based upon them i s p r e s e n t e d i n Cha p t e r V. T a b l e 15 T e s t C h a r a c t e r i s t i c s - - C o n c e p t s S u b t e s t Approach Number o f 1 terns T e s t Mean T e s t (S S.D. Range of Scores (%) Hoyt R e l i -a b i l i t y ' S.E.M. C o n v e n t i o n a l 1 8 5 4 . 4 4 2 0 . . 0 0 0 . , 0 0 - 1 0 0 . 0 0 0 . 7 3 . 1 . 8 3 Rasch 1 14 4 8 . 6 4 1 9 . , 6 4 0 . , 0 0 - 1 0 0 . 0 0 0 . 6 1 1 . 6 5 Rasch 11 21 5 7 . 1 0 1 7 . . 5 7 4 . , 7 6 - 1 0 0 . 0 0 0 . 7 1 1 . 9 3 Rasch 111 2 7 5 5 . 7 4 1 7 . 5 6 7 . , 4 1 - 9 6 . 3 0 0 . 7 7 2 . 2 1 Rasch IV 14 5 0 . 6 4 1 9 . 71 0 . , 0 0 - 9 2 . 8 6 0 . 6 1 1 . 6 7 Rasch V 1 9 5 9 . 1 1 1 7 . 7 9 5 . , 2 6 - 9 4 . 7 4 0 . 6 8 1 . 8 5 Rasch VI 2 6 5 6 . 6 9 1 7 . 8 5 3 . 8 5 - 9 6 . 1 5 0 . 7 7 2 . 1 6 T a b l e 1 6 T e s t C h a r a c t e r i s t i c s - - P r o c e s s e s S u b t e s t Approach Number o f T e s t Mean Test S.D. Range o f Hoyt R e l i - S.E.M 1 terns ( * ) Scores {%) a b i 1 i t y Convent i ona1 2 3 5 5 . 0 9 2 0 . 7 4 4 , • 5 3 - 1 0 0 . 0 0 0 . 8 1 2 . 0 5 Rasch 1 1 9 5 7 - 3 2 2 1 . 4 7 0 . , 0 0 - 1 0 0 . 0 0 0 . 7 8 1 . 8 5 Rasch 11 2 5 6 3 . 1 6 1 8 . 6 4 8 . , 0 0 - 1 0 0 . 0 0 0 . 8 1 2 . 0 1 Rasch 111 3 0 6 3 . 4 5 1 7 - 9 3 1 0 . , 0 0 - 1 0 0 . 0 0 0 . 8 3 2 . 1 9 Rasch IV 1 8 6 1 . 1 1 2 1 . 2 2 0 . , 0 0 - 1 0 0 . 0 0 0 . 7 8 1 . 7 6 Rasch V 2 7 6 4 . 8 1 1 7 - 7 4 7 - , 4 1 - 1 0 0 . 0 0 0 . 8 1 2 . 0 3 Rasch VI 31 6 4 . 1 6 1 7 . 8 7 9 -, 6 8 - 1 0 0 . 0 0 0 . 8 3 2 . 2 2 - 81 -T a b l e 17 C o r r e l a t i o n s Between S u b j e c t s ' Scores Concepts S u b j e c t Approach I II I I I IV V VI Convent i onal • 92 .94 .96 • 90 • 92 .96 Rasch I .95 • 91 .92 • 90 • 92 Rasch I I • 97 .88 • 93 .96 Rasch I I I • 91 .95 • 99 Rasch IV .96 • 92 Rasch V .96 Rasch IV T a b l e 18 C o r r e l a t i o n s Between Subj e c t s 1 Scores P r o c e s s e s Subtes t Approach I I I 1 1 1 IV V VI C o n v e n t i o n a l 97 .97 • 99 • 95 .96 .98 Rasch I • 98 • 97 • 97 .96 • 97 Rasch I I • 99 .96 .98 .98 Rasch I I I • 95 .98 • 99 Rasch IV .97 • 97 Rasch V .98 Rasch VI T a b l e 19 C o r r e l a t e d t - t e s t s — Concepts S u b t e s t s Approach I II I I I IV V VI C o n v e n t i o n a l Con. II I I I IV V VI Rasch I II I I I IV V VI Rasch II II II V * Rasch I I I I I I - V VI Rasch IV V VI Rasch V V Rasch VI - 8 2 -T a b l e 20 C o r r e l a t e d t - t e s t s - - P r o c e s s e s S u b t e s t Approach 1 II I I I IV V VI C o n v e n t i o n a l 1 11 III IV V VI Rasch 1 I I II I IV V VI Rasch 11 1 1 V VI Rasch 111 1 1 1 V VI Rasch IV V V.I. Rasch V V Rasch VI T a b l e 21 Items R e j e c t e d — C o n c e p t s S u b t e s t s I tern Number Approaches Con. I 11 I I I IV V VI 1 3 X X X X 4 X X X X 5 X X 6 * X X X 7 -10* X 1 2 * X X 1 3 * 19* 2 0 * 2 1 * X X X 2 4 * X X X X X X X 2 7 * X X 2 8 * 3 0 * X X X 3 4 * X X 3 8 * X X X X X 4 0 * 4 2 * X X X 4 5 » 4 6 * X X X X X X X 5 1 * 5 2 * X X X 5 3 * X X X X X X X 5 4 * X X X 5 5 * 5 8 * X X X 6 7 * X X X X X 7 2 * X X Note: Items marked w i t h an a s t e r i s k were t h o s e i n c l u d e d i n the o r i g i n a l assessment t e s t . - 8 3 -T a b l e 2 2 Items R e j e c t e d — P r o c e s s e s S u b t e s t 1 tern Number Approach Con. I I I III IV V VI 8 * 9* X X X X 1 1 * X X X 1 4 * X 1 6 * X X X 1 7 * X X X X 1 8 * 22* X X X X 23* 26 X X 29* 3 1 * X X X X 32* X 33* 35* X X X 3 6 * 3 7 * 4 1 * 4 4 * X X 4 8 * 4 9 * X X X 5 6 * X X 59* bO* X X 62* 6 3 * X X X 6 5 * 7 0 * 7 1 * X X X 7 3 X X X X X X X 7 4 7 6 Note: I terns marked w i t h an a s t e r i i s k were t h o s e i n c l u d e d i n th e o r i g i n a l assessment t e s t . T a b l e 2 3 P e r c e n t a g e o f I terns i n the F i n a l Assessment T e s t R e t a i n e d by Each Approach Category App roach Con. I I I I I I IV V VI Concepts 5 4 4 6 73 8 8 5 0 6 9 8 5 P r o c e s s e s 71 61 7 9 9 6 5 7 8 6 1 0 0 - 8 4 -CHAPTER V SUMMARY AND CONCLUSIONS Summa ry Purpose and P r o c e d u r e The purpose o f t h i s s t u d y was t o compare the r e s u l t s o f a p p l y i n g conven-t i o n a l and Rasch item a n a l y s i s p r o c e d u r e s t o a pool o f t e s t i tems. The item pool used was a 76-item m o d i f i e d v e r s i o n o f t h e - p i l o t t e s t s used i n c o n s t r u c -t i n g t he B r i t i s h C olumbia Grade 4 S c i e n c e Assessment T e s t a d m i n i s t e r e d i n 1978. The item pool was a d m i n i s t e r e d t o 527 grade 4 s t u d e n t s i n 15 s c h o o l s l o c a t e d i n t h r e e a d j a c e n t s c h o o l d i s t r i c t s i n the South Okanagan V a l l e y . To c o n t r o l f o r p o s s i b l e e f f e c t s o f speededness, s u b j e c t s ' t e s t b o o k l e t s were e l i m i n a t e d f rom :the i t e m a n a l y s i s u s i n g t he two c r i t e r i a out 1ined i n Ch a p t e r I I I . E l e v e n b o o k l e t s were dropped, l e a v i n g 516 b o o k l e t s f o r item a n a l y s i s . P r e l i m i n a r y a n a l y s e s i n d i c a t e d problems as re g a r d s Rasch a n a l y s i s o f items i n s h o r t s u b t e s t s o f f o u r o r f i v e i t e m s . For t h i s r e a s o n , i t was de-c i d e d t o r e s t r i c t i t e m a n a l y s i s t o t h e two l a r g e s t s u b t e s t s i n the it e m p o o l . These were the Concepts and P r o c e s s e s s u b t e s t s c o n s i s t i n g o f 30 and 32 i t e m s , r e s p e c t i v e l y . To de t e r m i n e t he u n i d i m e n s i o n a l i t y o f each o f the two s u b t e s t s , c o e f f i c i e n t a l p h a i n t e r n a l c o n s i s t e n c y i n d i c e s as w e l l as f a c t o r a n a l y s i s d a t a ( T a b l e 3) were c o n s i d e r e d . C o e f f i c i e n t a l p h a i n f o r m a t i o n i n d i c a t e d s t r o n g s u b t e s t homogeneity. For the f a c t o r a n a l y s e s , use o f t h e ' K a i s e r -Guttman c r i t e r i o n i n d i c a t e d a g r e a t e r number o f f a c t o r s t o be r e t a i n e d than d i d the s c r e e t e s t . In a l l c a s e s , more than one common f a c t o r was found on w h i c h t h e r e were s a l i e n t l o a d i n g s . I n s p e c t i o n o f c l u s t e r s o f items w i t h s a l i e n t p a t t e r n c o e f f i c i e n t s f a i l e d , however, t o p r o v i d e c o n c e p t u a l l y c l e a r d e f i n i t i o n s o f p o s s i b l e d i f f e r e n t t r a i t s b e i n g measured by the s u b t e s t s . I t was f e l t t h a t use o f dichotomous t e s t d a t a and t e t r a c h o r i c c o r r e l a t i o n s - 8 5 -among t e s t items l e d t o t h e l a r g e number o f f a c t o r s i n each s u b t e s t . I t was d e c i d e d t h a t t h e s u b t e s t s were e s s e n t i a l l y u n i d i m e n s i o n a 1 and t h a t a p p l i c a t i o n o f the Rasch model would be a p p r o p r i a t e . Four c o n v e n t i o n a l c r i t e r i a r e l a t i n g t o i t e m - d i f f i c u l t y , i t e m - s u b t e s t p o i n t - b i s e r i a 1 c o r r e l a t i o n s , d i s t r a c t o r - s u b t e s t p o i n t - b i s e r i a l c o r r e l a t i o n s , and d i s t r a c t o r d i f f i c u l t y were a p p l i e d i n the C o n v e n t i o n a l Approach. The i t e m - s u b t e s t p o i n t - b i s e r i a 1 c o r r e l a t i o n s were c o r r e c t e d f o r s p u r i o u s n e s s u s i n g G u i l f o r d ' s f o r m u l a . Three Rasch item a n a l y s i s c r i t e r i a r e l a t i n g t o mean square f i t , item d i s c r i m i n a t i o n , and i t e m d i f f i c u l t y were used i n Rasch Approach I. For Rasch Approach I I , the i t e m d i f f i c u l t y c r i t e r i o n was o m i t t e d , and i n Rasch Approach I I I the o n l y c r i t e r i o n used was mean square f i t . The c r i t e r i a used i n Rasch Approaches IV, V, and VI p a r a l l e l e d t h o s e used i n Rasch Approaches I, I I , and I I I , r e s p e c t i v e l y . For Rasch Approaches I, I I , and I I I , Panchapakesan's c o r r e c t i o n f o r g u e s s i n g f o r m u l a was used t o e l i m i n a t e s u b j e c t s w i t h lower s c o r e s from t h e c a l i b r a t i o n sample. For Rasch Approaches IV, V, and V I , the random g u e s s i n g l e v e l f o r m u l a was employed t o d e t e r m i n e the minimum s c o r e s o f s u b j e c t s a l l o w e d i n the c a l i b r a t i o n sample. The i t e m a n a l y s i s c r i t e r i a used i n the C o n v e n t i o n a l Approach and the s i x Rasch Ap-proaches a r e summarized i n T a b l e s 6 and 7- The LERTAP and BICAL computer programs were used i n the a n a l y s i s o f t e s t items f o r the C o n v e n t i o n a l and Rasch A p p r o a c h e s , r e s p e c t i v e l y . A s e r i e s o f e i g h t a n a l y s e s were c a r r i e d out t o compare the one Conven-t i o n a l and s i x Rasch Approaches t o t e s t item a n a l y s i s . Four o f the a n a l y s e s were f o c u s s e d at the item l e v e l . The p e r c e n t a g e o f items r e j e c t e d by each approach as w e l l as t h e p e r c e n t a g e o v e r l a p o f r e j e c t e d items among the d i f f e r -e n t approaches was d e t e r m i n e d . The e f f i c a c y o f the Rasch Approaches i n i d e n -t i f y i n g items i l l u s t r a t i n g problems as d e t e c t e d by the C o n v e n t i o n a l Approach - 86 -was i n v e s t i g a t e d . For each o f the Rasch A p p r o a c h e s , the p e r c e n t a g e o f items r e j e c t e d s o l e l y on t h e b a s i s o f the Rasch c r i t e r i a was d e t e r m i n e d . Three comparisons were aimed at the s u b t e s t s c o n s i d e r e d as e n t i t i e s . These a n a l y s e s i n v o l v e d - - ( a ) comparisons o f numbers o f items i n each s u b t e s t , s u b t e s t means, s t a n d a r d d e v i a t i o n s , and range o f s c o r e s , as w e l l as Hoyt e s t i m a t e s o f r e l i a b i l i t y and s u b t e s t s t a n d a r d e r r o r s o f measurement; (b) c o r r e l a t i o n s o f s u b j e c t s ' s c o r e s among a l l Concepts s u b t e s t s and among a l l P r o c e s s e s s u b t e s t s , and (c) c o r r e l a t e d t - t e s t s among a l l p o s s i b l e p a i r s o f Concepts s u b t e s t means and a l l p o s s i b l e p a i r s o f P r o c e s s e s s u b t e s t means. The e i g h t h a n a l y s i s i n v o l v e d a comparison o f the f i t o f each o f the Concepts and P r o c e s s e s s u b t e s t s r e s u l t i n g from the seven d i f f e r e n t approaches t o the o r i g i n a l assessment t e s t . In t h i s i n v e s t i g a t i o n , t h e items i n the Concepts and P r o c e s s e s c a t e g o r i e s o f the f i n a l v e r s i o n o f t h e Grade k S c i e n c e Assessment T e s t as d e t e r m i n e d by the C o n t r a c t Team ( B r i t i s h Columbia M i n i s t r y o f E d u c a t i o n , 1979) were used as the b a s i s f o r r o u g h l y d e t e r m i n i n g c o n t e n t v a l i d i t y o f t h e t e s t s produced by t h e approaches employed i n t h i s s t u d y . D i s c u s s i o n o f R e s u l t s 1. P e r c e n t a g e o f items r e j e c t e d by each approach. An i n s p e c t i o n o f the r e s u l t s summarized i n T a b l e 8 i n d i c a t e s t h a t the most s t r i n g e n t s e t o f Rasch c r i t e r i a a p p l i e d i n Rasch Approaches I and IV l e d t o the r e j e c t i o n o f the h i g h e s t p e r c e n t a g e o f items. A p p l i c a t i o n o f the c o n v e n t i o n a l c r i t e r i a l e d t o the next h i g h e s t p e r c e n t a g e r e j e c t i o n o f i t e m s , f o l l o w e d by Rasch Approaches II and V. Rasch Approaches I I I and V I , e m p l o y i n g o n l y the mean s q u a r e f i t c r i t e r i o n , l e d t o t h e l o w e s t p e r c e n t a g e r e j e c t i o n o f i t e m s . In a l l approach-e s , a h i g h e r p e r c e n t a g e o f Concepts items as compared t o P r o c e s s e s items was r e j e c t e d . For the Concepts s u b t e s t , Rasch Approaches V and V I , em p l o y i n g the ran-dom g u e s s i n g l e v e l f o r m u l a , l e d t o a s l i g h t l y h i g h e r p e r c e n t a g e o f items - 8 7 -r e j e c t e d than d i d t h e p a r a l l e l Rasch Approaches II and I I I , u s i n g Panchapak-e s a n 1 s f o r m u l a t o d e t e r m i n e the make-up o f the c a l i b r a t i o n sample. For the P r o c e s s e s s u b t e s t , however, the r e v e r s e i s t r u e , w i t h Rasch Approaches II and I I I r e j e c t i n g a somewhat h i g h e r p e r c e n t a g e o f items than t h e p a r a l l e l Rasch Approaches V and VI. In C o n v e n t i o n a l and Rasch i t e m a n a l y s i s , the p e r c e n t a g e o f items r e j e c t e d appears t o be s t r o n g l y r e l a t e d t o the s t r i n g e n c y o f the c r i t e r i a employed. For the Rasch A p p r o a c h e s , t h e r e appears t o be no a p p r e c i a b l e d i f f e r e n c e i n the p e r c e n t a g e o f items r e j e c t e d by approaches u s i n g P a n c h a p a k e s a n 1 s f o r m u l a and p a r a l l e l approaches e m p l o y i n g the random g u e s s i n g l e v e l f o r m u l a . 2. P e r c e n t a g e o v e r l a p o f r e j e c t e d i t e m s . I n s p e c t i o n o f T a b l e s 9 and 10 i n d i c a t e s t h a t the h i g h e s t p e r c e n t a g e o v e r l a p o f items r e j e c t e d by the Con-v e n t i o n a l Approach and the Rasch Approaches was w i t h Rasch Approaches I and IV, i n w h i c h t h e most s t r i n g e n t Rasch c r i t e r i a were a p p l i e d . In g e n e r a l , the p e r c e n t a g e o v e r l a p o f items r e j e c t e d by the C o n v e n t i o n a l and Rasch Approaches becomes reduced as the Rasch c r i t e r i a used i n item r e j e c t i o n become l e s s s t r i n g e n t . There i s , i n g e n e r a l , a somewhat h i g h e r p e r c e n t a g e o v e r l a p o f r e j e c t e d items between t h e C o n v e n t i o n a l Approach and Rasch Approaches u s i n g P a n c h a p a k e s a n 1 s f o r m u l a f o r d e t e r m i n i n g t h e c a 1 i b r a t i o n sample compared t o Rasch Approaches e m p l o y i n g the random g u e s s i n g l e v e l f o r m u l a . F i g u r e s ex-t r a c t e d from T a b l e s 9 and 10 i l l u s t r a t e the above t r e n d s i n T a b l e 2k. When the Rasch Approaches a r e c o n s i d e r e d a l o n e , the h i g h e s t p e r c e n t a g e o v e r l a p o f r e j e c t e d items o c c u r s between approaches u s i n g P a n c h a p a k e s a n 1 s f o r m u l a and t h e i r p a r a l l e l approaches e m p l o y i n g the random g u e s s i n g l e v e l f o r m u l a - - R a s c h I and IV; Rasch II and V; Rasch I I I and V I . Thus, i n d e t e r -m i n i n g s i m i l a r i t y between Rasch a p p r o a c h e s , the number o f item r e j e c t i o n c r i t e r i a i n the approaches appeared t o o v e r r i d e the e f f e c t o f t h e g u e s s i n g - 88 -T a b l e 2k P e r c e n t a g e O v e r l a p of R e j e c t e d I terns C o n v e n t i o n a l and Rasch Approaches Approach. Convent i ona1 Concepts P r o c e s s e s Rasch 1 56 57 Rasch 11 ko 23 Rasch 111 25 22 Rasch IV kl kk Rasch V 35 8 Rasch VI 33 11 11 a app 1ied i n d e t e r m i n i n g the c a l i b r a t i o n sample. These t r e n d s a r e summarized i n T a b l e 25. T a b l e 25 P e r c e n t a g e O v e r l a p o f R e j e c t e d Items P a r a l l e l Rasch Approaches (Concepts) IV V VI Rasch I 68 Rasch I I k3 Rasch I I I 75 ( P r o c e s s e s ) Rasch I 69 Rasch II 33 Rasch I I I 50 There i s a h i g h p e r c e n t a g e o v e r l a p o f r e j e c t e d items between Approaches I and II i n w h i c h Panchapakesan 1s f o r m u l a was a p p l i e d and i n g e n e r a l between Approaches IV and V i n w h i c h the random g u e s s i n g l e v e l f o r m u l a was used. The - 8 9 -p e r c e n t a g e o v e r l a p o f r e j e c t e d items between Approaches II and I I I , and between Approaches V and VI i s moderate. The lowest p e r c e n t a g e o v e r l a p o f r e j e c t e d items o c c u r s when Approaches I and M l , and when Approaches IV and VI a r e compared. When the f o l l o w i n g comparisons a c r o s s d i f f e r e n t methods o f d e t e r m i n i n g the c a l i b r a t i o n sample a re made--Approaches I and V; Approaches I and V I ; Approaches II and V l - - t h e p e r c e n t a g e o v e r l a p o f r e j e c t e d items tends t o drop as compared t o p a r a l l e l c omparisons w i t h i n the same methods o f d e t e r -m i n i n g the c a l i b r a t i o n samp 1e--Approaches I and I I ; Approaches f and I I I ; Approaches II and M l . These t r e n d s a r e summarized i n T a b l e 2 6 . T a b l e 2 6 P e r c e n t a g e O v e r l a p o f R e j e c t e d I terns Rasch Approaches W i t h i n (a) and A c r o s s (b) Methods o f D e t e r m i n i n g the C a l i b r a t i o n Sample (Concepts) II I I I V VI Rasch I 5 6 a 1 9 a 4 2 b 2 5 b Rasch I I 3 3 a . 3 0 b Rasch IV 6 9 a 2 5 a Rasch V 3 6 a ( P r o c e s s e s ) Rasch I 5 4 a 1 5 a 2 0 b 8 b Rasch I l 2 9 3 \k h Rasch IV 3 6 a 7 a Rasch V 2 0 a Two f a c t o r s appear t o i n f l u e n c e the p e r c e n t a g e o v e r l a p o f r e j e c t e d items among the Rasch approaches. A s t r o n g f a c t o r i s the number o f c r i t e r i a i n v o l v -ed. Approaches e m p l o y i n g p a r a l l e l c r i t e r i a but d i f f e r e n t g u e s s i n g l e v e l form-u l a s g e n e r a l l y i l l u s t r a t e a h i g h p e r c e n t a g e o v e r l a p o f r e j e c t e d i t e m s . A l s o , comparisons i n v o l v i n g one c r i t e r i o n / t w o c r i t e r i a approaches and two c r i t e r i a / t h r e e c r i t e r i a approaches r e j e c t e d a l a r g e r p e r c e n t a g e o f items i n common - 90 -than d i d c o m p a r i s o n s i n v o l v i n g one c r i t e r i o n / t h r e e c r i t e r i a a p proaches. Given the n u m b e r - o f - c r i t e r i a f a c t o r , c o m p a r i s o n s i n v o l v i n g one c r i t e r i o n / two c r i t e r i a approaches and two c r i t e r i a / t h r e e c r i t e r i a approaches r e j e c t e d a l a r g e r p e r c e n t a g e o f items i n common when the same g u e s s i n g f o r m u l a was a p p l i e d t o d e t e r m i n e the c a l i b r a t i o n sample than when p a r a l l e l c o m parisons a c r o s s d i f f e r e n t t y p e s o f g u e s s i n g f o r m u l a s were made. The n u m b e r - o f - c r i t e r i a r e l a t i o n s h i p between a p p r o a c h e s , however, appears t o be much s t r o n g e r than e f f e c t s caused by d i f f e r e n t g u e s s i n g l e v e l f o r m u l a s . 3- Rasch e l i m i n a t i o n o f items not meeting c o n v e n t i o n a l c r i t e r i a . The s p e c i f i c items r e j e c t e d by the C o n v e n t i o n a l Approach a r e summarized i n T a b l e 11. The p e r c e n t a g e s o f t h e s e items a l s o r e j e c t e d by the s i x Rasch approaches were computed and a r e summarized i n T a b l e s 12 and 13-The o v e r a l l i d e n t i f i c a t i o n o f items d i s p l a y i n g problems r e l a t e d t o the c o n v e n t i o n a l c r i t e r i a was h i g h e s t f o r the t h r e e c r i t e r i a Rasch Approaches I and IV. The p e r c e n t a g e s p r o g r e s s i v e l y dropped f o r the two c r i t e r i a Ap-proaches II and V, and the one c r i t e r i o n Approaches I II and V I . P a r t i c u l a r -l y f o r the P r o c e s s e s s u b t e s t , approaches e m p l o y i n g the random g u e s s i n g l e v e l f o r m u l a i d e n t i f i e d fewer items h a v i n g c o n v e n t i o n a l problems t h a n d i d p a r a l l e l approaches e m p l o y i n g P anchapakesan 1s f o r m u l a . As r e g a r d s s p e c i f i c problems w i t h i t e m s , i t seemed c l e a r t h a t removal o f the Rasch item d i f f i c u l t y c r i t e r i o n lowered the p e r c e n t a g e i d e n t i f i c a t i o n o f items h a v i n g a p - v a l u e o u t s i d e the range .20 t o .80 ( c o n v e n t i o n a l c r i t e r i o n a ) . In g e n e r a l , the approaches e m p l o y i n g o n l y the mean square f i t c r i t e r i o n were i n s e n s i t i v e as regards the i d e n t i f i c a t i o n o f items w i t h too h i g h o r too low p - v a l u e s . P a r t i c u l a r l y f o r the P r o c e s s e s s u b t e s t , approaches e m p l o y i n g P a nchapakesan 1s f o r m u l a seemed t o be somewhat more e f f e c t i v e i n i d e n t i f y i n g items w i t h p - v a l u e problems. A c l e a r t r e n d was not d i s c e r n a b l e as r e g a r d s Rasch i d e n t i f i c a t i o n o f - 91 -items h a v i n g i t e m - s u b t e s t p o i n t - b i s e r i a 1 c o r r e l a t i o n c o e f f i c i e n t s l e s s than .20 ( c o n v e n t i o n a l c r i t e r i o n b ) . In the case o f the P r o c e s s e s s u b t e s t , a l l approaches were e f f e c t i v e i n i d e n t i f y i n g the s i n g l e item d i s p l a y i n g t h i s problem. In the case o f the Concepts s u b t e s t , i t c o u l d not be d e t e r m i n e d whether removal o f t h e item d i f f i c u l t y c r i t e r i o n o r removal o f the d i s c r i m -i n a t i o n c r i t e r i o n had a l a r g e r e f f e c t i n r e d u c i n g the power o f the a n a l y s e s t o i d e n t i f y items w i t h p o i n t - b i s e r i a 1 problems. Approaches u s i n g the random g u e s s i n g l e v e l f o r m u l a appeared t o be somewhat more e f f e c t i v e i n i d e n t i f y i n g p o i n t - b i s e r i a l problems than d i d approaches u s i n g P a n c h a p a k e s a n 1 s f o r m u l a . The Rasch approaches seemed r a t h e r i n s e n s i t i v e as r e g a r d s the i d e n t i f i c a -t i o n o f items d i s p l a y i n g problems w i t h d i s t r a c t o r s ( c o n v e n t i o n a l c r i t e r i a c and d ) . For t h e Concepts s u b t e s t , o n l y Rasch Approaches I and II i d e n t i f i e d items h a v i n g d i s t r a c t o r s w i t h a p o i n t - b i s e r i a 1 c o e f f i c i e n t g r e a t e r t h a t z e r o . A l l approaches r e j e c t e d some items h a v i n g d i s t r a c t o r problems when t h e s e oc-c u r e d i n c o n j u n c t i o n w i t h p - v a l u e and/or i t e m - s u b t e s t p o i n t - b i s e r i a 1 problems ( c o n v e n t i o n a l c r i t e r i a c o m b i n a t i o n s b, c; a, b, c, and a, c, d ) . In g e n e r a l , approaches e m p l o y i n g P a n c h a p a k e s a n 1 s f o r m u l a were somewhat more e f f e c t i v e i n i d e n t i f y i n g items w i t h d i s t r a c t o r p roblems. k. Rasch s p e c i f i c i t e m s . P e r c e n t a g e s o f items r e j e c t e d by t h e d i f f e r -e nt Rasch approaches s o l e l y on the b a s i s o f Rasch c r i t e r i a as compared t o the t o t a l number o f items r e j e c t e d by each approach a r e summarized i n T a b l e 14. The f i g u r e s i n d i c a t e t h a t a l l items r e j e c t e d by the one c r i t e r i o n Approaches II I and VI d i s p l a y e d c o n v e n t i o n a l problems as w e l l . In g e n e r a l , the two c r i t e r i a Approaches II and V r e j e c t e d the l a r g e s t p e r c e n t a g e s o f items not d i s p l a y i n g c o n v e n t i o n a l problems e x c e p t i n the case o f t h e Rasch II Approach a p p l i e d t o the Concepts s u b t e s t . G e n e r a l l y , a somewhat s m a l l e r p e r c e n t a g e o f items r e j e c t e d s p e c i f i c a l l y by Rasch c r i t e r i a was i n d i c a t e d f o r the t h r e e c r i t e r i a Approaches I and IV e x c e p t i n the case o f Rasch Approach I a p p l i e d - 92 -t o t h e c o n c e p t s s u b t e s t . Approaches u s i n g the random g u e s s i n g l e v e l f o r m u l a appeared t o r e j e c t more R a s c h - s p e c i f i c items than d i d approaches emp l o y i n g Panchapakesan's f o r m u l a . 5- S u b t e s t c h a r a c t e r i s t i c s . S u b t e s t c h a r a c t e r i s t i c s a r e summarized i n T a b l e s 15 and 16. The l e n g t h s o f the Concepts s u b t e s t s ranged from 14 t o 27 i t e m s , w h i l e the l e n g t h s o f the P r o c e s s e s s u b t e s t s ranged from 18 t o 31 i t e m s . As c o u l d be e x p e c t e d , approaches w i t h l e s s s t r i n g e n t c r i t e r i a y i e l d e d the l o n g e s t s u b t e s t s . S u b t e s t s y i e l d e d by the one c r i t e r i o n Approaches I I I and VI were l o n g e s t f o l l o w e d by s u b t e s t s y i e l d e d by the two c r i t e r i a Approaches II and V. These were f o l l o w e d i n l e n g t h by the s u b t e s t s b u i l t u s i n g the Con-v e n t i o n a l A p proach, and f i n a l l y by s u b t e s t s c o n s t r u c t e d u s i n g the t h r e e c r i t e -r i a Rasch Approaches I and IV. The range i n s u b t e s t means was not T a r g e — C o n c e p t s 48.64% t o 59-11%; P r o c e s s e s 55-09% t o 64.81%. Making comparisons among Rasch approaches i n w h i c h the same g u e s s i n g l e v e l f o r m u l a was a p p l i e d , the two c r i t e r i a approaches y i e l d e d s u b t e s t s w i t h the h i g h e s t means. These were f o l l o w e d by the means o f s u b t e s t s c o n s t r u c t e d u s i n g the one c r i t e r i o n a p p r o a c h , s u b t e s t s c o n s t r u c -t e d u s i n g the C o n v e n t i o n a l A pproach, and f i n a l l y s u b t e s t s b u i l t u s i n g the t h r e e c r i t e r i a approach. In the case o f t h e C o n v e n t i o n a l A p p r o a c h , the low s u b t e s t means r e s u l t e d from t h e f a c t t h a t , o f the 14 items w i t h a p - v a l u e o u t s i d e the range o f .20 t o .80, 12 were easy items w i t h a p - v a l u e g r e a t e r than .80. E l i m i n a t i o n o f t h e s e items tended t o lower s u b t e s t means. Rasch Approaches I and IV, u s i n g an item d i f f i c u l t y c r i t e r i o n , tended t o e l i m i n a t e some o f t h e s e easy items as w e l l , w h i l e Approaches I I , I I I , V, and VI e l i m i -n a t e d fewer o r no items w i t h c o n v e n t i o n a l d i f f i c u l t y r e l a t e d p r o b l e m s . T h i s r e s u l t e d i n h i g h e r s u b t e s t means f o r the l a t t e r Rasch a p p r o a c h e s . T h i s e f f e c t i s a l s o e v i d e n t i n i n s p e c t i n g the lower ranges o f s c o r e s o b t a i n e d on s u b t e s t s - 9 3 -b u i l t u s i n g Rasch Approaches I and IV, w h i c h e l i m i n a t e d some o f the v e r y easy it e m s . S u b t e s t means were g e n e r a l l y somewhat h i g h e r f o r s u b t e s t s c o n s t r u c t e d u s i n g Rasch approaches i n w h i c h t h e random g u e s s i n g l e v e l f o r m u l a was used compared t o s u b t e s t s b u i l t u s i n g p a r a l l e l item r e j e c t i o n c r i t e r i a but a p p l y -ing P a n c h a p a k e s a n 1 s f o r m u l a t o d e t e r m i n e the make-up o f the c a l i b r a t i o n sam-p l e . T h i s t r e n d i s i l l u s t r a t e d i n F i g u r e 3 -In g e n e r a l , the Hoyt i n t e r n a l c o n s i s t e n c y e s t i m a t e s f o r t e s t s c o n s t r u c -t e d u s i n g Rasch approaches was g r e a t e s t f o r Approaches I I I and V I . These s u b t e s t s had the g r e a t e s t number o f i t e m s . Then f o l l o w e d i n t e r n a l c o n s i s t e n c y r e l i a b i l i t i e s o b t a i n e d f o r s u b t e s t s c o n s t r u c t e d u s i n g the C o n v e n t i o n a l Ap-p r o a c h , f o l l o w e d by c o e f f i c i e n t s f o r s u b t e s t s b u i l t u s i n g Rasch Approaches II and V, a l t h o u g h s u b t e s t s c o n s t r u c t e d u s i n g t h e C o n v e n t i o n a l Approach had some-what fewer items than d i d s u b t e s t s c o n s t r u c t e d u s i n g Approaches II and V. The s u b t e s t s w i t h the l e a s t number o f i t e m s , c o n s t r u c t e d u s i n g Rasch Approaches I and IV, had the lowest Hoyt r e l i a b i l i t y i n d i c e s . There appeared t o be no major d i f f e r e n c e s i n Hoyt r e l i a b i l i t y e s t i m a t e s f o r s u b t e s t s c o n s t r u c t e d us-i n g t h e random g u e s s i n g l e v e l f o r m u l a and t h o s e c o n s t r u c t e d u s i n g Panchapa-kesan's f o r m u l a . It was e v i d e n t t h a t the number o f items i n the s u b t e s t s r e s u l t i n g from each approach was h a v i n g an e f f e c t on t h e Hoyt r e l i a b i l i t y e s t i m a t e s . In an e f f o r t t o c o n t r o l f o r t h i s f a c t o r , stepped-up r e l i a b i l i t y e s t i m a t e s f o r sub-t e s t s o f equal l e n g t h produced by each approach were c a l c u l a t e d . The f o r m u l a used was the Spearman-Brown prophecy f o r m u l a (Magnusson, 1 9 6 7 ) : £ = N ' r  1 + (NT - 1)£ where = the number o f t i m e s the t e s t i s l o n g e r o r s h o r t e r than the o r i g i n a l t e s t , and £ = the r e l i a b i l i t y o f the t e s t w h i c h i s b e i n g l e n g t h e n e d o r s h o r t e n e d . - Sh -F i g u r e 3 S u b t e s t Means (%) 7 0 + Concepts P r o c e s s e s 6 5 + + o 6 0 + 5 5 + + o 5 0 + 4 5 ^ + C o n v e n t i o n a l x Rasch (Panchapakesan) o Rasch (Random) + Con. IV/I V / l l V l / l l l Con. IV/I V / l l V l / l l l - 95 -The r e l i a b i l i t i e s o f the Concepts s u b t e s t s were stepped-up t o a t e s t o f 30 items i n l e n g t h — t h e maximum number o f items i n t h e Concepts i t e m p o o l . The r e l i a b i l i t y o f the P r o c e s s e s s u b t e s t s were stepped-up t o a t e s t o f 32 items i n l e n g t h . The r e s u l t s a r e summarized i n T a b l e 27- The r e s u l t s empha-s i z e the l a c k o f any r e a l d i f f e r e n c e s among the r e l i a b i l i t i e s o f the s u b t e s t s c o n s t r u c t e d by a p p l y i n g the d i f f e r e n t approaches. T a b l e 27 Stepped-up Hoyt R e l i a b i l i t y E s t i m a t e s Approach S u b t e s t Concepts P r o c e s s e s Convent i ona1 .82 .86 Rasch 1 • 77 .86 Rasch 11 • 78 .85 Rasch 111 • 79 .8k Rasch IV • 77 .86 Rasch V • 77 .83 Rasch VI • 79 .83 Lord (1959) and Cureton e t a l , (1973) have shown e m p i r i c a l l y t h a t the s t a n d a r d e r r o r o f measurement, SEM, o f a t e s t i s d i r e c t l y p r o p o r t i o n a l t o V k_ , the square r o o t o f the number o f items on a t e s t . An attempt was made t o d e t e r m i n e t h e n a t u r e o f t h i s r e l a t i o n s h i p f o r the s u b t e s t s c o n s t r u c t e d us-ing the approaches i n t h i s s t u d y . S u b t e s t SEM's were c o r r e l a t e d w i t h the c o r r e s p o n d i n g v V ' s , and t h e l i n e a r r e g r e s s i o n of the SEM's on the V k_ 's was a l s o o b t a i n e d . A n a l y s e s were performed f o r a l l \k s u b t e s t s (two conven-t i o n a l ; 12 R a s c h ) , f o r the 12 Rasch s u b t e s t s , f o r the s i x Rasch s u b t e s t s con-s t r u c t e d u s i n g Panchapakesan's f o r m u l a , and f o r the s i x Rasch s u b t e s t s con-s t r u c t e d u s i n g approaches e m p l o y i n g t h e random g u e s s i n g l e v e l f o r m u l a . The r e s u l t s o f t h e s e a n a l y s e s a r e summarized i n T a b l e 28. - 96 -T a b l e 2 8 R e l a t i o n s h i p s B e t w e e n S u b t e s t SEM's a n d / ~ k s A n a l y s i s C o r r e l a t i o n SI ope A l l s u b t e s t s .965 .314 A l 1 R a s c h s u b t e s t s • 971 .320 R a s c h s u b t e s t s ( P a n c h a p a k e s a n ) R a s c h s u b t e s t s ( r a n d o m g u e s s i n g ) .976 .968 • 329 • 311 A h i g h c o r r e l a t i o n b e t w e e n s u b t e s t SEM's and t h e s q u a r e r o o t s o f t h e number o f i t e m s on t h e s u b t e s t i s e v i d e n t . T h e v a l u e s o f t h e r e g r e s s i o n c o e f f i c i e n t s a r e c o n s i d e r a b l y l o w e r t h a n t h e f i n d i n g s o f L o r d , o r C u r e t o n e t a l . In a n y c a s e , s u b t e s t SEM's a p p e a r t o be s o l e l y a f f e c t e d by t h e num-b e r o f i t e m s i n t h e s u b t e s t s and n o t by t h e d i f f e r e n t a p p r o a c h e s u s e d i n t h e i r c o n s t r u c t i o n . 6. C o r r e l a t i o n s among s u b j e c t s ' s c o r e s on s u b t e s t s . The c o r r e l a t i o n s among s u b j e c t s ' s c o r e s on e a c h p o s s i b l e p a i r o f s u b t e s t s a r e shown i n T a b l e s 17 a n d 18. As c a n be s e e n , r e g a r d l e s s o f t h e i t e m a n a l y s i s a p p r o a c h u s e d , s u b t e s t s c o r e s a r e h i g h l y c o r r e l a t e d . S u b t e s t s b u i l t u s i n g t h e d i f f e r e n t a p p r o a c h e s t e n d t o o r d e r s u b j e c t s i n much t h e same f a s h i o n . 7. C o m p a r i s o n o f s u b t e s t means. The means o f t h e s u b t e s t s p r o d u c e d by t h e d i f f e r e n t a p p r o a c h e s h a v e b e e n d i s c u s s e d u n d e r s e c t i o n 5 a b o v e . The r e s u l t s o f c o r r e l a t e d t - t e s t s c o m p a r i n g e a c h p o s s i b l e p a i r o f s u b t e s t means a r e p r e s e n t e d i n T a b l e s 19 a n d 20. T a b l e s 15 a n d 16 i n d i c a t e t h a t t h e d i f f e r -e n c e s b e t w e e n many o f t h e s u b t e s t means a r e n o t g r e a t . T h e r e s u l t s o f t h e t - t e s t s , h o w e v e r , show t h a t o n l y two s u b t e s t m e a n s , t h o s e f o r R a s c h A p p r o a c h e s II a n d VI ( C o n c e p t s ) , a n d t h o s e f o r R a s c h A p p r o a c h e s II a n d I I I ( P r o c e s s e s ) d o n o t d i f f e r s i g n i f i c a n t l y a t t h e a = .05 l e v e l . I t i s f e l t t h a t t h e l a r g e n_ of 516 i n v o l v e d i n t h e t - t e s t s , a s w e l l a s t h e h i g h c o r r e l a t i o n s b e t w e e n - 97 -s u b j e c t s ' s c o r e s on t h e s u b t e s t s i n c r e a s e d t h e p o w e r o f t h e t - t e s t s t o t h e e x t e n t t h a t s i g n i f i c a n t d i f f e r e n c e s b e t w e e n s u b t e s t means w e r e i n d i c a t e d i n t h e m a j o r i t y o f t h e c o m p a r i s o n s . 8 . D e t e r m i n a t i o n o f f i t t o o r i g i n a l a s s e s s m e n t t e s t . M o s t o f t h e i t e m s u s e d i n t h e p i l o t t e s t i n g f o r t h e G r a d e k S c i e n c e A s s e s s m e n t T e s t w e r e r e t a i n e d i n t h e f i n a l v e r s i o n o f t h a t t e s t . S i n c e t h e f i n a l v e r s i o n o f t h e a s s e s s m e n t t e s t was t h e s t a n d a r d by w h i c h t h e c o n t e n t s a m p l i n g o f t h e C o n c e p t s a n d P r o c e s s e s s u b t e s t s c o n s t r u c t e d i n t h i s s t u d y was j u d g e d , a p p r o a c h e s u s i n g l e s s s t r i n g e n t c r i t e r i a a n d t h u s r e j e c t i n g f e w e r i t e m s s a m p l e d t h e c o n t e n t d o m a i n m o s t e f f e c t i v e l y . As shown i n T a b l e 2 3 , s u b t e s t s c o n s t r u c t e d u s i n g R a s c h A p p r o a c h e s I I I a n d VI r e t a i n e d t h e h i g h e s t p e r c e n t a g e o f i t e m s u s e d i n t h e o r i g i n a l t e s t . T h e s e w e r e f o l l o w e d by t h e s u b t e s t s b u i l t u s i n g t h e two c r i t e r i a R a s c h A p p r o a c h e s II a n d V. T h e C o n v e n t i o n a l A p p r o a c h a n d R a s c h A p p r o a c h e s I a n d I V , e m p l o y i n g t h e m o s t s t r i n g e n t s e t s o f c r i t e r i a y i e l d e d s u b t e s t s w i t h t h e l o w e s t p e r c e n t a g e s o f i t e m s u s e d i n t h e o r i g i n a l t e s t . Ob-v i o u s d i f f e r e n c e s i n s a m p l i n g t h e c o n t e n t d o m a i n f o r s u b t e s t s b u i l t by R a s c h a p p r o a c h e s u s i n g t h e random g u e s s i n g l e v e l f o r m u l a as c o m p a r e d t o t h o s e u s i n g P a n c h a p a k e s a n ' s f o r m u l a w e r e n o t d i s c e r n a b l e . C o n c l u s i o n s T h e one C o n v e n t i o n a l and s i x R a s c h a p p r o a c h e s u t i l i z e d i n t h i s s t u d y p r o d u c e d q u i t e d i f f e r e n t s u b t e s t s a s r e g a r d s t h e s p e c i f i c i t e m s s e l e c t e d . The n u m b e r s o f i t e m s i n t h e s u b t e s t s and t h e s u b t e s t c h a r a c t e r i s t i c s , h o w e v e r , w e r e more a f u n c t i o n o f t h e s t r i n g e n c y o f t h e c r i t e r i a a p p l i e d , r a t h e r t h a n o f t h e p a r t i c u l a r a p p r o a c h , C o n v e n t i o n a l o r R a s c h , u s e d t o b u i l d t h e s u b t e s t s . I n d e e d , t h e h i g h e s t p e r c e n t a g e o v e r l a p o f i t e m s r e j e c t e d f o r t h e C o n v e n t i o n a l A p p r o a c h was w i t h R a s c h A p p r o a c h e s I and IV. T h e s e t h r e e a p p r o a c h e s a p p l i e d t h e s t r i c t e s t c r i t e r i a f o r i t e m r e j e c t i o n . The number o f i t e m s i n e a c h s u b t e s t , i n t u r n , a f f e c t e d t h e r e l i a b i l i t y - 98 -o f t h e i n s t r u m e n t s . When t h e S p e a r m a n - B r o w n p r o p h e c y f o r m u l a was a p p l i e d i n o r d e r t o c o m p a r e t h e H o y t i n t e r n a l c o n s i s t e n c y e s t i m a t e s o f t h e s u b t e s t s on a p a r w i t h e a c h o t h e r , no a p p a r e n t d i f f e r e n c e s w e r e f o u n d i n t h e a d j u s t e d c o e f -f i c i e n t s . T e s t SEM's and t h e s q u a r e r o o t s o f t h e number o f i t e m s on t h e s u b -t e s t s w e r e f o u n d t o be h i g h l y c o r r e l a t e d . R e g r e s s i o n o f s u b t e s t SEM's on t h e r e s p e c t i v e <l. k_ ' s p r o v i d e d f a i r l y c o n s i s t e n t r e s u l t s f o r a l l s u b t e s t s , a l l R a s c h s u b t e s t s , R a s c h s u b t e s t s b u i l t u s i n g P a n c h a p a k e s a n 1 s f o r m u l a , a n d R a s c h s u b t e s t s b u i l t u s i n g t h e random g u e s s i n g l e v e l f o r m u l a . The f i t o f t h e s u b t e s t s t o t h e o r i g i n a l a s s e s s m e n t t e s t was a l s o a p p a r -e n t l y a f u n c t i o n o f t h e s t r i n g e n c y o f t h e c r i t e r i a u s e d i n t h e d i f f e r e n t i t e m a n a l y s i s a p p r o a c h e s . A p p l i c a t i o n o f s t r i n g e n t c r i t e r i a , w h e t h e r C o n v e n t i o n a l o r R a s c h , p r o d u c e d s u b t e s t s w i t h f e w e r i t e m s and t h e s e s h o r t e r s u b t e s t s p r o -v i d e d a p o o r e r s a m p l i n g o f t h e c o n t e n t d o m a i n t h a n d i d s u b t e s t s c o n s t r u c t e d u s i n g more l e n i e n t a p p r o a c h e s . S e l e c t i o n o f i t e m s s o l e l y on t h e b a s i s o f s t a t i s t i c a l c r i t e r i a , a s d o n e i n t h i s s t u d y , i s r a r e l y c a r r i e d o u t i n a c t u a l t e s t c o n s t r u c t i o n t a s k s . When f a c e d w i t h a l i m i t e d number o f i t e m s t o t e s t g i v e n g o a l s a n d o b j e c t i v e s , as w e r e t h e t e s t b u i l d e r s i n c o n s t r u c t i n g t h e G r a d e k S c i e n c e A s s e s s m e n t T e s t , t h e d a t a p r o v i d e d by c o n v e n t i o n a l i t e m a n a l y s i s i s more h e l p f u l t h a n i t e m d a t a a r r i v e d a t t h r o u g h R a s c h i t e m a n a l y s i s . C o n v e n t i o n a l i t e m a n a l y s i s c r i t -e r i a c a n be a p p l i e d t o i d e n t i f y f a u l t y i t e m s w h i c h c a n t h e n be i m p r o v e d u p o n . S u c h i t e m s c a n be r e w r i t t e n a n d t h e s h r i n k a g e o f i t e m s a v a i l a b l e t o t e s t t h e d e s i r e d k n o w l e d g e and s k i l l s w i l l b e m i n i m a l . S t r i n g e n t a p p l i c a t i o n o f R a s c h i t e m a n a l y s i s c r i t e r i a , h o w e v e r , w o u l d a p p e a r t o l e a d t o a d e a d e n d . The i t e m i n d i c e s do n o t a p p e a r t o c a r r y t h e i n f o r m a t i o n n e e d e d f o r r e f i n e m e n t o f f a u l t y i t e m s . T h e r e a p p e a r s t o be an a n a l a g o u s r e l a t i o n s h i p b e t w e e n t h e c o n v e n t i o n a l p - v a l u e a n d t h e R a s c h i t e m d i f f i c u l t y i n d e x . I d e n t i f i c a t i o n o f i t e m s h a v i n g l ow i t e m - s u b t e s t c o r r e l a -- 99 -t i o n s i s somewhat l e s s e f f e c t i v e when Rasch item a n a l y s i s p r o c e d u r e s a r e used. A l s o , Rasch i t e m a n a l y s i s appears t o be p a r t i c u l a r l y impotent i n i d e n -t i f y i n g d i s t r a c t o r problems as r e v e a l e d by c o n v e n t i o n a l item a n a l y s i s . A l t h o u g h Rasch approaches seem t o produce s u b t e s t s o f equal r e l i a b i l i t y compared t o the c o n v e n t i o n a l a p p r o a c h , and Rasch s u b t e s t s o r d e r s u b j e c t s i n v e r y much the same f a s h i o n as s u b t e s t s b u i l t u s i n g c o n v e n t i o n a l methods, t h e f i n d i n g s o f t h i s s t u d y i n d i c a t e t h a t the c o n v e n t i o n a l approach s h o u l d be the method o f c h o i c e e s p e c i a l l y when s m a l l numbers o f items are a v a i l a b l e f o r p i l o t t e s t i n g . The Rasch approach t o item a n a l y s i s might f i n d e f f e c t i v e ap-p l i c a t i o n when l a r g e numbers o f items a r e a v a i l a b l e f o r p i l o t t e s t i n g . Use of Rasch item a n a l y s i s , however, would i n c u r the expense o f r e j e c t i o n o f v a l -u a b l e items which c o u l d be improved upon and used e f f e c t i v e l y . S h o u ld the c l a i m s of s p e c i f i c o b j e c t i v i t y o f item and person a b i l i t y p arameters t e n d e r e d by proponents o f the Rasch model someday be proved un-e q u i v o c a l l y , the importance o f the Rasch model would be g r e a t l y enhanced. At such a t i m e , a h y b r i d approach t o item a n a l y s i s u s i n g a c o m b i n a t i o n of conven-t i o n a l and Rasch c r i t e r i a might be most p o t e n t i n c a p i t a l i z i n g on the unique q u a l i t i e s o f both approaches. As y e t , however, t h e Rasch approach appears t o p r o v i d e no unique b e n e f i t s t o the f i e l d o f t e s t item a n a l y s i s . W h i l e the Rasch model f a i l s as a c r e a t i v e t o o l i n t e s t c o n s t r u c t i o n , i t i s p o s s i b l e t h a t i t c o u l d be employed as a c o n f i r m a t o r y t o o l once t e s t s have been c o n s t r u c t e d u s i n g c o n v e n t i o n a l item a n a l y s i s ( R o g e r s , 1981). The d i f f i -c u l t i e s i n f a c t o r a n a l y z i n g dichotomous t e s t d a t a are w e l l known (G o r s u c h , 1974). F a c t o r a n a l y s i s o f t e s t items u s u a l l y l e a d s t o a s p u r i o u s l y h i g h number o f f a c t o r s even though o t h e r e v i d e n c e i n d i c a t e s t h a t the t e s t i s i n a c t u a l i t y u n i d i m e n s i o n a l . A p p l i c a t i o n o f t h e Rasch model i n d e t e r m i n i n g whether a group o f items measures a s i n g l e t r a i t might be l e s s p r o b l e m a t i c a l - 100 -than use o f f a c t o r a n a l y s i s f o r t h e same purpose. L i m i t a t i o n s o f T h i s Study T h i s s t u d y d i d not ad d r e s s i t s e l f t o the c l a i m s o f s a m p l e - f r e e i t e m i n d i c e s and measures o f person a b i l i t y a s s o c i a t e d w i t h the Rasch model. The model was t r e a t e d as an i t e m a n a l y s i s approach t o be used i n a s t r a i g h t f o r -ward t e s t c o n s t r u c t i o n t a s k . The outcomes o f i t s a p p l i c a t i o n were compared w i t h r e s u l t s o b t a i n e d u s i n g c o n v e n t i o n a l item a n a l y s i s . In i n t e r p r e t i n g t he r e s u l t s o f t h i s s t u d y , i t must be borne i n mind t h a t the t e s t items used r e p r e s e n t e d an a l r e a d y h i g h l y r e f i n e d c o l l e c t i o n o f t e s t i t ems. Many o f them had been o b t a i n e d from p r e v i o u s l y c o n s t r u c t e d t e s t s by the C o n t r a c t Team. A l l had undergone f u r t h e r r e f i n e m e n t i n t h e s t e p s used t o c o n s t r u c t the Grade 4 S c i e n c e Assessment T e s t . The sample o f s t u d e n t s used i n t h i s study was e s s e n t i a l l y one o f con-v e n i e n c e . The s u b j e c t s came from semi r u r a l and s m a l l urban a r e a s . G e n e r a l -i z a t i o n o f the f i n d i n g s t o o t h e r p o p u l a t i o n s o f i n t e r e s t , f o r i n s t a n c e t h o s e l i v i n g i n l a r g e urban a r e a s where E n g l i s h may o f t e n be the second language spoken i n t h e home, must be viewed w i t h c a u t i o n . D i r e c t i o n s f o r F u t u r e Research As mentioned above, the items used i n t h i s s t u d y were h i g h l y r e f i n e d . I t i s p o s s i b l e t h a t c o m parison o f Rasch and c o n v e n t i o n a l i t e m a n a l y s i s ap-p r o a c h e s , a p p l i e d t o a pool o f newly c o n s t r u c t e d t e s t i t e m s , would produce r e s u l t s more c l e a r l y p o i n t i n g t o the s u p e r i o r i t y o f one approach o v e r the o t h e r . The Rasch item a n a l y s i s approaches were d e f i c i e n t as r e g a r d s the i d e n t i -f i c a t i o n o f items h a v i n g c o n v e n t i o n a l p r o b l e m s , e s p e c i a l l y as r e g a r d s d i s -t r a c t o r a n a l y s i s . R e s e a r c h d i r e c t e d towards e x p l o i t i n g t he p o s s i b l e i n f o r m a -t i o n c a r r i e d i n Rasch i t e m i n d i c e s t o i d e n t i f y items h a v i n g s p e c i f i c problems would be v a l u a b l e i n t h a t the Rasch approach c o u l d be a p p l i e d i n i t e m a n a l y s i s - 101 -w i t h a v i e w t o w a r d s i t e m i m p r o v e m e n t r a t h e r t h a n r e j e c t i o n . As m e n t i o n e d i n t h e r e v i e w o f t h e l i t e r a t u r e i n C h a p t e r I I , t h e t w o -p a r a m e t e r a n d t h r e e - p a r a m e t e r l a t e n t t r a i t m o d e l s a r e a l s o a v a i l a b l e f o r t e s t i t e m a n a l y s i s . A c o m p a r i s o n o f t h e r e s u l t s o b t a i n e d w i t h t h e s e a p p r o a c h -e s as c o m p a r e d w i t h t h o s e f o r c o n v e n t i o n a l i t e m a n a l y s i s w o u l d be i l l u m i n a t i n g as r e g a r d s t h e r e l a t i v e e f f i c i e n c y o f l a t e n t t r a i t i t e m a n a l y s i s a s c o m p a r e d t o t h e c o n v e n t i o n a l a p p r o a c h . G i v e n t h e d i f f i c u l t i e s i n v o l v e d i n u s i n g f a c t o r a n a l y s i s t o d e t e r m i n e t e s t h o m o g e n e i t y , a w o r t h w h i l e a v e n u e o f r e s e a r c h m i g h t be t o i n v e s t i g a t e t h e e f f i c a c y o f t h e R a s c h model a s a c o n t e n d i n g t o o l t o be u s e d i n d e t e r m i n -i n g t h e u n i d i m e n s i o n a 1 i t y o f t e s t s . - 102 -BIBLIOGRAPHY A l e a m o n i , L. M. , & Spencer, R.E. A comparison o f b i s e r i a l d i s c r i m i n a -t i o n , p o i n t b i s e r i a l d i s c r i m i n a t i o n , and d i f f i c u l t y i n d i c e s i n item a n a l y s i s d a t a . E d u c a t i o n a l and P s y c h o l o g i c a l Measurement, 1 9 6 9 , 2 9 , 3 5 3 - 3 5 8 . A n d e r s e n , E. B. A goodness o f f i t t e s t f o r the Rasch model. Psycho- met r i k a , 1 9 7 3 , 3 8 , 1 2 3-140. A n d e r s o n , J . , Kearney, G. E., & E v e r e t t , A. V. An e v a l u a t i o n o f Rasch's s t r u c t u r a l model f o r t e s t i t e m s . B r i t i s h J o u r n a l o f M a t h e m a t i c a l and  S t a t i s t i c a l P s y c h o l o g y , 1 9 6 8 , • 2 J _ , 2 3 1 - 2 3 8 . Berk, R. A. E m p i r i c a l e v a l u a t i o n o f f o r m u l a e f o r c o r r e c t i o n o f item-t o t a l p o i n t - b i s e r i a l c o r r e l a t i o n s . E d u c a t i o n a l and P s y c h o l o g i c a l  Measurement, 1 9 7 8 , 3 8 , 6 4 7 - 6 5 2 . Bloom, B. S., H a s t i n g s , J . T., S Madaus, G.F. Handbook o f F o r m a t i v e and  Summatiye E v a l u a t i o n . New York: M c G r a w - H i l l , 1971-B r i t i s h Columbia M i n i s t r y o f E d u c a t i o n . Assessment P l a n n i n g : B.C. A s s e s s - ment Program. V i c t o r i a , B.C., 1 9 7 5 • B r i t i s h Columbia M i n i s t r y o f E d u c a t i o n . General Report: B r i t i s h Columbia  S c i e n c e Assessment, 1 9 7 8 , V o l . 1 . V i c t o r i a , B.C., 1 9 7 9 Brown, F. G. P r i n c i p l e s o f E d u c a t i o n a l and P s y c h o l o g i c a l T e s t i n g . New York: H o l t , R i n e h a r t and W i n s t o n , 1 9 7 6 . C a r t l e d g e , C. McC. A comparison o f e q u i p e r c e n t i 1 e and Rasch e q u a t i n g metho-d o l o g i e s ( D o c t o r a l d i s s e r t a t i o n , N o r t h w e s t e r n U n i v e r s i t y , 1 9 7 6 ) . P i s- s e r t a t i o n A b s t r a c t s I n t e r n a t i o n a l , 1 9 7 6 , 3_7, 5141A. ( U n i v e r s i t y M i c r o f i 1 m s No. 7 6 - 2 2 1 5 ) • Cohen, L. A m o d i f i e d l o g i s t i c r e sponse model f o r item a n a l y s i s . U n p u b l i s h -ed m a n u s c r i p t , 1 9 7 6 . Cronbach, L. J . C o e f f i c i e n t a l p h a and the i n t e r n a l s t r u c t u r e o f t e s t s . P s y c h o m e t r i k a , 1 9 5 1 , J 6 , ' 2 9 7 - 3 3 4 . Cronbach, L. J . E s s e n t i a l s o f Psycho 1 o t i c a 1 T e s t i n g . ( 3 r d . e d . ) . New York: H a r p e r and Row, 1 9 7 0 . Cronbach, L. J . T e s t V a l i d a t i o n . In R. L. T h o r n d i k e ( E d . ) , Educat iona1  Measurement ( 2 n d . e d . ) . Washington: American C o u n c i l on E d u c a t i o n , 1 9 7 1 -C u r e t o n , E. E. C o r r e c t e d i t e m - t e s t c o r r e l a t i o n s . P s y c h o m e t r i k a , 1 9 6 6 , 3 1 , 9 3 - 9 6 . - 1 0 3 -C u r e t o n , E. E., Cook, J . A., F i s c h e r , R. T., L a s e r , S. A., R o c k w e l l , N. J . , S Simmons, J . W. Length o f t e s t and s t a n d a r d e r r o r o f measure-ment. E d u c a t i o n a l and P s y c h o l o g i c a l Measurement, 1 9 7 3 , 3 3 , 6 3 - 6 8 . D i n e r o , T. E. , £ H a e r t e l , E. A p p l i c a b i l i t y o f the Rasch model w i t h v a r y i n g item d i s c r i m i n a t i o n s . A p p l i e d P s y c h o l o g i c a l Measurement, 1 9 7 7 , 1, 5 8 1 - 5 9 2 . D o u g l a s s , F. M. , K h a v a r i , K. A., £ F a r b e r , P. D. A comparison o f c l a s s -i c a l and l a t e n t t r a i t item a n a l y s i s p r o c e d u r e s . E d u c a t i o n a l and  P s y c h o l o g i c a l Measurement, 1 9 7 9 , 3 9 , 3 3 7 ~ 3 5 2 . Du B o i s , P. H. V a r i e t i e s o f p s y c h o l o g i c a l t e s t homogeneity. Ameri can  P s y c h o l o g i s t , 1 9 7 0 , 2 5 , 5 3 2 - 5 3 6 . E b e l , R. L. E s s e n t i a l s of E d u c a t i o n a l Measurement. Englewood C l i f f s , N. J . : P r e n t i c e - H a l 1, 1 9 7 2 a . E b e l , R. L. Why i s a l o n g e r t e s t u s u a l l y a more r e l i a b l e t e s t ? E d u c a t i o n - a l and P s y c h o l o g i c a l Measurement. 1 3 7 2 b , 3_2, 2 4 9 - 2 5 3 . E d u c a t i o n a l Research I n s t i t u t e o f B r i t i s h C olumbia. I tern Bank Development f o r S c i e n c e h - ~J. Edge, J a n u a r y - F e b r u a r y , 1 9 8 0 , p. 4 . E n g e l h a r t , M. D. A comparison o f s e v e r a l itenr d i s c r i m i n a t i o n i n d i c e s . J o u r n a l o f E d u c a t i o n a l Measurement, 1 9 6 5 , 2_,- 6 9 _ 7 6 . F i n d l e y , W. G. A r a t i o n a l e f o r e v a l u a t i o n o f item d i s c r i m i n a t i o n s t a t i s -t i c s . E d u c a t i o n a l and P s y c h o l o g i c a l Measurement, 1 9 5 6 , J_6, 1 7 5 _ 1 8 0 . F o r b e s , D. W. The use o f Rasch l o g i s t i c s c a l i n g p r o c e d u r e s i n the deve-lopment o f s h o r t m u l t i - l e v e l a r i t h m e t i c achievement t e s t s f o r p u b l i c s c h o o l measurement. Paper p r e s e n t e d a t the annual m e e t i n g o f t h e American E d u c a t i o n a l Research A s s o c i a t i o n , San F r a n c i s c o , 1 9 7 6 . ... (ERIC document R e p r o d u c t i o n S e r v i c e No. E D 1 2 8 4 0 0 ) . F o r s t e r , F., Ingebo, G., £ Wolmut, P. Monographs on Improving Measure- ment and E v a l u a t i o n . U n p u b l i s h e d m a n u s c r i p t , undated. Fryman, J . G. A p p l i c a t i o n o f the Rasch s i m p l e l o g i s t i c model to a mathe-m a t i c s placement e x a m i n a t i o n ( D o c t o r a l d i s s e r t a t i o n , U n i v e r s i t y o f K e ntucky, 1 9 7 6 ) . D i s s e r t a t i o n A b s t r a c t s I n t e r n a t i o n a l , 1 9 7 6 , 3 7 , 5 6 2 A . ( U n i v e r s i t y M i c r o f i 1 m s No. 7 7 " 5 6 8 9 ) . Gardner, P. L. T e s t l e n g t h and the s t a n d a r d e r r o r o f measurement. J o u r -n a l o f E d u c a t i o n a l Measurement. 1 9 7 0 , ] _ , 2 7 1 - 2 7 3 . G l a s s , G. V. £ S t a n l e y , J . C. S t a t i s t i c a l Methods i n E d u c a t i o n and Psy- c h o l o g y . Englewood C l i f f s , N.J.: P r e n t i c e - H a l l , 1 9 7 0 . G o r s u c h , R. L. F a c t o r A n a l y s i s . Ph i 1ade 1ph i a: Saunders, 1 9 7 4 . - 104 -Green, S. B., L i s s i t z , R. W., & M u l a i k , S. A. L i m i t a t i o n s o f c o e f f i c i e n t a l p h a as an in d e x o f t e s t u n i d i m e n s i o n a 1 i t y . E d u c a t i o n a l and Psycho- l o g i c a l Measurement, 1 9 7 7 , . 3 7 , 8 2 7 - 8 3 8 . G u i l f o r d , J . P. The c o r r e l a t i o n o f an i t e m w i t h a com p o s i t e o f the r e -m a i n i n g items i n a t e s t . E d u c a t i o n a l and P s y c h o l o g i c a l Meausrement, 1 9 5 3 , 1 3 , 8 7 - 9 3 -G u s t a f s s o n , J-E. , A r e v i e w o f e s t i m a t i o n p r o c e d u r e s f o r the Rasch model w i t h an eye toward l o n g i s h t e s t s . J o u r n a l o f E d u c a t i o n a l S t a t i s t i c s , 1 9 8 0 , 5, 3 5-64. H a k s t i a n , R. A., S Bay, K. S. User's Manual t o Accompany the A l b e r t a Gen- e r a l F a c t o r A n a l y s i s Program (AGFAP). D i v i s i o n o f E d u c a t i o n a l Re-s e a r c h S e r v i c e s , U n i v e r s i t y o f A l b e r t a , 1 9 7 3 -Hambelton, R. K., & Traub, R. E. I n f o r m a t i o n c u r v e s and e f f i c i e n c y o f t h r e e l o g i s t i c t e s t models. B r i t i s h J o u r n a l o f M a t h e m a t i c a l and  S t a t i s t i c a l P s y c h o l o g y , . 1971 , 2k_, 2 7 3 - 2 8 1 . Hambleton, R. K. , Traub, R. E. A n a l y s i s o f e m p i r i c a l d a t a u s i n g two l o g i s t i c l a t e n t t r a i t models. B r i t i s h J o u r n a l o f M a t h e m a t i c a l and  S t a t i s t i c a l P s y c h o l o g y , 1 9 7 3 , 2 6 , 1 9 5 - 2 1 1 . Hambleton, R. K. , & Cook, L. L. L a t e n t t r a i t models and t h e i r use i n the a n a l y s i s o f e d u c a t i o n a l t e s t d a t a . J o u r n a l o f E d u c a t i o n a l Meas- urement , 1 9 7 7 , 14, 7 5 - 9 6 . Hambleton, R. K., Swaminathan, H., Cook, L. L., E i g n o r , D. R., & G i f f o r d , J . A. Developments i n l a t e n t t r a i t t h e o r y : models, t e c h -n i c a l i s s u e s , and a p p l i c a t i o n s . Review o f E d u c a t i o n a l R e s e a r c h , 1 9 7 8 , 48, 4 6 7 - 5 1 0 . Hashway, R. M. A comparison o f t e s t s d e r i v e d u s i n g Rasch and t r a d i t i o n a l p s y c h o m e t r i c paradigms ( D o c t o r a l d i s s e r t a t i o n , Boston C o l l e g e , 1 9 7 7 ) . D i s s e r t a t i o n A b s t r a c t s I n t e r n a t i o n a l , 1 9 7 7 , 3 8 , 7 4 4 A . ( U n i v e r s i t y M i c r o f i 1 m s No. 7 7 - 1 7 , 5 9 4 ) . H e n r y s s e n , S. C o r r e c t i o n o f i t e m - t o t a l c o r r e l a t i o n s i n itern ana 1 y s i s . P s y c h o m e t r i k a , 1 9 6 3 , 2 8 , 2 1 1 - 2 1 8 . H e n r y s s e n , S. G a t h e r i n g , a n a l y z i n g , and u s i n g d a t a on t e s t i t e m s . In R. L. T h o r n d i k e ( E d . ) , E d u c a t i o n a l Measurement ( 2 n d . e d . ) . Washing-t o n : American C o u n c i l on E d u c a t i o n , 1 9 7 1 . Hoyt, C. T e s t r e l i a b i l i t y e s t i m a t e d by a n a l y s i s o f v a r i a n c e . Psycho- m e t r i k a , 1 9 4 1 , 6 , 1 5 3 - 1 6 0 . K a r a b i n u s , R. A. The r - p o i n t b i s e r i a l l i m i t a t i o n . E d u c a t i o n a l and  P s y c h o l o g i c a l Measurement, 1 9 7 5 , 3 5 , 2 7 7 - 2 8 2 . Kuder, G. F., £ R i c h a r d s o n , M. W. The t h e o r y o f t h e e s t i m a t i o n o f t e s t r e l i a b i l i t y . P s y c h o m e t r i ka, 1 9 3 7 , 2 , 1 5 1 - 1 6 0 . - 1 0 5 -Lange, A., Lehmann, I. J . , & Mehrens, W. A. U s i n g item a n a l y s i s t o improve t e s t s . J o u r n a l o f E d u c a t i o n a l Measurement, 1 9 6 7 , ii> 6 5 - 6 8 . L o r d , F. M. T e s t s o f the same l e n g t h do have the same s t a n d a r d e r r o r o f measurement. E d u c a t i o n a l and P s y c h o l o g i c a l Measurement, 1 9 5 9 , 1 9 , 2 3 3 - 2 3 9 . L o r d , F. M., & N o v i c k , M. R. S t a t i s t i c a l T h e o r i e s o f Mental T e s t S c o r e s . R e a d i n g , Mass.: Addi son-Wes1ey, 1 9 6 8 . Lumsden, J . The c o n s t r u c t i o n o f u n i d i m e n s i o n a 1 t e s t s . P s y c h o l o g i ca1 B u i l e t i n , . 1 9 6 1 , 5 8 , 1 2 2 - 1 3 1 . Magnusson, D. T e s t Theory. R e a d i n g , Mass.: Addison-Wes 1ey, 1 9 6 7 -N e l s o n , L. R. Guide t o LERTAP Use and I n t e r p r e t a t i o n . Dunedin, New Z e a l a n d : Department o f E d u c a t i o n , U n i v e r s i t y o f Otago, 1 9 7 4 . N i l e , N. H., H u l l , C. H. , J e n k i n s , J . G., S t e i n b r e n n e r , K., S Bent, D. H. S t a t i s t i c a l Package f o r t h e S o c i a l S c i e n c e s ( 2 n d . e d . ) . New York: M c G r a w - H i l l , 1975-N u n n a l l y , J . C. P s y c h o m e t r i c Theory. New York: M c G r a w - H i l l , 1 9 6 7 -O o s t e r h o f , A. C. S i m i l a r i t y o f v a r i o u s item d i s c r i m i n a t i o n i n d i c e s . J o u r n a l o f E d u c a t i o n a l Measurement, 1 9 7 6 , J_3, 1 4 5 - 1 5 0 . O'Shea, T. J . An a p p l i c a t i o n o f the Rasch l o g i s t i c model t o the a s s e s s -ment o f change i n mathematics achievement. ( D o c t o r a l d i s s e r t a t i o n , U n i v e r s i t y o f B r i t i s h C o lumbia, 1 9 7 9 ) -Panchapakesan, N. The s i m p l e l o g i s t i c model and mental measurement.(Un-p l u b l i s h e d d o c t o r a l d i s s e r t a t i o n , U n i v e r s i t y o f C h i c a g o , 1 9 6 9 ) . Rasch, G. P r o b a b i l i s t i c Models f o r Some I n t e l l i g e n c e and A t t a i n m e n t T e s t s . Copenhagen: Danish I n s t i t u t e f o r E d u c a t i o n a l R e s e a r c h , i 9 6 0 . Rasch, G. An item a n a l y s i s w h i c h t a k e s i n d i v i d u a 1 d i f f e r e n c e s i n t o . a c -count. B r i t i s h J o u r n a l o f M a t h e m a t i c a l and S t a t i s t i c a l P s y c h o l o g y , 1 9 6 6 , _19 , 4 9 - 5 7 . -R e n t z , R. R. A Short G l o s s a r y o f Rasch Model T e r m i n o l o g y . U n p u b l i s h e d m a n u s c r i p t , 1 9 7 9 -R e n t z , R. R., & Bashaw, W. L. The n a t i o n a l r e f e r e n c e s c a l e f o r r e a d i n g : an a p p l i c a t i o n o f t h e Rasch model. J o u r n a l o f E d u c a t i o n Measure- ment, 1 9 7 7 , 14, 1 6 1 - 1 7 9 . R e n t z , R. R., & R e n t z , C. C. Does the Rasch Model R e a l l y Work? A D i s c u s s i o n f o r P r a c t i t i o n e r s (TM # 6 7 ) • P r i n c e t o n , N.J.: ERIC C l e a r i n g h o u s e on T e s t s , Measurement, and E v a l u a t i o n , E d u c a t i o n a l T e s t i n g S e r v i c e , 1 9 7 8 . - 106 -Rogers, W. T. B r i t i s h C olumbia l e a r n i n g assessment program. Stud i es  i n E d u c a t i o n a l E v a l u a t i o n , 1979, 5_, 3 9 - 5 3 -Rogers, W. T. P e r s o n a l c o m m u n i c a t i o n , August 12, 1981. Ryan, J . P., 6 Hamm, D. W. P r a c t i c a l p r o c e d u r e s f o r i n c r e a s i n g r e l i a b i l -i t y o f c l a s s r o o m t e s t s by u s i n g the Rasch model. Paper p r e s e n t e d at the annual meeting o f the N a t i o n a l C o u n c i l on Measurement i n Educa-t i o n , San F r a n c i s c o , 1976. (ERIC document R e p r o d u c t i o n S e r v i c e No. EDI 29901 ) . S t a n l e y , J . C. R e l i a b i l i t y , In R. L. T h o r n d i k e (Ed.) Educat i o n a l Measurement (2nd. e d . ) . Washington: American C o u n c i l on E d u c a t i o n , 1971. S t a n l e y , J . C , S H o p k i n s , K. D. E d u c a t i o n a l and P s y c h o l o g i c a l Meas- urement and E v a l u a t i o n ( 5 t h . ed. ). Eng 1 ewood C l i f f s , \T. J . : P r e n t i c e - H a l l , 1972. T i n s l e y , H. E. A., & Dawis, R. V. An i n v e s t i g a t i o n o f the Rasch s i m p l e l o g i s t i c model: sample f r e e i t e m and t e s t c a l i b r a t i o n . Educat i ona1 and P s y c h o l o g i c a l Measurement, 1975, 35_, 3 2 5 - 3 3 9 -W h i t e l y , S. E. Models, meanings and m i s u n d e r s t a n d i n g s : some i s s u e s i n a p p l y i n g Rasch's t h e o r y . J o u r n a l o f E d u c a t i o n a l Measurement, 1 9 7 7 , _1_4, 2 2 7 - 2 3 5 -W h i t e l y , S. E., & Dawis, R. V. The n a t u r e o f o b j e c t i v i t y w i t h the Rasch model. J o u r n a l o f E d u c a t i o n a l Measurement, 197**, 11, 163~ 1 78. W o l f , R. E v a l u a t i o n o f s e v e r a l f o r m u l a e f o r c o r r e c t i o n o f i t e m - t o t a l c o r r e l a t i o n s i n item a n a l y s i s . J o u r n a l o f E d u c a t i o n a l Measurement, 1967, 4., 2 1 - 2 6 . W r i g h t , B. D. S o l v i n g measurement problems w i t h the Rasch model. J o u r n a l o f E d u c a t i o n a l Measurement, 1977a, J_4, 9 7 " 1 l 6 . W r i g h t , B. D. M i s u n d e r s t a n d i n g the Rasch model. J o u r n a l o f E d u c a t i o n a l  Measurement, 1977b, T4, 219 -225 -W r i g h t , B. D., & Do u g l a s , G. A. Best p r o c e d u r e s f o r s a m p l e - f r e e item a n a l y s i s . A p p l i e d P s y c h o l o g i c a l Measurement, 1977a, J_, 281-295-W r i g h t , B. D., & Dougl a s , G. A. C o n d i t i o n a l v e r s u s u n c o n d i t i o n a l ' p r o -c e d u r e s f o r s a m p l e - f r e e item a n a l y s i s . E d u c a t i o n a l and P s y c h o l o g i c a l  Measurement, 1977b, 3_7, 4 7 - 6 0 . W r i g h t , B. D., S Mead, R. J . BICAL: C a l i b r a t i n g items and s c a l e s w i t h the Rasch model. Research Memorandum No. 23A. S t a t i s t i c a l Labor-a t o r y , Department o f E d u c a t i o n , U n i v e r s i t y o f C h i c a g o , 1978. W r i g h t , B. D., & Panchapakesan, N. A p r o c e d u r e f o r s a m p l e - f r e e i t e m a n a l y s i s . E d u c a t i o n a l and P s y c h o l o g i c a l Measurement, 1969, 29, 2 3 - 4 8 . - 107 -APPENDIX A TEST AND ADMINISTRATION MATERIALS - 109 -MODIFIED VERSION - P.L.A.P. SCIENCE ACHIEVEMENT PILOT TEST (1977) GRADE/YEAR FOUR ADMINISTRATION DIRECTIONS December 3-7, 1979 Thank you f o r your c o - o p e r a t i o n i n a d m i n i s t e r i n g t h i s Science P i l o t T e s t a t t h i s very busy time of year. The t e s t i s a m o d i f i e d v e r s i o n o f the Year/Grade 4 P.L.A.P. Science Achievement P i l o t T e s t which was a d m i n i s t e r e d i n November/December 1977. The purpose of t h i s t e s t i s NOT to measure i n d i v i d u a l , s c h o o l or d i s t r i c t achievement. Rather, item a n a l y s i s data r e s u l t i n g from t h i s t e s t i n g w i l l be used to b u i l d a s e r i e s of t e s t s u s i n g both the c o n v e n t i o n a l and Rasch approaches to t e s t b u i l d i n g . The u s e f u l n e s s and end products o f the two t e s t c o n s t r u c t i o n models w i l l then be compared. BEFORE STARTING THE TEST 1. You may wish to examine a copy o f the t e s t . I t c o n s i s t s o f two p a r t s : P a r t One - pages 1 to 16; P a r t Two - pages 17 - 34. I d e a l l y , the t e s t should be a d m i n i s t e r e d w i t h a r e s t p e r i o d between the two p a r t s . NOTE: If a vest period i s taken, i t i s essential that each student r e t a i n the same booklet for both -parts of the test. Some of the q u e s t i o n s asked may not seem to f i t every s c i e n c e program because s c i e n c e programs i n Grade/Year 4 as w e l l as primary s c i e n c e programs va r y . However, the t e s t i s a r e f l e c -t i o n of the o r i g i n a l (1977) t e s t c o n s t r u c t i o n committee's d e s i r e to t e s t f i v e b a s i c s c i e n c e s k i l l areas: Concepts, Processes, A p p l i c a t i o n , S a f e t y and S c i e n t i f i c L i t e r a c y . It i s essential that students attempt a l l questions. 2. Before s t a r t i n g the t e s t , p l e a s e w r i t e the f o l l o w i n g sample q u e s t i o n on the b l a c k b o a r d : How many months are t h e r e i n a year? 10 • 12 • 24 • 52 • I don't know L7J 3. Ensure t h a t a l l students have a sharpened p e n c i l or two, a c l e a n e r a s e r , and a centimeter r u l e r (needed f o r q u e s t i o n 37). You may wish to have a supply of sharpened p e n c i l s handy to take care of emergencies. 4. When the p u p i l s are s e t t l e d , d i s t r i b u t e the b o o k l e t s , t e l l i n g the p u p i l s not to open the b o o k l e t s u n t i l t o l d t o do so, and to p l a c e t h e i r r u l e r s a t the top of t h e i r desks, ready f o r use when needed. .:;v Students are to answer d i r e c t l y in the test booklets. over - 110 -STARTING THE TEST 1 . S t u d e n t s s h o u l d be i n f o r m e d t h a t t h e p u r p o s e o f t h i s t e s t i s t o p r o v i d e i n f o r m a t i o n w h i c h w i l l e n a b l e a r e s e a r c h e r t o b u i l d and compare d i f f e r e n t t y p e s o f s c i e n c e t e s t s . The s t u d e n t s can h e l p by w o r k i n g as c a r e f u l l y and q u i c k l y as t h e y c a n , and by a t t e m p t i n g a l l q u e s t i o n s . 2 . Have t h e s t u d e n t s open t h e i r b o o k l e t s t o t h e B a c k g r o u n d I n f o r m a t i o n s e c t i o n on page 1 . I t em 1 c a n be u sed t o e n s u r e t h a t s t u d e n t s o b t a i n t h e i r o r i g i n a l t e s t b o o k l e t a f t e r a r e s t p e r i o d . You may w i s h t o have them p u t t h e i r name o r c l a s s l i s t number i n t h e b o x . I f t h i s i s done i n p e n c i l , i t c an be e r a s e d a f t e r t h e t e s t i s c o m p l e t e d t o e n s u r e c o n f i d e n t -i a l i t y o f t h e s t u d e n t ' s i d e n t i f i c a t i o n . P l e a s e remember, i t i s i m p o r t a n t t h a t s t u d e n t s c o m p l e t e b o t h p a r t s o f t h e t e s t i n t h e same b o o k l e t . You may w i s h t o d e v i s e y o u r own method o f i d e n t i f i c a t i o n t o e n s u r e t h a t e a c h s t u d e n t r e c e i v e s h i s own t e s t b o o k l e t upon r e s u m p t i o n o f t h e t e s t a d m i n i s t r a t i o n a f t e r a r e s t p e r i o d . E n s u r e t h a t t h e s t u d e n t s have t i m e t o c o m p l e t e i t e m s 2 , 3 , 4 and 5 o f t h e B a c k g r o u n d I n f o r m a t i o n . T h i s i n f o r m a t i o n on age and gender may l e a d t o v a l u a b l e i n s i g h t s i n t h e i t e m a n a l y s i s t o be p e r f o r m e d . S t u d e n t s a r e t o p l a c e a check mark i n t h e box t o show t h e i r a n s w e r s . Be s u r e a n u m e r a l and n o t t h e name o f a day i s p r o v i d e d f o r q u e s t i o n 5 . 3 . Read t h e f o l l o w i n g i n s t r u c t i o n s t o t h e s t u d e n t s : FOR EACH QUESTION, SELECT THE ONE BEST ANSWER AND PUT. A CHECK MARK ( s/) IN THE BOX OPPOSITE YOUR ANSWER. I F YOU HAVE NO IDEA OF THE CORRECT ANSWER, PLACE YOUR MARK IN THE BOX THAT SHOWS " I DON'T KNOW. " Have t h e c l a s s r e a d t h e example you have w r i t t e n on .the b o a r d and have a s t u d e n t s u g g e s t t h e c o r r e c t a n s w e r . I l l u s t r a t e where t h e check mark s h o u l d g o . Remind t h e s t u d e n t s o f t h e ' I d o n ' t know 1 o p t i o n i f t h e y have no i d e a o f t h e c o r r e c t answer and t e l l them t o be s u r e t o mark o n l y one answer space f o r e v e r y q u e s t i o n . I f t h e y change an a n s w e r , t h e y s h o u l d e r a s e t h e i r o r i g i n a l check marks c a r e f u l l y . 4 . T e l l t h e s t u d e n t s t o b e g i n w i t h Q u e s t i o n 1 a t t h e b o t t o m o f page 1 . T e l l them t o s t o p a f t e r t h e y c o m p l e t e q u e s t i o n 4 0 on page 1 6 . I f t h e y f i n i s h e a r l i e r t h a n o t h e r s t u d e n t s , t h e y may go back and check t h e i r a n s w e r s , b u t t h e y a r e n o t t o go a h e a d . 5 . When t h e second s i t t i n g r e sumes , have p u p i l s t u r n i m m e d i a t e l y t o page 1 7 , Q u e s t i o n 4 1 and c o n d u c t P a r t Two i n t h e same manner as P a r t One . TIME ALLOCATION D i s t r i b u t i o n o f m a t e r i a l s ; c o m p l e t i o n o f b a c k g r o u n d i n f o r m a t i o n 5 t o 1 0 m i n u t e s P a r t 1 o f T e s t 3 0 t o 3 5 m i n u t e s ( a v e r a g e ) * P a r t 2 o f T e s t 3 0 t o 3 5 m i n u t e s ( a v e r a g e ) * * I f a t a l l p o s s i b l e p l e a s e e n s u r e t h a t a l l , o r a t l e a s t most ( 9 5 % ) o f t h e s t u d e n t s i n y o u r c l a s s have t i m e enough, t o c o m p l e t e a l l i t e m s . I f a s t u d e n t i s a b s e n t f o r p a r t o f t h e t e s t , p l e a s e a l l o w h i m / h e r t o make up t h a t p a r t , i f a t a l l p o s s i b l e . - 112 -- 113 -G R A D E / Y E A R 4 SCIENCE BACKGROUND INFORMATION !• Please put your name or class number i n the box. I I I i j 2. Are you a boy or a girl? Boy - • i Girl •. • , 3. How old are you today? 8 years of age ,- • 1 9 years of age • -j 10 years of age • 3 11 years of age • 4 more than 11 years of age.— • 5 4. When were you born? Month: January — • 01 F e b r u a r y — O 02 March D o s Apri l - • 04 May.— -- D o ; June - • OG J u l y — • 07 August • os September.. C 09 October.— • 10 November. D 11 December.. D 12 5. Day STOP! WAIT FOR YOUR TEACHER TO READ THE INSTRUCTIONS 1. In order to open a can of tomato juice, Sue punched TWO holes. Why do you think she did this? To l e t the juice pour out of the can more slowly O i To l e t a i r get into the can as the juice was poured Q 2 To l e t juice pour out of both holes • 3 To watch how the juice was pouring out . . . 0 4 I don 1 1 know • 5 -1-- 114 -A B C D 2 . Which container has a sign which means THIS CONTAINER M A Y EXPLODE? A • J B.- - - - O 2 C O 3 D. . _ — l o 4 I don't know - Q 5 M o s t o f t h e e a r t h i s c o v e r e d b y : F r e s h w a t e r S a l t w a t e r • £ Ice , • • -D3 S o i l I d o n ' t know . . . O 5 2 - 1 1 5 -4 . A boy and a girl both left school at the same time and arrived at their homes at the same time. They both went straight home. B o y ' s house S c h o o l Who moved more quickly? The girl moved more quickly ...... .D' The boy moved more quickly , L J i They both moved at the same speed. . D-3 I don't know. . . ["J^. Why does a compass needle p o i n t North? the E a r t h has onl y one moon D i E a r t h has a magnetic f i e l d D i E a r t h r e f l e c t s the Sun's l i g h t E a r t h ' s temperature i s not always the same.D* I don ' t know D 5 3 - 1 1 6 -A B c - : — 6 . The three solid objects shown above are the same size. If they float as shown, which one weighs the most? Object A - • i Object B • 2 Object C - • 3 They all weigh the same. Q 4 I don't know... Q 5 7 . Each year the Earth moves once around Mars • j Saturn. Q 2 the Sun - Q 3 the Moon —- Q 4 all of the other p l a n e t s . . . — Q 5 I don't know • e g Which would be easiest to measure with a metric ruler that is 30 cm long? The length of a pencil. - • 1 The thickness of a sheet of paper Q 2 The distance from your home to the school • 3 The distance from your home to the nearest grocery store.. O 4 I don't know - • 5 4 - 1 1 7 " All of the leaves in the first row are DICOTS ' " y c o r n p l a n t a i n b u l r u s h O N E of the leaves in the third row is a monocot d o g w o o d p o n d w e e d a s p e n g e r a n i u m A B C D 9 . Which leaf is a monocot? A • 1 B O 2 C O 3 D~~ - - - - • 4 I don't know - Q 5 5 - 118 -10 . When Tom threw his rubber ball into the air, it came back to the ground because the air pushed it back • i rubber always bounces back.._ • 2 the earth pulled it back • 3 the air is very light • 4 the earth is a large magnet • r, I don't know.... • c mountain E A and B ... • : A and C .— — • 2 B and C • 3 A, D, and E • 4 I don't know _ • 5 12 . The picture below shows a marble and a ruler. If the marble rolls from A to B in 2 seconds at a steady speed, how fast is it going? 24 cm 1 I I I I I I I I I I I 1 I 1 I I I I 1 I I I I I I I 1 I II 12 3 4 5 6 7 8 9 »6 II l i U H I S I U 7 l » l 9 J 0 i m 2 5 J 4 - 3 J M 1 7 M Z ' M I B 12 cm per 2 seconds — • 1 24 cm per second • 2 24 mm per second - • 3 240 ml_ per second.. • 4 12 cm per second - • 5 I don't know — - • 6 6 - 1 1 9 -• in many parts of British Columbia, a noon temperature of 2°C is most likely in January __ Q 1 May Q 2 J u n e - - • 3 Ju'y - -- — o 4 I don't know Q 3 The highest temperature recorded each day for a week is shown on the graph below. B 20°C-Ji CD M o n . Tues . W e d . Thurs . 14 . Which of the following statements is correct? Fri. Sat. Monday had the lowest temperature.... Q Highest temperature was recorded on Thursday • 0 It snowed on Tuesday : Q On Wednesday the temperature was higher than on Sunday.. • 4 I don't know ._ r-j . Some children were trying to find out which of three light bulbs was brightest. The children said the following things. Which of these gives the best start toward finding the answer? One bulb looks brightest to me so I already know the answer • A l l the bulbs look bright to me so there can't be an answer • It would help if we had a way to measure the brightness of a light bulb. • Let's take a vote; each person will vote for the bulb he thinks is the brightest Q I don't know Q 7 - 1 2 0 -16 . This is a graph of how tall 6 children are (in centimetres). Which child is shortest? CD E c CD 1 6 0 ' 150 -140-130-120. J I L L S U E K | M B O B . T E D J O E J i l l - • 1 Bob - • •> Sue • :i Ted • 4 Kim • 5 I don't know • c Look at the graph again. Which two children are the same height? Ted and Joe • i Sue and T e d — Q 2 J i l l and K i m — • 3 J i l l and Bob - - • 4 I don't know • s IS If you want to find out how much a person grew in one year, which of the following MUST you know about the person? His age • The type of food he e a t s — • His height at the start of the year.... • The height of his mother and father • I don't know • 19 . Which of the following have probably been on Earth the shortest time? Alligators.. - • 1 Dragonflies - D 2 Fish • a M e n — - - • 4 Snails .-— D 5 I don't know — Q G 8 - 1 2 2 -21 - it is most likely to snow when the weather is warm and clear.— - • 1 warm and foggy • 2 cold and clear • 3 warm and cloudy • 4 cold and cloudy — • •-, I don't know • c 22 . Alice has a flat tire on her bicycle. She pumps up the tire with an air pump and begins to ride. In a few minutes, the tire is flat again. To fix her tire, Alice must find a better air pump • whether the tire is made of rubber f j where the air leaks out of the tire • how many minutes it takes the tire to go flat • I don't know — • Look at the picture. 23 . Which choice tells all that you can see in the picture and no more? A hand is holding a match which is burning and giving off smoke. • 1 Someone has just lit a match [ J o A hand is holding a burning match • 3 Someone is about to be burned by the match he is holding.. • 4 I don't know • 5 24 . Three teaspoons of salt were mixed into a glass of water. How could you get back all the salt? Pour the water through a fine paper filter.. _ • Use a magnet on the liquid. • Boil the liquid until all the water is gone • Let the liquid stand overnight so that all of the salt goes to the bottom.... P i 4 I don't know • ,-, 3 10 - 1 2 3 " This is a picture of a room. Pretend that you are in this room and you want to measure its size, but you do not have a ruler. Which choice is NOT something you could use? Your foot lengths Floor tile lengths . '. U*-Chair lengths L J 3 Window lengths U H I don't know -Os-11 - 124 -Which of the following dissolves LEAST in water? Glass --- • i Salt - : - O 2 Soap - --~ • » Sugar - • 4 I don't know • .-, The caterpillar is an early stage in the life history of a f l y . - - - • i bee... - - • 2 grasshopper —- • a butterfly..— - • 4 mosquito - — - • -. I don't know - D c You could measure the distance from your home to the school in dozens — • i metres D 2 grams - - • 3 . litres - • 4 I don't know.. - • 5 30 . Germs sometimes make people sick because germs make poisons • 1 keep the blood from moving... • L. use up all of the water in the body. • ;; don't like people and want to hurt them M 4 I don't know - P I ,-, 12 - 1 2 5 -A class decided to show the temperature with the following shapes: Circle Diamond Triangle for over 20°C for 10° to 20°C for 5° to 10°C If the temperature at noon on Thursday is 7°C, which shape should be used? o I don't know. On each of three days the temperature at noon was: Monday Tuesday Wednesday 23°C 22°C 12°C Which of the following shows the temperatures at noon for the three days? Mon. Tues. Wed. o o A o o A o . o o 0 O A I don't know, - 126 -A doctor kept records of breathing rates of people when they were resting. He made the chart below. B R E A T H I N G R A T E S Persons Breaths in a minute B a b y boys 38 7-yr . -o ld g i r ls 25 7-yr . -o ld boys 25 10-yr . -o ld boys 20 Mothe rs 16 33 _ This chart suggests that boys breathe faster than girls • i girls breathe faster than boys Q 2 older people breathe faster than younger people • 3 younger people breathe faster than older people • 4 I don't know - D 5 The bones of dinosaurs that lived millions of years ago can be found in the ocean water • 1 ice on a pond • 2 trunks of trees • 3 rocks in the ground - • 4 I don't know •- • 3 35. Here is a picture of 4 objects. Which of the objects are the SAME? 1 and 4 • 1 2 and 3 - - - • 2 1, 2, and 4 - • 3 2 and 4 • 4 I don't know • 0 14 " 1 2 7 ~ Tom wanted to find out whether plants can grow better in the dark or in the light. He put a pot with 6 radish seeds in a dark room and a pot with 6 bean seeds on the window sill. A:;.'-dark' \ - radish, seeds"'/.,'< He added the same amount of water to both pots. The bean seeds grew better than the radish seeds, so Tom said his plants grow best in the light. 3 6 . To be able to say this, Tom should hare watered both pots more Q j watered the radish seeds more.r _ • 2 put the same kind of seeds in both pots • 3 grown the seeds in water instead of soil.... • 4 I don't know • 5 3 7 Use a r u l e r to measure the L O N G side of this rectangle. The length is 11 centimetres • 1 10 centimetres • 2 9 centimetres - • 3 8 centimetres.- • 4 I don't know • r. 3 8 • Which of the following is the SUREST sign that a dog is sick? Its fur is clean... r j , It sleeps at night _ _ • „ It is very playful _ f j 3 It has not eaten its food for two days • I don't know , _ _ __ r j 4 15 - 1 2 8 -The reason milk kept in a refrigerator does not go sour quickly is that the cold changes the water of the milk into ice — • separates the cream — D slows down the action of bacteria. • keeps flies away _ - - • causes a skin to form on the surface... ..- • I don't know.. • A litre of water is poured into each of the containers shown below and they are left uncovered in a warm room. After a day, which container will have the LEAST amount of water left in it? \ J A B C D E Container A. - - • Container B... ---- D Container C - - - - • Container D — • Container E — • I don't know.... • STOP WAIT FOR INSTRUCTIONS. 16 - 129 -SCIENCE ASSESSMENT —GRADE/YEAR 4 PART 2 A tiny rock is put into a measuring cup that has some water in it. Before the rock is put into the cup, the water is at the 50 mL line. After the rock is added, the water rises to the 60 mL line. 60 mL 50 m L . 40 m L . w .:. -.• -yj 60 mL _ mL 40 mL — Water 30 m L _ 20 mL _ 10 mL -Water 30 M|__ 20 m L -^ A ^ I O ) m l _ -B E F O R E A F T E R The space taken up by the rock is 10 mL.... • 50 mL • 60 mL • 1 10 mL • I don't know f j Seeds come from which one of the following parts of a plant? Flower - - • Leaf - • Root - • Stem • I don't know Q If all green plants died, what would be the most important effect on man? He would have to eat meat only • Sooner or later man would die of starvation- f j Man would get a little sick because he couldn't get vitamins—. • Man couldn't build houses because there would be no lumber.... [~J The land would be bare and not very pretty for man to look at.... • I don't know n 17 - 1 3 0 -Look at the imaginary tiny animals below. Wibbles look like those in the first row but none of those in the second row is a Wibble. 44 . Which one in the third row is NOT a Wibble? 4 5 . What is the MOST important thing that the lungs do? Hold the chest out_ ._ |_j x Protect against germs _ - • 2 Move different parts of the body .__ • 3 Pump the blood through the body.... - O 4 Provide a place for oxygen to enter the blood - • 0 I don't know _ — _ - • c 18 - 131 ~ 4 6 . What will happen to the balloon if it is left in the refrigerator? The balloon will crack _ • j The balloon will get bigger • 2 The balloon will stay the same..". • ;, The balloon will get smaller • 4 I don't know . f j 3 4 7 . About what topic do scientists know everything? Clouds • Stars • Trees. • No topic... • I don't know _. Q 4 8 . You could measure how heavy you are in kilograms - - • i litres • 2 kilometres —- - — • :s millilitres D •» I don't know - • o 19 49 . To which set of drawings does the egg belong? 1, 2, 9.... • , 3, 4, 5 - • 2 6, 7, 8 • 4, 5, 8 . - - . • 4 I don't know • a 20 ' 1 3 3 " At different times during a sunny day, a tree was seen to have cast z shadow of different lengths as shown in the diagram below. A B C D E Which diagram shows the shadow at midday? A • B - - - • C ..... - o D ...... - • E- - - - • I don't know.— Q Look at the diagrams below. To make the flashlight work, which way must we place the batteries? • • • I don't know.— . • - 1 3 4 -5 2 . Iron is most likely to rust when it is damp.. - - • i dry • 2 painted - - - • 3 covered with soap.. ~~ - D 4 covered with grease • 3 I don't know - - • c 5 3 • From the picture you can say that the scissors and the rock weigh about the same — • 1 the scissors weigh about two times as much as the rock.... Q o the rock weighs about two times as much as the scissors. • ;) the rock weighs about three times as much as the scissors • 4 I don't know. • 2 2 - 1 3 5 -54 . Which of the following COULD cause a rainbow? Fog and smog • j Rain and snow... • Clouds and i c e — • ;< Sunshine and rain _. • 4 I don't know _ ___ • s Look at Pictures 1 and 2 carefully. Picture 1 is a picture of two girls at the beginning of a foot race. Picture 2 is a picture of the girls as they cross the finish line. F in ish l ine 5 5 . Who ran more SLOWLY? The girl in the white T-shirt The girl in the dark T-shirt They were both travelling at the same speed I don't know . 23 • i • 2 • 3 • A - 1 3 6 -Someone said that if you mix salt and sugar with water and let the mixture stand, you get saltwater taffy—a kind "of candy. Which of the following would be the best way for you to test this idea? Take a vote among your friends • i Buy some saltwater taffy and see if it has salt in it _ .... Pj 2 Find out if salt and sugar have the same chemicals in them Pj a Grind up some saltwater taffy to see if you can get salt, sugar, and water... Pj 4 Try to mix salt, sugar, and water. Let them stand, and see what happens.. • 5 I don't know P r. Scientists would hare the most trouble testing which of the following? I have a fever Q 1 I am 2.1 metres t a l l — — • 2 My dog weighs 5 kilograms • 3 I can lift a 10 kilogram box • 4 My dog is better than your dog • 5 I don't know _ • c M I R R O R You hold a paper with the word POPSICLE facing the mirror, as shown in this picture. What does the word look like when you look into the mirror? q O q a i O J B - ° P O P S I C L E • BJoiaqoq --• ELCISPOP - -.• I don't know.-. -~ • 24 - 1 3 7 -Here are two pictures of a pot of water on a stove. Picture 2 was taken 5 minutes after picture 1. PICTURE 1 PICTURE 2 Which choice is the best way of telling that there has been a C H A N G E ? The water is boiling in picture 2 — • The gas is on in picture 2 —- - • The water gets hot when the gas is on • The water is not boiling in picture 1 - - Q The water is boiling in picture 2, but it is not boiling in picture 1 • I don't know - • It is a picture that Mary drew of a dog in a yard. Which of the following pictures shows an X in the same place in the yard as the dog in Mary's picture? 25 - 1 3 8 -51 • . If your teacher gave you a white powder and told you to try to find out what it is, which of the following must you NEVER do (unless told to by the teacher)? Put water on it • i Put iodine solution on it. • 2 Taste it... -— - • 3 Study it under a microscope. • 4 I don't know • 6 2 . Mary and Dave each bought the same kind of rubber ball. Mary said, "My ball bounces better than yours." Dave replied, "I'd like to see you prove that." What should Mary do? Drop both balls from the same height and see which bounces higher... f j j Throw both balls against a wall and see how far each ball bounces off the wall f j Drop the two balls from different heights and see which bounces higher • : ; Throw the balls down against the floor and see how high they bounce • 4 Feel the balls by hand to find which is harder Q 5 I don't know • o 6 3 . Which one of those shown below is the best container to measure the amount of rainfall in millimetres? 26 - 1 3 9 -How do we get scientific laws such as the law of gravity? Scientists vote on them . They are made by the police ..— They are passed by the government They come from many experiments I don't know _ Look at the pictures of imaginary pond water animals. Plaps look like all those in row 1. But N O N E of those in row 2 is a Plap. Which one in row 3 is a Plap? - 1 4 0 -o B a l l Metre st ick S i d e w a l k A ball is dropped on the sidewalk from a height of one metre as the picture above shows. n O o o o B D o 6 6 . Which of the above pictures shows how high the ball could bounce? Picture A.-.. B.... C-. D. ... E. .„ • i • 2 • a • 4 • 5 don't know H 28 - T 4 2 -6 8. After studying jars of pond water at your desk, it is important to wash your hands with water • x wash your hands and desk top with soap and water _ f j , spray the room with vinegar and water Q 3 stand under a hot light .. Q 4 I don't know Q 3 6 9 . While Joe was sitting under a tree, he watched a bird getting insects from between the cracks of the bark. Which drawing shows the kind of beak this bird likely had? C^*^ b - - - • * C • 3 D • 4 I don't know • r, 30 - 1 4 3 -A plant grows this way The first week it is 1 centimetre high. The second week it is 2 centimetres high. The third week it is 4 centimetres high. 70 . Which of the following graphs shows this growth? A cent imet res W e e k s B cent imet res W e e k s cent imet res A W e e k s W e e k s A;-— - - • i B • 2 C - :i D - • 4 I don't know... —. • o 31 - 1 4 4 -180 170 160 150 140 130 120 110 100 90 80 70 60 50 40 N u m b e r of Heartbeats per Minute W a l k i n g S w i m m i n g Wrest l ing R u n n i n g 71. Look at the graph abore. Your heart probably beats fastest when you are w a l k i n g - ... s w i m m i n g — w r e s t l i n g . . - . — r u n n i n g . . . ! I d o n ' t k n o w • i • 2 • 3 • 4 • .5 7 2 . When it is raining, which of the following M U S T be true? It is s p r i n g — - • i T h e S u n is s h i n i n g • 2 T h e w i n d is b l o w i n g 3 T h e r e a r e c l o u d s i n t h e s k y — • 4 I d o n ' t k n o w . . . — — - D 5 3 2 To find the temperature of the air outdoors, it would be best to put the thermometer at which of the places marked in the picture? A • , B D i c • * D. I don't know LZI5 Peter put some seeds on damp cotton in a dish. Jan put some seeds of the same kind into a glass full of water by the side of his. After two days. Peter's seeds sprouted but nothing seemed to happen to Jan's. o f h PETER 'S SEEDS JAN ' S SEEDS Which of the following is the most likely explanation? Jan 's seeds had been kept dry for too long . Jan did not allow her seeds enough air . . . Jan did not put the glass in a warm enough Jan should have used a different kind of seed Jan did not use any cotton I don't know D i Di place. •'• 'jjs 33 Look at these make-believe animals. Snorps look l i k e t h i s : But NONE of these i s a Snorp: ^\ 76. What do Snorps have i n common? i round bodies D' t a i l s -...Da-round bodies and t a i l s ...D 3 dots . CU t a i l s and dots .O5 I don't know Dfe 34 B l a n k i e a f - 148 -APPENDIX B CORRESPONDENCE ~ 1 4 9 -79-10-21 Dear S i r , I am a P e n t i c t o n i n t e r m e d i a t e teacher e n r o l l e d i n a M.A. program i n measurement and e v a l u a t i o n a t U.B.C. Completion of my program r e q u i r e s a r e s e a r c h p r o j e c t w i t h t h e s i s . The t o p i c I have chosen i n v o l v e s the c o n s t r u c t i o n and comparison of t e s t s r e s u l t i n g from two d i f f e r e n t approaches to item a n a l y s i s - the c o n v e n t i o n a l , o r t r a d i t i o n a l , procedure, and the Rasch procedure. The c o n v e n t i o n a l method o f t e s t c o n s t r u c t i o n i s w e l l researched but the Rasch model r e q u i r e s i n v e s t i g a t i o n as to i t s m e r i t s . My p r o j e c t i n v o l v e s a d m i n i s t r a t i o n o f a 76-item s c i e n c e p i l o t t e s t to approximately 500 grade four s t u d e n t s . In order to e x e r c i s e g r e a t e r c o n t r o l over the d i s t r i b u t i o n and c o l l e c t i o n of t e s t m a t e r i a l s , I wish to base my study i n the Okanagan V a l -l e y . I am r e q u e s t i n g your p e r m i s s i o n to c o n t a c t elementary sc h o o l p r i n c i p a l s i n your school d i s t r i c t by l e t t e r a s k i n g t h e i r a pproval to a d m i n i s t e r the s c i e n c e t e s t to the grade f o u r s t u -dents i n t h e i r s c h o o l s . The t e s t i n q u e s t i o n i s a m o d i f i e d v e r s i o n of the p i l o t t e s t s used i n 1977 to c o n s t r u c t the grade four s c i e n c e a c h i e v e -ment t e s t used i n the P r o v i n c i a l L e a r n i n g Assessment in; S p r i n g , 1978. The t e s t w i l l be group adm i n i s t e r e d w i t h students respond-i n g d i r e c t l y i n the t e s t b o o k l e t s . Only one a d m i n i s t r a t i o n of the t e s t i s r e q u i r e d w i t h a t e s t i n g time of approximately s i x t y minutes. I p l a n to have the t e s t s ready f o r a d m i n i s t r a t i o n i n l a t e November or e a r l y December t h i s year. The purpose of the p r o j e c t i s to compare two methods of t e s t c o n s t r u c t i o n and the products r e s u l t i n g from the two d i f f e r -ent approaches. I n d i v i d u a l , s c h o o l , and d i s t r i c t t e s t data w i l l not be computed. Consequently, s t r i c t e s t c o n f i d e n c e r e g a r d i n g these data i s assured. I would be very g r a t e f u l should you allow me to pursue t h i s e d u c a t i o n a l r e s e a r c h i n the elementary schools of your d i s t r i c t . I f your approval i s granted, I would l i k e t o r equest a l i s t of elementary schools i n your d i s t r i c t and t h e i r addresses. P l e a s e use the stamped, s e l f - a d d r e s s e d envelope f o r your r e p l y . I f you - 151 -79-11-08 Dear S i r or Madam, I am a P e n t i c t o n i n t e r m e d i a t e teacher e n r o l l e d i n a M.A. program i n measurement and e v a l u a t i o n a t U.B.C. Completion of my program r e q u i r e s a r e s e a r c h p r o j e c t w i t h t h e s i s . The t o p i c I have chosen i n v o l v e s c o n s t r u c t i o n and comparison of grade f o u r s c i e n c e t e s t s produced by a p p l y i n g c o n v e n t i o n a l and Rasch c r i t e -r i a of item a n a l y s i s . The nature o f t e s t s r e s u l t i n g from the a p p l i c a t i o n of the Rasch model and how they compare with conven-t i o n a l l y c o n s t r u c t e d t e s t s needs to be b e t t e r understood. I hope t h a t my r e s e a r c h w i l l c o n t r i b u t e t o t h i s understanding. For completion o f my p r o j e c t , I r e q u i r e a sample of approx-i m a t e l y 500 grade four students i n the Okanagan V a l l e y . I have r e c e i v e d p e r m i s s i o n from your Superintendent o f Schools to con-t a c t p r i n c i p a l s i n your d i s t r i c t a s k i n g f o r p e r m i s s i o n t o admin-i s t e r a 76-item grade f o u r s c i e n c e t e s t i n t h e i r s c h o o l s . I f you grant p e r m i s s i o n , and should your s c h o o l then be i n c l u d e d i n the sample of schools which w i l l be drawn f o r t e s t a d m i n i s t r a t i o n , t e s t b o o k l e t s w i l l be sent to your s c h o o l f o r group a d m i n i s t r a -t i o n d u r i n g the f i r s t week i n December. The t e s t to be administered i s a m o d i f i e d v e r s i o n o f the p i l o t t e s t s used i n 19 77 to c o n s t r u c t the grade f o u r s c i e n c e achievement t e s t used by the P r o v i n c i a l L e a r n i n g Assessment Pro-gram. The t e s t w i l l be group adm i n i s t e r e d t o c l a s s e s of grade fo u r students w i t h responses checked d i r e c t l y i n the t e s t book-l e t s . Only one a d m i n i s t r a t i o n o f the t e s t i s r e q u i r e d w i t h a t e s t i n g time of approximately s i x t y minutes, although slower workers may need a somewhat lon g e r time to f i n i s h . T e s t i n g can be s p l i t i n t o two s i t t i n g s , both on the same day. Tes t s w i l l be computer scored . The purpose of the r e s e a r c h i s to compare two methods o f t e s t c o n s t r u c t i o n , not t o measure d i f f e r e n c e s i n the c h i l d r e n t a k i n g the t e s t s . Consequently, the s t r i c t e s t c o n f i d e n t i a l i t y r e g a r d i n g i n d i v i d u a l and sc h o o l r e s u l t s i s assured. I w i l l be g r a t e f u l should you consent to grant me p e r m i s s i o n to i n c l u d e your s c h o o l i n the po o l from which I w i l l form my sample. T e s t i n g w i l l take p l a c e approximately the f i r s t week - 1 5 3 -APPENDIX C ITEM ANALYSIS DATA I t e m A n a l y s i s D a t a — C o n c e p t s S u b t e s t P a n c h a p a k e s a n R a n d o m M e a n S q . I t e m I t e m M e a n S q . I t e m I t e m I t e m p - v a l u e r * F i t D i s c . D i f f . F i t D i s c . D i f f . _ 1 .55 , 3 4 0 1 9 8 1 . 2 9 - 0 . 0 3 0 ^ 9 9 1 . 2 2 - 0 . 0 6 _ 3 ^ 5 5 , 3 8 C L 9 3 1 . 4 2 . - 0 . 0 9 0^92 1 . 3 9 - 0 . 0 3 _ 4 , 4 5 , 4 0 0 ^ 9 2 1 ^ 6 7 0_A0 0 ^ 9 3 1 ^ 4 4 0 . 3 9 _ 5 .30 , 3 6 0 L 9 6 1 . 3 3 1 . 1 0 0 ^ 8 8 1 ^ 3 9 1A6 _ 6 .88 , 2 8 1A0 0 . 9 6 - 2 . 2 5 O J 5 9 1 . 2 5 - 2 . 0 1 _ 7 , 6 1 . 3 1 LiOfj 0 . 9 4 - 0 . 3 9 0 , 9 7 1 . 1 2 - 0 . 3 3 JLO , 3 5 . 2 7 1 J U 1A5 0 . 9 1 1 . 0 1 0 J L % 0 . 8 7 12 .29 . 3 6 0^_96 1 ^ 3 6 1AA 0_ L 91 1 ^ 3 7 L J 7 _13 , 4 3 .25 1 ^ 0 8 ( L 9 3 0 ^ 5 7 L 0 7 0 ^ 9 3 0 . 5 1 _19 .39 , 2 9 1 J 0 7 ( h 8 2 0_M 1 . 0 1 1 . 0 4 0 . 6 8 2 0 ^ 4 6 . 2 8 L J J 4 L 0 3 0A1 1M K 0 1 0 . 3 6 2 1 . 8 5 . 2 2 C L 9 9 0 . 9 3 - 1 . 7 6 0 ^ 9 8 1 . 0 1 - 1 . 7 3 2 4 .17 .02 1A7 ( L 6 2 2A5 l _ i 3 4 0 ^ 3 7 1 J £ 2 7 A2 . 3 3 0 ^ 9 4 1 ^ 5 9 ( L 6 3 0 ^ 9 8 1 ^ 2 6 0 . 5 7 2 8 . 7 4 .29 1 ^ 0 2 0 . 7 7 - 1 . 0 5 L 0 1 0 . 9 8 - 1 . 0 1 ,30 . 3 4 , 1 8 1 J T J O J 5 4 L 0 2 1A1 0 ^ 4 0 . 9 6 3 4 .79 , 3 6 C L 8 8 1 . 1 7 - 1 . 5 0 C L 8 4 1 . 3 3 - 1 . 3 2 3 8 .83 , 3 6 0J9 1 . 3 9 - 1 . 7 6 0 J 3 0 1 . 3 8 - 1 . 6 1 4 0 . 5 5 . 2 1 1 . 0 6 0 . 9 3 0 . 0 7 1 . 0 9 0 . 8 5 - 0 . 0 3 I t e m A n a l y s i s D a t a — C o n c e p t s S u b t e s t P a n c h a p a k e s a n R a n d o m M e a n S q . I t e m I t e m M e a n S q . I t e m I t e m I t e m £ - v a l u e F i t D i s c . D i f f . F i t D i s c . D i f f . 4 2 , 4 8 , 3 0 0 _ L 9 5 3 ^ 4 5 C L 3 9 0 , 9 9 IAS 0 . 3 1 4 5 .60 .32. 0 ^ 8 1 . 1 8 - 0 . 2 5 C L 9 9 1 . 1 2 - 0 . 2 6 4 6 , 5 0 , 0 6 1A9 0_ L 29 0A1 L 2 7 0 ^ 2 8 0 . 2 1 5 1 , 5 0 .23 1 ^ 0 9 0 ^ 7 2 0 ^ 2 4 1M 0 ^ 9 6 0 . 2 2 5 2 J3 A2 ( L 9 8 1 . 1 4 - 1 . 2 8 ( L 8 9 1 . 4 2 - 1 . 0 1 5 3 A9 . 0 7 1A1 0_AI 1^_90 1 ^ 2 2 0A3 L 7 5 5 4 .86 . 3 3 0 ^ 1 . 3 8 - 1 . 9 8 C L 7 9 1 . 2 6 - 1 . 8 4 5 5 .69 , 2 4 0 J 9 5 1 . 1 9 - 0 . 5 9 U J Q 0 . 9 5 - 0 . 6 7 5 8 .36 AS 1 L 0 7 0J2 0 ^ 9 3 IAO O J 5 9 0 . 8 2 6 7 . 3 1 .16 1 ^ 0 7 C L 8 3 K 2 1 LI6 0 . 6 8 1 . 1 0 7 2 .16 . 3 1 0 ^ 9 7 0 . 9 5 - 1 . 2 2 0M 1 . 0 6 - 1 . 1 8 M e a n 1 . 0 0 IM 0 ^ 0 0 1 . 0 1 1 . 0 4 0 . 0 0 S t a n d a r d D e v i a t i o n 0 . 1 0 0 . 3 4 1 . 1 6 0 . 1 3 0 . 3 1 1 . 0 8 * I t e m / s u b t e s t p o i n t - b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t s c o r r e c t e d f o r s p u r i o u s n e s s u s i n g G u i l f o r d ' s ( 1 9 5 3 ) f o r m u l a . I t e m A n a l y s i s D a t a — P r o c e s s e s S u b t e s t P a n c h a p a k e s a n R a n d o m M e a n S q . I t e m I t e m M e a n S q . I t e m I t e m I t e m p _ - v a l u e r ^ * F v t D i s c . D i f f . F i t D i s c . D i f f . 8 . 6 5 . 4 0 1A2 0 . 1 0 C L 9 5 1A9 0 . 0 7 9 .60 . 2 4 1A8 OJl OAl L 2 3 O J 5 4 0 . 3 1 11 .92 .27 ( L 9 3 0 . 6 9 - 2 . 1 0 0 J _ 8 8 0 . 9 5 - 2 . 0 3 14 .82 . 3 4 ( L 8 8 1 . 1 2 - 0 . 9 8 ( L 8 8 1 . 0 8 - 0 . 9 9 1 6 , 9 5 JB 0 . 8 1 0 . 9 5 - 2 . 9 0 0 ^ 6 9 1 . 0 6 - 2 . 6 4 17 .97 . 3 1 0 . 4 7 1 . 3 1 - 3 . 9 1 0 ^ 4 1 1 . 1 9 - 3 . 4 5 1 8 . 6 4 . 3 7 0_,9S L 0 4 0_A3 (L96 L J J 7 0 . 1 2 2 2 . 8 5 .44 1 . 3 9 1 . 2 1 - 1 . 4 9 1A5 1 . 2 7 - 1 . 3 2 2 3 J3 . 3 0 L 0 5 0 . 8 5 - 0 . 3 0 L 0 9 0 . 8 4 - 0 . 3 8 2 6 . 3 2 . 3 2 C L 9 7 IAA L 8 1 IAI 1 ^ 0 2 1 ^ 7 3 2 9 . 6 9 . 2 9 1 ^ 0 9 0 . 8 7 - 0 . 1 1 IAO 0 . 9 0 - 0 . 1 6 3 1 . 6 9 .50 ( K 7 8 1 . 5 3 - 0 . 1 5 0J9 1 . 4 8 - 0 . 1 4 3 2 .36 . 3 2 C L 9 8 IA6 L 5 9 U ) 3 1A0 K 5 3 3 3 . 6 9 . 4 5 0 ^ 9 0 1 . 1 3 - 0 . 2 3 0J& 1 . 2 5 - 0 . 1 6 3 5 . 9 2 . 3 1 0J2 1 . 0 3 - 2 . 1 9 0J2 1 . 0 7 - 2 . 1 6 3 6 . 4 4 . 3 7 0 J _ 9 6 L J 3 1AZ C L 9 9 1A3 L 0 8 3 7 . 8 0 , 3 2 ( L 9 3 0 . 9 3 - 0 . 8 4 C L 9 2 1 . 0 0 - 0 . 8 4 4 1 . 6 7 . 4 3 0 . 9 0 1 . 1 9 - 0 . 0 3 0 ^ 8 9 1 . 2 3 - 0 . 0 4 I t e m A n a l y s i s D a t a — P r o c e s s e s S u b t e s t P a n c h a p a k e s a n R a n d o m M e a n S q . I t e m I t e m M e a n S q . I t e m I t e m I t e m p _ - v a l u e r ^ F i t D i s c . P i f f . F i t D i s c . D i f f . 4 4 . 6 1 . 2 4 \ A \ CL7_9 0 ^ 3 7 L 1 8 0_J>9 0 . 2 6 4 8 .57 . 4 0 ( L 9 5 L J J L C L 4 9 ( L 9 7 1 J L 2 0 . 4 4 4 9 . 9 0 . 2 5 LP_7 0 . 8 3 - 1 . 7 9 1 . 0 3 0 . - 9 3 - 1 . 7 9 5 6 . 4 9 . 4 5 0 ^ 9 2 K 2 2 0M ( L 9 1 L 2 9 0 . 8 4 5 9 . 3 8 . 3 4 L 0 2 1 ^ 0 4 1A2 L 0 5 U00 L J 8 6 0 . 7 3 .30 L 0 8 0 . 7 3 - 0 . 3 7 1 ^ 0 7 0 . 8 2 - 0 . 4 0 6 2 .60 . 2 9 K 0 9 ( h 8 3 0 ^ 3 7 IA2 C L 7 8 0 . 3 0 6 3 .21 .24 IA3 0 . 7 4 2 . 1 0 K 2 2 0J7 2_JD4 6 5 . 4 4 . 3 6 0^99 1 . 0 6 . 1 . 1 3 UO0 1A5 L J T J 7 0 A l ^ 3 6 l_iPJ y > 4 lAO K O I 1M2 0 . 9 8 _71 .89 . 3 4 0 ^ 9 1 . 0 7 - 1 . 7 9 0 . 9 3 1 . 0 8 - 1 . 7 4 7 3 .56 . 0 9 2 ^ 0 6 C L I O 3 L 8 5 2 J 5 3 C L J 3 3,69 7 4 . 3 9 . 4 1 C L 9 5 K 2 1 1 ^ 3 7 ( L 9 1 L 2 6 1 ^ 3 5 7 6 . 4 5 A3 0^95 1/2A lAH 0 J K 3 1 ^ 2 0 1 ^ 0 4 M e a n 1 . 0 1 U H ( h O O UQg L 0 2 0 . 0 0 S t a n d a r d D e v i a t i o n 0 . 2 5 0 . 2 5 1 . 5 9 0 . 3 2 0 . 2 5 1 . 5 0 * I t e m / s u b t e s t p o i n t - b i s e r i a l c o r r e l a t i o n c o e f f i c i e n t s c o r r e c t e d f o r s p u r i o u s n e s s u s i n g G u i l f o r d ' s ( 1 9 5 3 ) f o r m u l a . 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            data-media="{[{embed.selectedMedia}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.831.1-0095022/manifest

Comment

Related Items