UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

The Henmon-Nelson : computerized Harley, Dwight David 1979

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.

Item Metadata

Download

Media
831-UBC_1979_A8 H37.pdf [ 3.29MB ]
Metadata
JSON: 831-1.0094815.json
JSON-LD: 831-1.0094815-ld.json
RDF/XML (Pretty): 831-1.0094815-rdf.xml
RDF/JSON: 831-1.0094815-rdf.json
Turtle: 831-1.0094815-turtle.txt
N-Triples: 831-1.0094815-rdf-ntriples.txt
Original Record: 831-1.0094815-source.json
Full Text
831-1.0094815-fulltext.txt
Citation
831-1.0094815.ris

Full Text

THE HENMON-NELSON: COMPUTERIZED by DWIGHT DAVID HARLEY B.Ed., U n i v e r s i t y o f B r i t i s h Columbia,. ..1 THESIS SUBMITTED IH PARTIAL FULFILLMENT THE REQUIREMENTS FOB THE DEGREE OF . HASTES OF ARTS i n THE FACOLTI OF GRADUATE STUDIES (De p a r t m e n t o f E d u c a t i o n a l P s y c h o l o g y ) He a c c e p t t h i s t h e s i s a s c o n f o r m i n g t o t h e r e q u i r e d s t a n d a r d THE UNIVERSITY OF BRITISH COLOMBIA June, 1979 @ Dwight D a v i d H a r l e y , 1979 I n p r e s e n t i n g t h i s t h e s i s i n p a r t i a l f u l f i l l m e n t o f t h e r e q u i r e m e n t s f o r an a d v a n c e d d e g r e e a t t h e U n i v e r s i t y , o f B r i t i s h C o l u m b i a , I a g r e e t h a t t h e L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r r e f e r e n c e and s t u d y . I f u r t h e r a g r e e t h a t p e r m i s s i o n f o r e x t e n s i v e c o p y i n g o f t h i s t h e s i s f o r s c h o l a r l y p u r p o s e s may be g r a n t e d by t h e Head o f my D e p a r t m e n t o r by h i s r e p r e s e n t a t i v e s . I t i s u n d e r s t o o d t h a t c o p y i n g o r p u b l i c a t i o n o f t h i s t h e s i s f o r f i n a n c i a l g a i n s h a l l n o t be a l l o w e d w i t h o u t my w r i t t e n p e r m i s s i o n . D e p a r t m e n t o f E d u c a t i o n a l P s y c h o l o g y The U n i v e r s i t y o f B r i t i s h C o l u m b i a 2075 i f e s b r o o k P l a c e V a n c o u v e r , Canada V6T 1H5 J u n e , 1979-A b s t r a c t C o m p u t e r i z e d t e s t a d m i n i s t r a t i o n i s a r e c e n t a p p l i c a t i o n o f computer t e c h n o l o g y t o t h e a r t o f t e s t i n g t h a t i s j u s t now coming i n t o vogue. I t was t h e p u r p o s e o f t h i s s t u d y t o d e t e r m i n e what e f f e c t , i f any, c o m p u t e r i z e d a d m i n i s t r a t i o n h a d on t h e s c o r e s o b t a i n e d f r o m t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4. An i n t e r a c t i v e c o m p u t e r p r o g r a m was d e s i g n e d t o a d m i n i s t e r t h i s t e s t . C o m p u t e r i z a t i o n o f s t a n d a r d i z e d p s y c h o l o g i c a l t e s t s s u c h a s t h e S l o s s o n I n t e l l i g e n c e T e s t , t h e M i n n e s o t a M u l t i p h a s i c P e r s o n a l i t y I n v e n t o r y , and t h e C a l i f o r n i a A c h i e v e m e n t T e s t h a s d e m o n s t r a t e d t h e t e c h n i c a l f e a s i b i l i t y o f c o m p u t e r i z e d t e s t i n g , b ut l i t e r a t u r e s e a r c h e s have shown t h a t l i m i t e d r e s e a r c h h a s been done w i t h t h e s e p r o g r a m s . T h i s s t u d y f o l l o w e d t h e c l a s s i c a l , s i n g l e g r o u p , two t r i a l , r e p e a t e d measure d e s i g n . Each o f t h e 36 s u b j e c t s w r o t e one o f t h e two p a r a l l e l f o r m s o f t h e Henmon N e l s o n i n t h e c o n v e n t i o n a l manner and r e s p o n d e d t o t h e o t h e r f o r m t h r o u g h t h e c o m p u t e r . The o r d e r and medium by wh i c h e a c h s u b j e c t r e s p o n d e d t o t h e d i f f e r e n t f o r m s was r a n d o m i z e d t o c o n t r o l f o r s u c h f a c t o r s a s f a t i g u e and p r e t e s t s e n s i t i z a t i o n . D e p e n d e n t t - t e s t s t a t i s t i c s were used t o compare ( i ) t h e mean w r i t t e n v e r b a l s c o r e w i t h t h e mean c o m p u t e r i z e d v e r b a l s c o r e ; ( i i ) t h e mean w r i t t e n q u a n t i t a t i v e s c o r e w i t h t h e mean c o m p u t e r i z e d q u a n t i t a t i v e s c o r e ; and ( i i i ) t h e mean w r i t t e n t o t a l s c o r e w i t h t h e mean c o m p u t e r i z e d t o t a l s c o r e . A l l t h r e e t - t e s t s t a t i s t i c s were f o u n d t o be n o n s i g n i f i c a n t , i n d i c a t i n g t h a t t h e n u l l h y p o t h e s e s s h o u l d n o t be r e j e c t e d . T h e s e r e s u l t s s u g g e s t e d t h a t c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson can p r o v i d e r e s u l t s c o m p a r a b l e t o t h o s e o b t a i n e d f r o m a c o n v e n t i o n a l a d m i n i s t r a t i o n . R e s e a r c h S u p e r v i s o r : D r . H. R a t z l a f f i v TABLE OP CONTENTS C h a p t e r 1 - The P r o b l e m 1 C h a p t e r 2 - The L i t e r a t u r e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 C h a p t e r 3 - M e t h o d o l o g y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 C h a p t e r 4 - R e s u l t s 34 C h a p t e r 5 - D i s c u s s i o n 46 B i b l i o g r a p h y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - . . . . . . . . . . . 54 A p p e n d i x A - D e m o g r a p h i c Q u e s t i o n a i r e . . . . . . . . . . . . . . . . . . . . . 56 A p p e n d i x B - L i s t i n g O f Computer Program 58 V L I S T OF TABLES T a b l e I - S p l i t - h a l f B e l i a b i l i t i e s (Henmon-Nelson) ..... 19 T a b l e I I - C o e f f i c i e n t s Of E q u i v a l e n c e (Henmon-Nelson) - . 2 0 T a b l e I I I - S t a n d a r d E r r o r s O f Measurement (Henmon-Nelson) T a b l e IV - P o s s i b l e T e s t i n g A s s i g n m e n t s . . . . . . . . . . . . . . . . . 28 T a b l e V - F r e q u e n c y Of Gender . . . . . . . . . . . . . . . . . . . . . . . . . . 34 T a b l e VI - F r e g u e n c y Of Age .34 . T a b l e V I I - F r e g u e n c y Of Major ...........-.-........-.<.-. 35 T a b l e V I I I - F r e g u e n c y Of Y e a r Of E n r o l l m e n t . . . . . . . . . . . . . . 35 T a b l e IX - F r e g u e n c y Of P r i o r T e r m i n a l Use . . . . . . . . . . . . . . 35 T a b l e X - Means, S t a n d a r d D e v i a t i o n s And S t a n d a r d E r r o r s 36 T a b l e XI - P r o p o r t i o n a l Means . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 T a b l e X I I - P o s t Hoc Means And ( S t a n d a r d D e v i a t i o n s ) . . . . . 40 T a b l e X I I I - P o s t Hoc Summary .............................41 v i L I S T OF FIGURES F i g u r e 1 - T w o - t r i a l R e p e a t e d Measure D e s i g n 26 v i i Acknowledgement I w i s h t o e x t e n d my a p p r e c i a t i o n t o G a r r y Forwood f o r h i s t e c h n i c a l e x p e r t i s e i n t h e d e s i g n and i m p l e m e n t a t i o n o f t h e computer program u s e d f o r t h e a d m i n i s t r a t i o n o f t h e Henmon N e l s o n . I w o u l d l i k e t o e x p r e s s my g r a t i t u d e t o my r e s e a r c h s u p e r v i s o r Dr. H. fiatzlaff, and c o m m i t t e e members Dr. 0. G l d r i d g e and D r . fl. Westrom, f o r t h e i r h e l p and c o u n s e l . I w i s h t o thank my p a r e n t s D a v i d and J u n e H a r l e y , f o r a l l t h e i r h e l p , s u p p o r t , and e n c o u r a g e m e n t t h r o u g h o u t t h e c o u r s e o f my s t u d i e s . F i n a l l y t o my w i f e M a r g a r e t , my s i n c e r e t h a n k s f o r h e r u n d e r s t a n d i n g and encouragement w h i c h were n e c e s s a r y i n c o m p l e t i n g t h i s s t u d y . 1 CHAPTER 1 The P r o b l e m Computer t e c h n o l o g y was f i r s t i n t r o d u c e d t o t h e t e s t i n g s c e n e d u r i n g t h e e a r l y 1950's. Computers were u s e d p r i m a r i l y t o r e p l a c e c l e r k s , a n d t h e i r s c o r i n g m a c h i n e s , who c o u l d mark and r e c o r d o n l y a few h u n d r e d t e s t f o r m s i n a one h o u r p e r i o d . The c o m p u t e r s t h a t were a v a i l a b l e d u r i n g t h i s p e r i o d p r o d u c e d a name l i s t ( i d e n t i f i c a t i o n l i s t ) w i t h c o r r e s p o n d i n g s c o r e s and a few summary s t a t i s t i c s s u c h a s t h e mean, mode, and s t a n d a r d d e v i a t i o n . As t e c h n o l o g y a d v a n c e d , s o d i d t h e r o l e o f c o m p u t e r s i n t e s t i n g . C o m p u t e r s now p r o d u c e n o t o n l y a l i s t o f names and s c o r e s b u t c o m p r e h e n s i v e s t a t i s t i c a l d i s t r i b u t i o n s and a n a l y s e s d u r i n g t h e t i m e t h a t t h e answer s h e e t s a r e s c a n n e d and s c o r e d . W ith t h e use o f o p t i c a l s c a n n e r s f o r s e n s i n g t h e r e s p o n s e s h e e t s , and c o m p u t e r s f o r s c o r i n g and p r o c e s s i n g , t h e t i m e r e g u i r e d f o r t e s t a n a l y s i s has been d e c r e a s e d by w e l l o v e r 100 t i m e s . However, i n t h e t e s t i n g e n v i r o n m e n t c o m p u t e r s can p e r f o r m t a s k s o t h e r t h a n t h o s e r e l a t e d t o d a t a a n a l y s i s and r e t r i e v a l f r o m d a t a s u p p l i e d t o them by answer s h e e t s . More r e c e n t l y c o m p u t e r s a r e b e i n g used i n t h e p r e p a r a t i o n o f t h e t e s t i t s e l f . T e s t p r e p a r a t i o n c a n be s u p p o r t e d by c o m p u t e r s i n a v a r i e t y o f ways r a n g i n g f r o m i t e m b a n k i n g and i t e m g e n e r a t i o n t o i t e m s e l e c t i o n and t e s t p r i n t i n g . C o n s i d e r i n g t h e i n f o r m a t i o n 2 p r o c e s s i n g c a p a b i l i t i e s of p r e s e n t day c o m p u t e r s , CATC 1 i s a most n a t u r a l d e v e l o p m e n t i n t h e a r t o f t e s t i n g . ( L i p p e y , 1974) To d a t e c o m p u t e r s have been used i n t e s t p r e p a r a t i o n , s c o r i n g , and a n a l y s i s . E ach o f t h e s e t a s k s r e q u i r e s c l e r i c a l o p e r a t i o n s and s t r a i g h t f o r w a r d d e c i s i o n s . T h e s e t a s k s a r e a l g o r i t h m i c , and h ence machine e x e c u t a b l e . A n o t h e r t a s k w h i c h i s a l g o r i t h m i c i n n a t u r e a s w e l l a s a l o g i c a l e x t e n s i o n o f c o m p u t e r t e c h n o l o g y i n t o t h e r e a l m o f t e s t i n g i s c o m p u t e r i z e d -t e s t - a d m i n i s t r a t i o n . I n o r d e r t o r e d u c e t h e d i f f e r e n t i a l e f f e c t s o f t h e t e s t i n g e n v i r o n m e n t , e x a m i n e r ' s manuals go t o g r e a t l e n g t h s i n an a t t e m p t t o a c h i e v e u n i f o r m i t y a n d c o n t r o l o v e r t h e p r o c e d u r e s t o be f o l l o w e d i n t h e a d m i n i s t r a t i o n o f t e s t s . A b e t t e r way t o e n s u r e u n i f o r m i t y i s t o program t h e s e p r o c e d u r e s and h a v e t h e computer f o l l o w t h i s p r o c e s s i n t h e a d m i n i s t r a t i o n o f t h e t e s t . C o m p u t e r i z e d t e s t a d m i n i s t r a t i o n g r e a t l y r e d u c e s t h e e f f e c t o f t h e e n v i r o n m e n t on t h e t e s t i n g s i t u a t i o n . I t i s w i t h t h i s c o n c e p t o f c o m p u t e r i z e d t e s t a d m i n i s t r a t i o n t h a t t h i s s t u d y d e a l t . 1 . v The P r o b l e m C o m p u t e r i z e d t e s t a d m i n i s t r a t i o n i s a r e c e n t a p p l i c a t i o n o f computer t e c h n o l o g y t o t h e a r e a o f t e s t i n g t h a t i s j u s t now c oming i n t o vogue. I t h a s been u t i l i z e d , t o a l a r g e e x t e n t , i n 1 Computer A s s i s t e d T e s t C o n s t r u c t i o n 3 c o n j u n c t i o n w i t h c o m p u t e r i z e d a d a p t i v e t e s t i n g ( F e r g u s o n , 1969) . C o m p u t e r i z e d a d a p t i v e t e s t i n g (CAT) i s a f o r m o f t e s t i n g d u r i n g w h i c h t h e t e s t e e r e s p o n d s t o a b r a n c h i n g o r programmed t e s t i n t e r a c t i v e l y t h r o u g h t h e use o f a c o n v e r s a t i o n a l t e r m i n a l . R e s e a r c h b a s e d on t h e t h e o r e t i c a l a n d e m p i r i c a l a s p e c t s o f CAT h a s s o f a r been i n c o n c l u s i v e , b u t p s y c h o m e t r i c r e s e a r c h h a s p r o v e d p r o m i s i n g and h a s e n c o u r a g e d f u r t h e r work i n t h i s a r e a . However, most i f n o t a l l r e s e a r c h done w i t h CAT, as w e l l a s w i t h o t h e r a p p l i c a t i o n s o f c o m p u t e r i z e d t e s t a d m i n i s t r a t i o n , has a p p a r e n t l y g l o s s e d o v e r one n o t - s o - m i n o r d e t a i l - T h a t i s , d o e s t h e t e s t e e , u s i n g a c o m p u t e r t e r m i n a l t o r e s p o n d t o a t e s t , a c h i e v e t h e same s c o r e as he would r e s p o n d i n g t o t h e same t e s t i n t h e c o n v e n t i o n a l manner? T h i s was t h e g e n e r a l p r o b l e m t o which t h e ' s t u d y a d d r e s s e d i t s e l f . A p r o b l e m o f t h i s d i m e n s i o n i s c l e a r l y u n a n s w e r a b l e i n t h e c o n t e x t o f j u s t one s t u d y . C o n s i d e r i n g a l l t h e t y p e s o f t e s t s p r e s e n t l y i n use a s w e l l a s t h e number o f c o n v e n t i o n a l r e s p o n s e f o r m s , t h e r e s e a r c h e r i s c o m p e l l e d t o r e d u c e t h e p r o b l e m t o a managahle s i z e by s e l e c t i n g one t e s t and e x a m i n i n g t h e e f f e c t t h a t c o m p u t e r i z e d a d m i n i s t r a t i o n has on t h a t p a r t i c u l a r i n s t r u m e n t . I n k e e p i n g w i t h t h i s l i n e o f t h o u g h t , t h i s s t u d y a t t e m p t e d t o d e t e r m i n e what e f f e c t , i f any, c o m p u t e r i z e d a d m i n i s t r a t i o n had on t h e s c o r e s o b t a i n e d f r o m t h e Henmon-Helson T e s t o f M e n t a l A b i l i t y , l e v e l 4 ( N e l s o n , Lamke, and K e s l o , 1 9 6 1 ) . 4 2. D e f i n i t i o n Of Terms The f o l l o w i n g i s a c o l l e c t i o n o f key t e r m s a c c o m p a n i e d by d e f i n i t i o n s t h a t i n d i c a t e t h e p r e c i s e s e n s e i n w h i c h t h e s e t e r m s a r e used i n t h i s s t u d y . a) Computer a d m i n i s t r a t i o n ; p r e s e n t i n g t e s t s t i m u l i a n d a c c e p t i n g r e s p o n s e s t o t h e s e s t i m u l i f r o m t e s t e e s v i a c o n v e r s a t i o n a l t e r m i n a l s . b) C o n v e n t i o n a l ad mi n i s t r a t i o n : p r e s e n t i n g t e s t s t i m u l i and a c c e p t i n g r e s p o n s e s t o t h e s e s t i m u l i f r o m t e s t e e s i n t h e medium ( p e n c i l and p a p e r , o r a l , e t c . ) by w h i c h t h e t e s t was o r i g i n a l l y i n t e n d e d t o be a d m i n i s t e r e d . c) C o n v e r s a t i o n a l t e r m i n a l : a t y p e w r i t e r l i k e d e v i c e , s u c h a s t h e Snn A r b o r t e r m i n a l s which were u s e d i n t h i s s t u d y , w i t h which t h e u s e r c a n communicate i n t e r a c t i v e l y , i n t e r m s of r e a l t i m e , w i t h a t i m e s h a r i n g c o m p u t e r s y s t e m . In t h e c a s e o f t h e Ann A r b o r t e r m i n a l s , i n f o r m a t i o n was d i s p l a y e d t o t h e u s e r on a c a t h o d e - r a y - t u b e (CRT) s c r e e n . d) Page mode: a t e r m i n a l mode by w h i c h e n t i r e " s c r e e n s - f u l l " o f i n f o r m a t i o n a r e d i s p l a y e d on t h e t e r m i n a l s CRT s c r e e n . T h e s e " p a g e s " a r e c h a n g e d i n t h e i r e n t i r e t y , p a r a l l e l i n g t h e a c t i o n o f t u r n i n g a page i n a book. Pages may be r e d i s p l a y e d a s many t i m e s a s d e s i r e d i n t h e e x a c t f o r m a t a s t h e y were o r i g i n a l l y . e) S c o r e s : i ) q u a n t i t a t i v e : t h e number o f c o r r e c t r e s p o n s e s g i v e n on t h e g u a n t i t a t i v e s u b t e s t o f t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4-i i ) v e r b a l : t h e number o f c o r r e c t r e s p o n s e s g i v e n on t h e v e r b a l s u b t e s t o f t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , 5 l e v e l 4. i i i ) t o t a l : t h e s u a o f t h e q u a n t i t a t i v e and v e r b a l s c o r e s on t h e Henmon-Nelson T e s t o f d e n t a l A b i l i t y , l e v e l 4. f ) W r i t t e n f o r m : t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4, a d m i n i s t e r e d and t a k e n i n t h e c o n v e n t i o n a l , p a p e r and p e n c i l manner. 3. R e s e a r c h Q u e s t i o n s G i v e n t h a t t h e p r o b l e m o f t h e s t u d y was t o d e t e r m i n e whether o r n o t c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4, h a s any e f f e c t on t h e o b t a i n e d s c o r e s , t h r e e r e s e a r c h q u e s t i o n s emerged. 1. I s t h e r e a s i g n i f i c a n t d i f f e r e n c e between t h e a v e r a g e raw q u a n t i t a t i v e s c o r e o b t a i n e d f r o m a c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4 and t h e a v e r a g e raw q u a n t i t a t i v e s c o r e o b t a i n e d f r o m a c o n v e n t i o n a l a d m i n i s t r a t i o n o f t h e t e s t ? 2. I s t h e r e a s i g n i f i c a n t d i f f e r e n c e between t h e a v e r a g e raw v e r b a l s c o r e o b t a i n e d f r o m a c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4 and t h e a v e r a g e raw v e r b a l s c o r e o b t a i n e d f r o m a c o n v e n t i o n a l a d m i n i s t r a t i o n o f t h e t e s t ? 3. I s t h e r e a s i g n i f i c a n t d i f f e r e n c e between t h e 6 a v e r a g e r a n t o t a l s c o r e o b t a i n e d f r o m a c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson T e s t o f M e n t a l a b i l i t y , l e v e l 4 and t h e a v e r a g e raw t o t a l s c o r e o b t a i n e d f r o m a c o n v e n t i o n a l a d m i n i s t r a t i o n o f t h e t e s t ? P o s t hoc a n a l y s e s were c a r r i e d o u t w i t h r e s p e c t t o o t h e r v a r i a b l e s o f i n t e r e s t , s u c h a s s e x and age. a n a l y s e s o f t h i s n a t u r e t o o k t h e f o r m o f r e p e a t e d measure ANOVAs w i t h one g r o u p i n g f a c t o r . A f i v e p e r c e n t s i g n i f i c a n c e l e v e l was u s e d due t o t h e e x p l o r a t o r y n a t u r e o f t h i s s t u d y . I t was i n t e n d e d t o show t h a t c o m p a r a b l e r e s u l t s c a n be o b t a i n e d t h r o u g h c o m p u t e r a d m i n i s t r a t i o n . 4. D e l i m i n a t i o n o f t h e S t u d y The s t u d y was c a r r i e d o u t u s i n g s t u d e n t v o l u n t e e r s e n r o l l e d i n f i r s t y e a r u n i v e r s i t y t h r o u g h f i r s t y e a r g r a d u a t e s c h o o l a t t h e U n i v e r s i t y o f B r i t i s h C o l u m b i a . The s t u d y e m p l o y e d o n l y Ann A r b o r CET t e r m i n a l s w h i c h a r e s u p p o r t e d by a PDP-1t/40 m i n i - c o m p u t e r a t t a c h e d t o an Amdahl 470 V/6, M odel I I c o m p u t e r . The use o f o t h e r t y p e s o f t e r m i n a l s a v a i l a b l e a t t h e U n i v e r s i t y o f B r i t i s h C o l u m b i a ' s c o m p u t e r i n s t a l l a t i o n w o u l d have n e c e s s i t a t e d c o m p l e x I/O ( i n p u t / o u t p u t p r o c e d u r e s ) f o r m a t c h a n g e s i n t h e c omputer program w h i c h a d m i n i s t e r e d t h e t e s t . I n t h e c o n t e x t o f t h e g l o b a l p r o b l e m c o n c e r n i n g t e s t s a s a 7 whole, i t i s o n l y p o s s i b l e t o g e n e r a l i z e t h e f i n d i n g s o f t h e s t u d y t o o t h e r t e s t s i n s o f a r a s t h e y a r e s i m i l a r t o t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4. When c o m p a r i n g t e s t s t o t h e Henmon-Nelson f o r t h e p u r p o s e s o f e x t e n d i n g any g e n e r a l i z a t i o n s o f t h e f i n d i n g s , i t i s i m p o r t a n t t o n o t e t h a t t h e Henmon-Nelson c o n s i s t s t o t a l l y o f m u l t i p l e c h o i c e f o r m a t e d r e s p o n s e s t o v e r b a l and q u a n t i t a t i v e s t i m u l i . E x t e n d i n g any g e n e r a l i z a t i o n s t o i n c l u d e t r u e and f a l s e r e s p o n s e s s h o u l d n o t p r e s e n t a p r o b l e m a s t r u e and f a l s e i s a m o d i f i e d v e r s i o n o f t h e m u l t i p l e c h o i c e f o r m a t . However, m a t c h i n g r e s p o n s e f o r m a t s a r e e n t i r e l y d i f f e r e n t and s h o u l d become t h e s u b j e c t o f o t h e r r e s e a r c h . 5. J u s t i f i c a t i o n J u s t i f i c a t i o n f o r t h i s s t u d y was b e s t s e e n i n t h e l i g h t o f t h e g e n e r a l p r o b l e m . S o l u t i o n o f t h e im m e d i a t e p r o b l e m was o n l y o f m i n o r p r a c t i c a l i m p o r t a n c e i n i t s e l f . However, as a b a s i s from which t o e x t e n d f u t u r e r e s e a r c h i n t o t h e i n t e r f a c e between t e s t e e s a n d co m p u t e r a s s o c i a t e d d e v i c e s used f o r t e s t i n g , t h i s s t u d y was o f s i g n i f i c a n c e . When i t i s d e m o n s t r a t e d t h a t c o m p a r a b l e r e s u l t s c a n be o b t a i n e d f r o m c o m p u t e r a d m i n i s t e r e d t e s t s and t h e i r c o r r e s p o n d i n g w r i t t e n f o r m s , t h e e n t i r e t e s t i n g e n v i r o n m e n t c a n be r e v o l u t i o n i z e d i n t o a more e f f i c i e n t a nd p r a c t i c a l p r o c e s s . As d i s c u s s e d p r e v i o u s l y , c o m p u t e r a d m i n i s t e r e d t e s t i n g a l l o w s f o r t h e s t a n d a r d i z a t i o n of t h e a d m i n i s t r a t i v e p r o c e d u r e s . H a v i n g t e s t e e s r e s p o n d d i r e c t l y t o a co m p u t e r e l i m i n a t e s 8 c l e r i c a l e r r o r s i n t r a n s p o s i n g answer s h e e t s i n t o a m a c h i n e r e a d a b l e f o r m . A l s o u s i n g c o m p u t e r s t o s c o r e t h e t e s t s f u r t h e r r e d u c e s c l e r i c a l e r r o r s and a c h i e v e s g r e a t e r o b j e c t i v i t y i n s c o r i n g . The p r o b l e m o f t e s t s e c u r i t y i s r e d u c e d s i n c e t h e r e w i l l n o t be t e s t b o o k l e t s a n d answer s h e e t s t o l o s e o r m i s p l a c e . However t h i s p r o b l e m c a n n o t be e n t i r e l y e l i m i n a t e d s i n c e i t now becomes c o n f o u n d e d w i t h t h e new p r o b l e m o f c omputer s e c u r i t y . Computer a d m i n i s t e r e d t e s t i n g e l i m i n a t e s t h e e x a m i n e r e f f e c t a s w e l l a s p r o b l e m s i n v o l v e d w i t h t e s t m a t e r i a l s , p r i n t i n g , s t o r a g e , d i s t r i b u t i o n , and answer s h e e t p r o c e s s i n g . Computer a d m i n i s t e r e d t e s t i n g s h o u l d g r e a t l y i n c r e a s e t h e e f f i c i e n c y o f t h e t e s t i n g p r o c e s s . L o g i s t i c a l a d v a n t a g e s a r e n o t t h e s o l e o f f e r i n g s o f c o m p u t e r i z e d t e s t a d m i n i s t r a t i o n . Computers u s e d i n t h i s f a s h i o n e n a b l e us t o o b t a i n i n f o r m a t i o n t h a t would be i m p o s s i b l e o r i m p r a c t i c a l t o c o l l e c t u s i n g t h e c o n v e n t i o n a l modes o f r e s p o n s e ( e g . i t e m r e s p o n s e l a t e n c y ; t h e amount o f t i m e i n w h i c h i t t a k e s t h e t e s t e e t o answer e a c h i t e m ) . O t h e r e x a m p l e s i n c l u d e t h e number o f t i m e s p a g e s a r e v i e w e d , f r e q u e n c y o f c h a n g e d r e s p o n s e s p e r i t e m , a l i s t o f p r e v i o u s r e s p o n s e s , and s o on. T h e s e k i n d s o f i n f o r m a t i o n w i l l a l l o w f o r f u r t h e r e m p i r i c a l r e s e a r c h t o be done i n t h e a r e a o f r e s p o n s e s t y l e . C l e a r l y t h e r e i s a need f o r a u t o m a t i o n o f t h e t e s t i n g p r o c e s s . The f o l l o w i n g c h a p t e r e x a m i n e s t h e work done i n t h e a r e a o f c o m p u t e r i z e d t e s t i n g and p r o v i d e s a c r i t i q u e o f t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4. 9 CHAPTER 2 The L i t e r a t u r e I n t h e p a s t few y e a r s , c o m p u t e r p r o g r a m s h a v e been d e v e l o p e d which a l l o w f o r t h e a d m i n i s t r a t i o n o f s t a n d a r d i z e d p s y c h o l o g i c a l t e s t s . The p u r p o s e o f t h i s s t u d y was t o d e t e r m i n e whether p e r f o r m a n c e on t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4, was a f f e c t e d by t h i s new mode o f p r e s e n t a t i o n . I n o r d e r t o p r o v i d e a f o u n d a t i o n f o r t h i s s t u d y , t h e l i t e r a t u r e r e v i e w w i l l c o v e r t h e f o l l o w i n g a r e a s : (1) a u t o m a t e d t e s t i n g ; (2) c o m p u t e r i z e d t e s t i n g ; and (3) Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4. 1. Automated T e s t i n g A u tomated t e s t i n g i s t h e f o r e r u n n e r o f c o m p u t e r i z e d t e s t a d m i n i s t r a t i o n . I t e n j o y e d t h e same a d v a n t a g e s o v e r c o n v e n t i o n a l t e s t i n g t h a t c o m p u t e r i z e d a d m i n i s t r a t i o n now e n j o y s . T h e s e a d v a n t a g e s i n c l u d e g r e a t e r s t a n d a r d i z a t i o n o f t e s t a d m i n i s t r a t i o n p r o c e d u r e s , e l i m i n a t i o n o f c l e r i c a l e r r o r s e n c o u n t e r e d i n t h e r e c o r d i n g and s c o r i n g o f r e s p o n s e s , a s w e l l a s t h e c a p a b i l i t y o f g a t h e r i n g d a t a r e g a r d i n g r e s p o n s e s t y l e s t h a t would o t h e r w i s e be i m p r a c t i c a l t o c o l l e c t . One o f t h e e a r l i e s t d e m o n s t r a t i o n s o f t h e u s e f u l n e s s o f a t e c h n o l o g i c a l a p p r o a c h t o t e s t a d m i n i s t r a t i o n was made by E l w o o d (1969) who d e v e l o p e d a n a u t o m a t e d t e s t i n g b o o t h programmed t o a d m i n i s t e r t h e H e c h s l e r A d u l t I n t e l l i g e n c e S c a l e (HAIS). The d e v i c e a d m i n i s t e r e d a l l p e r f o r m a n c e and v e r b a l s u b t e s t s o f t h e 3AXS and r e c o r d e d e a c h r e s p o n s e made t o t h e i n d i v i d u a l t e s t 10 i t e m s . Two s u b t e s t s were a u t o m a t i c a l l y s c o r e d and r e s p o n s e s t y l e d a t a c o l l e c t e d . A u t omated t e s t i n g i s a m e c h a n i c a l a p p r o a c h t o t e s t a d m i n i s t r a t i o n . E l w o o d ' s bo o t h c o n s i s t e d o f a p o r t a b l e s o u n d p r o o f room c o n t a i n i n g a r e s p o n s e p a n e l and two c a s e s o f t w e l v e d r a w e r s . The d r a w e r s c o n t a i n e d t h e m a t e r i a l s f o r t h e v a r i o u s s u b t e s t s o f t h e WAIS. Ea c h d r a w e r was s p r i n g l o a d e d , s o l e n o i d o p e r a t e d , and r e m o t e l y c o n t r o l l e d by an e i g h t - t r a c k p a p e r t a p e r e a d e r and a BSE e l e c t r o n i c d i g i t a l l o g i c s y s t e m l o c a t e d o u t s i d e t h e b o o t h . The BBS s y s t e m r e c o r d e d t h e amount o f t i m e t h e s u b j e c t s p e n t on e a c h i t e m . S u b j e c t s were t a k e n i n t o t h e b o o t h by a r e s e a r c h a s s i s t a n t who e x p l a i n e d t h e o p e r a t i o n s o f t h e b o o t h . S u b j e c t s were t h e n g i v e n p r a c t i c e a t p r e s s i n g t h e "BEADY" b u t t o n w h i c h opened a d e m o n s t r a t i o n d r a w e r c o n t a i n i n g m a t e r i a l s w h i c h gave t h e s u b j e c t e x p e r i e n c e a t w o r k i n g w i t h t h e t e s t p i e c e s . , Shen t h e drawer was r e p l a c e d t h e r e s e a r c h a s s i s t a n t would l e a v e t h e b o o t h and t h e s u b j e c t would p r e s s t h e "BEADY" b u t t o n t o b e g i n t h e t e s t . R e s e a r c h a s s i s t a n t s s c o r e d t h e s u b j e c t s p e r f o r m a n c e by i n s p e c t i n g t h e c o n t e n t s o f t h e d r a w e r s a n d n o t i n g t h e t i m e p r i n t - o u t r e c e i v e d f r o m t h e t e l e t y p e w hich was a t t a c h e d t o t h e BRS l o g i c s y s t e m . S c o r i n g s t a n d a r d s d e s c r i b e d i n t h e WAIS manual were f o l l o w e d . I n v e s t i g a t i o n s i n t o t h e t e s t - r e t e s t r e l i a b i l i t y o f E l w o o d ' s a u t o m a t e d WAIS were c a r r i e d o ut by G r i f f i t h and Elwood ( c i t e d i n H e d l e t a l . , 1 9 6 9 ) . V e r b a l s u b s c o r e r e l i a b i l i t y was e s t i m a t e d a t r=0.99 w h i l e t h e r e l i a b i l i t y o f t h e p e r f o r m a n c e s c a l e was r=0.97. R e l i a b i l i t y o f t h e f u l l s c a l e was a p p r o x i m a t e l y r=Q. 98. 11 A c o m p a r i s o n o f t h e a u t o m a t e d WAIS t o t h e t r a d i t i o n a l WAIS was made by O r r ( c i t e d i n H i t t i e t a l . , 1 9 69). He f o u n d r e l i a b i l i t i e s o f r=0.94, r=0.82, a n d r=0.93 f o r t h e v e r b a l , p e r f o r m a n c e , and f u l l s c a l e s c o r e s r e s p e c t i v e l y . Such was t h e e x t e n t o f E l w o o d ' s a u t o m a t e d MAIS i n t h e e a r l y 1 9 7 0 » s . E x c e p t i o n a l r e s u l t s were o b t a i n e d i n t h e r e l a t e d s t u d i e s , b u t p r o b l e m s w h i c h r e d u c e d t h e d e v i c e s e f f i c i e n c y e x i s t e d . O n l y h a l f o f t h e t e s t i n g p r o c e s s was a u t o m a t e d . P s y c h o l o g i s t s would s t i l l have t o s c o r e t h e t e s t which was a c o s t l y p r o c e d u r e b o t h i n t e r m s o f t i m e and l a b o u r . C l e a r l y t h e n e x t g o a l f o r r e s e a r c h e r s i n a u t o m a t e d t e s t i n g s h o u l d be t h e c o m p l e t e and t o t a l a u t o m a t i o n o f t h e e n t i r e t e s t i n g p r o c e s s . The t i m e was now r i g h t f o r t h e u n i o n o f computer t e c h n o l o g y and t h e p r o c e d u r e s o f t e s t a d m i n i s t r a t i o n . 2. C o m p u t e r i z e d T e s t i n g One o f t h e f i r s t a t t e m p t s a t c o m p l e t e l y a u t o m a t i n g a s t a n d a r d i z e d p s y c h o l o g i c a l t e s t was t h e c o m p u t e r i z a t i o n o f t h e S l o s s o n I n t e l l i g e n c e T e s t ( S I T ) . H e d l , O ' K e i l , and Hansen (1971) d e s i g n e d a computer program t o a d m i n i s t e r t h e S I T t o s u b j e c t s v i a C B T - g r a p h i c t y p e t e r m i n a l s . The program r e c o r d e d a l l s t u d e n t r e s p o n s e s and r e s p o n s e l a t e n c i e s . I n t a k i n g t h e t e s t t h e s u b j e c t f i r s t r e a d t h e i t e m w h i c h was d i s p l a y e d on t h e CBT t e r m i n a l s and t h e n r e s p o n d e d by t y p i n g o n t h e t e r m i n a l * s k e y b o a r d . The co m p u t e r t h e n e v a l u a t e d t h e r e s p o n s e a s t o t a l l y c o r r e c t , t o t a l l y i n c o r r e c t , o r p a r t i a l l y c o r r e c t . I f t h e r e s p o n s e was p a r t i a l l y c o r r e c t t h e s u b j e c t was 12 t h e n prompted by t h e c o m p u t e r t o e l a b o r a t e . The s u b j e c t e l a b o r a t e d i n t h e same way by t y p i n g h i s r e s p o n s e i n t o t h e c o m p u t e r . The p r o c e d u r e f o l l o w e d by t h e c o m p u t e r i n t h e a d m i n i s t r a t i o n o f t h e S I T p a r a l l e l s t h a t o f t h e p r o c e d u r e s u g g e s t e d by t h e a u t h o r s o f t h e S I T . A l l s c o r i n g was done a u t o m a t i c a l l y by t h e c o m p u t e r . V e r b a l r e s p o n s e s t o t h e t e s t were s c o r e d t h r o u g h t h e u s e o f a l g o r i t h m s . E a c h r e s p o n s e was s c a n n e d f o r c o m b i n a t i o n s o f keywords w h i c h were matched t o l i s t s s t o r e d i n t h e machine. I f c e r t a i n c o m b i n a t i o n s o f words a p p e a r e d i n t h e r e s p o n s e , i t was s c o r e d a s c o r r e c t w h i l e o t h e r c o m b i n a t i o n s o f words c o n s t i t u t e d an i n c o r r e c t r e s p o n s e . Q u a n t i t a t i v e s c o r e s were matched a g a i n s t answer s e t s w h i c h were a l s o s t o r e d on d i s k i n t h e c o m p u t e r . I n t h e t e s t i n g o f t h e s c o r i n g mechanisms o f t h e program, 25 c o l l e g e s t u d e n t s w r o t e t h e c o m p u t e r b a s e d v e r s i o n o f t h e S I T . The c o m p u t e r c a l c u l a t e d a s c o r e and p r o d u c e d an e x a c t c o p y o f t h e d i a l o g u e c a r r i e d on between i t and t h e s u b j e c t . T h i s d i a l o g u e was t h e n g i v e n t o p s y c h o l o g i s t s t o e v a l u a t e and s c o r e . The l i s t o f 25 p a i r s o f s c o r e s (computer s c o r e and p s y c h o l o g i s t s c o r e ) was t h e n c o r r e l a t e d and a P e a r s o n r o f 0.83 was o b t a i n e d . T h i s would seem t o i n d i c a t e t h a t t h e s c o r i n g p r o c e d u r e s o f t h e SIT program were r e a s o n a b l y r e l i a b l e . The s t u d y went on t o d e t e r m i n e whether c o m p u t e r a d m i n i s t r a t i o n o f t h e S I T p r o v i d e d r e s u l t s e q u i v a l e n t t o r e s u l t s f r o m c o n v e n t i o n a l a d m i n i s t r a t i o n . An a t t e m p t was a l s o made t o compare t h e r e s u l t s o f t h e c o m p u t e r - b a s e d SIT t o t h e WAIS. O s i n g a 3x3 l a t i n s q u a r e ( c o u n t e r b a l a n c e d ) d e s i g n (n=48) 13 s u b j e c t s were t e s t e d w i t h t h e WAIS, t h e co m p u t e r v e r s i o n o f t h e S I T , and t h e c o n v e n t i o n a l S I T . T h e r e was a one week i n t e r v a l between e a c h o f t h e t h r e e t e s t i n g s , fiandom a s s i g n m e n t was u s e d t o c o n t r o l f o r e x a m i n e r e f f e c t s . The P e a r s o n product-moment c o r r e l a t i o n c o e f f i c i e n t between t h e s c o r e s o f t h e c o m p u t e r - b a s e d S I T and t h e s c o r e s o f t h e c o n v e n t i o n a l S I T was 0.75, s i g n i f i c a n t a t t h e .01 l e v e l . N o n - s i g n i f i c a n t P e a r s o n c o r r e l a t i o n s were f o u n d between t h e c o m p u t e r i z e d S I T a n d t h e WAIS (r=Q. 54) a s w e l l a s between t h e c o n v e n t i o n a l S I T and t h e c o n v e n t i o n a l WAIS ( r = 0 . 5 2 ) . I t s h o u l d be n o t e d t h a t P e a r s o n * s c o r r e l a t i o n between t h e au t o m a t e d WAIS and t h e c o n v e n t i o n a l WAIS (r=0.93) i s much g r e a t e r t h a n t h e c o r r e l a t i o n between t h e c o m p u t e r i z e d S I T and t h e c o n v e n t i o n a l S I T (r=0.75)- T h i s i s i n p a r t due t o t h e f a c t t h a t b o t h f o r m s o f t h e WAIS were hand s c o r e d w h i l e one f o r m o f th e S I T was s c o r e d by hand a n d t h e o t h e r f o r m s c o r e d b y co m p u t e r . By c o m p a r i s o n , t h e P e a r s o n r between hand s c o r i n g o f t h e c o m puter b a s e d S I T and co m p u t e r s c o r i n g o f t h e c o m p u t e r b a s e d S I T was 0.83. I t s h o u l d a l s o be n o t e d t h a t t h e t e s t -r e t e s t r e l i a b i l i t y c o e f f i c i e n t o f t h e WAIS was 0.96. Auto m a t e d t e s t i n g h a s n o t been l i m i t e d s o l e y t o i n d i v i d u a l i z e d t e s t s . G r o u p t e s t s have a l s o been c o m p u t e r i z e d . As o f 1973 Hoye and Wang (1973) r e p o r t t h a t c o m p u t e r i z e d v e r s i o n s o f t h e C a l i f o r n i a A c h i e v e m e n t T e s t , t h e M i n n e s o t a M u l t i p h a s i c P e r s o n a l i t y I n v e n t o r y (MMPI), and t h e Kuder V o c a t i o n a l I n t e r e s t s I n v e n t o r y h a v e b e e n d e v e l o p e d . L e k a n (1970) i n d i c a t e s t h a t t h e s e c o m p u t e r i z e d t e s t s were a v a i l a b l e 14 p r i o r t o 1970. However l i t e r a t u r e s e a r c h e s h a v e shown t h a t l i m i t e d r e s e a r c h has been done w i t h t h e s e p r o g r a m s . One p a p e r (L u s h e n e , O ' N e i l , a n d Dunn 1972) d i d s t a t e t h a t c o r r e l a t i o n s between t h e s c o r e s o f t h e c o m p u t e r i z e d MMPI and t h e s c o r e s o f t h e c o n v e n t i o n a l HMPI were a s h i g h a s t h o s e o b t a i n e d f r o m t e s t ^ r e t e s t o r p a r a l l e l f o r m s . E m p i r i c a l d a t a c o n f i r m i n g t h i s was n o t r e p o r t e d . More r e c e n t l y ( J B i t t i , S i f t e r , a n d S t u c k l e s s , 1972) t h e Ravens P r o g r e s s i v e M a t r i c i e s T e s t (RPMT), a t e s t o f n o n - v e r b a l g e n e r a l a p t i t u d e , was c o m p u t e r i z e d and t e s t e d on a sample o f 76 d e a f , p o s t s e c o n d a r y s t u d e n t s . One o f t h e o b j e c t i v e s o f t h e s t u d y was t o d e t e r m i n e i f p e r f o r m a n c e was a f f e c t e d by mode o f t e s t p r e s e n t a t i o n . S i n c e t h e r e i s a s e x b i a s on t h e RPMT a s t r a t i f i e d random p r o c e s s was u s e d t o a s s i g n t h e s t u d e n t s t o t h e two g r o u p s . One g r o u p was t e s t e d u n d e r c o m p u t e r i z e d c o n d i t i o n s w h i l e t h e s e c o n d g r o u p was t e s t e d i n t h e u s u a l f a s h i o n . The a n a l y s i s showed t h a t t h e r e was no s i g n i f i c a n t d i f f e r e n c e on p e r f o r m a n c e w i t h r e s p e c t t o t h e mode o f p r e s e n t a t i o n . However a s i g n i f i c a n t (<x=.05 ) s e x d i f f e r e n c e was f o u n d , b u t t h i s was c o n s i s t a n t w i t h t h e d a t a a l r e a d y a v a i l a b l e on t h e RPMT. k s i g n i f i c a n t i n t e r a c t i o n e f f e c t was n o t f o u n d , e v e n t h o u g h v i s u a l i n s p e c t i o n o f t h e means would s u g g e s t one. Th u s , s t u d i e s h a v e d e m o n s t r a t e d t h e f e a s i b i l i t y o f u s i n g c o m p u t e r s t o a d m i n i s t e r t e s t s . R e s u l t s s o f a r have been e n c o u r a g i n g b u t i t s t i l l r e m a i n s t o be c o n f i r m e d w h e t h e r computer a d m i n i s t e r e d t e s t s p r o v i d e c o m p a r a b l e r e s u l t s t o t h o s e o b t a i n e d u n d e r c o n v e n t i o n a l t e s t i n g -15 3. Henmon MeIson T e s t Of M e n t a l A b i l i t y The Henmon-Nelson T e s t o f M e n t a l A b i l i t y i s a t i m e d , g r o u p a d m i n i s t e r e d t e s t which i s d e s i g n e d t o "measure t h o s e a s p e c t s o f m e n t a l a b i l i t y w h ich a r e i m p o r t a n t f o r s u c c e s s i n a c a d e m i c work and i n s i m i l a r e n d e a v o r s o u t s i d e t h e c l a s s r o o m . " ( N e l s o n e t a l . , 1 9 6 1 ) F o u r l e v e l s o f t h e t e s t h a v e been d e v e l o p e d t o measure t h e r a n g e o f a b i l i t y f o u n d i n g r a d e s K t h r o u g h f i r s t y e a r g r a d u a t e s c h o o l . The i n t e r e s t o f t h e p r e s e n t s t u d y i s l i m i t e d t o t h e uppe r m o s t l e v e l which i s a p p r o p r i a t e f o r g r a d e s 13 t o g r a d u a t e s c h o o l . The t e s t p a c k a g e c o n s i s t s o f two t e s t f o r m s , A and B, a l o n g w i t h s e p a r a t e answer s h e e t s . A manual a p p r o p r i a t e f o r b o t h f o r m s i s a l s o s u p p l i e d . T h r e e k i n d s o f answer f o r m s c an be u s e d w i t h Henmon-Nelson: IBM 805 and IBM 1230 machine o r hand s c o r i n g answer s h e e t s and a s e l f - s c o r i n g answer s h e e t w h i c h e l i m i n a t e s t h e need f o r a s c o r i n g key. A d m i n i s t r a t i o n o f t h e t e s t i s s i m p l e . S p e c i a l t r a i n i n g i s n o t r e q u i r e d , however f a m i l i a r i z a t i o n w i t h t h e e x a m i n e r * s manual i s n e c e s s a r y t o o b t a i n more r e l i a b l e m e a s u r e s . . The a u t h o r s f e e l t h a t t h e i r t e s t , a s w e l l a s a l l o t h e r g r o u p m e n t a l a b i l i t y t e s t s , s h o u l d be c a t e g o r i z e d a s L e v e l B a s d e f i n e d by t h e A m e r i c a n P s y c h o l o g i c a l A s s o c i a t i o n ' s Code o f S t a n d a r d s f o r T e s t D i s t r i b u t i o n . T h i s would r e q u i r e t h e a d m i n i s t r a t o r s o f su c h t e s t s t o be w e l l v e r s e d i n e d u c a t i o n a l measurement, s t a t i s t i c s , g u i d a n c e , a s w e l l a s human l e a r n i n g and 16 d e v e l o p m e n t . However, t h e a u t h o r s do c o n c e d e t h a t r e a s o n a b l e r e s u l t s can be o b t a i n e d by a c o m p e t e n t i n s t r u c t o r who c a r e f u l l y f o l l o w s t h e i n s t r u c t i o n s p r o v i d e d i n t h e e x a m i n e r ' s manual. Haw s c o r e i s d e f i n e d a s t h e t o t a l number o f c o r r e c t r e s p o n s e s w i t h t h e u n d e r l y i n g s c a l e c o n t i n u o u s and i n t e r v a l i n n a t u r e . Saw s c o r e s f o r t h e q u a n t i t a t i v e and v e r b a l s u b s e t s a r e d e t e r m i n e d w i t h t h e t o t a l s c o r e s e e n a s t h e sum o f t h e two s u b t e s t s c o r e s . & t a b l e i s p r o v i d e d t o c o n v e r t e a c h o f t h e t h r e e raw s c o r e s t o p e r c e n t i l e r a n k s . T h e s e norms were g a t h e r e d on c o l l e g e f r e s h m e n who were t e s t e d e a r l y i n t h e s c h o o l y e a r . Norms f o r o t h e r s u b g r o u p s o f t h e t a r g e t p o p u l a t i o n h a v e n o t been p r o v i d e d . The a u t h o r s a t t e m p t t o j u s t i f y t h i s o m i s s i o n by c l a i m i n g t h a t l o c a l norms a r e more u s e f u l t h a n n a t i o n a l norms s i n c e t h e c a l i b e r o f s t u d e n t a t t r a c t e d t o s p e c i f i c i n s t i t u t i o n s v a r i e s . T h i s may be t r u e , but i t r e n d e r s t h e t e s t l e s s t h a n u s e f u l , u n l e s s t h e u s e r i s p r e p a r e d t o go t o t h e t r o u b l e o f e s t a b l i s h i n g h i s own norms. C a r e s h o u l d be t a k e n when i n t e r p r e t i n g p e r c e n t i l e r a n k s c o r e s . The u s e r s h o u l d be aware t h a t p e r c e n t i l e r a n k s a r e n o t e q u a l . They i n f l a t e raw s c o r e d i f f e r e n c e s n e a r t h e c e n t r e o f t h e d i s t r i b u t i o n and r e d u c e d i f f e r e n c e s t o w a r d the e x t r e m e s . I t must be n o t e d t h a t p e r c e n t i l e r a n k s s h o u l d be u s e d o n l y a s an o r d i n a l s c a l e r a t h e r t h a n a s an i n t e r v a l s c a l e . The m anual i n c l u d e s one b r i e f p a r a g r a p h r e g a r d i n g t h e i n t e r p r e t a t i o n o f t h e p e r c e n t i l e t a b l e s , b u t c a u t i o n s r e g a r d i n g o v e r ^ i n t e r p r e t a t i o n o f t h e s e s c o r e s a r e n o t made. I n c l u s i o n o f e q u i v a l e n t s t a n i n e s c o r e s would h e l p a l l e v i a t e t h e p r o b l e m s e n c o u n t e r e d when 17 w o r k i n g w i t h p e r c e n t i l e r a n k s . They would a l s o h e l p a v o i d a s i n g l e s c o r e f r o m b e i n g m i s - i n t e r p r e t e d . The e x a m i n e r ' s manual i s w e l l - d e s i g n e d w i t h p a r t i c u l a r a t t e n t i o n g i v e n t o t h e c l a r i t y o f i n s t r u c t i o n s a s w e l l a s t o t h e p o s s i b l e u s e s o f t h e Henmon-Nelson t e s t s . T h i s s h o u l d make t h e t e s t v e r y u s e f u l t o t h o s e u n f a m i l i a r w i t h s t a n d a r d i z e d t e s t i n g , once r e l i a b l e norms have been o b t a i n e d . The a c c o m p a n y i n g t e s t b o o k l e t s had a somewhat crowded f o r m a t . The t e s t was c o n s t r u c t e d f r o m a p o o l o f 400 i t e m s w h i c h h a d been e x t e n s i v e l y p i l o t t e s t e d . Host o f t h e i t e m s were t a k e n f r o m t h e Iowa T e s t o f M e n t a l a b i l i t y and t h e p r e v i o u s e d i t i o n o f t h e Henmon-Nelson. T h i s p o o l was a g a i n t e s t e d and t h e r e s u l t s a n a l y z e d w i t h r e s p e c t t o d i f f i c u l t y i n d e x , v a r i a n c e , and c o n t r i b u t i o n t o o v e r a l l r e l i a b i l i t y . By a n a l y z i n g t h e s e s t a t i s t i c s and t h e c o n t e n t o f e a c h i t e m , two p a r a l l e l f o r m s were c o n s t r u c t e d . a v a r i e t y o f d i f f e r e n t s t y l e s o f answer s h e e t s were u s e d d u r i n g t h e p i l o t t e s t i n g . E a c h t y p e o f r e s p o n s e f o r m was u n i f o r m l y d i s t r i b u t e d t h r o u g h o u t t h e p o p u l a t i o n . The manual d o e s n o t p r o v i d e any s t a t i s t i c s d e a l i n g w i t h t h e e f f e c t o f t h e d i f f e r e n t answer f o r m s on t h e i t e m o r t e s t r e s u l t s . Sex b i a s o f t h e r e s u l t i n g f o r m s was i n v e s t i g a t e d . S amples of 100 men and 100 women were r a n d o m l y drawn f r o m t h e n o r m i n g p o p u l a t i o n . A t t h e t e n p e r c e n t a l p h a l e v e l , t h e means and s t a n d a r d d e v i a t i o n s o f g r o u p s a c r o s s f o r m s d i d n o t s i g n i f i c a n t l y d i f f e r . 18 The n o r m a t i v e sample c o n s i s t e d o f 2004 A m e r i c a n c o l l e g e f r e s h m e n . 1002 s u b j e c t s wrote e a c h f o r m o f t h e Henmon-Nelson w i t h i n t h e f i r s t two months o f t h e i r f i r s t s e m e s t e r o f c o l l e g e a t t e n d a n c e . The s a m p l e was s t r a t i f i e d by l o c a t i o n , s i z e , and t y p e o f i n s t i t u t i o n s u c h t h a t i t r e f l e c t e d t h e g e n e r a l c o l l e g e p o p u l a t i o n a t t h a t t i m e ( F a l l , 1 9 5 8). As p r e v i o u s l y n o t e d , t h e t e s t was n o t normed f o r o t h e r s u b s e t s o f t h e t a r g e t p o p u l a t i o n a s t h e a u t h o r s f e l t l o c a l norms would be more u s e f u l t h a n n a t i o n a l norms. D a t a r e g a r d i n g t h e r e a d i n g l e v e l o f t h e c o l l e g e v e r s i o n o f t h e Henmon-Nelson i s n o t p r o v i d e d . A s u b j e c t i v e e x a m i n a t i o n o f t h e t e s t l e a d s one t o b e l i e v e t h a t i t i s h e a v i l y d e p e n d e n t upon r e a d i n g s k i l l s . The manual p r o v i d e s c o r r e l a t i o n a l d a t a between t h e Henmon-Nelson and t h e N e l s o n - D e n n y B e a d i n g T e s t . The c o r r e l a t i o n between t o t a l raw s c o r e s i s a p p r o x i m a t e l y 0.78 ( P e a r s o n r ) . T h i s e m p i r i c a l l y i n d i c a t e s t h a t t h e Henmon-Nelson i s d e p e n d e n t upon r e a d i n g s k i l l s . The t e s t i n g e n e r a l , i s h e a v i l y b i a s e d t o w a r d t h e v e r b a l f a c t o r o f m e n t a l a b i l i t y . A p p r o x i m a t e l y 60% o f t h e t o t a l number of i t e m s b e l o n g t o t h e v e r b a l s c a l e whereas o n l y 40X o f t h e i t e m s a r e q u a n t i t a t i v e i n n a t u r e . T h i s a f f e c t s t h e r e l i a b i l i t y o f t h e q u a n t i t a t i v e s c a l e and w e i g h t s the t e s t i n f a v o r o f v e r b a l a b i l i t y . A l s o t h e two t y p e s o f i t e m s a r e n o t u n i f o r m l y d i s t r i b u t e d t h r o u g h o u t t h e t e s t . T h i s o b v i o u s l y w i l l a f f e c t t h e s c o r e o f t h e s l o w e r s u b j e c t who d o e s n o t f i n i s h w i t h i n t h e a l l o t t e d t i m e . 19 The Henmon-Nelson, a s n o t e d p r e v i o u s l y , i s a s p e e d t e s t w i t h a f o r t y m i n u t e t i m e l i m i t . The e x a m i n e r ' s manual d o e s n o t r e p o r t any d a t a r e g a r d i n g t h e e f f e c t o f t i m e upon an i n d i v i d u a l ' s s c o r e . However t h e a u t h o r s a r e g u i c k t o c o n c e d e t h a t s p e e d w i l l a d v e r s e l y a f f e c t t h e s c o r e o f s u b j e c t s w i t h a s l o w r e s p o n s e s t y l e . Odd-even r e l i a b i l i t y c o e f f i c i e n t s a r e l i s t e d f o r b o t h f o r m s A and B. These v a l u e s a r e g i v e n i n T a b l e I . Q, V, and T o t a l r e p r e s e n t t h e q u a n t i t a t i v e s c a l e s r e s p e c t i v e l y , ^ The manual i s u n c l e a r a s t o w h e t h e r o r n o t t h e s e v a l u e s have been c o r r e c t e d by t h e Spearman-Brown P r o p h e c y f o r m u l a . C o m p u t a t i o n s were b a s e d upon a random sample o f 100 s u b j e c t s s e l e c t e d f r o m t h e n o r m a t i v e s a m p l e . T a b l e I - S p l i t - H a l f R e l i a b i l i t i e s (Henmon-Nelson) N=100 Form A Form B Q -921 -889 V -925 .934 T o t a l .950 .940 I t s h o u l d be n o t e d t h a t t h e s p l i t h a l f method o f e s t i m a t i n g r e l i a b i l i t y l e a d s t o i n f l a t e d v a l u e s when a p p l i e d t o s p e e d t e s t s . A l t e r n a t e f o r m s r e l i a b i l i t y e s t i m a t e s a r e g i v e n i n T a b l e I I . T h i s i s a more r e a l i s t i c a p p r o a c h t o o b t a i n i n g r e l i a b i l i t y e s t i m a t e s o f s p e e d t e s t s t h a n by u s i n g t h e s p l i t - h a l f method (Hagnusson, 1967). T e s t f o r m s A and B were a d m i n i s t e r e d t o t h e same s u b j e c t s w i t h a t i m e i n t e r v a l o f 35 d a y s between t e s t i n g s . 20 J u s t i f i c a t i o n i s n o t g i v e n f o r t h e r a t h e r l o n g i n t e r v a l between t h e s u c c e s s i v e t e s t i n g s . The r e t e s t i n g s h o u l d h a v e been c a r r i e d o u t w i t h i n a much s h o r t e r t i m e p e r i o d ( e g . l e s s t h a n 10 days) s i n c e l o n g e r p e r i o d s a l l o w f o r s o u r c e s o f i n v a l i d i t y s u c h a s c o n t e m p o r a r y h i s t o r y , m a t u r a t i o n and e x p e r i m e n t a l m o r t a l i t y t o weaken t h e i n t e r n a l v a l i d i t y o f t h e a l t e r n a t e f o r m s s t u d y . , T a b l e I I - C o e f f i c i e n t s o f E q u i v a l e n c e (Henmon-^ Nelson) Form h Form B N=52 Mean S- P. Mean s. p. r Q 19-58 6.00 23.46 6.08 .840 28.50 10.91 28.69 8.68 .876 T o t a l 48.08 14.09 52. 15 12.44 .887 From T a b l e I I i t can be s e e n t h a t t h e r e i s a f o u r p o i n t raw s c o r e i n c r e a s e between f o r m A and B w h i c h i s most l i k e l y due t o a p r a c t i c e e f f e c t on t h e q u a n t i t a t i v e s u b t e s t . T h i s e f f e c t c o u l d h a v e been c o n t r o l l e d i f a c o u n t e r b a l a n c e d d e s i g n would have been employed i n t h e t e s t i n g . C o e f f i c i e n t s o f r a t i o n a l e - e q u i v a l e n c e ( i n t e r n a l c o n s i s t e n c y ) r e l i a b i l i t y a r e n o t p r o v i d e d i n t h e manual. T h e s e n o n - c o r r e l a t i o n a l e s t i m a t e s o f r e l i a b i l i t y would h a v e g i v e n some i n d i c a t i o n o f t h e h o m o g e n i e t y o f t h e Henmon-Nelson a s a whole. S t a n d a r d e r r o r s o f measurement a r e g i v e n i n T a b l e I I I . The manual p r o v i d e s an e x c e l l e n t s e c t i o n on t h e i n t e r p r e t a t i o n o f t h i s s t a t i s t i c . 21 T a b l e I I I - S t a n d a r d E r r o r s o f Measurement (Henmon-Nelson) N=100 Form A Form B Q 1.98 2.40 V 3.05 2.86 T o t a l 3.64 3.57 T i e a u t h o r s o f t h e Henmon-Nelson have a t t e m p t e d t o e s t a b l i s h c r i t e r i o n - r e l a t e d v a l i d i t y by c o r r e l a t i n g i t w i t h o t h e r m e asures o f m e n t a l a b i l i t y , s u c h a s g r a d e p o i n t a v e r a g e (GPA), t h e N e l s o n - D e n n y B e a d i n g T e s t and o t h e r l e v e l s o f t h e Henmon-Nelson t e s t s . C o r r e l a t i o n s w i t h GPA r a n g e from 0.417 ( q u a n t i t a t i v e s u b s c o r e ) t o 0.783 ( t o t a l s c o r e ) . . The c o r r e l a t i o n s between o t h e r l e v e l s o f t h e Henmon-Nelson and i t s c o l l e g e l e v e l r a n g e f r o m 0.696 t o 0.791. I t i s i n t e r e s t i n g t o n o t e t h a t i n a l l c a s e s c o r r e l a t i o n s between t h e q u a n t i t a t i v e s c o r e and t h e c r i t e r i o n s c o r e were l o w e s t w h i l e c o r r e l a t i o n s between t h e t o t a l s c o r e and t h e c r i t e r i o n were a l w a y s t h e h i g h e s t . A t t e m p t s t o e s t a b l i s h c o n s t r u c t v a l i d i t y were n o t r e p o r t e d i n t h e manual. The Henmon-Nelson T e s t o f M e n t a l A b i l i t y i s p r a c t i c a l i n f o r m a t and e a s y t o a d m i n i s t e r , s c o r e , and i n t e r p r e t . The t e s t p o s s e s s e s good r e l i a b i l i t y and r e a s o n a b l e v a l i d i t y . T h i s makes t h e t e s t u s e f u l t o c l a s s r o o m t e a c h e r s a n d c o u n s e l o r s who make d e c i s i o n s r e g a r d i n g a c a d e m i c c o n s i d e r a t i o n s f o r a v a r i e t y o f p u p i l s . The t e c h n i c a l f e a s i b i l i t y o f c o m p u t e r i z e d t e s t i n g h a s been d e m o n s t r a t e d , however i t r e m a i n s t o be shown t h a t c o m p a r a b l e 22 r e s u l t s c a n be o b t a i n e d f r o m computer b a s e d t e s t i n g . The f o l l o w i n g c h a p t e r d e s c r i b e s t h e m e t h o d o l o g y i n v o l v e d i n d e t e r m i n i n g t h e e f f e c t o f c o m p u t e r i z a t i o n upon t h e Henmon-N e l s o n . 23 CHAPTER 3 M e t h o d o l o g y The p u r p o s e o f t h i s s t u d y was t o a s s e s s t h e e f f e c t o f co m p u t e r a d m i n i s t r a t i o n on t h e s c o r e s r e s u l t i n g f r o m a m u l t i p l e -c h o i c e t e s t t h a t was a d m i n i s t e r e d i n t e r a c t i v e l y v i a c o n v e r s a t i o n a l , CRT t e r m i n a l s . M e t h o d o l o g i c a l l y , t h e s t u d y was s t r a i g h t f o r w a r d . I n b r i e f , e a c h r e s p o n d e n t c o m p l e t e d two a l t e r n a t e f o r m s o f a m u l t i p l e - c h o i c e f o r m a t t e d , s t a n d a r d i z e d t e s t . One t e s t i n g was done v i a computer w h i l e t h e o t h e r was c a r r i e d o u t i n t h e c o n v e n t i o n a l manner. The r e s u l t i n g two s e t s o f s c o r e s were a n a l y z e d t o d e t e r m i n e t h e e f f e c t o f c o m p u t e r i z e d a d m i n i s t r a t i o n on a s t a n d a r d i z e d t e s t . 1. M e a s u r i n g I n s t r u m e n t s  Henmon-Nelson The t e s t c h o s e n t o be u s e d i n t h e s t u d y was t h e Henmon-N e l s o n T e s t o f M e n t a l A b i l i t y , l e v e l 4. I t was s e l e c t e d on t h e b a s i s o f t h r e e c r i t e r i a . F i r s t ; t h e t e s t u s e d must have p a r a l l e l f o r m s w i t h a d e q u a t e p a r a l l e l f o r m s r e l i a b i l i t y . As t a b l e d i n a p r e v i o u s c h a p t e r , t h e Henmon-Nelson h a s a n o v e r a l l r e l i a b i l i t y o f a p p r o x i m a t e l y 0.86 {Pearson r ) . T h i s l e v e l was c o n s i d e r e d a c c e p t a b l e f o r t h e p u r p o s e s o f t h i s s t u d y . S e c o n d l y , t h e c h o s e n t e s t must be t r u e / f a l s e , m u l t i p l e -c h o i c e , o r m a t c h i n g i n f o r m a t . T h i s was r e q u i r e d i n o r d e r t o f a c i l i t a t e t h e use o f answer k e y s i n machine s c o r i n g . The Henmon-Nelson c o n s i s t s e x c l u s i v e l y o f m u l t i p l e - c h o i c e t y p e 24 i t e m s . F i n a l l y , t h e t e s t mast c o n t a i n o n l y v e r b a l o r n u m e r i c a l s t i m u l i and r e s p o n s e s s o a s t o m i n i m i z e t h e c o m p l e x i t y o f t h e com p u t e r program w h i c h was used t o a d m i n i s t e r t h e t e s t . G r a p h i c a l s t i m u l i would n e c e s s i t a t e t h e use o f t e r m i n a l s w i t h p o w e r f u l d i s p l a y c a p a b i l i t i e s . A v a i l a b i l i t y o f t h e s e t y p e s o f t e r m i n a l s a t t h e U n i v e r s i t y o f B r i t i s h C o l u m b i a ' s c o m p u t e r i n s t a l l a t i o n i s l i m i t e d a n d v e r y e x p e n s i v e . V a l i d i t y was n o t used a s a c r i t e r i o n i n t h e s e l e c t i o n o f t h e t e s t . O b t a i n e d s c o r e s were n o t i n t e r p r e t e d . , The s o l e c o n c e r n o f t h e p r o j e c t was t h e c o n s i s t e n c y o f measure a c r o s s mode o f p r e s e n t a t i o n . D e m o g r a p h i c Q u e s t i o n n a i r e A d e m o g r a p h i c g u e s t i o n n a i r e was a d m i n i s t e r e d t o a l l r e s p o n d e n t s b e f o r e t h e t e s t i n g p r o c e d u r e s began. A copy o f t h i s g u e s t i o n n a i r e c a n be f o u n d i n A p p e n d i x A. T h i s i n f o r m a t i o n was s t o r e d i n t h e c o m p u t e r , and u s e d as r e q u i r e d f o r a n a l y s i s . C omputer Progra m V i t a l t o t h i s s t u d y was a co m p u t e r program t h a t was d e v e l o p e d t o a d m i n i s t e r t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y . I t i s a program w r i t t e n i n PL/1 d e s i g n e d t o be u s e d on t h e U n i v e r s i t y o f B r i t i s h C o l u m b i a ' s Amdahl 470 V/6, H o d e l I I computer w h i c h i s u n d e r c o n t r o l o f a M i c h i g a n T e r m i n a l System (MTS) o p e r a t i n g s y s t e m . The p r o g r a m i s i n t e n d e d t o be a c c e s s e d t h r o u g h t h e u s e o f t h e Ann A r b o r c o n v e r s a t i o n a l t e r m i n a l s . A c o m p l e t e l i s t i n g o f t h e program c a n be f o u n d i n Ap p e n d i x B. Hhen t h e s u b j e c t f i r s t s a t down a t t h e t e r m i n a l t h e s u b j e c t 25 i d e n t i f i c a t i o n number was e n t e r e d . The program t h e n d i s p l a y e d a s e r i e s o f s i m p l e d i r e c t i o n s r e g a r d i n g t h e use o f t h e t e r m i n a l . The s u b j e c t was i n s t r u c t e d by t h e c o m p u t e r on how t o e n t e r r e s p o n s e s , p r o c e e d t o t h e n e x t page, r e t u r n t o a p r e v i o u s page, and how t o change a n s w e r s . The p r o g ram t h e n d i s p l a y e d t h e i n s t r u c t i o n s f o r t a k i n g t h e t e s t i n much t h e same manner a s t h e y a p p e a r i n t h e e x a m i n e r ' s manual., The t h r e e s a m p l e i t e m s t h a t were u s e d w i t h t h e w r i t t e n f o r m were p r e s e n t e d t o g i v e t h e s u b j e c t an o p p o r t u n i t y t o " t r y o u t " t h e t e r m i n a l . The c o m p u t e r t h e n g i v e s t h e s u b j e c t t h e o p t i o n o f r e v i e w i n g t h e i n s t r u c t i o n s o r b e g i n n i n g t h e t e s t . Sfhen t h e r e s p o n d e n t d e c i d e d he was r e a d y t o b e g i n t h e t e s t and i n d i c a t e d t h i s t o t h e c o m p u t e r , t h e program s e l e c t e d t h e a p p r o p r i a t e f o r m f o r a d m i n i s t r a t i o n on t h e b a s i s o f t h e s u b j e c t ' s i d e n t i f i c a t i o n number-I n d e s i g n i n g t h e p r o g r a m , e v e r y a t t e m p t was made t o d u p l i c a t e t h e p r o c e d u r e s used when t a k i n g t h e t e s t i n t h e c o n v e n t i o n a l manner. I t e m s were p r e s e n t e d by t h e page ( t e r m i n a l s a r e s e t i n page mode) i n a c r owded f o r m a t , p a r a l l e l i n g t h a t o f t h e t e s t b o o k l e t . I t e m s c o u l d be a n s w e r e d i n any o r d e r , r e s p o n s e s c o u l d be c h a n g e d any number o f t i m e s and p ages c o u l d be ' t u r n e d * a t w i l l . E m p h a s i s was p l a c e d upon k e e p i n g t h e t e r m i n a l s * o p e r a t i n g i n s t r u c t i o n s t o a minimum. The program h a s an a u t o m a t i c f e a t u r e which k e p t t r a c k o f e l a p s e d t i m e f r o m t h e moment t h e f i r s t page was p r e s e n t e d u n t i l t h e 40 m i n u t e t i m e l i m i t h a d e x p i r e d . Time r e m a i n i n g was d i s p l a y e d i n t h e u p p e r r i g h t hand c o r n e r o f t h e t e r m i n a l ' s s c r e e n and c h a n g e d e a c h t i m e a page was f l i p p e d . 26 The program h a s s e v e r a l f e a t u r e s u s e f u l i n d a t a c o l l e c t i o n . A r e c o r d o f t o t a l t i m e s p e n t on e a c h page was k e p t a l o n g w i t h number o f t i m e s e a c h page was d i s p l a y e d . R e s p o n s e s f o r a l l i t e m s were s t o r e d and t h e number o f r e s p o n s e c h a n g e s p e r i t e m was r e c o r d e d . The p r o g r a m s c o r e s t h e t e s t , r e c o r d s t h e t h r e e s c o r e s , and s t o r e s a l l c o l l e c t e d d a t a i n a f i l e f o r f u r t h e r a n a l y s i s . , D e s i g n The s t u d y f o l l o w e d t h e c l a s s i c a l , s i n g l e g r o u p , two t r i a l , r e p e a t e d measure d e s i g n . I t was a one-way d e s i g n , i n t e n d e d o n l y t o a s s e s s d i f f e r e n c e s between t h e two t r i a l s . , F i g u r e 1 - T w o - T r i a l R e p e a t e d Measure D e s i g n The two o b s e r v a t i o n s . A, and A 1 # a r e t h e s c o r e s o b t a i n e d f r o m t e s t i n g t h r o u g h t h e c o mputer and t e s t i n g i n t h e c o n v e n t i o n a l f a s h i o n . Most o f t h e c u r r e n t l i t e r a t u r e t h a t d e a l s w i t h r e s e a r c h d e s i g n ( C a m p b e l l and S t a n l e y , 1966; Ker.ling.er, 1973) l i s t h i s t o r y , m a t u r a t i o n , and s t a t i s t i c a l r e g r e s s i o n a s p o s s i b l e t h r e a t s t o t h e i n t e r n a l v a l i d t y o f t h i s d e s i g n . However, i n t h i s s t u d y t h e s e t h r e a t s a r e c o n t r o l l e d . B o t h h i s t o r y and m a t u r a t i o n a r e l a r g e l y c o n t r o l l e d by t h e e x t r e m e l y s h o r t t i m e 27 i n t e r v a l between t e s t i n g s (10 m i n u t e s ) . S t a t i s t i c a l r e g r e s s i o n was n o t a c o n c e r n s i n c e o n l y one g r o u p was u s e d and s u b j e c t s were n o t s e l e c t e d on t h e b a s i s o f e x t r e m e s c o r e s on p r e v i o u s t e s t i n g s . The o r d e r and medium by w h i c h e a c h s u b j e c t r e s p o n d e d t o t h e d i f f e r e n t f o r m s was r a n d o m i z e d . T h i s c o n t r o l l e d f a c t o r s s u c h a s f a t i g u e and p r e t e s t s e n s i t i z a t i o n . R a n d o m i z a t i o n o f o r d e r a l s o c o n t r o l l e d t h e p r a c t i c e e f f e c t t h a t was f o u n d i n t h e o r i g i n a l n o r m i n g p r o c e d u r e s o f t h e t e s t . T h e r e was an e q u a l number o f s u b j e c t s who were f i r s t o b s e r v e d under t h e A , c o n d i t i o n t o t h o s e who were o b s e r v e d f i r s t u n der A a . R a n d o m i z a t i o n a l s o h e l p e d i n t h e c o n t r o l o f h i s t o r y a n d m a t u r a t i o n f o r e x a c t l y t h e same r e a s o n . The p r o c e d u r e employed f o r t h e r a n d o m i z a t i o n i s d i s c u s s e d i n d e t a i l l a t e r i n t h i s c h a p t e r . T h i s c h o i c e o f d e s i g n a l l o w e d s u b j e c t s t o answer b o t h f o r m s o f t h e Henmon-Nelson u s i n g e a c h mode o f r e s p o n s e i n random c o m b i n a t i o n s , t h u s a l l o w i n g f o r i s o l a t i o n o f t h e e f f e c t o f t h e d i f f e r e n t r e s p o n s e modes w i t h o u t i n t e r f e r e n c e f r o m e x t r a n e o u s v a r i a b l e s . T h i s i n c r e a s e d t h e power o f t h e s t u d y and made t h e c h o s e n d e s i g n a most e f f i c i e n t v e h i c l e f o r t h e s o l u t i o n o f t h e p r o b l e m . 3. P o p u l a t i o n The Henmon-Nelson, l e v e l 4 i s a p p r o p r i a t e f o r s t u d e n t s e n r o l l e d i n f i r s t y e a r u n d e r g r a d u a t e work t h r o u g h f i r s t y e a r g r a d u a t e s c h o o l . The s t u d y u s e d v o l u n t e e r s f r o m t h e O n i v e r s t i y o f B r i t i s h C o l u m b i a n s g e n e r a l s t u d e n t p o p u l a t i o n who f e l l i n t o 28 t h e above c a t e g o r y . O n d e r g r a d u a t e c l a s s e s i n t h e F a c u l t y o f E d u c a t i o n were s e a r c h e d f o r v o l u n t e e r s . A maximum o f 90 m i n u t e s per s u b j e c t was a l l o w e d t o c o m p l e t e t h e t e s t i n g . , A sample s i z e o f 36 s u b j e c t s was u s e d . 4. Data C o l l e c t i o n E a c h s u b j e c t was g i v e n a c o p y o f t h e d e m o g r a p h i c g u e s t i o n n a i r e t o c o m p l e t e when he f i r s t came t o t h e l a b . Bhen t h e g u e s t i o n n a i r e was c o m p l e t e d he g a v e t h e f o r m t o t h e e x p e r i m e n t e r who t h e n r a n d o m l y a s s i g n e d an i d e n t i f i c a t i o n number t o t h e s u b j e c t . E a c h o f t h e i d e n t i f i c a t i o n numbers had p r e v i o u s l y been r a n d o m l y a s s i g n e d t o a t e s t i n g p r o c e s s . E a c h a s s i g n m e n t gave t h e o r d e r i n w h i c h t h e t e s t f o r m s were a d m i n i s t e r e d a n d w h i c h r e s p o n s e mode was used f o r e a c h o f t h e t e s t i n g s . , T h e r e were f o u r p o s s i b l e a s s i g n m e n t s . T a b l e IV - P o s s i b l e T e s t i n g A s s i g n m e n t s A s s i g n m e n t P r o c e s s 1 A c o m p u t e r B w r i t t e n 2 B w r i t t e n A c o mputer 3 B c o m p u t e r A w r i t t e n 4 A w r i t t e n B computer T a b l e IV i n d i c a t e s t h e o r d e r a s w e l l a s t h e c o m b i n a t i o n s o f mediums a n d t e s t f o r m s i n w h i c h t h e t e s t s were a d m i n i s t e r e d . The e x p e r i m e n t e r d i r e c t e d t h e s u b j e c t t o e i t h e r a t e r m i n a l o r t h e e x a m i n a t i o n room d e p e n d i n g upon t h e a s s i g n e d p r o c e s s . T e r m i n a l s The e x p e r i m e n t e r s i g n e d t h e s u b j e c t on a t t h e t e r m i n a l and 29 t h e n i n v i g i l a t e d t h e t e r m i n a l s e s s i o n . I n v i g i l a t i o n r e q u i r e d t h a t t h e a s s i s t a n t be p r e s e n t i n t h e t e r m i n a l room i n t h e e v e n t o f h a r d w a r e / s o f t w a r e m a l f u n c t i o n d u r i n g t h e t e s t i n g . O n l y t h r e e t e r m i n a l s were i n use a t any one t i m e , due t o t h e f a c i l i t i e s a v a i l a b l e . I n s t r u c t i o n s and d i r e c t i o n s were g i v e n by t h e c o m p u t e r . The i n v i g i l a t o r was a v a i l a b l e t o c l a r i f y i n s t r u c t i o n s . I n t h e e v e n t t h a t t h e i n v i g i l a t o r was r e q u i r e d t o c l a r i f y i n s t r u c t i o n s , he made c a r e f u l n o t e o f t h e n a t u r e o f t h e p r o b l e m and s u b j e c t and s u b m i t t e d t h i s d a t a f o r s u b j e c t i v e a n a l y s i s . Once t h e t e s t i n g p a r t o f t h e s t u d y h a d begun, t h e d i r e c t i o n s g i v e n by t h e c omputer were not m o d i f i e d . The c o m p u t e r f i r s t p r o v i d e d an i n t r o d u c t i o n t o i t s o p e r a t i o n s and t h e n p r e s e n t e d t h r e e s a m ple i t e m s f o r t h e s u b j e c t t o t r y . T h e s e i t e m s were t a k e n f r o m t h e Henmon-Nelson*s a d m i n i s t r a t i o n manual. The s u b j e c t was g i v e n t h e o p p o r t u n i t y t o r e v i e w t h e i n s t r u c t i o n s and s a m ple q u e s t i o n s once a g a i n i f he d e s i r e d . I f he wanted t o c o n t i n u e on t o t h e t e s t , t h e f i r s t page o f t h e t e s t was t h e n p r e s e n t e d and t h e p r o g r a i ' s 40 m i n u t e t i m e l i m i t was begun. A t t h e end o f 40 m i n u t e s t h e p r o g r a m s t o p p e d t h e a d m i n i s t r a t i o n o f t h e t e s t , s t o r e d t h e n e c e s s a r y d a t a , p r e s e n t e d a c o n c l u d i n g message t o t h e s u b j e c t t h a n k i n g him f o r h i s c o o p e r a t i o n and t h e n " s i g n e d o f f . " The s u b j e c t t h e n r e p o r t e d t o t h e r e s e a r c h a s s i s t a n t f o r f u r t h e r i n s t r u c t i o n s . B r i t t e n Form The e x p e r i m e n t e r a s s i g n e d t h e s u b j e c t t o one o f t h r e e c h a i r s a t a t a b l e i n t h e e x a m i n a t i o n room and a d m i n i s t e r e d t h e a p p r o p r i a t e f o r m o f t h e Henmon-Nelson f o l l o w i n g t h e p r e s c r i b e d p r o c e d u r e o u t l i n e d i n t h e e x a m i n e r ' s manual. The s u b j e c t was 30 s u p p l i e d w i t h a t e s t b o o k l e t , s e l f s c o r i n g answer s h e e t , and a p e n c i l . E a c h s u b j e c t was a l l o t t e d t h e p r e s c r i b e d 40 m i n u t e t i m e l i m i t t o c o m p l e t e t h e t e s t . A t t h e end o f 40 m i n u t e s e a c h p a p e r was c o l l e c t e d . T h i s d a t a was k e y p u n c h e d on computer c a r d s i n p r e p a r a t i o n f o r m a c h i n e s c o r i n g and d a t a s t o r a g e . I n between t h e w r i t i n g o f t h e two t e s t f o r m s , s u b j e c t s were g i v e n a 10 m i n u t e r e s t p e r i o d , d u r i n g w h i c h t h e y were n o t a l l o w e d t o c o n v e r s e w i t h o t h e r s u b j e c t s ; c o f f e e was a v a i l a b l e . T e s t i n g was c a r r i e d o u t i n t h e E d u c a t i o n a l R e s e a r c h S e r v i c e C e n t e r (ERSC) l o c a t e d i n t h e basement o f t h e S c a r f e b u i l d i n g a t t h e U n i v e r s i t y o f B r i t i s h C o l u m b i a . T i m e s f o r t e s t i n g were a r r a n g e d a t t h e c o n v e n i e n c e o f t h e s u b j e c t s . , , T e s t i n g was c a r r i e d o u t o v e r an 8 day p e r i o d . D a t a C o l l e c t e d D a t a was c o l l e c t e d on e a c h s u b j e c t f r o m t h r e e d i f f e r e n t s o u r c e s . D a t a f r o m t h e f i r s t s o u r c e , t h e d e m o g r a p h i c g u e s t i o n n a i r e , was c o d e d , k e y p u n c h e d , and r e a d i n t o t h e c o m p u t e r f o r machine s t o r a g e and u s e i n l a t e r a n a l y s i s . A l l t h e d a t a t h a t was c o l l e c t e d d u r i n g t h e c o m p u t e r a d m i n i s t r a t i o n o f t h e t e s t was a u t o m a t i c a l l y s t o r e d by t h e program a t t h e end o f t h e 40 m i n u t e s . The t y p e s o f d a t a c o l l e c t e d d u r i n g t h i s p r o c e d u r e i n c l u d e d i n d i v i d u a l i t e m r e s p o n s e s , t h e t o t a l t i m e s p e n t on e a c h page, t h e number o f t i m e s e a c h page was d i s p l a y e d , and t h e number o f r e s p o n s e c h a n g e s p e r q u e s t i o n . The s c o r e s on t h e v e r b a l and q u a n t i t a t i v e s u b t e s t s a s w e l l a s t h e t o t a l t e s t s c o r e were s t o r e d . 31 R e s p o n s e s from t h e w r i t t e n f o r m were k e y - p u n c h e d i n p r e p a r a t i o n f o r m a c h i n e s c o r i n g . S i n c e i t e m d a t a was c o l l e c t e d and s t o r e d on d i s k , machine s c o r i n g o f t h e c o n v e n t i o n a l t e s t i n g was done i n t h e i n t e r e s t s o f t i m e and e f f i c i e n c y . The o n l y t y p e s o f d a t a t h a t were f e a s i b l e t o c o l l e c t f r o m t h e w r i t t e n f o r m were r e s p o n s e d a t a and s c o r e s on s u b t e s t s and t o t a l s . T h i s d e m o n s t r a t e d one o f t h e a d v a n t a g e s o f c o m p u t e r a d m i n i s t r a t i o n o v e r t h e c o n v e n t i o n a l method o f t e s t a d m i n i s t r a t i o n . The f i n a l d a t a f i l e c o n t a i n e d t h r e e r e c o r d s p e r s u b j e c t , e a c h headed by s u b j e c t i d e n t i f i c a t i o n . The f i r s t r e c o r d c o n t a i n e d t h e d e m o g r a p h i c d a t a , t h e s e c o n d r e c o r d t h e d a t a c o l l e c t e d d u r i n g computer a d m i n i s t r a t i o n , w h i l e t h e t h i r d r e c o r d c o n s i s t e d o f d a t a c o l l e c t e d d u r i n g t h e c o n v e n t i o n a l a d m i n i s t r a t i o n . A l l d a t a and p r o g r a m f i l e s were b a c k e d up on m a g n e t i c t a p e i n o r d e r t o p r o t e c t a g a i n s t a s y s t e m m a l f u n c t i o n . 5 S t a t i s t i c s W i t h t h e d e s i g n f o l l o w i n g a two t r i a l , r e p e a t e d measure p a r a d i g m , t h e most e f f i c i e n t means o f a n a l y s i s was t h e p a i r e d (dependent) t - t e s t (Winer, 1971). S i n c e a c o v a r i a n c e t e r m i s f o u n d i n t h e d e n o m i n a t o r o f t h e t e x p r e s s i o n , i t i s s o m e t i m e s r e f e r r e d t o a s t h e c o r r e l a t e d t . * R e p e a t e d measures i s a c t u a l l y a m a t c h i n g p r o c e s s where e v e r y s u b j e c t i s matched w i t h h i m s e l f ; * R e c a l l t h a t c o v a r i a n c e i s p r o p o r t i o n a l t o t h e c o r r e l a t i o n c o e f f i c i e n t 32 h e n c e dependency a c r o s s measures. T h i s dependency however, i s u s e f u l . I t h e l p s i n t h e c o n t r o l o f some o f t h e e x t r a n e o u s v a r i a b l e s which c a n i n f l u e n c e t h e d e p e n d e n t v a r i a b l e . I n o t h e r words, t h e d e p e n d e n t s a m p l e s r e d u c e t h e s u b j e c t t o s u b j e c t v a r i a b i l i t y ( r e s i d u a l v a r i a n c e ) . The u s u a l a s s u m p t i o n s o f n o r m a l i t y o f p o p u l a t i o n and h o m o s c e d a s t i c l t y were made. The t h r e e h y p o t h e s e s , s t a t e d i n t h e n u l l f o r m were a s f o l l o w s . 1. T h e r e i s n o t a s t a t i s t i c a l l y s i g n i f i c a n t d i f f e r e n c e between t h e a v e r a g e raw q u a n t i t a t i v e s c o r e o b t a i n e d f r o m a c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson T e s t o f M e n t a l a b i l i t y , l e v e l A and t h e a v e r a g e raw q u a n t i t a t i v e s c o r e o b t a i n e d f r o m a c o n v e n t i o n a l a d m i n i s t r a t i o n o f t h e t e s t , a t t h e <*=.05 l e v e l o f s i g n i f i c a n c e . 2. T h e r e i s n o t a s t a t i s t i c a l l y s i g n i f i c a n t d i f f e r e n c e between t h e a v e r a g e raw v e r b a l s c o r e o b t a i n e d f r o m a c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson T e s t o f M e n t a l a b i l i t y , l e v e l A and t h e a v e r a g e raw v e r b a l s c o r e o b t a i n e d f r o m a c o n v e n t i o n a l a d m i n i s t r a t i o n o f t h e t e s t , a t t h e <*=.05 l e v e l o f s i g n i f i c a n c e . 3. T h e r e i s n o t a s t a t i s t i c a l l y s i g n i f i c a n t 33 d i f f e r e n c e between t h e a v e r a g e raw t o t a l s c o r e o b t a i n e d f r o m a c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4 and t h e a v e r a g e raw t o t a l s c o r e o b t a i n e d f r o m a c o n v e n t i o n a l a d m i n i s t r a t i o n o f t h e t e s t , a t t h e <*>=.05 l e v e l o f s i g n i f i c a n c e . The a n a l y s i s c o n s i s t e d o f t h r e e n o n - r d i r e c t i o n a l t - t e s t s ; one f o r t h e q u a n t i t a t i v e s c o r e , one f o r t h e v e r b a l s c o r e a n d one f o r t h e t o t a l s c o r e . The s t a t i s t i c a l h y p o t h e s e s f o r e a c h o f t h e t h r e e t e s t s were: H 0 : u, = u t H, : u, # u,_ S c o r e s on e a c h s u b t e s t a s w e l l a s t h e t o t a l s c o r e were a n a l y z e d by means o f SPSS ( N i e , H u l l , J e n k i n s , S t e i n b r e n n e r , and B e n t , 1975), u s i n g t h e T - T e s t / P a i r s p r o c e d u r e c a r d . M i s s i n g d a t a was n o t e n c o u n t e r e d i n t h e a n a l y s i s . P e a r s o n product-moment c o r r e l a t i o n c o e f f i c i e n t s were a l s o computed t o g i v e an i n d i c a t i o n o f t h e m a g n i t u d e o f t h e r e l a t i o n s h i p between t h e s c o r e s o b t a i n e d f r o m t h e c o m p u t e r a d m i n i s t r a t i o n o f t h e Henmon-Nelson and t h e s c o r e s o b t a i n e d i n t h e t r a d i t i o n a l manner. SPSS was u s e d t o d e t e r m i n e t h i s c o e f f i c i e n t a s w e l l a s t e s t i n g i t s s i g n i f i c a n c e . 34 CHAPTER 4 R e s u l t s The a n a l y s i s o f t h e d a t a c o l l e c t e d i n t h e s t u d y was d i v i d e d i n t o f o u r c a t e g o r i e s : 1) d e s c r i p t i v e a n a l y s i s o f t h e s a m p l e ; 2) a n a l y s i s d i r e c t e d t o w a r d s a n s w e r i n g t h e r e s e a r c h q u e s t i o n s ; 3) p o s t hoc a n a l y s i s ; and 4) s u b j e c t i v e a n a l y s i s . Each o f t h e above d i v i s i o n s a r e d e a l t w i t h i n t u r n . 1. Sample A s a m p l e o f 36 s u b j e c t s was used i n t h e s t u d y . The s u b j e c t s were e g u a l l y d i v i d e d among t h e f o u r t e s t i n g p r o c e d u r e s a s l i s t e d i n T a b l e I V . T a b l e V t h r o u g h T a b l e I X g i v e f r e g u e n c y breakdowns w i t h r e s p e c t t o d e m o g r a p h i c v a r i a b l e s . T a b l e V - F r e g u e n c y Of Gender ( i ) G e n d e r H a l e 19 Female 17 T o t a l 36 T a b l e VI - F r e g u e n c y O f Age ( i i ) Age L e s s Than 18 1 18-23 22 24-29 11 30* 2 T o t a l 36 35 f a b l e V I I - F r e q u e n c y Of M a j o r ( i i i ) M a j o r F i n e a r t s 8 H u m a n i t i e s 3 S o c i a l S c i e n c e s 6 S c i e n c e 15 Phys Educ 4 T o t a l 36 T a b l e V I I I - F r e q u e n c y Of Year Of E n r o l l m e n t ( i v ) Y e a r o f e n r o l l m e n t 1 7 2 1 3 12 4 0 5 14 G r a d u a t e 2 T o t a l 36 T a b l e IX - F r e q u e n c y Of P r i o r T e r m i n a l Use (v) P r e v i o u s u s e o f c o m p u t e r t e r m i n a l s Yes 14 No 22 T o t a l 36 F i v e o f t h e 14 s u b j e c t s were f a m i l i a r w i t h c o m p u t e r t e r m i n a l s a s a r e s u l t o f t h e i r p r e v i o u s employment a s bank 36 t e l l e r s . F i v e o f t h e r e m a i n i n g n i n e had used t e r m i n a l s i n h i g h s c h o o l w h i l e t h e l a s t f o u r s u b j e c t s g a i n e d t h e i r e x p e r i e n c e a t t h e U n i v e r s i t y Of B r i t i s h C o l u m b i a . Of t h e 36 s u b j e c t s , 28 were e n r o l l e d i n t h e F a c u l t y o f E d u c a t i o n (18 e l e m e n t a r y ; 10 s e c o n d a r y ) w h i l e t h e r e m a i n i n g 8 were from t h e F a c u l t i e s o f A r t s and S c i e n c e . 2. D a t a A n a l y s i s T a b l e X p r o v i d e s a l i s t o f means, s t a n d a r d d e v i a t i o n s , and s t a n d a r d e r r o r s i n raw s c o r e m e t r i c f o r e a c h o f t h e s i x d e p e n d e n t v a r i a b l e s ; c o m p u t e r i z e d v e r b a l s c o r e {CV), w r i t t e n v e r b a l s c o r e (HV) , c o m p u t e r i z e d q u a n t i t a t i v e s c o r e (CQ), w r i t t e n q u a n t i t a t i v e s c o r e (HQ), c o m p u t e r i z e d t o t a l s c o r e (CT) , and w r i t t e n t o t a l s c o r e (HT). X - Means, S t a n d a r d D e v i a t i o n s And S t a n d a r d V a r i a b l e Mean S t a n d a r d D e v i a t i o n S t a n d a r d E r r o r CV 42.056 9. 940 1.657 HV 41.028 11.770 1-962 CQ 27.445 7.534 1.256 HQ 26.194 8.632 1.439 CT 69.500 15.890 2. 648 HT 67.222 18.509 3.085 M e a n i n g f u l c o m p a r i s o n s o f means and s t a n d a r d d e v i a t i o n s c a n o n l y be made between t h e two v e r b a l s c a l e s , t h e two g u a n t i t a t i v e s c a l e s , a n d t h e two f u l l s c a l e s due t o t h e raw s c o r e m e t r i c . 37 T a b l e XI p r o v i d e s t h e p r o p o r t i o n a l means t o f a c i l i t a t e i n t e r -s c a l e c o m p a r i s o n . T a b l e XI - P r o p o r t i o n a l Cleans C o m p u t e r i z e d W r i t t e n V 0.701 0.684 Q 0.686 0.655 T o t a l 0.695 0.672 E x a m i n a t i o n o f t h e means i n T a b l e s X and XI show t h a t t h e c o m p u t e r i z e d s c o r e s a r e c o n s i s t e n t l y h i g h e r t h a n t h e w r i t t e n s c o r e s a c r o s s t h e t h r e e s c a l e s . T h i s c o u l d s u g g e s t t h e p r e s e n c e o f a n o n s i g n i f i c a n t n o v e l t y e f f e c t . T a b l e X a l s o shows a s m a l l e r v a r i a n c e o f s c o r e s a c r o s s e a c h s c a l e f o r t h e c o m p u t e r i z e d f o r m s . H y p o t h e s i s J I n c o n s i d e r i n g t h e f i r s t n u l l h y p o t h e s i s (page 32) r e g a r d i n g d i f f e r e n c e s between t h e a v e r a g e , raw, c o m p u t e r i z e d v e r b a l s c o r e and t h e a v e r a g e , raw, w r i t t e n v e r b a l s c o r e , a d e p e n d e n t t - s t a t i s t i c was c a l c u l a t e d (t=0.71, df=35, p=0.485) which i n d i c a t e d t h a t t h e r e was no s i g n i f i c a n t d i f f e r e n c e between t h e a v e r a g e , raw, c o m p u t e r i z e d v e r b a l s c o r e and t h e a v e r a g e , raw, w r i t t e n v e r b a l s c o r e and t h a t t h e n u l l h y p o t h e s i s ( H 0 : u l = u l ) s h o u l d n o t be r e j e c t e d . A P e a r s o n p r o d u c t moment c o r r e l a t i o n c o e f f i c i e n t o f 0.688 was a l s o c a l c u l a t e d and f o u n d t o be s i g n i f i c a n t l y d i f f e r e n t f r o m z e r o (p<0.001). T h i s s u g g e s t e d a s i g n i f i c a n t c o r r e l a t i o n between t h e raw c o m p u t e r i z e d v e r b a l s c o r e s and t h e raw w r i t t e n v e r b a l s c o r e s . 38 H y p o t h e s i s 2 I n c o n s i d e r i n g t h e s e c o n d n u l l h y p o t h e s i s (page 32) r e g a r d i n g d i f f e r e n c e s between t h e a v e r a g e , raw, c o m p u t e r i z e d q u a n t i t a t i v e s c o r e and t h e a v e r a g e , raw, w r i t t e n q u a n t i t a t i v e s c o r e , a d e p e n d e n t t - s t a t i s t i c was c a l c u l a t e d (t=0.96, df=35, p=0. 342) w h i c h i n d i c a t e d t h a t t h e r e was no s i g n i f i c a n t d i f f e r e n c e between t h e a v e r a g e , raw, c o m p u t e r i z e d q u a n t i t a t i v e s c o r e and t h e a v e r a g e , raw, w r i t t e n g u a n t i t a t i v e s c o r e and t h a t t h e n u l l h y p o t h e s i s (H 0:u,=u i) s h o u l d n o t be r e j e c t e d . & P e a r s o n p r o d u c t moment c o r r e l a t i o n c o e f f i c i e n t o f 0.544 was a l s o c a l c u l a t e d and f o u n d t o be s i g n i f i c a n t l y d i f f e r e n t f r o m z e r o (p<0.001)., T h i s s u g g e s t e d a s i g n i f i c a n t c o r r e l a t i o n between t h e raw c o m p u t e r i z e d g u a n t i t a t i v e s c o r e s and t h e raw w r i t t e n g u a n t i t a t i v e s c o r e s . H y p o t h e s i s 3 I n c o n s i d e r i n g t h e t h i r d n u l l h y p o t h e s i s (page 32) r e g a r d i n g d i f f e r e n c e s between t h e a v e r a g e , raw, c o m p u t e r i z e d t o t a l s c o r e a n d t h e a v e r a g e , raw, w r i t t e n t o t a l s c o r e , a d e p e n d e n t t - s t a t i s t i c was c a l c u l a t e d (t=0.88, d f = 3 5 , p=0.384). T h i s r e s u l t i n d i c a t e d t h a t t h e r e was no s i g n i f i c a n t d i f f e r e n c e between t h e a v e r a g e , raw, c o m p u t e r i z e d t o t a l s c o r e and t h e a v e r a g e , raw, w r i t t e n t o t a l s c o r e and t h a t t h e n u l l h y p o t h e s i s (H 0:u,=u 1) s h o u l d n o t be r e j e c t e d . k P e a r s o n p r o d u c t moment c o r r e l a t i o n c o e f f i c i e n t o f 0.604 was c a l c u l a t e d and f o u n d t o be s i g n i f i c a n t l y d i f f e r e n t f r o m z e r o (p<0.001). T h i s s u g g e s t e d a s i g n i f i c a n t c o r r e l a t i o n between t h e raw c o m p u t e r i z e d t o t a l s c o r e s and t h e raw w r i t t e n t o t a l s c o r e s . 39 I n a l l t h r e e c a s e s t h e n u l l h y p o t h e s i s was s u p p o r t e d as t h e a v a i l a b l e l i t e r a t u r e had s u g g e s t e d . Thus i t a p p e a r s r e a s o n a b l e t o c o n c l u d e t h a t c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-N e l s o n T e s t o f M e n t a l A b i l i t y , l e v e l 4, c a n p r o d u c e r e s u l t s e q u i v a l e n t t o t h o s e o b t a i n e d f r o m c o n v e n t i o n a l a d m i n i s t r a t i o n s . T h e s e r e s u l t s a l s o s u g g e s t t h a t c o m p u t e r i z e d a d m i n i s t r a t i o n o f t e s t s s i m i l a r i n n a t u r e and f o r m a t t o t h e Henmon-Nelson, d o e s n o t a f f e c t t h e r e s u l t i n g s c o r e s . 3. P o s t Hoc A n a l y s i s P o s t h o c a n a l y s e s t o o k the f o r m o f a one t r i a l f a c t o r r e p e a t e d m e asures ANOVAs w i t h e a c h o f t h e d e m o g r a p h i c v a r i a b l e s a c t i n g i n t u r n a s g r o u p i n g f a c t o r s . T a b l e X I I c o n t a i n s t h e means and s t a n d a r d d e v i a t i o n s f o r e a c h o f t h e v a r i a b l e s u s e d i n t h e a n a l y s i s . The v a r i a b l e " y e a r " r e c o r d s the s u b j e c t ' s p r e s e n t e n r o l l m e n t s t a t u s a t t h e U n i v e r s i t y o f B r i t i s h C o l u m b i a w h i l e "comp" s i g n a l s p r e v i o u s c o m p u t e r t e r m i n a l e x p e r i e n c e . " M a j o r " i n d i c a t e s t h e s u b j e c t ' s a c a d e m i c major o r major a r e a o f s t u d y . A summary o f t h e p o s t hoc f i n d i n g s w i t h r e s p e c t t o t h e v a r i o u s g r o u p i n g v a r i a b l e s i s p r e s e n t e d i n T a b l e X I I I . F r a t i o s r e g a r d i n g t h e w i t h i n f a c t o r (mode o f p r e s e n t a t i o n ) and i n t e r a c t i o n s a r e n o t p r o v i d e d s i n c e t h e y were a l l f o u n d t o be n o n s i g n i f i c a n t . M a r g i n a l means f o r t h e w i t h i n f a c t o r c a n be f o u n d i n T a b l e X. I n T a b l e X I I I an F r a t i o f o l l o w e d by a s i n g l e a s t e r i s k (*) i n d i c a t e s 5% s i g n i f i c a n c e w h i l e a d o u b l e a s t e r i s k (**) i n d i c a t e s a .156 s i g n i f i c a n c e l e v e l . T a b l e X I I - P o s t Hoc Beans And ( S t a n d a r d D e v i a t i o n s ) CV SV CQ HQ CT HT GENDER H a l e 44.32 43.68 n=19 (7.39) (10.75) Female 39.52 38.06 n=17 (11.91) (12,46) 29.58 28.05 73.89 71.74 (7.54) (9.20) (13.64) (18.28) 25.06 24.12 64.59 62.18 (6.98) (7.69) (17.16) (17.96) AGE -23 n=23 24+ n=13 38.83 (9.67) 47.77 (7.84) 37.87 (12.37) 46.62 (8-39) 25.39 (7.38) 31.08 (6.59) 24.52 (8.69) 64.22 (15.45) 62.39 (18.96) 29.15 78.85 75.77 (7.99) (12.27) (14.71) HAJOfi A r t s n=8 42. ( 9 . 50 27) 44.63 (9.67) 22.75 (8.79) 20. (8. 63 48) 65-25 (16.85) 65.25 (16-59) Hamants n=3 43. (11-33 59) 34.67 (13.05) 31-33 (3.78) 20. (9-67 Q7) 74.67 (15.37) 55.33 (22. 12) Soc S c n=6 40. (10. 17 59) 40.33 (10.50) 26.83 (8.23) 29. (7-00 29) 67.00 (16.57) 69.33 (16-52) S c i e n c e n=15 42. (11. 60 63) 40.80 (14.42) 30-20 (6.23) 29. (8. 67 53) 72-80 (16.96) 70.74 (21.93) Phys Ed n=4 41. (5. 00 66) 40.50 (7.33) 24.50 (7.33) 24. (<*-25 86) 65-50 (12-77) 64-75 (10-34) YEAR COMP - 2 n=8 41.38 (11-89) 36.88 (17.23) 28.75 (7.36) 26. ( 9 . 13 57) 70.13 (18.75) 63.00 (25-81) i - 4 n=12 40.00 (8.59) 38.83 (10.56) 23.58 (6.79) 20. (6. 58 65) 63.58 (12.89) 59-42 (14.87) 5* n=16 43.94 (10. 15) 44-75 (8.68) 29-69 (7-43) 30. ( 7 . 43 36) 73.63 (16-03) 75.19 (14.15) Y e s n=14 47.86 (6-71) 48. 14 C»-95) 32.57 (4.94) 33. (4. 3 5 85) 80.4 3 (10.55) 81.50 (8-45) No 38.36 36-50 24.18 21.64 62.55 58-14 n=22 (10.00) (12.67) (7.13) (7.33) (14.88) (17.45) 41 T a b l e X I I I - P o s t Hoc Summary V e r b a l Q u a n t i t a t i v e T o t a l Gender F=2.55G F=3.392 F=3-612 (df=34,p=. 120) (df=34,p=.074) (df=34,p=. 066) M a j o r F=0.14Q F= 1.206 F=0.172 (df=33,p=.392) (df=33,p=.328) (df =33 ,p=. 131) Y e a r F=0.965 F=4.555 * F=2-166 (df=33,p=-392) (df=33,p=.018) (df=33 ,p=. 131) Age F=7.792 ** F=4.860 * F=8.284 ** (d£=34,p=.009) (df=34,p=.034) (df=34,p=.007) Comp F=12.845 ** F=32.586 ** F= 26.468 ** (df=34,p=.001) (df=34,p=.Q01) {df=34 ,p=.001) From T a b l e 4.9 i t i s e v i d e n t t h a t t h e r e a r e n o n s i g n i f i c a n t g e n d e r and m a j o r - a r e a - o f - s t u d y e f f e c t s . & g e n d e r e f f e c t was n o t e x p e c t e d s i n c e t h e Henmon-Nelson i s r e p o r t e d n o t t o have a s e x b i a s . However, a s i g n i f i c a n t " m a j o r " e f f e c t was a n t i c i p a t e d due to t h e h i g h l y v e r b a l n a t u r e o f t h e t e s t , b u t none i n f a c t m a t e r i a l i z e d . The v a r i a b l e " y e a r " had a s i g n i f i c a n t F r a t i o when a n a l y z e d w i t h t h e g u a n t i t a t i v e s c o r e s a s t h e c r i t e r i o n measure. A n a l y s i s u s i n g S c h e f f e ' s m u l t i p l e c o n t r a s t method was c a r r i e d o u t and t h e a v e r a g e raw g u a n t i t a t i v e s c o r e o f s u b j e c t s i n t h e i r t h i r d and f o u r t h y e a r a t t h e U n i v e r s i t y o f B r i t i s h C o l u m b i a ( g r o u p 2) was f o u n d t o be s i g n i f i c a n t l y l o w e r t h a n t h a t o f t h e s u b j e c t s i n t h e i r f i f t h and g r a d u a t e y e a r s ( g r o u p 3 ) . T h i s i s most l i k e l y due t o t y p e I e r r o r and a r t i f a c t s o f s a m p l i n g . The r e m a i n i n g two p a i r w i s e c o m p a r i s o n s were b o t h n o n s i g n i f i c a n t . , The age f a c t o r was b l o c k e d i n t o two l e v e l s ; 23 y e a r s and y o u n g e r o r 24 y e a r s and o l d e r . T h i s was done t o o b t a i n s u f f i c i e n t c e l l s i z e s t o f a c i l i t a t e t h e a n a l y s i s o f v a r i a n c e p r o c e d u r e s . S i g n i f i c a n c e was f o u n d on a l l t h r e e c r i t e r i o n v a r i a b l e s and e x a m i n a t i o n o f t h e c e l l means i n d i c a t e d t h e a v e r a g e raw s c o r e s on a l l s c a l e s f o r t h e younger s u b j e c t s were s u b s t a n t i a l l y l o w e r t h a n t h o s e o f t h e o l d e r group.,., The f i n a l v a r i a b l e w h i c h was u s e d i n t h e a n a l y s i s , p r e v i o u s - c o m p u t e r - u s e , p r o v i d e d i n t e r e s t i n g r e s u l t s . S i g n i f i c a n t F r a t i o s (p<.001) were o b t a i n e d f o r e a c h o f t h e t h r e e s c a l e s . S u b j e c t s who had p r e v i o u s l y u s e d c o m p u t e r t e r m i n a l s c o n s i s t e n t l y s c o r e d h i g h e r on e a c h s c a l e t h a n d i d t h o s e s u b j e c t s w i t h o u t any p r e v i o u s c o m p u t e r e x p o s u r e . I n t e r a c t i o n s were n o n s i g n i f i c a n t a s was t h e w i t h i n f a c t o r (mode o f p r e s e n t a t i o n ) e f f e c t . T h i s c o u l d i m p l y t h a t t h e s u b j e c t s who h a v e u s e d c o m p u t e r s a r e more " i n t e l l i g e n t 1 ! , a s d e f i n e d and measured by t h e Henmon-Nelson T e s t o f M e n t a l a b i l i t y , l e v e l 4, t h a n t h o s e who h a v e n o t . I t c o u l d a l s o s u g g e s t t h a t c o m p u t e r s c i e n c e d e v e l o p s s k i l l s w h i c h s e n s i t i z e s u b j e c t s t o t h e Henmon-N e l s o n . T h e s e a r e two a r e a s f o r f u t u r e s t u d i e s t o e x p l o r e . 4. S u b j e c t i v e a n a l y s i s A t t h e end o f t h e f i r s t day o f t e s t i n g a m i n o r p r o b l e m w i t h t h e c o m p u t e r a d m i n i s t r a t i o n program was a c c i d e n t l y d i s c o v e r e d d u r i n g d a t a - f i l e b a c k up p r o c e d u r e s . The p r o b l e m was t r a c e d t o a r o u t i n e t h a t c h e c k e d f o r d u p l i c a t e s u b j e c t i d e n t i f i c a t i o n numbers. The r o u t i n e t o o k t h e f i r s t t h r e e d i g i t s o f t h e i d e n t i f i c a t i o n number and c h e c k e d t h e c o r r e s p o n d i n g e l e m e n t i n an a r r a y c o n s i s t i n g e x c l u s i v e l y o f z e r o s and o n e s . I f t h e e l e m e n t was a " 0 M , i t was t h e n c h a n g e d t o a " I " and t h e s u b j e c t c o n t i n u e d w i t h t h e t e s t i n g program. However, i f t h e e l e m e n t was n o n - z e r o ( i e . " I " ) , a d u p l i c a t e i d e n t i f i c a t i o n number had b e en e n t e r e d and t h e s u b j e c t was a s k e d t o r e e n t e r h i s number. I f t h e same c o n d i t i o n o c c u r r e d , t h e s u b j e c t was i n s t r u c t e d by t h e c o m puter t o s e e t h e e x p e r i m e n t e r . The p r o b l e m was t h a t , u n d e r f r e a k c o n d i t i o n s , p o r t i o n s o f t h e c h e c k i n g a r r a y was b l a n k e d o u t , t h u s n o t a l l o w i n g a s u b j e c t whose number c o r r e s p o n d e d w i t h a b l a n k e d o u t e l e m e n t t o be t e s t e d . F o r t u n a t e l y , t h e p r o b l e m was d i s c o v e r e d b e f o r e any c o m p l i c a t i o n s a r o s e . The C f t l * p o r t i o n o f t h e p r o g ram t h a t i n s t r u c t e d t h e s u b j e c t s i n t h e p r o c e d u r e s r e q u i r e d i n t h e o p e r a t i o n o f t h e t e r m i n a l worked r e m a r k a b l y w e l l . The program d e t e r m i n e d , t h a t s u b j e c t s t o o k a p p r o x i m a t e l y 5 t o 6 m i n u t e s t o c o m p l e t e t h i s segment w h i c h i n c l u d e d t h e p r a c t i c e t e s t i t e m s . I n s t r u c t i o n s were a p p a r e n t l y c l e a r enough a s p r o b l e m s r e g a r d i n g t e r m i n a l p r o c e d u r e s d i d n o t a r i s e . T h e s e i n s t r u c t i o n s however, s h o u l d have made c l e a r t o s u b j e c t s t h a t t h e program would a u t o m a t i c a l l y t e r m i n a t e a t t h e end o f t h e 40 m i n u t e s o f t e s t i n g , a s one o r two s u b j e c t s became " b e f u d d l e d " when t h e program t e r m i n a t e d . One s u b j e c t who was a s s i g n e d t o g r o u p 3 (B c o m p u t e r and a w r i t t e n ) c o m p l e t e d t h e c o m p u t e r i z e d B f o r m and began t o t a k e * Computer A s s i s t e d I n s t r u c t i o n 44 t h e w r i t t e n A f o r m . A f t e r a v e r y s h o r t t i m e t h e s u b j e c t became f r u s t r a t e d and l e f t t h e e x a m i n a t i o n room. T h r o u g h d i s c u s s i o n w i t h t h e e x p e r i m e n t e r , i t was d i s c o v e r e d t h a t t h e f r u s t r a t i o n was due t o E n g l i s h b e i n g t h e s u b j e c t ' s s e c o n d l a n g u a g e c a u s i n g g r e a t d i f f i c u l t y i n c o m p l e t i n g t h e h i g h l y v e r b a l Henmon-Nelson. D a t a c o l l e c t e d on t h i s s u b j e c t was t r e a t e d a s i n c o m p l e t e and e x c l u d e d f r o m t h e a n a l y s i s . I t i s i n t e r e s t i n g t o n o t e t h a t t h e s u b j e c t became f r u s t r a t e d w h i l e t a k i n g t h e w r i t t e n f o r m and n o t t h e c o m p u t e r i z e d f o r m . I n g e n e r a l , t h e a t t i t u d e s o f t h e s u b j e c t s t o w a r d s c o m p u t e r i z e d t e s t i n g , a s j u d g e d t h r o u g h i n f o r m a l i n t e r v i e w s , were f a v o u r a b l e . Most s u b j e c t s r e f e r r e d t o t h e c o m p u t e r i z e d v e r s i o n a s " f u n " . E v e r y s u b j e c t p r e f e r r e d t h e c o m p u t e r i z e d v e r s i o n t o t h e c o n v e n t i o n a l mode o f a d m i n i s t r a t i o n . Some s a i d i t was " b e t t e r " , i n t e r m s o f s p e e d and e a s e o f r e s p o n s e t o t h e v e r b a l i t e m s , b u t s l o w e r f o r g u a n t i t a t i v e i t e m s as t h e y had t o be c o p i e d o n t o p a p e r b e f o r e t h e y c o u l d be s o l v e d . One s u b j e c t f o u n d i t d i f f i c u l t t o l o o k a t a CBT s c r e e n f o r 45 m i n u t e s w h i l e a n o t h e r s u b j e c t f o u n d t h e r e a d i n g d i s t a n c e ( i e . d i s t a n c e f r o m eye t o f o c a l p l a n e ) t o be t o o g r e a t . C o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson was r e l a t i v e l y i n e x p e n s i v e . The c o s t o f a d m i n i s t r a t i o n was a p p r o x i m a t e l y $4.27 (computer d o l l a r s ) p e r p e r s o n . T h i s i n c l u d e s t h e $2.25 f e e f o r c o n n e c t t i m e t and a n o m i n a l c h a r g e * b a s e d upon a r a t e o f $3.00 p e r h o u r a t a p p r o x i m a t e l y 45 m i n u t e s p e r p e r s o n 45 f o r d i s k s t o r a g e . T h i s c o n c l u d e s t h e a n a l y s i s phase o f t h i s s t u d y . The f o l l o w i n g c h a p t e r w i l l p r e s e n t a b r i e f summary, a d i s c u s s i o n o f t h e s t u d y * s l i m i t a t i o n s , and s u g g e s t i o n s f o r f u t u r e r e s e a r c h . 46 CHAPTER 5 D i s c u s s i o n JL Summary The g e n e r a l p r o b l e m t o which t h i s s t u d y a d d r e s s e d i t s e l f , was t o d e t e r m i n e what e f f e c t c o m p u t e r i z e d a d m i n i s t r a t i o n had on t h e s c o r e s o b t a i n e d f r o m a s t a n d a r d i z e d t e s t . C l e a r l y , a p r o b l e m o f s u c h s i z e would be v e r y d i f f i c u l t i f n o t i m p o s s i b l e t o answer a d e g u a t e l y f r o m t h e r e s u l t s o f j u s t one s t u d y . I t was n e c e s s a r y t o r e d u c e t h e s c o p e o f t h e p r o b l e m t o a manageable s i z e . T h u s i t became t h e p u r p o s e o f t h i s s t u d y t o d e t e r m i n e what e f f e c t , i f a n y , c o m p u t e r i z e d a d m i n i s t r a t i o n had on t h e s c o r e s o b t a i n e d from t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4. From t h i s p r o b l e m t h e s t u d y ' s n u l l h y p o t h e s e s emerged. 1. T h e r e i s n o t a s t a t i s t i c a l l y s i g n i f i c a n t d i f f e r e n c e between t h e a v e r a g e raw q u a n t i t a t i v e s c o r e o b t a i n e d f r o m a c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4 and t h e a v e r a g e raw q u a n t i t a t i v e s c o r e o b t a i n e d f r o m a c o n v e n t i o n a l a d m i n i s t r a t i o n o f t h e t e s t , a t t h e °\=.05 l e v e l o f s i g n i f i c a n c e . 2. T h e r e i s n o t a s t a t i s t i c a l l y s i g n i f i c a n t d i f f e r e n c e between t h e a v e r a g e raw v e r b a l s c o r e o b t a i n e d f r o m a c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4 and t h e a v e r a g e raw v e r b a l s c o r e o b t a i n e d f r o m a c o n v e n t i o n a l a d m i n i s t r a t i o n o f t h e t e s t , a t t h e <X=.Q5 l e v e l o f 47 s i g n i f i c a n c e . 3. T h e r e i s n o t a s t a t i s t i c a l l y s i g n i f i c a n t d i f f e r e n c e between t h e a v e r a g e raw t o t a l s c o r e o b t a i n e d f r o m a c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y , l e v e l 4 and t h e a v e r a g e raw t o t a l s c o r e o b t a i n e d f r o m a c o n v e n t i o n a l a d m i n i s t r a t i o n o f t h e t e s t , a t t h e ^ =.05 l e v e l o f s i g n i f i c a n c e . Dependent t - t e s t s t a t i s t i c s and P e a r s o n p r o d u c t moment c o r r e l a t i o n c o e f f i c i e n t s were c a l c u l a t e d w i t h r e s p e c t t o e a c h o f t h e t h r e e h y p o t h e s e s and i n e a c h c a s e t h e n u l l h y p o t h e s i s was n o t r e j e c t e d . The P e a r s o n r c o e f f i c i e n t s i n d i c a t e d s i g n i f i c a n t c o r r e l a t i o n s between t h e c o m p u t e r i z e d and w r i t t e n s c o r e s . T h e s e r e s u l t s s u p p o r t e d t h e t h r e e n u l l h y p o t h e s e s and s u g g e s t e d t h a t c o m p u t e r i z e d a d m i n i s t r a t i o n o f t h e Henmon-Nelson can p r o v i d e r e s u l t s c o m p a r a b l e t o t h o s e o b t a i n e d f r o m a c o n v e n t i o n a l a d m i n i s t r a t i o n . P o s t hoc a n a l y s e s were c a r r i e d o u t t o a s s e s s t h e e f f e c t o f t h e d i f f e r e n t d e m o g r a p h i c v a r i a b l e s . N o n s i g n i f i c a n t F r a t i o s were f o u n d f o r g e n d e r and m a j o r - a r e a - o f - s t u d y w h i l e s i g n i f i c a n c e r e s u l t e d f o r age and p r e v i o u s - c o m p u t e r - u s e . S u b j e c t s o l d e r t h a n 23 y e a r s s c o r e d c o n s i d e r a b l y h i g h e r t h a n t h o s e 23 and y o u n g e r . T h o s e s t u d e n t s w i t h p r e v i o u s c o m p u t e r e x p e r i e n c e s c o r e d h i g h e r t h a n t h o s e w i t h o u t . A " y e a r - o f - e n r o l l m e n t " e f f e c t was f o u n d , b u t i t was a t t r i b u t e d t o s a m p l i n g e r r o r . 48 2. Weaknesses To Be C o n s i d e r e d The s t r u c t u r e o f t h e p r o p o s e d s t u d y was c o n s i d e r e d s o u n d * E x t r a n e o u s v a r i a b l e s t h a t c o u l d i n f l u e n c e t h e d e p e n d e n t m e a s u r e s were c o n t r o l l e d b o t h by d e s i g n and t h e use o f r a n d o m i z a t i o n . The s t a t i s t i c a l model on w h i c h t h e a n a l y s i s was b a s e d was s t r a i g h t f o r w a r d and f o r t h e p u r p o s e s o f t h i s s t u d y , e l e g a n t . However, a s i n most i f n o t a l l r e s e a r c h , d e s i g n s do f a l l s h o r t o f t h e i d e a l and t h i s s t u d y was no e x c e p t i o n . , One o r two m i n o r s h o r t c o m i n g s e x i s t e d b u t t h e s e d i d n o t r a d i c a l l y a f f e c t t h e f i n d i n g s . The most s e r i o u s p r o b l e m was one r e l a t i n g t o t h e n a t u r e o f t h e s a m p l e . S i n c e t h e s a m p l e c o n s i s t e d t o t a l l y of v o l u n t e e r s , a b i a s was i n t r o d u c e d . T h i s b i a s was a r e s u l t o f t h e sample n o t b e i n g s t r i c t l y r e p r e s e n t a t i v e o f t h e t a r g e t p o p u l a t i o n . The s u b j e c t s were v o l u n t e e r s and p o s s i b l y had u l t e r i o r m o t i v e s f o r p a r t i c i p a t i n g i n t h e s t u d y . T h e s e m o t i v e s c o u l d have i n f l u e n c e d t h e o b s e r v e d m e a s u r e s under t h e e x p e r i m e n t a l c o n d i t i o n s . Members o f t h e t a r g e t p o p u l a t i o n who d i d n o t v o l u n t e e r a l s o b i a s t h e s t u d y . : P o r example, i t i s u n l i k e l y t h a t " c o m p u t e r -p h o b i c s " v o l u n t e e r e d a s s u b j e c t s f o r a s t u d y which u s e d c o m p u t e r - s u b j e c t i n t e r a c t i o n a s a means o f c o l l e c t i n g d a t a . P e o p l e b e l o n g i n g t o s u c h s u b p o p u l a t i o n s s h o u l d be i n c l u d e d i n t h e t e s t i n g s i n c e g e n e r a l i z a t i o n s w i l l be made a b o u t t h e p o p u l a t i o n t o w h i c h t h e y b e l o n g . A r a n d o m l y s e l e c t e d sample i s i d e a l f o r a l m o s t any s t u d y b u t t h e p o s s i b i l i t y f o r t h a t h e r e , s i m p l y d i d n o t e x i s t . C l e a r l y , i n c l u s i o n ( i n t h e p r o p e r p r o p o r t i o n ) o f p e o p l e who do n o t v o l u n t e e r would e l i m i n a t e t h e 49 b i a s t h a t r e s u l t e d f r o m u s i n g o n l y v o l u n t e e r s . U s i n g v o l u n t e e r s a s s u b j e c t s i n s t u d i e s o f t h i s n a t u r e d i d p r e s e n t p r o b l e m s w i t h r e s p e c t t o e x t e r n a l v a l i d i t y , b u t t h e y p r o b a b l y were n o t s e r i o u s enough t o d r a s t i c a l l y a f f e c t t h e r e s u l t s o f t h e s t u d y . . O t h e r weaknesses i n t h e s t u d y were r e l a t e d t o t h e c o m p u t e r program t h a t was used t o a d m i n i s t e r t h e t e s t . I n p r e s e n t i n g t h e t e s t i t was i n t e n d e d t o d i s p l a y e a c h page i n a f o r m a t p a r a l l e l i n g t h a t o f t h e t e s t b o o k l e t . U n f o r t u n a t e l y t h e 37 l i n e CRT d i s p l a y does n o t p e r m i t e x a c t d u p l i c a t i o n . . Hence, t h e i t e m s were n o t p r e s e n t e d i n t h e same page g r o u p i n g s a s t h e y a p p e a r e d i n t h e t e s t b o o k l e t . S i n c e d i f f e r e n t i t e m g r o u p i n g s t e n d t o c a u s e s c o r e f l u c t u a t i o n s , i t i s p r o b a b l e t h a t t h i s s o u r c e o f u n c o n t r o l l a b l e v a r i a n c e c a u s e d a s l i g h t i n c r e a s e i n t h e e r r o r v a r i a n c e . T h i s i n c r e a s e was n o t c o n s i d e r e d l a r g e enough t o a f f e c t t h e f i n a l r e s u l t s o f t h e s t u d y . A m i n o r p o i n t b u t s t i l l w o r t h n o t i n g , i s t h a t a l l a l p h a b e t i c c h a r a c t e r s were d i s p l a y e d e x c l u s i v e l y i n u p p e r c a s e . I t was n o t p r a c t i c a l i n t e r m s o f c o m p u t e r t i m e and r e a l t i m e t o have t h e d i s p l a y u t i l i z i n g b o t h u p p e r a n d l o w e r c a s e s . C o n s i d e r i n g t h e age o f t h e s u b j e c t s , c o n f u s i o n a r i s i n g f r o m t h i s , was n o t a n t i c i p a t e d and d i d n o t o c c u r . A n o t h e r c o m p u t e r r e l a t e d weakness i s t h a t a w a r n i n g was n o t g i v e n t o t h e u s e r when an i l l e g a l c h a r a c t e r ( i e . n o t a " 1 " o r "2" o r "3" o r , , 4 M o r n 5 " ) was g i v e n as a r e s p o n s e . T e c h n i c a l l y , i t was a s i m p l e a d d i t i o n t o t h e pro g r a m , b u t p r o b l e m s r e g a r d i n g t h e 37 l i n e CRT d i s p l a y l i m i t a t i o n o f t h e Ann A r b o r t e r m i n a l s made t h e c h a n g e i m p r a c t i c a l . The p r o g r a m i g n o r e d a l l i l l e g a l 50 c h a r a c t e r s i n t h e i n p u t . I n r e v i e w i n g h i s a n s w e r s , t h e s u b j e c t p r o c e e d e d page by pa g e . He had t o go t o e i t h e r t h e page i m m e d i a t e l y p r e c e e d i n g o r i m m e d i a t e l y ahead o f t h e p r e s e n t page. ; He d i d n o t h a v e t h e f r e e d o m t o t u r n d i r e c t l y t o a g i v e n page. T h i s t e c h n i c a l l y , was a g a i n e a s y t o c h a n g e b u t i t was a l s o n e c e s s a r y t o keep t h e t e r m i n a l s o p e r a t i n g p r o c e d u r e s down t o a b a r e minimum. As a r e s u l t t h e c h a n g e was r e g a r d e d a s an u n n e c e s s a r y l u x u r y -3. F u t u r e R e s e a r c h H i t h c o m p u t e r i z e d t e s t a d m i n i s t r a t i o n b e i n g a r e l a t i v e l y r e c e n t d e v e l o p m e n t i n t h e a r t o f t e s t i n g , many new a r e a s e x i s t f o r f u t u r e r e l a t e d r e s e a r c h . To f a c i l i t a t e d i s c u s s i o n t h e s e a r e a s have been d i v i d e d i n t o t h r e e g r o u p s . C l e a r l y t h e s e g r o u p s a r e n o t m u t u a l l y e x c l u s i v e ; t h e y s e r v e o n l y as a means o f i d e n t i f y i n g t h e f o c u s o f e a c h g r o u p o f s u g g e s t e d s t u d i e s . The f i r s t g r o u p o f s t u d i e s i n c l u d e s a l l t h o s e w h i c h a r e d i r e c t l y r e l a t e d t o t h e c o m p u t e r i z a t i o n o f t h e Henmon-Nelson. T h e s e w o u l d i n c l u d e r e l i a b i l i t y s t u d i e s i n w h i c h s u b j e c t s t o o k b o t h c o m p u t e r i z e d v e r s i o n s w i t h p a r a l l e l f o r m s r e l i a b i l i t y c o e f f i c i e n t s c a l c u l a t e d a n d compared t o t h a t o f t h e manual <ie. 0 . 8 8 ) . C o m p u t e r i z e d t e s t - r e t e s t r e l i a b i l i t y s t u d i e s s h o u l d a l s o be done i n o r d e r t o f u l l y a p p r e c i a t e t h e e f f e c t of c o m p u t e r i z e d a d m i n i s t r a t i o n on t h e Henmon-Nelson. F u r t h e r s t u d i e s s h o u l d be c a r r i e d o u t t o d e t e r m i n e i f t h e 51 s i g n i f i c a n t F r a t i o c o r r e s p o n d i n g t o y e a r o f e n r o l l m e n t a c r o s s t h e g u a n t i t a t i v e s c a l e , i s i n f a c t a t t r i b u t a b l e t o s a m p l i n g e r r o r . A l a r g e s a m ple drawn a t random s h o u l d be u s e d i n a s t u d y o f t h i s n a t u r e . , A r e p e a t e d measure d e s i g n w i t h a s i n g l e , s i x l e v e l g r o u p i n g f a c t o r would be t h e b e s t model t o f o l l o w i n t h e a n a l y s i s . The s i g n i f i c a n c e o f " p r e v i o u s - c o m p u t e r - u s a g e " s u g g e s t s two s e r i e s o f s t u d i e s t o be u n d e r t a k e n . F i r s t , one s e r i e s s h o u l d e x p l o r e t h e p o s s i b i l i t y t h a t p e o p l e who u s e c o m p u t e r s do s c o r e s i g n i f i c a n t l y h i g h e r on t h e Henmon-Nelson's t h r e e s c a l e s t h a n t h o s e who have n o t u s e d a c o m p u t e r . S e c o n d l y , s t u d i e s s h o u l d be done t o d e t e r m i n e i f t h e s k i l l s t e s t e d by t h e Henmon-Nelson a r e t h o s e w h i c h a r e d e v e l o p e d i n t h e s t u d y o f c omputer s c i e n c e . And t h i r d l y , c o u l d t h e Henmon-Nelson T e s t o f M e n t a l A b i l i t y be u s e d a s an a p t i t u d e t e s t f o r c o m p u t e r s c i e n c e ? T h i s t h i r d g r o u p o f s t u d i e s p r e s u p p o s s e s p o s i t i v e r e s u l t s f r o m t h e s e c o n d s e r i e s . S t u d i e s c o u l d be done t o a s s e s s t h e e f f e c t s o f i m p l e m e n t i n g t h e t e c h n i c a l programming r e f i n e m e n t s l i s t e d p r e v i o u s l y i n t h i s c h a p t e r . T h e s e s t u d i e s would a t t e m p t t o d e t e r m i n e i f t h e r e s u l t s o f t h i s p a p e r a r e s t i l l v a l i d . C hanges i n t h e p r e s e n t a t i o n f o r m a t o f t h e t e s t c o u l d a l s o be made and t h e i r e f f e c t s a n a l y z e d . F o r e x a m p l e , i t e m s c o u l d be p r e s e n t e d i n g r o u p s o f f i v e p e r s c r e e n - p a g e . T h i s would d r a s t i c a l l y a l t e r t h e a p p e a r a n c e o f t h e t e s t and p e r h a p s r e s u l t i n t h e s c o r e s b e i n g r a d i c a l l y d i f f e r e n t . The s e c o n d g r o u p o f s t u d i e s a d d r e s s e s i t s e l f t o t h e more g e n e r a l t o p i c o f c o m p u t e r i z e d t e s t i n g . T h i s s t u d y l i m i t e d 52 i t s e l f t o t h e a d m i n i s t r a t i o n o f one p a r t i c u l a r t e s t on one p a r t i c u l a r t y p e o f t e r m i n a l . S t u d i e s s h o u l d be c a r r i e d o u t t h a t examine t h e d i f f e r e n t ways t h e d i f f e r e n t t e s t s b ehave on d i f f e r e n t t e r m i n a l t y p e s . P e r h a p s d i f f e r e n t r e s u l t s would have been o b t a i n e d i f , f o r e x a m p l e , t h e Henmon-Nelson was a d m i n i s t e r e d on a Vu-Com t e r m i n a l o r an IBM 3270 t y p e t e r m i n a l . A g a i n , r e p e a t e d measure d e s i g n s w i t h g r o u p i n g f a c t o r s would be t h e most e f f i c i e n t means o f a n a l y s i s . A n o t h e r s e r i e s o f s t u d i e s would examine t h e d i f f e r e n t methods o f p r e s e n t i n g t h e t e r m i n a l ' s o p e r a t i n g i n s t r u c t i o n s . The i n s t r u c t i o n s i n t h i s s t u d y were p r e s e n t e d v i a a CAI t y p e r o u t i n e a t t h e b e g i n n i n g o f t h e t e r m i n a l s e s s i o n . T h i s worked r e m a r k a b l y w e l l a s t h e r e were no p r o b l e m s o r g u i r i e s a r i s i n g f r o m t h i s p r o c e d u r e . However t h e r e a r e o t h e r ways i n w h i c h t h e s e i n s t r u c t i o n s c o u l d h a v e been p r e s e n t e d and i t i s i m p o r t a n t t o e xamine t h e e f f i c i e n c i e s o f t h e s e a l t e r n a t e methods i n o r d e r t o i m p r o v e c o m p u t e r i z e d t e s t i n g . The f i n a l g r o u p o f s t u d i e s would c o n c e r n i t s e l f w i t h t h e p s y c h o m e t r i c p r o p e r t i e s o f t h e d i f f e r e n t k i n d s o f i n f o r m a t i o n w h i c h now i s s e e n a s p r a c t i c a l t o c o l l e c t u s i n g c o m p u t e r i z e d t e s t i n g . T h e s e s t u d i e s w i l l a nswer s u c h q u e s t i o n s a s : Can r e s p o n s e l a t e n c y b e r e l a t e d t o s c o r e , r e l i a b i l i t y , o r e v e n measurement e r r o r ? Hhat i s t h e r e l a t i o n s h i p between t h e number o f t i m e s t h e r e s p o n s e t o an i t e m was c h a n g e d and t h e d i f f i c u l t y i n d e x and i t e m v a r i a n c e ? The a n s w e r s t o t h e s e q u e s t i o n s and o t h e r s w i l l open new d o o r s i n t h e a r e a s o f t e s t s c o r e t h e o r y and r e s p o n s e s t y l e a n a l y s i s . 53 With c o m p u t e r i z e d t e s t i n g b e i n g t e c h n i c a l l y f e a s i b l e , i t i s hoped t h a t f u t u r e s t u d i e s w i l l d e m o n s t r a t e t h e m e r i t s o f t h i s new f i e l d and t h u s p r o v e i t t o be a u s e f u l and r e v o l u t i o n a r y d e v e l o p m e n t i n t h e a r t o f t e s t i n g . A m e r i c a n P s y c h o l o g i c a l A s s o c i a t i o n . S t a n d a r d s f o r e d u c a t i o n a l and p s y c h o l o g i c a l t e s t s . W a s h i n g t o n , D. C : A u t h o r , 1974. C a m p b e l l , D. T., & S t a n l e y , J . C. E x p e r i m e n t a l and g u a s i - e x p e r i m e n t a l d e s i g n s f o r . r e s e a r c h . . C h i c a g o : Band M c N a l l y , 1966. E l w o o d , 0. L. A u t o m a t i o n o f p s y c h o l o g i c a l t e s t i n g . A m e r i c a n  P s y c h o l o g i s t . 1969, 24 , 287-289. F e r g u s o n , B. L . Computer a s s i s t e d c r i t e r i o n r e f e r e n c e d t e s t i n g .  H o r k i n g p a p e r 49. P i t t s b u r g h D h n i v e r s i t y , P., A. 1966. (EBIC Document No. ED040061) U e d l , J . , O ' N e i l , H. F . , S Hansen, D. N. Computer b a s e d i n t e l l i g e n c e t e s t i n g . F l o r i d a S t a t e U n i v e r s i t y , 1971. (ERIC Document No. ED050146) H i t t i , F. J . , B i f f e r , R. L., & S t u c k l e s s , E. R. Computer managed t e s t i n g : A f e a s i b i l i t y s t u d y w i t h d e a f s t u d e n t s . R o c h e s t e r I n s t i t u t e o f T e c h n o l o g y , 1971. (ERIC Document No. ED0605981) Hoye, R, E - , 6 Hang, A. C. (E.D.) I n d e x t o c o mputer b a s e d l e a r n i n g . Englewood C l i f f s , Ne* J e r s e y : E d u c a t i o n a l T e c h n o l o g y P u b l i c a t i o n s , 1973. K e r l i n g e r , F^ N. F o u n d a t i o n s o f b e h a v i o r a l r e s e a r c h . (2nd. e d . ) . New Y o r k , New Y o r k : H o l t B i n e h a r t and H i n s t o n , 1973. L i p p e y , G. Computer a s s i s t e d t e s t c o n s t r u c t i o n . E n g l e w o o d C l i f f s , New J e r s e y : E d u c a t i o n a l T e c h n i c a l P u b l i c a t i o n s , 1974. 55 L e k a n , H. A. (ED) I n d e x t o computer a s s i s t e d i n s t r u c t i o n . (2nd e d . ) . B o s t o n : S t e r l i n g I n s t i t u t e P u b l i c a t i o n s , 1970. L u s h e n e , B., 0 * N e i l , H. f . , 6 Dunn, T. E q u i v a l e n t v a l i d i t y o f a c o m p l e t e l y c o m p u t e r i z e d MBPI. T e c h memo number 48. F l o r i d a S t a t e U n i v e r s i t y , T a l l a h a s s e , 1972., (ERIC Document No. ED072657) Hagnusson, D. T e s t t h e o r y . R e a d i n g , M a s s a c h u s e t t s : A d d i s o n W e s e l y , 1967. N e l s o n , M. J - , Lamke, T., & K e l s o , P. C. The Henmon N e l s o n T e s t Of M e n t a l A b i l i t y , C o l l e g e L e v e l . (Rev. Ed.) New Y o r k : M i f f l i n , 1961. N i e , N. H., H u l l , C. H-, J e n k i n s , J . G., S t e i a b r e n n e r , K., & B e n t , D. H S t a t i s t i c a l P a ckage F o r The S o c i a l S c i e n c e s (2nd ed.) New Y o r k : McGraw H i l l , ~ 1 9 7 5 . H i n e r , B . . J . S t a t i s t i c a l p r i n c i p l e s i n e x p e r i m e n t a l d e s i g n (2nd e d ) . New Y o r k : McGraw H i l l , 1 9 7 U 56 A p p e n d i x A The f o l l o w i n g q u e s t i o n a i r e was u s e d t o c o l l e c t g e n e r a l i n f o r m a t i o n on e a c h o f t h e s u b j e c t s p a r t i c i p a t i n g i n t h e s t u d y -I t was d i s t r i b u t e d t o e a c h s u b j e c t p r i o r t o t h e b e g i n n i n g o f h i s p a r t i n t h e t e s t i n g . 57 S u b j e c t Number...---, f i me.-.-.-........... D a t e . . . . . . . . . . . . . . . . . 1. SEX: O M a l e O Female 2. AGE: O ~ 1 8 O 18-23 O 24-29 O 30+ 3. HftJOfi: O F i n e a r t s O H u m a n i t i e s O S o c i a l s c i e n c e s O S c i e n c e O P h y s i c a l E d u c a t i o n o r B e c r e a t i o n O O t h e r ( s p e c i f y - . . . . . . . - - - - . - . . - - . . . - . ) 4. A r e you e n r o l l e d i n t h e F a c u l t y o f E d u c a t i o n ? 0 No O E l e m e n t a r y O S e c o n d a r y 5. , B h i c h y e a r a r e you e n r o l l e d i n ? 01 02 0 3 Q> H O 5 O G r a d 6. Have y o u e v e r used a computer t e r m i n a l b e f o r e ? O * e s o N o I f y e s , where? O H i q n S c h o o l O U n i v e r s i t y O Work ( s p e c i f y - . . . - - . - . . . . - . . . . . ) O O t h e r ( s p e c i f y . . . , , , , . , . , . . , . . , , ) 58 A p p e n d i x B The f o l l o w i n g p a g e s r e p r e s e n t a s o u r c e l i s t i n g o f t h e computer program w h i c h was used t o a d m i n i s t e r t h e Henmon-Nelson T e s t Of M e n t a l A b i l i t y , l e v e l 4. I t i s w r i t t e n i n PL/1 and d e s i g n e d t o be used on t h e O n i v e r s t i y o f B r i t i s h C o l u m b i a * s Amdahl 460 V/6, Model I I com p u t e r . The program i s d e s i g n e d t o use o n l y t h e Ann A r b o r CRT t e r m i n a l s a t t h e U n i v e r s i t y o f B r i t i s h C o l u m b i a w h i c h a r e s u p p o r t e d by a PDP-11/40 m i n i computer a t t a c h e d t o t h e Amdahl 470. MAIN: FRCCECUKE CPTICNS ( K M K ) ; /* THIS PROGRAM PRESENTS AN INTELLIGENCE TEST TC /* GROLPS hE/SFE IS IN. A STUDENT BASED UPCN WHICH OF /* /* /* /* /* /* GROUP 1 TAKES TEST A BY COMPUTER t THEN TAKES TEST B MANUALLY. GRCUP 2 TAKES TEST B MANUALLY, THEN TAKES TEST A BY COMPUTER. GROUP i TAKES TEST B BY COMPUTER, THEN TAKES TEST A MANUALLY. GRCUP 4 TAKES TEST A MANUALLY, THEN TAKES TEST 6 BY COMPUTER. EACH STIOENT IS GIVEN A LNICUE I.O. BY THE EXPERIMENTER. THIS I .C. IS PECORDEC IN THE STUDENT F I L E . IF THE EXPERIMENTER ATTEMPTS TO SPECIFY AN FOUR */ */ */ */ */ */ */ / * l . D . WHICH HAS ALREADY BEEN ALLOCATED, THE PROGRAM PRCMPTS HIM FOP A NEW ONE*/ /* AS THE STUDENT TAKES THE TEST, HIS RESPONSES AND SOME FREQUENCY AND TIME */ /* DATA ARE RECORDED. WHEN THE TEST IS COKPLETEO OR TIME IS UP, THE TEST IS */ /* SCOREC ANC ALL COLLECTED DATA IS STORED IN THE STUDENT FILE AT MTS LINE */ /* l . D . * 100. */ /* RESPONSES TO THE MANUAL TEST ARE KEY-PUNCHED, THEN READ BY A SECOND PROGRAM */ /* WHICH ECITS THEM FOR ERRCRS, SCORES THEM AND STORES THEM IN THE STUDENT FILE*/ /* AT MTS LINE l . D . * 100 • 1. */ /* STLDENT FILE RECCROS: */ DCL 1 HEADER_PECCPC, 2 I_D_FLAG (3001 CHAR ( l i ; DCL 1 TESTEE.RECCRC, 5 I_D 5 GROCP 5 AN ShEK_OATA, 10 ANSWERS 10 ANSWER_FREC 1100) 5 SCORE_DATA, 10 NUMERIC.SCOPE 10 VER8Al_SCCRE BIN 10 TCT AL_S COR E BIN 5 PAGE_FREO (15) BIN 5 TIME_ON_PAGE (15) SIN 5 TO T AL_ TI WE BIN PIC "599 ' INIT (0 ) , CHAR (11, CHAR (100) INIT ( • • ) , BIN FIXED (15) IN IT 1(100)0), BIN FIXED (15) INIT (0 ) , FIXEC F IXEC FIXEO FLOAT FLCAT (15) ( 15) (15) (21) (21) INIT IN IT INIT INIT IN IT (0 ), ( C), ( (15101, ( (15)0 ) , (0) ; /* TEST FILE RECORDS: */ CCL I TEST_ INFO, 5 CORPECT.ANSWER (100) CHAR ( I I , 5 OUESTION.TYPE (ICO) CHAR I I ) , 5 L A ST_P A C- E PIC ' 9 9 ' ; CCL 1 PACE_INFO, 5 LINES_PER_PAGE 5 C_PER_PAGE 5 FIRS T_OL E S T PIC ' 9 9 « , PIC 'W, PIC « 9 9 9 « ; CCL TESTLINE CHAR (ICC) VARYING: /* FILE DECLARATIONS: */ CCL TESTFILE DCL STUDFILE FILE DIRECT KEYEO ENVIRONMENT (UI350I INDEXED! FILE DIRECT KEYED ENVIRONMENT (UI350) INDEXED( /* MISCELLANEOUS DECLARATIONS: */ CCL 1 PAGE_FCR, 5 HDR_FIL L_1 5 PAG E_NBR 5 HDR_FILL_2 5 HDR_F ILL_3 5 TIH E_R E M 5 HOR_FlLL_4 CCL FAGE_HDR_STRING CHAR (6) IN IT ( « PAGE ' It PIC « Z 9 « , CHAR (44 1 IN IT ( • • J , CHAR ( 2 2 » INIT ('TIME REMAINING: ABOUT •), PIC « Z 9 ' , CHAR (41 INIT (• KIN' I; ChAR (801 DEFINED PAGE_HDR; CCL ELAPSED SIGNCFF ENTRY RETURNS (BIN FLCAT), ENTRY ; DCL 1 ANSWER_SCALE, 5 ANS_SCALE_FILL 5 ANS_SCALE_QUEST (14) DCL ANSWER_SCALE_STRING CCL ANSWER (100) AN SWE R_ S T R INGl AN SW ER_CFAR 1 (37) *NSUER_STRIN02 AN SWER_CHAR 2 (37) PEVIEW_PEPLY CHANGE_PAGE_CHAR KEY STUDENT_KEY DCL FAGENO OUEST_NBR SLCT_NER CHAR (18) IN IT COOUESTION NUMBERS:'). P I C 'zzzz'; CHAR (74) DEFINED ANSWER_SCALE: CHAR (1) DEFINED ANSWERS, CHAR (37) , CHAR (1) DEFINED ANShER_STRINGI, CHAR (37) , CHAR (I) DEFINED ANSWER_STRING2, CHAR (10 1, CHAR (II , CHAR (4) E A S E D (PTR ) , CHAR (4) BASED I STUD ENT_PTRI; BIN F IXED (15) IN IT BIN FIXEC (15), BIN FIXEC 115); II), DCL LINE STUDENT LINE BIN BIN FIXED F IXED (31) , (31) ; DCL STARTT IME PAGE_ TIfE BIN FLOAT BIN FLCAT (21) , (21) ; /* SET UP FILE KEYS FCR REACS AND WRITES: */ P TR = ACCR(LINE); /* TESTFILE KEY */ STLOENT_PTR = ACCR(STUDENT_LINEJ ; /* STUCFILE KEY */ /* SET LP TERMINAL FCR CUTFUT */ CALL CNTL ('*S I NK*« , •FlSTCRY=0' ) ; CALL CNTL ( •*SINK*•,•ROLL • I ; CALL CNTL (•*S I NK*• ,•TABI = CFF; , 5 , 9 , 13 , 17,2 1, 25, 29, 33, 37,41 •); /* ATTACH AND CPEN OUTPUT F ILE : */ CALL INSTRUCTIONS; CALL CNTL ( '*S1NK*•,•L INE • ) ; CALL GET.TEST: STARTTIfE = ELAFSEC; /* PRESENT TEST TO SUBJECT: */ CO WHILE (ELAPSED - STARTTIME < 2400 £ PAGENO <= LA S T_PAGE): CALL 01SPLAY_FAGE(PAGENOI; CALL GET .ANSWERS(PAGENO» ; IF EL APS E C - STARTTIME < 2400 THEN CALL SELECT_FAGE; END; /* DO */ /* CALCULATE ANC SAVE TEST DATA AND CUIT: */ CALL CALC_SCCPES; /* SAVE STUCENT DATA: */ REWRITE FILE (STUDF ILE i FROM (HEACER_RECCRD ) KEY (STUDENT_KEY); STUDENT_L INE = I_D * ICOOOO: WRITE FILE (STUCFILEJ FRCP (TESTEE_RECORDI KEYFRCM (STUDEN T_K E Y) CALL TERMINATE; FILE DIRECT BIT( l ) , S I T U ) , B I T ( l » ; CHAR ( 1 ) , CHAR (3) IN IT (•152') , FIXEO BIN (15) ; INSTRUCTIONS: PRCC: DCL IN ST FIL E DCL ANSFLAG KFLAG GOF LAG CCL REPLY (3) <NSWERS NBR_WRONG CALL C NT L ( ' * S I N K * , , , L I N E ' I; CALL ATTACH! 'STLDFILE = STUDFILE' ) ; CPEN FILE (STUDFILE) UPDATE: STLDENT_LINE = 1C00; /* MTS LINE I V READ FILE (STUDFILE) INTO (HEADER_RECCRD) CALL ATTACHl>INSTFILE=TEST-INTRC•): OPEN FILE I INSTF ILE ) INPUT ; PAGENO=l; DO I = 1 TO 3; CALL D ISPLAY_ INSTR II) ; END ; ICFLAG = ' C ' B ; DC ViFILEI ICFLAG = • 0 ' B I ; CALL DI SPLAY_IN'STR( 1CCB) ; IF ANS^sER_CFAR2(l ) >= 'O ' ANS WER_CHAR 2(2) >= « C « ANS W ER_ CF AR 2 < 3) >= « C ' THEN I_C = SUeSTR(ANSWER_STRING2,1,3) GKCLP = ANS)«ER_ChAR2(4 J; IF I_D > 0 £ I_0 < 3C1 £ GROUP > 'O* THEN IF I_0_FLAG(I_D) = THEN DC; I_C_FLAG(I_ ICFLAG = • 1 £ N C ; IF ICFLAG = 'C 'B KEYED ENV IRONMENT(U(35 C) INDEXED); KEY (STUDENT_KEY); /* PAGE 4 */ £ ANSWER_CFAR2( 1) <= ' 9 " £ ANSWER_CHAR2(2) •9« £ ANSWER_ChAR2(3) <= « 9 ' £ GROUP < >5' • C . D ) B; 1' THEN CO; CALL C I S F C A Y _ I N S T R « 1 0 1 B ) ; /* PAGE 5*/ IF ANS l» E R_ CH AR 2 ( I ) >= '0* 6 AN S WER_ CHAR2( 1) <= » 9 ' 6 ANSWER_CHAR2(2) >= « 0 « £ ANSWER.CHAR2(2) <= ' 9 ' 6 ANSWER_CHAR2(3) >= ' O ' 6 ANSWER.CFAR2(3) <= ' 9 ' THEN I_D = SUBSTR(ANSWER_STRING2,1,3); GROUP = ANSWER_CHAP2<4); IF I_D > 0 £ I_C < 301 £ GROUP > ' C £ GROUP < •5' THEN IF I_0_FLAG( I_0) = ' C THEN DO; I_D_FLAG(I_OI = ' l ' ; IC FLAG = ' l ' B i END ; END; IF ICFLAG = •0 * B THEN CALL D I SPLAY_INSTR(11 OB); /* PAGE 6 • / END; CALL U I S F L A Y _ I N £ T R ( 1 1 1 E ) ; /* PAGE 7 •/ CCFLAG = ' O ' B ; DO WHILE (GCFLAG = « 0 ' E ) ; CALL OISPLAY_INSTR(1CCCB); /* PAGE 8 */ CALL C N T L ( » * S I N K * « , ' T A B l = O N ' I ; CALL OISPLAY_lNSTRl 1001BK /* PAGE 9 */ CALL C M L I « » S I M < * ' , ' T A E I = O F F « ); CO 1 = I TO 3; REPLY( I 1 = ANSUER_CHAR2(4*1-3) ; IF REPLY(I) < ' 1 ' I REPLY(I) > « 5 » THEN REFLY(I) = •_ • ; END ; ANSWER_CHAR211) = ' ' ; DO WHILE ( AN SWER_CH*R2 (I) -•= « P' ) ; CALL D I S P L A Y _ I N S T R ( I C 1 C B » ; /* PAGE 10 */ ENC; ANSFLAG = ' C ' B ; CC WhILE (ANSFLAG = ' O ' B ) : CALL DISPLAY_INSTR(1011B); /* PAGE 11 */ DISPLAY!' PREVIOUS ANSWERS: • I I RE FLY(1)I I• ' I I REPLY(2)I I• CALL CNTL( ••SINK*' , 'TABI = CN' ); DISPLAY!'EENTER ANY CHANGES: •) RE PLY!ANSWER_STRING2 ): CALL CNTL('*S I N K * ' T A B I=OFF') : NBR_WRONG = 3; CO I = 1 TC 3; IF ANSWER_CHAR2l4*I-3) = ' • THEN REPLY(I) = ANSWER_CHAR2(4*I-3); IF RE FLY (I ) < I REPLY! I ) > ' 5 ' THEN REPLYIH = IF REPLYIII = SUBSTR!ANSWERS, I, 1) THEN NBR_WRONG = NBR_WRONG END; IF NBR_ WRONG = C THEN ANSFLAG = ' i ' B ; ELSE CO; DO I = 1 TC 8; DISPLAY ( •- ' ) ; END; /* 24 BLANK LINES */ DISPLAY!' ' I INBR_WRONGI I• OF YGUR ANSWERS ARE INCORRECT • I ; CALL OISPLAY_INSTR< 1100BI; /* PAGE 12 */ IF AN S hER_ CHAR 2( II = ' N ' THEN ANSF LAG = • l 'B ; END; ENC ; ANSwER_CHAR2(1) = • •; CO WHILE ( ANSWEK_CFAR2< It ' T ' £ A N SWE R_CH AR2 (1) ' R ' ) ; CALL 01SFLAY_INSTR(1101B); / * PAGE 13 */ IF ANSWER_CHAR2( 1) = ' T ' THEN GQFLAG = ' l ' B ; ENC; END ; CC WHILE IANSWER_CFAR2( 1) -= ' N ' l i CALL 01 SFLAY_INSTR(lUOB); /* PAGE 14 */ ENC; D ISPLAY! ' - ' ) ; C ISP-LAY I ' I ; / * 6 BLANK LINES */ CLOSE FILE ( INSTF I LE ) ; CALL CNTL ( • « S I N K * • , • R C L L • I; D ISPL AY_ INST R : PRCC (P ) ; DCL F FIXED BIN (15) ; CCL 1 PAGE_REC , 5 NUKLINES PIC " 9 9 " , 5 TITLE_FILLER CHAR 18); LINE = pnooooo; READ FILE < INSTFILE) INTO (PAGE_REC) KEY (KEY); DO LINE = P*1CCCCC*1CCC TO (P*1CC+NUNLINES-1)*10C0 EY 1000; REAC FILE (INSTFILE) INTO (TESTLINE) KEY (KEY); DISPLAY (TESTLINE I ; ENC; IF P -= 11 THEN CO; LINE = (P*100 + NU^L INES)*1000: READ FILE (INSTFILE) INTO (TESTLINE) KEY (KEY) ; CISPLAY (TESTLINE) REPLY (ANSWER.STRING2 I; ENC; ENC CISPLAY_INSTR; END INSTRUCTIONS; GET_TEST: PROC ; CCL TEST.NAPE CHAR ( 2 0 ) ; IF GROUP < ' 3 ' THEN CALL ATTACH(•TESTF ILE=TEST.A' ) ; /+ GROUPS 1 £ 2 TAKE TEST A */ ELSE CALL ATTACH('TESTFILE = TEST.B ' I ; /* GROUPS 3 £ 4 TAKE TEST 8 */ OPEN FILE (TESTFILE) INFUT; /* REAC HEADER LINE FRGM TESTFILE: */ LINE = 100C; /* M S LINE 1 */ READ FILE (TESTFILE) INTO (TEST_ INFO) KEY (KEY); ENC GET_TEST; OA 0ISPLAY_FAGE: PFCC(P) : DCL F BIN F I X E C U 5 ) ; PAGE_ FR EG(P ) = PAGE_FREC(PI «• 1; PAGE_TIPE = ELAFSEC; PAG E_N BR = P; TIME.REM = TRUNCU24CC - (ELAPSED - ST A R T T IVE I ) / 6 0 » ; DISPLAY(PAGE.HOR.STR IN G ) ; /* READ PAGE HEADER FRCf TESTFILE: */ LINE = PAG ENO + 1CCOCC; /* HT S LINE = PAGE NC * 100 */ RE#0 FILE (TESTFILE) IMG (PAGE_ INFO I KEY (KEY): /* PRINT THE CUESTICNS: */ DC LINE = PAG ENC * 100COO 4 1000 TO PAGENO * 100COO • LINES_PER_PAGE * 1000 BY 1000; READ FILE (TESTFILE) INTO (TE STL INE J KEY (KEY)I DISPLAY(TESTLINE); ENC ; /• PRINT ThE ANSWER SCALE: */ CALL DISPLAY.ANSWER.SCALE; CALL CET.PREV_ANSWERS • IF PAGE.FREC(P) * 1 THEN CISPLAYC "I I ANSWER_STR ING1I; ELSE CISPLAY(' PREVIOUS ANSWERS: • I I A N S W E R . S T R I N G 1 » ; END DISPLAY_PAGE; D[SPLAY_ANSWER.SCALE: PRCC; /* FILL IN AND PRINT THE ANSWER SCALE: */ ANS_SCALE_CUEST = 0 : J = FIRST_CLEST; DO I = I TC C_PER_PAGE; ANS_SCALE_CLEST(I) * J ; J = J * l ; ENC; DISPLAY(AN SWER.SCALE.ST RING) ; ENC CISPLAY_ANSWER_SCALE ; GET_PREV_ANSWERS: PRCC; ANSwER.STRINGl = • •; DO I = 1 TO C_PER_PAGE; IF ANSWER(F IRST.QUEST•I-1 ) ANSWER.CHAR 1(( I-1>*4*1) ELSE ANSWER_CHAR1((I-l1*4*1) END; END GET.PREV.ANSWERS; GET.ANSw ER S : PRCC ( F ) ; DCL P BIN FIXEC (15) ; CALL CNTL ( • * S I NK * •, • T AS I =0N •) ; CFs IF PAGE.FREC(F) = 1 THEN CISPLAY ('CENTER YOUR ANSWERS:') REPLY (ANSWER.STRING2); ELSE DISPLAY ( 'CENTER CCFFECTICNS: ' ) REPLY (ANSWER.STR ING2 ); CALL CNTL ('*SINK*' , 'TAEI=CFF' ) ; PAGE.T l fE = ELAPSED - PAGE.TIME; TIME.ON.PAGE (P) = TINE.CN.PAGE(P) + PAGE.TIME; = • • THEN = '_*: = ANSwER(FIRST_QUEST+I-l>; DC I = 1 TC C_PER_PACE: SLC T_NBR = 4*1 - 3; CUEST_NBR = FIRST_CUEST + 1 - 1 ; IF ANSWER_CHAR2(SLCT_NBP) -= • • THEN DC; *NSWER_FR£Q(QUEST_N8F I = ANSWER_FREQ<CUEST_NBR) • 1; IF ANSWER_CFAR2!SLCT_NBR) < ' 1 ' I AN S W E P_CHAR2 < SLQT_NB R) > « 5 ' THEN ANSWER_CF/R2(SLQT_N8R) = • ' ; ANSWER(QLEST_NBR» = ANSWER_CHAR2 1SLCT_NBF); END ; END; END GET_ANSWERS ; SELECT_PAGE : PRCC; IF PAGENO < LAST_PAGE THEN DC ; IF PAGENC = 1 THEN CISPLAY!'CTYPE "N" TO SEE THE NEXT PAGE: ') REPLY!CHANGE_PAGE_CHAR); ELSE CI SPLAY(•6TY PE "N" TO SEE THE NEXT PAGE, OR " P " TO SEE THE PREVICUS PAGE AGAIN: •) REPLY ICHANGE_FAGE_CHAP» ; IF CHANG£_PAGE_CHAR = 'N• THEN PAGENO = PAG ENO •» 1; ELSE IF CHANG E_ PA C-E_ CHAR = ' P ' THEN PACENC = PACENC - 1; END : ELSE IF ELAPSED - STARTTIME < 2400 /* 40 MINUTES */ THEN CO; DISPLAY C t Y C U ARE FINISHED THE TEST. DO YOU WANT TO REVIEW? (YES,NO) •) REPLY (R EV IEW_R EPL V ); IF SUBSTR(REVIEW_REPLY,1,1) = 'N * THEN PAGENC = PAGENC • 1: ELSE PACENC = l ; END; ELSE DO; CISPL/Y (• YCU ARE FINISHED THE T E S T . ' ) ; PAGENO = PAGENC • 1; ENC ; END SELECT PAGE; CT« CALC_SCCPES: PRCC; TO TAL_ TI ME = ELAPSED - STARTTIME; CC I = 1 TC 100; IF ANSWER(I) = CCRRECT_ANSWER(It THEN CC; TOTAL.SCCPE = TCTAL_SCCRE • 1; IF QUEST IGN_TYPE( II = • V THEN VEReAL_ S C C P E = V EREAI SCORE + 1; ELSE NUM ER IC_SCORE = NUMERIC.SCCRE • 1; END; END ; ENC CALC_SCC,RES ; TERMNATE: FRCC; CLCSE FILE (TESTFILE! , FILE ISTUCFILE); DISPLAY (*-TFANKYCU FOR PARTICIPATING IN THIS PART OF THE EXPERIMENT.' DISPLAY ( • OPLEA SE SEE THE ATTENDANT FCR FLRTFER INSTRUCTIONS.'): CALL SIGNOFF; END TERMINATE; END MAIN; $SIG 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            data-media="{[{embed.selectedMedia}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.831.1-0094815/manifest

Comment

Related Items