UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Conversational computer terminals in psychological testing Longbottom, Ronald Arthur 1974

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.

Item Metadata

Download

Media
831-UBC_1974_A4_5 L65.pdf [ 4.15MB ]
Metadata
JSON: 831-1.0092983.json
JSON-LD: 831-1.0092983-ld.json
RDF/XML (Pretty): 831-1.0092983-rdf.xml
RDF/JSON: 831-1.0092983-rdf.json
Turtle: 831-1.0092983-turtle.txt
N-Triples: 831-1.0092983-rdf-ntriples.txt
Original Record: 831-1.0092983-source.json
Full Text
831-1.0092983-fulltext.txt
Citation
831-1.0092983.ris

Full Text

CONVERSATIONAL COMPUTER TERMINALS IN PSYCHOLOGICAL TESTING by RONALD ARTHUR LONGBOTTOM B.Com., University of B r i t i s h Columbia, 1969 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF BUSINESS ADMINISTRATION i n the Faculty of Commerce and Business Administration We accept t h i s thesis as conforming to the required standard THE UNIVERSITY OF BRITISH COLUMBIA A p r i l , 1974 In p r e s e n t i n g t h i s t h e s i s i n p a r t i a l f u l f i l m e n t o f the requirements f o r an advanced degree a t the U n i v e r s i t y o f B r i t i s h Columbia, I agree t h a t the L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r r e f e r e n c e and study. I f u r t h e r agree t h a t p e r m i s s i o n f o r e x t e n s i v e c o p y i n g o f t h i s t h e s i s f o r s c h o l a r l y purposes may be g r a n t e d by the Head o f my Department or by h i s r e p r e s e n t a t i v e s . I t i s understood t h a t c o p y i n g or p u b l i c a t i o n o f t h i s t h e s i s f o r f i n a n c i a l g a i n s h a l l not be a llowed without my w r i t t e n p e r m i s s i o n . Department o f The U n i v e r s i t y o f B r i t i s h Columbia Vancouver 8, Canada ABSTRACT The concept of CCTT (conversational computer terminal testing) i s investigatedas a p r a c t i c a l and advantageous solution to the problem of administrating a s t a t i s t i c a l l y -complex testing model, which w i l l minimize the time losses and delays of conventional psychological t e s t i n g . In the process of personnel selection, larger organiz-ations often i n v i t e certain of t h e i r applicants to undergo psychological t e s t i n g . This usually involves inconvenience for the applicant, additional expense fo r the organization, and time losses f o r both. The testing procedure w i l l t y p i -c a l l y be of a fix e d treatment, group testing nature whereas for many applicants an adaptive testing procedure would be more appropriate. Although testing mechanisms fo r adaptive treatments exist, they tend to be awkward and lim i t e d . More sophisticated t h e o r e t i c a l models have been developed but tend to lack appropriate means of administration. In t h i s investigation a Fortran IV CCTT probability/seq-uent i a l model i s developed and comparisons are made between CCTT and conventional test administration of verbal and numerical analogy tests to 180 subjects. The CCTT model presents an applicant with a test item, scores the response, and s t a t i s t i c a l l y computes and predicts group membership within pre-definable accuracy l e v e l s . The three possible c l a s s i f i c a t i o n s and a s s o c i a t e d t r e a t m e n t s a r e : h i g h s c o r e r , a d m i n i s t e r next t e s t ; low s c o r e r , t e r m i n a t e t e s t i n g ; u n c l a s s -i f i e d , c o n t i n u e p r e s e n t t e s t . F i n d i n g s o f t h e i n v e s t i g a t i o n i n d i c a t e t h a t t h e r e a r e no s i g n i f i c a n t d i f f e r e n c e s i n o v e r a l l t e s t s c o r e s between the two methods o f t e s t a d m i n i s t r a t i o n . The CCTT model i s a b l e t o c l a s s i f y i n d i v i d u a l s w i t h an a c c u r a c y l e v e l e x c e e d i n g $0% on bo v e r b a l and n u m e r i c a l t e s t s a f t e r o n l y 25% o f the t e s t i t e m s have been a d m i n i s t e r e d , w i t h a r e s u l t a n t r e d u c t i o n i n t e s t i n g t i me o f 65%. A d d i t i o n a l l y , CCTT o f f e r s t h e b e n e f i t s o f i n d i v i d u a l i z e d demand t e s t i n g , a u t o m a t i c s c o r i n g and r e c o r d i n g , complex and t a i l o r e d d e c i s i o n b r a n c h i n g , and immediate i n t e r -p r e t a t i o n o f r e s u l t s . S u g g e s t i o n s f o r f u r t h e r r e s e a r c h i n t h i s and a s s o c i a t e d a r e a s a r e p r o v i d e d . TABLE OF CONTENTS Page LIST OF TABLES v i i CHAPTER I INTRODUCTION 1 The Personnel Selection Process 1 The Problem 2 Proposed Solution 6 II REVIEW OF THE PERTINENT LITERATURE 9 S t a t i s t i c a l Decision Theory 9 Application to Psychological Testing 10 III POSSIBLE COMPUTERIZED TESTING MODELS 14 Models 14 Probability/sequential single stage 14 Probabili t y multi-stage 15 Branching multi-stage. 15 Complexity/branching 16 Summary 16 IV COMPUTERIZED DEMONSTRATION MODEL 17 Purpose 17 Limitations 17 Design 18 C l a s s i f i c a t i o n procedure 18 V Page Psychological tests 20 Computer model 21 Discussion 24 V APPLICATION OF THE MODEL 26 Sample 26 Methodology .: 26 Results and interpretation 30 Test order 30 Learning effects 30 Test administration 32 Scores 32 Times 35 Items 35 Accuracy 37 Costs 37 Anxiety 37 VI CONCLUSIONS AND SUGGESTIONS FOR FURTHER RESEARCH 39 Discussion 39 Advantages 39 Disadvantages 40 Limitations 41 Conclusions 41 Suggestions f o r further research 42 Summary 45 BIBLIOGRAPHY 4-6 v i APPENDICES Page APPENDIX A - Materials used i n Paper Test Administration 50 Shortened Verbal Form Minnesota Multimodel Analogy Test - 1971 Revision 51 Shortened Numerical Form Minnesota Multimodal Analogy Test - 1971 57 Answer Sheet f o r Paper Tests 63 APPENDIX B - Fortran IV CCTT Model... 65 APPENDIX C - Printed Instructions f o r 'sign-on' to CCTT 70 APPENDIX D - Computer Verbal and Numerical Analogy Tests 72 APPENDIX E - V a l i d Student Numbers and Section - P a r t i a l Computer F i l e 77 APPENDIX F - Subject's Test Record - P a r t i a l Computer F i l e 79 APPENDIX G - Relative Cost Comparison of CCTT and Paper Testing 82 LIST OF TABLES TABLE PAGE I . T e s t Item P r o b a b i l i t i e s - C a l i b r a t i o n Sample.... 28 I I . Comparison o f T e s t i n g Order S c o r e s - CCTT and Paper T e s t i n g 31 I I I . Comparison o f T e s t S c o r e s - CCTT and Paper T e s t i n g 33 IV. Comparison o f T e s t Times - CCTT and Paper T e s t i n g 34 V. Time and Number o f Item R e q u i r e d f o r CCTT t o C l a s s i f y S u b j e c t s 36 V I . CCTT S u b j e c t C l a s s i f i c a t i o n T a b l e s 38 CHAPTER I INTRODUCTION • I THE PERSONNEL SELECTION PROCESS T y p i c a l l y , t h e p e r s o n n e l s e l e c t i o n p r o c e s s o f today i n v o l v e s a sequence o f events w h i c h an a p p l i c a n t must s u r v i v e i f he i s t o be s u c c e s s f u l i n h i s b i d f o r a p a r t -i c u l a r p o s i t i o n . A l t h o u g h t h e o r d e r , e x t e n t and emphasis of t h e s e events may v a r y , t h e r e a r e s e v e r a l t h a t can be c o n s i d e r e d r e p r e s e n t a t i v e o f t h e medium t o l a r g e s i z e d o r g a n i z a t i o n . These a r e the a p p l i c a t i o n b l a n k , p e r s o n a l i n t e r v i e w s , p s y c h o l o g i c a l t e s t s , r e f e r e n c e c h e c k s , and m e d i c a l e x a m i n a t i o n . What each o f t h e s e s e l e c t i o n t o o l s i s u l t i m a t e l y a t t e m p t i n g t o a s s i s t i n i s the p r e d i c t i o n o f an a p p l i c a n t ' s performance o r p o t e n t i a l f o r performance w i t h i n a g i v e n t a s k range. F a i l u r e t o pass any one o f t h e s e o b s t a c l e s w i l l u s u a l l y mean e l i m i n a t i o n f r om t h e b a l a n c e o f the s e l e c t i o n p r o c e s s , p a r t i c u l a r l y where a l a r g e number o f o t h e r a p p l i c a n t s a r e a v a i l a b l e . Many v a r i a t i o n s o f t h i s t y p i c a l p r o c e s s e x i s t but most a r e s e q u e n t i a l w i t h fewer a p p l i c a n t s b e i n g c o n s i d e r e d a t each subsequent s t a g e . 2 I I THE PROBLEM The p e r s o n n e l s e l e c t i o n p r o c e s s i s a t i m e consuming and c o s t l y p r o c e d u r e both from the p o i n t o f view o f the s e l e c t i n g o r g a n i z a t i o n and of the a p p l i c a n t . When t h e s e p r o c e s s e s a r e conducted i n a s e q u e n t i a l , s t e p by s t e p manner, v a r y i n g time l a p s e s w i l l u s u a l l y o c c u r between each s t e p . F o r example, when an a p p l i c a n t completes and submits an a p p l i c a t i o n he i s not n o r m a l l y g i v e n an immediate i n t e r v i e w but r a t h e r i s r e q u i r e d t o a w a i t a p o s s i b l e i n v i t a t i o n f o r one. Once i n t e r v i e w e d he may next be i n v i t e d t o complete a few p s y c h o l o g i c a l t e s t s but t h i s o f t e n i n v o l v e s a r e t u r n t r i p f o r a s c h e d u l e d group t e s t i n g s e s s i o n . A f t e r t e s t i n g i t can be a n o t h e r day o r two b e f o r e the r e s u l t s have been s c o r e d and i n t e r p r e t e d and the s e i n t u r n may l e a d t o a f u r t h e r i n t e r v i e w , r e f e r e n c e c h e c k i n g , and u l t i m a t e l y a job o f f e r ( u s u a l l y s u b j e c t t o p a s s i n g a m e d i c a l ) . E l i m i n a t i o n o f the a p p l i c a n t may come a f t e r any o f the above s t e p s w i t h an e a r l y r e j e c t i o n f a v o r a b l e t o b o t h p a r t i e s . From t h e o r g a n i z a t i o n ' s p o i n t o f view, each a d d i t i o n a l s t e p l e a d i n g t o an u l t i m a t e r e j e c t i o n o n l y adds t o t h e c o s t o f f i n a l s e l e c t i o n . Time l o s t between s t e p s a l s o t e n d s t o reduce the p r o b a b i l i t y t h a t the a p p l i c a n t w i l l remain a v a i l a b l e f o r h i r e , p a r t i c u l a r l y i f he i s a c t i v e l y s e e k i n g employment e l s e w h e r e . From the a p p l i c a n t ' s i d e , e a r l y r e j e c t i o n e l i m i n a t e s f u r t h e r and u n n e c e s s a r y consumption of h i s t i m e . T h i s i s o f p a r t i c u l a r i m p o r t a n c e when he i s c u r r e n t l y employed and must t a k e time o f f f o r i n t e r v i e w s and t e s t s . Of the a c t u a l s t e p s i n the above s e l e c t i o n p r o c e s s p s y c h o l o g i c a l t e s t i n g i s perhaps the most time consuming and u n c o m f o r t a b l e f o r t h e job a p p l i c a n t . The a p p l i c a t i o n b l a n k m e r e l y r e q u e s t s b i o g r a p h i c a l d a t a and about f i f t e e n minutes o f the a p p l i c a n t ' s t i m e . The i n t e r v i e w can v a r y but g e n e r a l l y d u p l i c a t e s a p p l i c a t i o n d a t a t o some e x t e n t , t a k e s f i f t e e n t o f i f t y m i n u t e s , and p r o v i d e s the a p p l i c a n t w i t h some i n f o r m a t i o n i n r e t u r n . R e f e r e n c e c h e c k i n g i s n o r m a l l y t r a n s p a r e n t t o t h e a p p l i c a n t and m e d i c a l s a r e u s u a l l y o n l y g i v e n once a c o n d i t i o n a l o f f e r o f employment has been extended. P s y c h o l o g i c a l t e s t s on the otherhand may r e q u i r e f r om o n e - h a l f t o f i v e o r s i x hours o f what many c o n s i d e r p r o b i n g , p r e s s u r e d , and even e x h a u s t i v e e x e r c i s e . O f t e n when t e s t r e s u l t s a r e f i n a l l y a n a l y z e d i t becomes e v i d e n t t h a t f u r t h e r t e s t i n g i n a s p e c i f i c a r e a would have been d e s i r a b l e o r t h a t c e r t a i n t e s t s were not r e a l l y r e q u i r e d . I t i s not uncommon f o r an a p p l i c a n t t o 4 be e l i m i n a t e d on the b a s i s o f s p e c i f i c t e s t s ( e g . a b i l i t i e s ) but because t h e s e t e s t s cannot be i m m e d i a t e l y s c o r e d and i n t e r p r e t e d t h e r e m a i n i n g t e s t s must a l s o be completed. F u r t h e r m o r e , many c o n v e n t i o n a l p s y c h o l o g i c a l t e s t s t e n d t o be somewhat i n e f f i c i e n t as t h e y a r e d e s i g n e d t o measure a g i v e n t r a i t a c r o s s a wide range o f l e v e l s . T h i s l e a d s t o h i g h e r - s c o r i n g i n d i v i d u a l s w a s t i n g time a n s w e r i n g easy i t e m s w h i l e l o w - s c o r e r s may guess a t d i f f i c u l t i t e m s . S e v e r a l t h e o r e t i c a l o r p a r t i a l s o l u t i o n s t o t h e s e problems have e x i s t e d f o r some t i m e . Among them a r e t h e t e c h n i q u e s o f 'programmed' and ' m u l t i - s t a g e ' t e s t i n g . W i t h programmed t e s t s the s u b j e c t i s g u i d e d t o i t e m s o f an a p p r o p r i a t e d i f f i c u l t y l e v e l t h r o u g h a system o f s e q u e n t i a l b r a n c h i n g . I f he g e t s an i t e m r i g h t he proceeds t o a more d i f f i c u l t one and i f wrong, t o an e a s i e r one. T y p i c a l l y , the range o f d i f f i c u l t y between s u c c e s s i v e i t e m s n o r m a l l y d e c r e a s e s . I n m u l t i - s t a g e t e s t i n g , a d m i n i s t r a t i o n and c o n t e n t o f s u c c e s s i v e t e s t s i s d e t e r m i n e d by performance on p r e c e d i n g t e s t s . T h i s a l l o w s subsequent t e s t s t o be matched t o t h e s u b j e c t ' s a b i l i t y l e v e l . The d i f f i c u l t i e s w i t h t h e s e and o t h e r s o l u t i o n s have been i n t h e a c t u a l a d m i n i s t r a t i o n o f t h e t e s t s . L o r d (1971a) s u g g e s t s a cumbersome p a p e r - a n d - p e n c i l p r o c e d u r e f o r two-stage t e s t i n g . The answer sheet f o r t h e f i r s t t e s t i s * 5 completed i n d u p l i c a t e w i t h t h e o r i g i n a l b e i n g immediately-c o l l e c t e d f o r l a t e r s c o r i n g as the o f f i c a l r e c o r d o f p e r -formance. The d u p l i c a t e i s s c o r e d by the examinee a c c o r d i n g t o i n s t r u c t i o n s p r o v i d e d by t h e examiner and t h i s s c o r e d e t e r m i n e s t h e second-stage t e s t t o be a d m i n i s t e r e d . Hubbard (1963), e n c o u n t e r i n g problems i n programmed t e s t a d m i n i s t r a t i o n d e v e l o p e d a r a t h e r unique s o l u t i o n . The answer sheet c o n s i s t e d o f a s e r i e s o f i n k e d b l o c k s , each numbered t o c o r r e s p o n d t o the p o s s i b l e answers. When the examinee made h i s c h o i c e he s i m p l y e r a s e d the appro-p r i a t e i n k b l o c k r e v e a l i n g t h e number o f t h e next t e s t q u e s t i o n t o be a t t e m p t e d . A n n e t t (1964) d e v e l o p e d a s i m i l a r s o l u t i o n u t i l i z i n g a g r i d d e v i c e w i t h numbers em-bossed onto a m e t a l o r p l a s t i c m a t r i x . The answer sheet was p a s t e d over t h e g r i d and the examinee rubbed a s o f t p e n c i l o v e r h i s chosen answer, thus e x p o s i n g the h i d d e n number d i r e c t i n g him t o the next t e s t i t e m . Such methods have p r o v i d e d l i m i t e d means of t a i l o r i n g and s h o r t e n i n g p s y c h o l o g i c a l t e s t s . What i s s t i l l r e q u i r e d however, i s a form o f demand t e s t i n g not r e q u i r i n g t h e s e r v i c e s o f a f u l l time t e s t a d m i n i s t r a t o r and p s y c h o l o g i s t , yet c a p a b l e o f i n s t a n t a n e o u s s c o r i n g , t a i l o r i n g , i n t e r p r e t -a t i o n , and feedback w h i l e s h o r t e n i n g t h e t e s t s . A d d i t i o n a l l y , such t e s t i n g s h o u l d be c a p a b l e o f a p p l y i n g t h e more s o p h i s t -6 i c a t e d s t a t i s t i c a l d e c i s i o n models t h a t a r e a v a i l a b l e t o d a y . I l l PROPOSED SOLUTION I n t h e f i e l d o f p s y c h o l o g i c a l t e s t s , computers have been put t o work i n such a r e a s as s c o r i n g , i n t e r p r e t i n g , v a l i d -a t i n g , and even d e v e l o p i n g t e s t s . L i m i t e d use however, has been made of the computer as a t e s t a d m i n i s t r a t o r . I t i s proposed t h a t t h rough t h e use o f c o n v e r s a t i o n a l computer t e r m i n a l t e s t i n g (CCTT) many o f t h e p r e c e d i n g problems c o u l d be s o l v e d . CCTT i s a p r o c e s s w h i c h p e r m i t s demand t e s t i n g w i t h t h e f a c i l i t y t o make immediate d e c i s i o n s , t o br a n c h t o o t h e r t e s t s where r e q u i r e d and t o p r o v i d e i n s t a n t a n e o u s t e s t i n t e r p r e t a t i o n and feedback. CCTT can e l i m i n a t e the time l a p s e s on e i t h e r s i d e of p s y c h o l o g i c a l t e s t i n g i n t h e p e r s o n n e l environment and appears t o have the c a p a c i t y t o s h o r t e n t e s t s and enhance t h e i r q u a l i t y . Under the concept o f CCTT, when an a p p l i c a n t i s i d e n t i f i e d f o r p s y c h o l o g i c a l t e s t i n g ( u s u a l l y d u r i n g t h e i n t e r v i e w ) he i s s i m p l y s e a t e d a t a OCT w i t h a few i n s t r u c t -i o n s ( v e r b a l , p r i n t e d o r d i s p l a y e d on t h e t e r m i n a l ) and the computer does t h e r e s t . CCTT w i l l t e a c h t h e a p p l i c a n t how to o p e r a t e the t e r m i n a l i n a m a t t e r o f seconds and w i l l 7 then administer the appropriate t e s t s . The possible te s t i n g models are v i r t u a l l y l i m i t l e s s as i s discussed i n Chapter three. As the terminal administers the test items the computer scores and analyzes each response. It constantly monitors performance, deciding whether further t e s t i n g i s required and i f so, i n what areas. Thus testing i s t a i l o r e d f o r the applicant as he proceeds and i t can be terminated as soon as s u f f i c i e n t decision information has been coll e c t e d . Even while the applicant i s s t i l l working at the CCT a complete test evaluation can be printed at a remote terminal. It i s conceivable that a l o g i c a l extension of the CCTT concept would be the development of a CCTS/P (Conversational Computer Terminal Selection/Placement) model. Such an integrated and comprehensive model could be used to c o l l e c t biographical and other data normally gathered from the application blank, interviews, testing, and even pre-medical. While attempting to decide i f an applicant was acceptable f o r employment the model would also be determining the most suitable areas of placement and whether such vacancies existed or were l i k e l y to i n the near future. This paper i s , however, confined to the development and testing of a single CCTT model. U t i l i z i n g one type of test i t primarily explores whether s i g n i f i c a n t differences i n performance occur between paper and computer adminis-trations and whether s i g n i f i c a n t reductions i n tes t i n g time can be re a l i z e d using CCTT. Relative costs of CCTT are also examined. CHAPTER I I REVIEW OF THE PERTINENT LITERATURE I STATISTICAL DECISION THEORY S t a t i s t i c a l d e c i s i o n t h e o r y , o r g i n a t e d by Abraham Wald i n the l a t e 1930's ( G i r s h i c k , 1954), i s s t i l l a r e l a t i v e l y young and i n c o m p l e t e s c i e n c e . Wald (1950) i n v e s t i g a t e d problems o f i n s p e c t i o n and q u a l i t y c o n t r o l where e i t h e r an ac c e p t a n c e o r r e j e c t i o n d e c i s i o n was r e q u i r e d . C l e a r l y , the p e r s o n n e l s e l e c t i o n p r o c e s s o u t l i n e d i n Chapter I r e q u i r e s a s i m i l a r u l t i m a t e d e c i s i o n and one o f t h e mechanisms by w h i c h t h a t d e c i s i o n may be reached i s p s y c h o l o g i c a l t e s t i n g . I n a r r i v i n g a t t h i s u l t i m a t e d e c i s i o n , each o f the above p r o c e s s e s may u t i l i z e a t h i r d a l t e r n a t i v e when the d e c i s i o n t o a c c e p t o r r e j e c t i s not c l e a r - c u t , t h a t i s , t o c o n t i n u e t h e p r o c e s s by c o l l e c t i n g more i n f o r m a t i o n . T y p i c a l l y t h i s i s what happens i n the o v e r a l l s e q u e n t i a l s e l e c t i o n p r o c e s s but not i n t h e t e s t i n g segment. Here, a p r e - d e t e r m i n e d amount o f t e s t d a t a i s f i r s t c o l l e c t e d and a n a l y z e d , and then an a c c e p t / r e j e c t d e c i s i o n i s made. T h i s i s i n e f f i c i e n t , however, as u s u a l l y more th a n the r e q u i r e d amount o f i n f o r m a t i o n t o r e a c h a d e c i s i o n i s c o l l e c t e d o r o c c a s i o n a l l y i t i s d i s c o v e r e d , a f t e r the f a c t , t h a t s u f f i c i e n t 10 test information was not gathered. Thus exists the case f o r sequential t e s t i n g strategies. A strategy must state what the decision maker w i l l do i n any possible contingency (Girshick, 1954, p. 464) a n u -normally consists of a set of conditional p r o b a b i l i t i e s , (Cronbach and Gleser, 1965, p. 19) each between 0 and 1. Although p r o b a b i l i t i e s may be c l a s s i f i e d as either objective or subjective (Levin and Kirkpatrick, 1965), the concern here i s with the former. These are referred to as assigned p r o b a b i l i t i e s that are supported by h i s t o r i c a l evidence and common experience, or are self-evident from the physical attribute of the system. They indicate the chance of an event happening. II APPLICATION TO PSYCHOLOGICAL TESTING Sequential testing One form of sequential t e s t i n g i s referred to as programmed tes t i n g . This involves a sequential system of branching which guides the subject to test items of an appropriate d i f f i c u l t y l e v e l . Cleary, Linn, and Rock (1968, p. 183) indicate that programmed tests have the capacity to s i g n i f i c a n t l y reduce te s t i n g time required f o r a b i l i t y assessment, to reduce the component of unreliable variance due to random guessing, and to increase the accuracy of 11 measurement near cut-offs by having subjects respond to additio n a l items. In i t s simplest form a programmed test consists of a routing section where the necessary branching takes place to direct the subject to the appropriate l e v e l of test items and a measurement section which concentrates these items. Lord (1968) has appropriately l a b e l l e d t h i s form of sequential t e s t i n g ' t a i l o r e d - t e s t i n g ' where an attempt i s made to ' t a i l o r ' the d i f f i c u l t y of the test items to the ' a b i l i t y ' of the i n d i v i d u a l being tested. He indicates (Lord, 1971 jb,p. k) that considerable work remains to be done i n determining what the i n i t i a l d i f f i c u l t y l e v e l should be, how much the d i f f i c u l t y l e v e l should change a f t e r any right or wrong answer, how responses should be scored, and how the effectiveness of the possible procedures should be compared. A ' l e s s - t a i l o r e d ' form of t a i l o r e d - t e s t i n g i s sequential two-stage testing. This procedure c a l l s f o r a routing test followed by one of several alternative second-stage convent-ional tests. Cronbach and Gleser (1965, Chapter 6) explore two-stage testing as a sequential decision theory technique whereby examinees may be c l a s s i f i e d a f t e r the f i r s t test into accept, r e j e c t , or continue testing categories. The main advantages of t h i s procedure are a savings i n testing time and elimination of unnecessary tests. 12 A sequential model that has been shown theoretically-capable of reducing the number of test items by one-half i s a p r o b a b i l i t y approach by Linn, Rock and Cleary (1972). This type of approach has, however, lacked an appropriate mechanism for i t s administration. CCTT appears to provide t h i s mechanism and thus a sequential p r o b a b i l i t y model i s explored more f u l l y i n subsequent chapters. P a i t i c h (1973) presents a computer program capable of generating automatic psychological reports f o r a battery of measures of i n t e l l i g e n c e and personality. The various tests are interpreted by rather simple procedures that involve p r i n t i n g given sentences or short paragraphs that correspond to scores f a l l i n g within a specified range. Hedl, O'Neil and Hansen (1971a) have developed an automated administration and scoring program for the Slosson Intelligence Test (Slosson, 1963). A considerable amount of work has also been done on computerized int e r p r e t a t i o n of the Minnesota Multiphasic Personality Inventory (MMPI) (Hansen, Hedl and O'Neil, 1971), but only recently has an attempt been made to computerize i t s administration (Dunn, Lushene and O'Neil, 1972). I t was found that the test could be successfully administered and scored i n t h i s manner although the study i t s e l f was mainly concerned with latency analysis. Waters (1970) i n a com-parison of computer-simulated branching and conventional 13 tests of varied lengths found branching tests had higher cor r e l a t i o n with underlying a b i l i t y than did any of the conventional t e s t s . Gedye and M i l l e r (1969, pp. 253-259) indicated automated tes t i n g i s probably j u s t i f i e d on economic grounds and also found subjects of a l l ages f a v o r a l l y disposed to automated t e s t i n g . Elwood (1972) found automated administration of the Wechsler Adult I n t e l l -igence Scale (WAIS) was indeed cheaper than face-to-face administration by a factor of one-half. Hedl, O'Neil and Hansen (1971b) have, however, found that computer tes t i n g procedures can lead to higher l e v e l s of anxiety. Several of the above findings have resulted from computer-assisted i n s t r u c t i o n (CAI) applicationsbut most have been generated by a need to improve selection and t e s t i n g procedures. Other i d e n t i f i e d horizons f o r related CCT applications are i n the f i e l d s of counselling (Veldman and Menaker, 1968; E l l i s and Tiedeman, 1970), and medical history and diagnosis (Edwards, 1968; Mayne, Weksel and Sholtz, 1968). CHAPTER III POSSIBLE COMPUTERIZED TESTING MODELS In the preceding discussion of s t a t i s t i c a l decision theory, three main classes of decisions were presented. These were sequential two-stage, prob a b i l i t y , and branching decisions. The present chapter i s intended to indicate some of the possible CCTT models that could r e s u l t from such decision patterns. I MODELS Single stage The t y p i c a l t e s t i n g model i n the f i e l d of personnel selection today i s one of fixed-treatment. The i n d i v i d u a l completes a series of tests, they are analyzed and a decision i s made to hire or not h i r e . As discussed i n Chapter I t h i s model i s i n e f f i c i e n t f o r several reasons. To computerize i t would be of minimal benefit. Sequential multi-stage In sequential multi-stage testing a decision i s possible and normally expected a f t e r each stage. T y p i c a l l y t h i s decision i s to either suspend testing or to continue and i s based on test performance to that point with fewer in d i v i d u a l s being tested at each stage. Simple cut-off scores can exist 15 f o r each t e s t and c u m u l a t i v e s c o r e s a r e a l s o p o s s i b l e . A l t h o u g h t h e two s t a g e v e r s i o n o f t h i s model i s sometimes a p p l i e d i n p s y c h o l o g i c a l t e s t i n g , i t i s awkward, and m u l t i -s t a g e a p p l i c a t i o n s o f i t a r e v e r y much more so. A l t h o u g h such a model i s r e a d i l y c o m p u t e r i z e d i t does l a c k some e f f i c i e n c y i n t h a t e n t i r e s t a g e s o f t h e t e s t must s t i l l be g i v e n . P r o b a b i l i t y / s e q u e n t i a l s i n g l e s t a g e T h i s model i s based on p r o b a b i l i t y o f group membership and i s v i r t u a l l y i m p o s s i b l e t o a d m i n i s t e r i n a c o n v e n t i o n a l s e t t i n g , due to the m o d e r a t e l y complex s t a t i s t i c a l c a l c u l a t i o n s r e q u i r e d a f t e r each r e s p o n s e . I t i s on the o t h e r hand v e r y r e a d i l y c o m p u t e r i z e d and t h e o r e t i c a l l y c a p a b l e o f a 50% r e d u c t i o n i n the number o f t e s t i t e m s r e q u i r e d as r e f e r e n c e d i n t h e p r e v i o u s c h a p t e r . The d e c i s i o n r eached by t h i s model i s dichotomous ( a c c e p t o r r e j e c t ) . P r o b a b i l i t y m u l t - s t a g e T h i s i s i d e n t i c a l t o the p r e c e d i n g s i n g l e s t a g e model ex c e p t t h a t a t h i r d a l t e r n a t i v e d e c i s i o n i s now p o s s i b l e and t h a t i s t o c o n t i n u e t e s t i n g w i t h t h e next s t a g e . T h i s t y p e o f model i s d e v e l o p e d i n C h a p t e r IV. B r a n c h i n g m u l t i - s t a g e The b r a n c h i n g m u l t i - s t a g e model adds two new d e c i s i o n 16 d i m e n s i o n s i n t h a t i t d e t e r m i n e s w h i c h t e s t o r item(§) w i l l be a d m i n i s t e r e d n e x t . Thus i t i s capable, o f t a i l o r i n g t e s t s w i t h i n and a c r o s s s u b j e c t a r e a s . C o m p l e x i t y / b r a n c h i n g T h i s t y p e o f model i n v o l v e s b r a n c h i n g t o o t h e r i t e m s , t e s t s o r t r e a t m e n t s based on what c o u l d be complex s t a t i s t i c a l c a l c u l a t i o n s r e s u l t i n g from answer p a t t e r n s , a s s o c i a t e d l a t e n c i e s , p r e v i o u s performance, b i o g r a p h i c a l and t e s t d a t a , i n t e r v i e w r e s e r v a t i o n s and so on. C o n s i d e r a b l e d e v e l o p m e n t a l work w i l l be r e q u i r e d b e f o r e t h i s t y pe o f model becomes a r e a l i t y . I t b e g i n s t o encompass a n a l y s i s o f d a t a c o l l e c t e d i n o t h e r a s p e c t s o f the s e l e c t i o n p r o c e s s and a t t e m p t s t o d u p l i c a t e o r make t h e P e r s o n n e l Manager's s e l e c t i o n f o r him. I I SUMMARY The above models were b r i e f l y d e s c r i b e d t o i l l u s t r a t e t h e apparent c a p a b i l i t i e s o f CCTT. C o n s i d e r a b l y more r e s e a r c h w i l l be r e q u i r e d on t h e concept t o examine i t s f e a s i b i l i t y , a p p l i c a b i l i t y , and a c c e p t a b i l i t y . CHAPTER IV COMPUTERIZED DEMONSTRATION MODEL I PURPOSE A m u l t i - s t a g e p r o b a b i l i t y / s e q u e n t i a l c o m p u t e r i z e d t e s t i n g model was de v e l o p e d f o r s e v e r a l purposes, namely: - t o demonstrate t h a t p s y c h o l o g i c a l t e s t s w i t h a m u l t i p l e - c h o i c e f o r m a t c o u l d r e a d i l y be a d m i n i s t e r e d , s c o r e d , i n t e r p r e t e d , and r e c o r d e d by means o f CCTT; - t o i n v e s t i g a t e what e f f e c t s i f any CCTT would have on o v e r a l l t e s t performance; - t o determine i f s i g n i f i c a n t t i m e s a v i n g s i n t e s t a d m i n i s t r a t i o n c o u l d be r e a l i z e d t h r o u g h use o f a p r o b a b i l i t y / s e q u e n t i a l CCTT model; - t o o b t a i n some r e l a t i v e measure o f t e s t i n g c o s t s u s i n g CCTT. I I LIMITATIONS As d i s c u s s e d i n the p r e c e d i n g c h a p t e r t h e degree o f c o m p l e x i t y o f CCTT model can be almost l i m i t l e s s . F o r purposes o f d e m o n s t r a t i o n however, a s i m p l e p r o b a b i l i t y / s e q u e n t i a l d e c i s i o n model was chosen. Only one t y p e o f psy-16-chological test (2 forms) was used and a simple p a s s / f a i l c r i t e r i o n was developed. To f a c i l i t a t e complete data c o l l e c t i o n , the t e s t - c u t o f f and multi-stage decision options of the model were not activated. Thus i n addition to predicting f i n a l performance, the model was used to a c t u a l l y determine a subject's f i n a l performance as well. I l l DESIGN C l a s s i f i c a t i o n procedure The design of the probability/sequential decision model incorporated a technique developed by Armitage (1950) and recently applied i n research on sequential t e s t i n g f o r dichotomous decisions (Linn, Rock, and Cleary, 1972). B a s i c a l l y the technique requires that a c a l i b r a t i o n sample be collected, scored, and s p l i t into high and low scoring groups. It i s then possible to prepare estimates of each item's d i f f i c u l t y f o r each group by simply c a l c u l a t i n g the proportion of subjects i n each group who answer that item c o r r e c t l y . L e t t i n g be the proportion of c a l i b r a t i o n subjects i n the high scoring group who answer item i of the test c o r r e c t l y and Pij_ be the proportion of c a l i b r a t i o n subjects i n the low scoring group who answer item i of the test correctly, additional subjects may be tested and c l a s s i f i e d 19 as follows: 1. test item i i s presented to and answered by the subject; 2. test item i i s scored; 3. the subject i s assigned to the high group a f t e r the nth item i f : n f (n) = £ log A, i = l and to the low group i f : f (n) = £ log Ri<-A i = l where R^  = (P ni./ Pl.:i.) fc^e r e s P o n s e t 0 test item i i s correct, R^  = ((1-Pj^)/(1-P-^)) i f the response to test item i i s incorrect, and A= a constant. I f assignment i s not made then another test item (i+1) i s presented to the subject. This process i s continued either u n t i l a subject i s c l a s s i f i e d , or the end of the test i s reached, at which point c l a s s i f i c a t i o n i s to the high group i f f(n)^- 0 and to the low group i f f(n)<0. I f test items were of equal d i f f i c u l t y then A values of 1.39, 2.30, 3.00, and 4.61 would correspond to upper bounds on the p r o b a b i l i t i e s of m i s c l a s s i f i c a t i o n of .25, .10, 20 .05, and .01 respectively (Linn, et a l . , p.89). As the test items used for t h i s research project were not of equal d i f f i c u l t y i t may only be stated that there i s an inverse r e l a t i o n s h i p between the value of A and the error probabil-i t i e s . This i s not a serious l i m i t a t i o n however, as the value of A may be e a s i l y adjusted to provide the desired l e v e l of accuracy i n c l a s s i f i c a t i o n . At any given value of A the accuracy l e v e l may be found by simply determining what percentage of subjects have been co r r e c t l y c l a s s i f i e d . The value of A may then be adjusted accordingly and the percentage recalculated, repeating t h i s process u n t i l an A value i s determined f o r the desired l e v e l of accuracy i n c l a s s i f i c a t i o n . As t h i s l e v e l increases however, so should the average number of test items required f o r c l a s s i f i c a t i o n . Psychological Tests Only two c r i t e r i a were used i n selecting tests for demonstrating the model. The f i r s t was that the tests should u t i l i z e multiple-choice format answers to f a c i l i t a t e input to the computer. Secondly, i t was desirable that the tests should be of a d i f f i c u l t y l e v e l such that there would be s u f f i c i e n t variance i n the observed test scores. What the tests a c t u a l l y measured was not c r i t i c a l to the design. As the model was only designed to predict o v e r a l l score on 21 the t e s t s b e i n g a d m i n i s t e r e d , o t h e r c r i t e r i a were not r e q u i r e d . The t e s t s chosen f o r the d e m o n s t r a t i o n were th e V e r b a l and N u m e r i c a l forms o f t h e M i n n e s o t a M u l t i m o d a l Analogy T e s t -1971 R e v i s i o n (Appendix A ) . Only the f i r s t f o r t y i t e m s from each form were used t o a l l o w f o r c l a s s r o o m a d m i n i s t r a t -i o n i n g a t h e r i n g the c a l i b r a t i o n sample. Computer model A F o r t r a n IV computer model (Appendix B) was d e v e l o p e d t o p e r f o r m the f o l l o w i n g f u n c t i o n s from a v i d e o s c r e e n con-v e r s a t i o n a l computer t e r m i n a l (IBM 3270); 1. d e t e r m i n e t h a t the s u b j e c t i s genuine and has not p r e v i o u s l y undergone the c o m p u t e r i z e d t e s t i n g ; 2. determine o r d e r o f t e s t a d m i n i s t r a t i o n a c c o r d i n g t o s u b j e c t ' s i d e n t i f i c a t i o n ; 3. a d m i n i s t e r t e s t i n s t r u c t i o n s and sample q u e s t i o n s ; 4. a d m i n i s t e r t e s t i t e m s , one a t a t i m e ; 5. p e r f o r m s t a t i s t i c a l a n a l y s i s a f t e r each response t o d e t e r m i n e i f s u b j e c t can be c l a s s i f i e d (and t e s t t e r m i n a t e d ) ; 6. s t o r e t e s t i n g d a t a i n a f i l e ( u n i q u e to t h a t s u b j e c t ) a f t e r each t e s t r e s p o n s e ; 7. p r e s e n t t e r m i n a t i o n message t o s u b j e c t . An i d e n t i f i c a t i o n f i l e i s created which consists of each subject's section and student number. A subject 'signs-on' to the conversational computer terminal using printed instructions (Appendix C) to operate the terminal's keyboard. The computer then requests the subject's student and section numbers by displaying instructions on the t e r -minal's video screen. Next a storage f i l e i s created by the computer using the subject's i d e n t i f i c a t i o n number as the f i l e name, and the calendar date and time of day are entered on the f i l e . I f the subject's number cannot be found i n the i d e n t i f i c a t i o n f i l e (which indicates he i s probably not a genuine subject) or i f a storage f i l e f o r that number already exists (which means the subject has previously undergone the computerized testing) an appropriate message i s returned to the screen and the subject i s automatically 'signed-off the terminal. The verbal and numerical analogy tests and associated instructions reside i n separate computer storage f i l e s (Appendix D). The order i n which the tests are administered i s determined by the subject's section number. . Instructions fo r the f i r s t tests are then displayed on the terminal screen along with a sample question. These remain on the screen u n t i l the subject depresses a key in d i c a t i n g he i s ready to • 23 proceed. At t h i s point a timing routine i s activated which computes and records ( i n milliseconds) the time taken by the subject to respond to each question, the CPU (central pro-cessing unit) time associated with administration and c a l -culations f o r each question, and the t o t a l elapsed time. Next, an analogy question and f i v e possible answers (only one of which i s correct) are displayed on the screen and remain there u n t i l a response between '0' and '5' is' entered on the terminal keyboard. A l l other responses are rejected and an appropriate message i s displayed. A '0' response indicates the answer i s not known and i s consid-ered an incorrect response to the question. The response i s then evaluated as discussed above under ' c l a s s i f i c a t i o n procedure' (p. 18). A l l t e s t i n g data associated with the test item are automatically placed on the subject's f i l e . These include: question i d e n t i f i c a t i o n , item response, correct/incorrect indicator, p r o b a b i l i t y l e v e l s f o r high and low groups, calculated Rj_ and A scores, A l e v e l i n e f f e c t , t o t a l number of items rig h t , t o t a l percent r i g h t , c l a s s i f i c a t -ion indicator ( i e . high, low, or none), interrupt and operating error counters, t o t a l CPU time, t o t a l elapsed time, and elapsed item response time. 24 I f c l a s s i f i c a t i o n i s not made the n t h e next q u e s t i o n i s p r e s e n t e d and the p r o c e s s i s r e p e a t e d u n t i l c l a s s i f i c a t i o n i s made. I f t h e s u b j e c t i s c l a s s i f i e d as b e l o n g i n g t o : t h e low group then t e s t i n g i s t h e o r e t i c a l l y t e r m i n a t e d . I f c l a s s i f i e d t o the h i g h group t h e n the next t e s t i s a d m i n i s -t e r e d f o l l o w i n g t h e same pr o c e d u r e as o u t l i n e d f o r t h e f i r s t t e s t . When t e s t i n g i s t e r m i n a t e d , t h e date and time a r e w r i t t e n on t h e end o f the s u b j e c t ' s f i l e , a t e r m i n a t i o n message i s d i s p l a y e d on the v i d e o s c r e e n , and t h e s u b j e c t i s a u t o m a t i c a l l y ' s i g n e d - o f f ' . F a c i l i t i e s a r e b u i l t i n t o t h e program t o p e r m i t the s u b j e c t t o s i g n h i m s e l f o f f i n the case o f an emergency and t o s i g n on a g a i n l a t e r , p i c k i n g up where he l e f t o f f . Any such a c t i o n i s r e c o r d e d on h i s f i l e i n c l u d i n g p l a c e and l e n g t h o f i n t e r r u p t i o n so t h a t t h e da t a can be e x c l u d e d from a n a l y s i s where d e s i r e d . IV DISCUSSION The CCTT model was i n t e n d e d t o s e r v e b o t h d e m o n s t r a t i o n and r e s e a r c h purposes and was t h u s kept r a t h e r s i m p l e i n terms o f d e c i s i o n d e s i g n and t y p e s o f t e s t s used. Demonstr-a t i o n purposes r e q u i r e d t h a t t e s t i n g be t e r m i n a t e d once a s u b j e c t was c l a s s i f i e d w h i l e the r e s e a r c h d e s i g n r e q u i r e d a d m i n i s t r a t i o n o f t h e complete t e s t s i n d i f f e r e n t sequences (see Chapter V, M e t h o d o l o g y ) . F o r t h e s e r e a s o n s , the 25 d e c i s i o n by t h e computer model t o c l a s s i f y a s u b j e c t p r i o r t o t e s t c o m p l e t i o n was s i m p l y r e c o r d e d on t h e s u b j e c t ' s f i l e and he was a l l o w e d t o f i n i s h t h e t e s t s , t h u s p r o v i d i n g • v a l u a b l e v a l i d a t i o n d a t a . T h i s a l s o p e r m i t t e d c o l l e c t i o n o f c l a s s i f i c a t i o n d a t a a t a v a r i e t y o f A v a l u e s . T i m i n g i n f o r m a t i o n was r e c o r d e d t o p e r m i t r e l a t i v e c o s t e s t i m a t e s o f CCTT but s e v e r a l o t h e r uses of t h i s d a t a a r e a l s o p o s s i b l e (see C hapter V I ) . CHAPTER V APPLICATION OF THE MODEL I SAMPLE The sample was comprised of a t o t a l of 180 University of B r i t i s h Columbia students from four sections of an undergraduate introductory organizational behavior course. P a r t i c i p a t i o n was on a voluntary basis and required approximately two hours of each subject's time, one hour of which was during regularly scheduled classes and the other at the subject's convenience within a given time period. Individual performance data and o v e r a l l findings were presented to the subjects upon completion of the study and bonus grade points were awarded. Testing wa-s conducted over a seven week period during the f a l l of 1973. II METHODOLOGY Two of the four sections were randomly selected and the paper version of the tests (Appendix A) was administered i n class. One section was given the verbal test f i r s t , followed by the numerical t e s t . The second section received the same tests i n the reverse order. Responses were scored for each test and res u l t s fo r the sections were combined into a 'paper l ' s t ' group. The median score f o r the group on the verbal test was then calculated (22/40) and used to divide the group into high and low scoring categories. Next, the responses of the 27 high category f o r each verbal test item were examined to determine what proportion answered the item r i g h t . This proportion was the equivalent to the p r o b a b i l i t y of a high verbal scorer getting the item r i g h t . S i m i l a r l y , item p r o b a b i l i t i e s were calculated for each verbal item when membership was i n the low category. The same procedure was used to determine item prob-a b i l i t i e s f o r high and low categories on the numerical paper test (median * 26/40). These verbal and numerical item p r o b a b i l i t i e s (Table I) were then placed on computer f i l e s (Appendix D) f o r input into the CCTT model as described i n the previous chapter. A period of approximately three weeks was allowed to elapse from when the f i r s t two sections had completed the paper version of the tests. Next the computer version of the tests (the CCTT model) was made available to a l l subjects (four sections) f o r a period of nine days. During t h i s period each subject was permitted to undergo the com-puter administration of the tests at his or her convenience using the instructions handed out i n class (Appendix C) to get 'signed-on' to the computer. Once 'signed-on', the video screen of the conversational computer terminal provided a l l necessary instructions f o r the subject to complete the tests with ' s i g n - o f f being handled automatically by the TABLE I TEST ITEM PROBABILITIES - CALIBRATION SAMPLE 28 Item Verbal Numerical P l i Phi p l i p h i 1 .973 .933 .463 .455 2 .622 .646 .561 .733 3 .676 .375 .000 .068 4 .243 .625 .829 .864 5 .216 .438 .171 .114 6 .189 .354 .780 .955 7 .784 .896 .073 .136 8 .351 .771 .659 .977 9 .459 .604 .244 .659 10 .622 .604 .878 1.000 11 .297 .563 .195 .432 12 .243 .167 .329 .977 13 .595 .958 .268 .432 14 .568 .896 .585 .818 15 .297 .521 .317 .432 16 .568 .750 .415 .682 17 .405 .542 .195 .364 18 .297 .396 .415 .477 19 .676 .396 .634 .773 20 .622 .792 .512 .864 21 .541 .563 .805 .955 22 .351 .729 .535 .341 23 .324 .646 .146 .227 24 .216 .479 .171 .568 25 .405 .833 .268 .568 26 .243 .750 .488 .795 27 .486 .813 .805 .909 28 .378 .667 .220 .159 29 .243 .521 .951 .959 30 .081 .183 .902 .977 31 .703 .875 .098 .227 32 .568 .792 .927 1.000 33 .405 .729 .927 .927 34 .459 .667 .878 1.000 35 .189 .354 .585 .386 36 .216 .563 .829 .955 37 .734 .875 .732 .955 38 .703 .833 .878 1.000 39 .541 .604 .317 .295 40 .595 .833 .805 1.000 p h i - proportion of subjects i n high scoring group who answer test item i co r r e c t l y P l i - proportion of subjects i n low scoring group who answer test item i c o r r e c t l y 29 computer. Details of how the CCTT model administered and scored the tests were provided i n the previous chapter under 'design' =(p. 18). The model was a r t i f i c i a l l y limited to a maximum of s i x subjects at any one time to avoid tying up too many conver-sational terminals f o r one project. An abandoned terminal automatically ' s i g n s - o f f a f t e r a short inactive period. Random v i s i t s were made (by this researcher) to the terminal area to check f o r collaboration of subjects and to provide assistance i f any d i f f i c u l t i e s were being encountered i n terminal operation. Neither were observed. For those subjects that had already completed the paper version of the tests, the CCTT model administered the two tests i n the opposite order. That i s , those subjects that did the verbal test f i r s t on the paper version of the tests were given the numerical test f i r s t by the computer. This also resulted i n a di f f e r e n t sample question being presented as only one sample question (either verbal or numerical) was presented f o r a complete te s t i n g . Once the computer phase of testing was complete, another three weeks were allowed to elapse. Then, the paper version of the tests was administered i n class to the two sections that had only completed the computer version. Again the order of tests was reversed from that of the previous admin-i s t r a t i o n . 30 I I I RESULTS AND INTERPRETATION T e s t o r d e r A d m i n i s t r a t i o n o f the v e r b a l and n u m e r i c a l forms o f the a n a l o g y t e s t was c o u n t e r b a l a n c e d t o o f f s e t any o r d e r i n g e f f e c t s . T a b l e I I i n d i c a t e s t h a t f o r the v e r b a l t e s t t h e r e was no s i g n i f i c a n t d i f f e r e n c e i n v e r b a l s c o r e s whether t h i s t e s t came b e f o r e o r a f t e r the n u m e r i c a l t e s t . T h i s a p p l i e d t o b o t h the paper and computer a d m i n i s t r a t i o n o f t h e t e s t s . F o r the n u m e r i c a l t e s t however, o r d e r o f a d m i n i s t r a t i o n d i d have a s i g n i f i c a n t e f f e c t w h i c h was i n opp o s i n g d i r e c t i o n s f o r paper and computer a d m i n i s t r a t i o n s . W i t h paper a d m i n i s t r a t i o n the n u m e r i c a l s c o r e was s i g n i f i -c a n t l y h i g h e r when t h e n u m e r i c a l t e s t f o l l o w e d t h e v e r b a l t e s t . W i t h computer a d m i n i s t r a t i o n however, n u m e r i c a l performance d e c l i n e d s i g n i f i c a n t l y when t h e n u m e r i c a l t e s t f o l l o w e d the v e r b a l t e s t . The r e a s o n s f o r t h e s e o b s e r v e d d i f f e r e n c e s a r e not c l e a r . L e a r n i n g e f f e c t s A p e r i o d o f a p p r o x i m a t e l y t h r e e weeks was l e f t between t h e f i r s t paper a d m i n i s t r a t i o n o f t h e t e s t s and t h e computer a d m i n i s t r a t i o n , and a n o t h e r t h r e e weeks between t h i s and the second paper a d m i n i s t r a t i o n so as to m i n i m i z e any l e a r n i n g e f f e c t s . N e v e r t h e l e s s , s i g n i f i c a n t l e a r n i n g e f f e c t s 31 TABLE I I COMPARISON OF TESTING ORDER SCORES-CCTT AND PAPER TESTING Group N T e s t A d m i n i s - Mean S t a n d a r d T T e s t t r a t i o n D e v i a t i o n Value Paper l ' s t 45 V V/N 22.222 5.464 40 N/V 23.625 5.16*2 1.200 45 N V/N 25.639 2.891 40 N/V 23.225 4.933 2.729 xxx CCTT l ' s t 26 V V/N 22.335 4.913 32 N/V 23.933 5.325 0.733 23 N V/N 21.733 6.082 33 N/V 25.379 3.577 2.839 xxx N- N u m e r i c a l V= V e r b a l S i g n i f i c a n c e l e v e l (2 t a i l ) x = .1 xx - .05 xxx = .01 32 d i d o c c u r on o v e r a l l t e s t s c o r e s (see T a b l e I I I ) . F o r t h o s e s u b j e c t s c o m p l e t i n g the paper a d m i n i s t r a t i o n f i r s t , t h e r e was a s i g n i f i c a n t l e a r n i n g e f f e c t on t h e v e r b a l t e s t and a p o s i t i v e though not s i g n i f i c a n t l e a r n i n g e f f e c t on t h e n u m e r i c a l t e s t . F o r t h o s e c o m p l e t i n g t h e computer a d m i n i s t r a t i o n f i r s t t h e r e were s i g n i f i c a n t l e a r n i n g e f f e c t s on b o t h t h e v e r b a l and n u m e r i c a l t e s t s w i t h the n u m e r i c a l l e a r n i n g e f f e c t b e i n g the s t r o n g e r . S i g n i f i c a n t l e a r n i n g e f f e c t s were a l s o e v i d e n t i n o v e r a l l t e s t t i m e s (see T a b l e I V ) . Those s u b j e c t s c o m p l e t i n g the paper t e s t s f i r s t took s i g n i f i c a n t l y l e s s t ime on the computer v e r s i o n o f b o t h t h e v e r b a l and n u m e r i c a l t e s t s t h a n d i d the computer f i r s t group. S i m i l a r l y , t h o s e comp-l e t i n g t h e computer t e s t s f i r s t took s i g n i f i c a n t l y l e s s time on t h e paper v e r s i o n s o f t h e v e r b a l and n u m e r i c a l t e s t s t h a n d i d the paper f i r s t group. T e s t a d m i n i s t r a t i o n S c o r e s . No s i g n i f i c a n t d i f f e r e n c e s were o b s e r v e d i n o v e r a l l t e s t s c o r e s between th e paper f i r s t and computer f i r s t a d m i n i s t r a t i o n s o f e i t h e r the v e r b a l o r n u m e r i c a l t e s t s (see T a b l e I I I ) . T h i s would t e n d t o i n d i c a t e t h a t t h e two methods o f t e s t a d m i n i s t r a t i o n , paper and computer, a r e comparable. 33 TABLE I I I COMPARISON OF TEST SCORES -CCTT AND PAPER TESTING Group N T e s t A d m i n i s -t r a t i o n Mean S t a n d a r d D e v i a t i o n T T e s t V a l u e Paper 1'st 69 V CCTT Paper 25.725 23.203 5.377 5.293 2.756 x x x 69 N CCTT Paper 26.058 25.319 2.950 3.367 1.361 CCTT l ' s t 48 V CCTT Paper 23.417 25.438 5.222 4.855 1.943 x 48 N CCTT Paper 23.896 26.375 5.203 3.279 2.763 x x x Paper l ' s t CCTT l ' s t 85 67 V V Paper CCTT 22.882 23.896 5.348 5.240 1.166 Paper 1'st CCTT l ' s t 85 63 N N Paper CCTT 24.529 24.333 4.151 5.016 -0.251 N= N u m e r i c a l V= V e r b a l S i g n i f i c a n c e l e v e l (2 t a i l ) x = .1 xx = .05 xxx = .01 34 TABLE IV COMPARISON OF TEST TIMES-CCTT AND PAPER TESTING Group N Test Adminis-t r a t i o n Mean Standard Deviation T Test Value Paper 1'st CCTT-I'st 70 60 V CCTT 762.228 920.089 172.698 312.669 3.454 xxxx Paper I'st CCTT I'st 70 60 N CCTT 062.058 1586.636 311.601 665.179 5.55S xxxx Paper I'st CCTT I'st 37 68 V Paper 763.800 633.540 155.340 15 8. 040 4.033 xxxx Paper I'st CCTT I'st 37 68 N Paper 1201.620 990.900 284.760 244.980 3.756 xxxx Paper 1'st CCS! I'st 37 60 V Paper CCTT 763.800 920.089 155.340 312.669 3.240 xxx Paper 1'st CCTT I'st 37 60 N Paper CCTT 1201.620 1586.636 284.760 665.179 3.399 xxxx N= Numerical V= Verbal Significance l e v e l (2 t a i l ) x = .1 xx = .05 xxx = .01 xxxx = .001 35 Times. For complete tests, computer administration was s i g n i f i c a n t l y slower than paper administration for both verbal and numerical testing (see Table IV). That i s , on an item for item comparison, subjects took longer on the computer administration with an average time per verbal item of 23.0 seconds compared with 19.1 f o r paper adminis-t r a t i o n and 39.7 seconds per numerical item compared with 30.0 seconds f o r paper. Some of thi s difference may be explained by the s l i g h t l y d i f f e r e n t instructions regarding times for each test administration (see Appendices A, D). It i s also l i k e l y that subjects did not experience the same pressure i n CCTT as they did i n the group test admin-i s t r a t i o n where t h e i r completion time was observable by t h e i r classmates. The mean time to c l a s s i f y a subject using CCTT was substa n t i a l l y less (V* 205 s e c , N * 478 s e c , Table V) than to completely test a subject using paper administration (V= 764 s e c , N * 1202 s e c , Table IV). This demonstrates that s i g n i f i c a n t time savings may be r e a l i z e d i n using a sequential CCTT model f o r c l a s s i f i c a t i o n with t h i s type of te s t . (Total time saving = 65$, V = 73% f N = 60%). Items. The number of items on the paper administra-t i o n of the verbal and numerical tests was f o r t y each (Ap-pendix A). Subjects were c l a s s i f i e d by CCTT af t e r an average of 9.0 verbal items and 9.9 numerical items (Table V). 36 TABLE V TIME AND NUMBER OF ITEMS REQUIRED FOR CCTT TO CLASSIFY SUBJECTS (A LEVEL = 1.39) CCTT I'st Paper tes t i n g I'st (N.=60) (N*70) Verbal Numerical Verbal Numerical Elapsed Time (sec) mean std. dev, minimum maximum per item 205.3 161.2 43 ^37 22.8 473.4 374.3 95 2068 47.3 184.0 125.1 49 636 19.4 3 1 7 . 9 233 .3 42 1299 2 8 . 4 CPU Time (sec) 1.101 1.200 1.151 1.359 Items mean std. dev. minimum maximum 9.0 5.3 4 23 9 .9 7.1 3 33 9 .5 5.3 4 27 11.2 6 . 4 3 35 37 This represents an average reduction of more than 75% i n the number of test items required using CCTT. Accuracy. With the substantial reduction i n number of test items required, CCTT was s t i l l able to c l a s s i f y subjects on the numerical test with an accuracy of $1.0% and on the verbal test with an accuracy of 8*5.1% (Table VI). Costs. Although several simplifying assumptions were made i t would appear that CCTT costs would be competitive with paper testing (Appendix G). In the example presented, actual CCTT costs were computed to be $.730 per subject compared with hypo-t h e t i c a l costs of $.675 per subject using paper testing, a difference of 8%. Anxiety. Although not s p e c i f i c a l l y part of the study design, reaction to CCTT was informally discussed with each of the four test groups i n post-test analysis sessions. Reaction was for the most part favorable to CCTT. Many indicated enjoyment of the novelty of CCTT and others indicated they f e l t much less tense than i n the group paper administration. Several noted f r u s t r a t i o n at not being able to change answers on the terminal. Only three subjects expressed objections to the con-cept of CCTT and two of these indiv i d u a l s were also against any form of psychological measurement. A l l subjects were i n favour of the convenient demand testing feature. TABLE VI CCTT SUBJECT CLASSIFICATION TABLES (A LEVEL = 1.39) N u m e r i c a l T e s t A c t u a l V e r b a l T e s t A c t u a l C l a s s i f i e d H i g h Low T o t a l H i g h 30 2 32 Low 10 21 31 T o t a l 40 23 63 A c c u r a c y = 81.0% C l a s s i f i e d H i g h Low T o t a l H i g h 39 8 47 Low 2 18 20 T o t a l 41 26 67 A c c u r a c y = 85. Ifo CHAPTER M CONCLUSIONS AND SUGGESTIONS FOR FURTHER RESEARCH I DISCUSSION Advantages -CCTT appears i d e a l l y suited to multi-stage and t a i l o r e d t e s t i n g . -CCTT f a c i l i t a t e s the application of s t a t i s t i c a l l y complex decision and testing models such as the pro b a b i l i t y sequential model presented here which cannot be applied i n a conventional te s t i n g envir-onment . -CCTT offers the convenience of demand, twenty-four hour tes t i n g to both examiner and examinee. -CCTT i s s e l f - i n s t r u c t i o n a l and eliminates the need for a formal test administrator. -CCTT i s capable of standardized test i n s t r u c t i o n . -CCTT can s i g n i f i c a n t l y reduce testing time and the number of test items required. -CCTT permits adjustment of tes t i n g accuracy l e v e l desired. -CCTT automatically, instantaneously and objectively scores, records, and interprets test r e s u l t s , pro-viding immediate feedback c a p a b i l i t i e s without the 40 intervention of a psychologist. -CCTT i s capable of accurate latency response recording. -CCTT data c o l l e c t i o n provides a broad, highly acces-s i b l e data base for further research and analysis. -CCTT i s capable of c o l l e c t i n g s e l f - v a l i d a t i o n data. -CCTT does not appear to a f f e c t o v e r a l l test scores. -CCTT does appear to be r e l a t i v e l y cost competitive with conventional t e s t i n g . -CCTT does not appear to generate undue anxiety. -CCTT i s portable using a suitcase type terminal that operates over any conventional telephone. -CCTT i s probably applicable to many other forms of test i n g . Disadvantages -CCTT may not be re a d i l y applicable to certain forms of testi n g , such as those requiring p i c t o r i a l representation. -CCTT may be r e l a t i v e l y expensive for some forms of testing, such as those requiring administration of the entire t e s t . -CCTT presently requires considerable development and test i n g . -CCTT i s dependent upon a r e l i a b l e computer system which i s not prone to frequent or prolonged f a i l u r e s . 40--CCTT may adversely a f f e c t the performance of some individuals such as those who f e e l threatened by a a computer. II LIMITATIONS Only one CCTT model was developed. Its emphasis was categorization of subjects rather than continuous measure-ment and i t predicted test performance rather than task (job) performance. Reactions and anxieties of subjects between the two modes of testing were not s p e c i f i c a l l y studied. Formal and detailed cost comparisons of the two methods of testing were not attempted. I l l CONCLUSIONS From the present investigation i t i s concluded that: -Various psychological tests may be administered via conversational computer terminals. -A variety of CCTT models are possible. -A sequential p r o b a b i l i t y CCTT model w i l l s i g n i f i c a n t l y shorten t e s t i n g time. -CCTT i s self-administering. -CCTT appears cost-competitive with conventional t e s t i n g . 42 IV SUGGESTIONS FOR FURTHER RESEARCH Further research of the CCTT concept i s suggested i n seven main areas; the present model, other models, tests, c r i t e r i a , anxiety, costs, and applications. With regard to the present model i t i s l i k e l y that the number of items required for c l a s s i f i c a t i o n may be substant-i a l l y reduced by arranging test items so that maximum- d i s -criminators are encountered early i n the t e s t . Such a procedure would most l i k e l y require c o l l e c t i o n of a further c a l i b r a t i o n sample to determine i f item ordering had any s i g n i f i c a n t e f f e c t on responses. Additional, more complex models such as those discussed i n Chapter III should be examined. Two types of branching multi-stage models p a r t i c u l a r l y warrant investigation. One i s a model that w i l l determine which tests should be administered. This would l i k e l y operate on the basis of pre-defined decision rules for ranges of scores on tests such as those measuring a b i l i t i e s . A more complex version of t h i s model would deal with tests i n areas such as personality measurement. The second branching multi-stage model i s one which would branch to test items of appropriate d i f f i c u l t y l e v e l s for the subject. This again could shorten tests,providing more f i n i t e measure-ment at the appropriate l e v e l . Another type of model requiring more s p e c i f i c , problem-oriented research i s the complexity/ 43 branching model described i n Chapter I I I . It i s comprised of. many possible segments, each of which may be researched on i t s own. Further, i t may have to be developed f o r s p e c i f i c applications which are perhaps unique to a given environment. Other forms and types of tests should also be attempted using CCTT, including various response modes. Alternative methods of response input should be researched f o r tests that do not lend themselves to multiple choice answers. Possible conversion of tests to convenient input formats such as 'multi-ple choice' should also be considered. Various methods of test presentation should be explored for tests that involve non-verbal items such as p i c t o r i a l representations. CCTT should be applied to the prediction of actual per-formance c r i t e r i a as well as known test scores. The present testing model was used to predict actual test scores when i n fact tests are t y p i c a l l y designed to predict certain performance c r i t e r i a . This represents a more d i f f i c u l t a pplication of the model as test v a l i d i t y becomes a determining factor i n i t s effectiveness. Formal measurement of objections to CCTT and resultant anxiety levels should be studied. It appears reasonable to assume that some individuals may either object to CCTT or experience a change i n anxiety l e v e l while undergoing CCTT. The nature and extent of these should be determined. For 44 example, they may be found to exist but not adversely a f f e c t test performance or may only exist within certain groups of individu a l s under certain circumstances. Comparative cost data should be collected i n the actual application of CCTT i n an appropriate environment. The present study was limited to hypothetical paper testing costs, and comparisons were on the basis of a short testing period i n an experimental environment. Additional applications and expansion of the CCTT concept such as outlined i n Chapter I should be explored more f u l l y . S p e c i f i c a l l y , t h i s e n t a i l s expansion of the CCTT concept to interview, select, and ultimately place applicants. 45; V SUMMARY CCTT has been demonstrated to be a viable substitute f o r at least one form of psychological testing. As such i t w i l l eliminate two timely delays i n the personnel selection process as described i n Chapter one, namely awaiting test administration and test i n t e r p r e t a t i o n while at the same time providing numerous other advantages. It has the added ad-vantagenof being capable of substantially reducing testing time. More-over the CCTT concept appears expandable to the entire selection process thus solving further time delay and o b j e c t i v i t y problems. Further research, preferably i n an org ganizational setting w i l l be required to explore t h i s and other questions raised. With the costs of computer f a c i l i t i e s constantly declining, hardware becoming more sophisticated, and complete computer systems available to the small user on a time sharing basis, i t appears highly probable that CCTT w i l l eventually replace conventional psychological t e s t i n g . BIBLIOGRAPHY Annett, J. 'A low-cost cheat-proof teaching system', Programmed Learning, 196/+., 1, pp. 155-8. Armitage, P. Sequential analysis with more than two a l t e r -native hypotheses, and i t s r e l a t i o n to discriminant function analysis. Journal of the Royal S t a t i s t i c a l  Society, 1950, 12, pp. 137-144. Cleary,~T. A., Linn, R. L., & Rock, D. A. Exploratory study of programmed tests. Educational and Psychological  Measurement, 1968, 28, pp. 345-360. Cronbach, L. J. and Gleser. G. C. Psychological tests and personnel decisions. (2nd ed.) Urbana, 111.: University of. I l l i n o i s Press, 1965. Dunn, T. G., Lushene, R. E., & O'Neil, H. F. Complete automation of the MMPI and a study of i t s response latencies. Journal of Consulting and C l i n i c a l Psychology, 1972, Vol. 39, No. 3, pp. 381-387. Edwards, D. A. W. C l i n i c a l analysis of dysphagia - steps towards automation and early diagnosis. In Symposium  on Carcinoma of the Oesophagus, 29th Annul General Meeting of The B r i t i s h Society of Gastroenterology, London. 1968. E l l i s , A. B. and Tiedeman, D. V. Can a machine counsel? In Computer-assisted Instruction, Testing, and Guidance, ed. Wayne H. Holtzman. New York, Evanston, & London: Harper and Row, Publishers, 1970, pp. 345-373. Elwood, D. L. Test retest r e l i a b i l i t y and cost analyses of automated and face to face i n t e l l i g e n c e t e s t i n g . Int. J . Man-Machine Studies, 1972, 4, pp. 1-22. Gedye, J. L. and M i l l e r , E. The automation of psychological assessment. Int. J. Man-Machine Studies, 1969, 1, pp. 258-259. Girshick, M. A. An elementary survey of s t a t i s t i c a l decision theory. Rev, educ. Res., 1954. 4 7 Hansen, D. N., Hedl, J. J., & O'Neil, H. F., J r . A review of automated te s t i n g . Paper presented at the meeting of the American Educational Research Association, New York, February 1971. Hedl, J., O'Neil, H. F., & Hansen, D. N. The e f f e c t i v e nature of computer-based testing procedure. Proceedings of the  Annual Convention of the American Psychological Associa-t i o n , 1971, Vol. 6, pp. 535-536. (a) O'Neil, H. F., & Hansen, D. N. Computer-based i n -t e l l i g e n c e t esting. Paper presented at the meeting of the American Educational Research Association, New York, February 1971. (b) Hubbard, J. P. 'Programmed testing i n the examinations of the National Board of Medical Examiners'. In: Anne Anastasi, ed. Testing Problems i n Perspective. Washing-ton D. C : American Council on Education, W66, pp. 1 9 5 - 2 0 7 . Linn, R. L.,'Rock, D. A., and Cleary, T. A. Sequential t e s t i n g for dichotomous decisions. Educational and Psychological  Measurement, 1972, 32, ppy. 85-95. Lord, F. M. A t h e o r e t i c a l study of two-stage testing. Psychometrika, Vol. 36, No. 3, September 1971. (a) . 'Some test theory for t a i l o r e d t e s t i n g ' . Paper presented at the Conference on Computer-Assisted Instruction, Testing and Guidance, Austin, Texas, October 1968. Research B u l l e t i n , 68-38 and ONR Technical  Report Contract Nonr. 2752 ( 0 0 ) , Princeton, N. J.: Educational Testing Service. . Robbins-monro procedures f o r t a i l o r e d t e s t i n g . Educational and Psychological Measurement, 1971, 31, pp.3-31. (b) Levin, R. I. and Kirkpatrick, C. A. Quantitative Approaches  to Management. New York: McGraw-Hill Book Company, 1965. Mayne, J.';G., Weksel, W. & Sholtz, P. N. Toward automating the medical history. Proc. Staff Meet. Mayo C l i n . 1968, 43, 1 P a i t i c h , D. Computers In behavioral science: A comprehensive automated psychological examination and report (CAPER). Behavioral Science. 1973, Vol. 18 (2), pp. 131-136. 48 Slosson, R. L. The Slosson Intelligence Test f o r Children  and Adults. East Aurora, 111.: Slosson Educational Publication, 1 9 6 3 . Veldman, D. J. and Menaker, S. >L. Computer application i n assessment and counselling. Journal of School Psychology, Spring 1968, Vol. 6, No. 3 . Wald, J . H., J r . The counselling assignment problem. Psychometrika, 1950, 2 3 , pp. 55-65. Waters, C. W. Comparison of computer-stimulated conventional and branching tests. U.S. Army BESRL Technical Research  Note, 1970 (Mar), No. 216, p. 38. APPENDICES APPENDIX A M a t e r i a l s Used i n Paper Test A d m i n i s t r a t i o n 51 Shortened Verbal Form Minnesota Multimodal Analogy Test 1971 Revision INSTRUCTIONS - .5.2 6n the following pages are some analogy questions ve would like you to answer. This test has bean designed to take approximately 35 minutes to complete, and you w i l l have to work quickly to.finish in the time allowed. The questions are arranged in two groups. The f i r s t 'Uo questions are in word form; the last UO questions are in the form of numbers. Here i s an example of a word analogy: Feather : ' :: Scale : Fish (1) Swim (2) Air (3) Water (U) Bird (5) Fly This i s the way i t should read: "Feather is to blank as Scale is to Fish?" You are to choose the word which best completes the analogy. In this example, you might reason that a scale grows on a fish - what does a feather grow on? Look at the five possible answers. "Bird" i s the best answer, so you would write "h" in the correct space on the answer sheet. The procedure for answering the number analogies is the same. You w i l l find that the reasoning required for each analogy question is different, and that the blank may appear i n any position. For each question you must determine the most appropriate reasoning and then select the word which best completes the analogy. For each analogy question, write the number of the correct answer on the answer sheet. I f you are not sure which is the correct answer, write the figure " 0 " in the space provided on the answer sheet and go on to the next question. Please work in pen, and do not change any answers. Do not make any marks on the test booklet i t s e l f . Be careful to put the answer i n the right space. Please answer every question. Begin with the word analogies and continue through the number analogies u n t i l you have answered a l l 80 questions. 3e careful that you do not skip any questions. Please record the times where indicated on the answer sheet and turn in a l l test materials as soon as you have finished. If you have any questions, please ask the person in charge. Please be assured that your responses w i l l be kept i n strictest confidence^ W O R D A N A L O G I E S i : Hunter :: Doctor : Lawyer (1) Engineer (2) Gentleman (3) Drunk (4) Fisherman (5) C i t i z e n D i v e : Swim :: Dove : ' (1) Tree (2) C o o (3) F ly (4) Swam (5) Swimming : C l o w n . :: T a l l : Dwarf (1) Short (2) Laugh (3) D u l l (4) Beard (5) Circus : Bacon :: Breast : Ham (1) Steak (2) Pig (3) Baby (4) Eggs (5) Wing : M i s s :: Husband : M r . (1) Housewife - (2), Lady (3) Nurse (4) Lass (5) Que D u l l : _ J . :: Shiny : Sharp (1) Dumb (2) Keen (3) Bright (4) Boring (5) D u l l C h e c k : . :: C a s h : Money (1) Mark (2) Go ld (3) Note (4) Giant (5) Goose Tree : M a n :: Sap : '  (1) Axe (2) Woman (3) Maple (4) Blood (5) Arm . Pick : V i o l i n :: : Banjo (1) Play (2) Sing (3) 3 o w ( v String (5) Fiddle . : Behind :: Drag : Ahead (1) Before (2) Beside (3) After (4) P u l l (5) Push I 54 1 1 . G a s o l i n e : C a r : : : C l o c k (1) O i l (2) W i n d (3) S p r i n g (4) S tem (5) G e a r 12. W a l k . : S t a n d : : M o w : • •  (1) C l i m b (2) S h a v e (3) G r o w (4) Run (5) L a w n 1 3 . : C a n o e : : C a n n o n : P i s t o l (1) B a t t l e s h i p (2) R o w b o a t (3) R i f l e (4) O c e a n (5) O a r 1 4 . S i n : S o n g : : S o n : .  (1) C h o r u s (2) S i n g l e (3) H u m (4) S i n g (5) D a u g h t e r 1 5 . M a r b l e : . : : E a r t h : S u n (1) F i r e (2) B r i g h t (3) R o u n d (4) B a s k e t b a l l (5) G a m e 1 6 . T o n : _ _ _ _ _ _ _ : : T i n : T i n k l e (1) P o u n d (2) T o n g u e ( 3 ) . T o n i g h t (4) W e t (5) S t e e l y 1 7 . • C o l l e g e : : P r a y : C h u r c h (1) B o o k s (2) T e a c h e r (3) L e a r n (4) C l a s s (5) S t u d y 1 8 . A s h : B e e c h : : O a k : . ( 1 ) P i n e (2) M a p l e (3) W a l n u t (4) H i c k o r y (5) E l m 1 9 . T r a i n : A u t o : : T r a c k : ' • (1) T i e (2) T i r e (3) C a r (4) Road (5) P a t h 20. : Y e s t e r d a y :: Tomorrow : T o d a y (1) T o m o r r o w (2) T o n i g h t (3) T o d a y (4) N o w (5) T h e n 55 •21. Unkind : Sweetheart :: ' : Guard (1) Care less . (2) C r u e l (3) Safe (4) Royal (5) Happy 22. Hire : Cent :: : Scent (1) Fire (2) Smel l (3) Dent (4) Work (5) Higher 23. Tomato : :: Carrot : Orange (1) Beet (2) Ce l ery (3) Peach (4) Potato (5) Pear 24. : Fai th :: Valuable : Trust (1) Believe (2) Honor (3) Sure (4) Heaven (5) Dear 25. Gobble : Person : : : Turkey -(1) Talk (2) Thanksgiving (3) November (4) Eat (5) Feed 26. Acorn : - :: Co l t : Horse (1) Nut (2) Mare (3) Seed (4) Oak (5) Saddle 27. Deer : Does :: Catt le : (1) D i d (2) Horses (3)- C a l f (4) Doesn't (5) Cows 28. Blue : :: Yellow : Orange (1) Purple (2) Red (3) Green (4) Brown (5) Gold 29 . Dog : M u s i c : Hymn (1) Bark (2) Hound (3) Church (4) Howl (5) Cat 30. Dot- : C i r c l e :: Marble : " (1) Ball (2) Tire (3) Square (4) Cigarette (5) Glass 56 31. School : Fish :: . : Tree (1) College (2) Turtle (3) Plant (4) Grove (5) Book 32. Odd : Proper :: Queer : (1). Correct (2) Reason (3) Ideal (4) Wise (5) Wrong 33. Who'd : Y/ould :: We've : (1) Went (2) Have (3) Will (4) They (5) Wouldn't 34. Cloth : :: Animal : Elephant (1) Shirt (2) Leather (3) Sail (4) Linen (5) Fashion 35. Nail : Screw :: : Nut (1) Tack (2) Clip (3). Bolt (4) Washer (5) Hammer 36. Sea .: Wave :: : Brook (1) Water (2) Ocean (3) Swell (4) River (5) Creek 37. Prune : Grape :: - : Plum (1) Orange (2) Raisin (3) Wine (4) Fruit (5) Grapefruit 38. Eager : :: Honor : Shame (1) Lively (2) Hurt (3) Pity (4) Dull (5) Spirit 39. Bold : Honest :: : Unfair (1) Forward (2) Unkind (3) Shy (4) Gentle (5) Trust 40. Pot : :: Pool : Loop (1) Round (2) Pan (3) Rope (4) Top (5) Pond Shortened Numerical Form Minnesota Multimodal Analogy Test 1971 Revision INSTRUCTIONS 58 On the following pages are some analogy questions we would like you to answer. This test has been designed to take approximately 35 minutes to complete, and you w i l l have to work quickly to finish in the time allowed. The questions are arranged in two groups. The f i r s t UO questions are in number form; the last kO questions are in word form. Here is an example of a number analogy: 7U6 : h :: 839 : ' (1) 9 (2) 1* (3) 76 ( M 8 (5) 3 This is the way i t should read: "First number is to Second number as Third number i s to blank." You are to choose the number which best completes the analogy. In this example, you might reason that the number "U" is in the middle of number 7^ 6. What number is in the middle of 839? Look at the five possible answers. Since the number "3" is in the middle of 839» you would write "5" in the correct space on the answer sheet. The procedure for answering the word analogies i s the same. You w i l l find that the reasoning required for each analogy question i s different, and that the blank may appear in any position. For each question you must determine the most appropriate reasoning and then select the word which best completes the analogy. For each analogy question, write the number of the correct answer on the answer sheet. If you are not sure which is the correct answer, write the figure "0" in the space provided on the answer sheet and go on to the next question. . Please work in pen, and do not change any answers. Do not make any marks on the test booklet i t s e l f . Be careful to put the answer in the right space. Please answer every question. Begin with the number analogies and continue through the word analogies u n t i l you have answered a l l 80 questions. Be careful that you do not skip any questions. Please record the times where indicated on the answer sheet and turn in a l l test materials as soon as you have finished. If you have any questions, please ask the person in charge. Please be assured that your responses w i l l be kept i n s t r i c t e s t confidence. . ' 59 N U M B E R A N A L O G I E S 1. 2 : 6 :: : 17 (1) 11 (2) 3 (3) 28 (4) 9 (5) 42 2 . 6 : ' :: 14 : 8143 (1) 93 (2) 108 (3) 4137 (4) 7 (5) 163 3. _ _ _ _ _ : 8 :: 37 : 16 (1) 4 (2) 13 (3) 26 (4) 419 (5) 3 4. \ : 827 :: 3 : 1963 (1) 7 (2) 28 (3) 2 (4) 8 (5) 14 5. 23 : 67 :: 13 : (1) 45 (2) 78 (3) 46 (4) 34 (5) 56 6. " : 3147 :: 6 : 367 (1) 17 (2) 34 (3) 31 (4) 14 (5) 7 7. 12 : • :: 456 : 345 (1) 431 (2) 13 (3) 64 (4) 235 (5) 57 8. 405050 : 7 :: 314151 : (1) 2 (2) 1 (3) 5 (4) 6 (5) 7 9. 4 : 1411 : : 2223 (1) 2 (2) 3 (3) 4 (4) 23 (5) 1 1 0 . 1345 : 2 :: 5679 : (1) 2 (2) 8 (3) 6 (4) 7 (5) 3 : 2478 :: 4 : 1354 (1) 7 (2) 4 (3) 2 (4) 32 (5) 78 : 4120 :: 7 : 470 (1) 2 (2) 24 (3) 12 (4) 1 (5) 4 : 3246 :: 3 : 1729 (1) 4 (2) 1 (3) 3 (4) 6 (5) 2 : 3, 7, 11 :: 13, 17, 21 (1) .7, 9, 11 (2) . 1, - 3, 5 • (3) 2 , 6 , 1 0 (4) 7, 11, 15 (5) 1, 5, 9 1 + 1 : l x l :: : 3 x 3 (1) 18 (2) 6 (3) 2 + 2 (4) (3 - 3 (5) 9 192837 : 4 :: 283746 : . (1) 4 (2) 6 (3) 9 (4) 8 (5) 5 1 / 2 , 1 / 4 , 1 / 8 : ' :: 1 6 , 8 , 4 : 2, 4, 8 (1) 1/3, 1/6, 1/12 (2) 0, 2 / 4 (3) 2 / 4 / 2 / 8 , 2/16 (4) 1/16, 1/32, 1/64 (5) 1/2, 1, 2 7 x 3 : 4 - 1 : (6/3) - 1 (1) (7/3) - 1 (2) 21 (3) 7 - 4 (4) 4 + 3 (5) (9/3) - 1 . ' : 2 + 2 :: 1 - 1 : 2 - 2 (1) 3 + 3 (2) 2 x 2 (3) l x l (4) 1 + 1 (5) 4 + 4 7717 : :: 6466 4 (1) 8 (2) 1 (3) 7 (4). 5 (5) 5 6 1 21 . : 789 :: 2 : 34 (1) 4 (2) 23 (3) 78 (4) 6 (5) 5 22 . 13 : 24 :: 24 : . (1) 47 (2) 13 (3) 35 (4) 15 (5) 36 23 . 283 : • :: 46 : 12 (1) 43 (2) 729 (3) 61 (4) 25 (4) 3818 24. 525 : 718 :: . : 26 (1) 191 (2) 8 (3) 3 (4) 7438 (5) 25 25 . 1792 : 3383 :: 6 : _ _ _ _ _ _ _ _ _ _ _ _ _ • • . (1) 1686 (2) 491 (3) 37 (4) 9 (5) 5566 26. 0 : 10 :: 3 : _ _ _ _ _ _ _ _ _ _ (1) 30 (2) 7 (3) 13 (4) 1 0 - 3 (5) .3 27 . 414 : :: 629 : 2 (1) 4 (2) 1 (3) 2 (4) 14 (5) 3 28 . . : 21 :: 9 : 14 (1) 7 (2) 8 (3) 9 (4) 5 (5) 3 2 9 . 23 : 32 :: ' • : 12 (1) 63 (2) 21 (3) 13 (4) 22 (5) 32 30 . 4197 : 7149 :: _ _ _ _ _ _ _ _ : 91 (1) 9247 (2) 19 (3) 29 (4) 49 (5) 74 (1) 7 (2) 36 (3) 198 (4) 104 (5) 6983 418 : 814 :: 27 : (1) 724 (2) 278 (3) 72 (4) 87 (5) 74 91 : 431 : 134 (1) 19 (2) 90 (3) 910 (4) 119 (5) 09 2 : :: 7 : 471 (1). 27 (2) 924 (3) 4 (4) 142 (5) 782 8 x 0 : ' :: 8 + 0 : 8 + 1 (1) 8 x 1 (2) 1 (3) 9 (4) 9 x 0 (5) 1 x 0 641 : 146 :: 717 : (1) 171 (2) 614 (3) 177 (4) 717 (5) 617 987 : 234 :: 876 : _ _ _ _ _ _ _ _ (1) 769 (2) 454 (3) 345 (4) 235 (5) 793 557 : :: 28 : 82 (1) 755 (2) 577 (3) 757 (4) 575 (5) 75 1 : 6 - 1 :: 2 x 2 : (1) 12 - 2 (2) 5 (3) 5 + 3 (4) 6 - 2 (5) 8 2 1345 :: : 689 (1) 5 (2) 8 (3) 4 (4) 7 (5) 1 63 Answer Sheet for Paper Tests -4-vO ANSWER SHEET NAME (PLEASE PRINT) I.D. NUMBER SECTION AGE SEX test a 1 11 21 31 enter time 2 3 _ 12 1 3 _ 22 2 3 _ 3 2 _ 3 3 _ 4. 1 4 . 34 5. 1 5 . 2 5 . 3 5 . 6. 1 6 . 2 6 . 3 6 . 7. 1 7 . 2 7. 3 7 8. 1 8 . 2 8 . 38 9 . 1 9 . 2 9 . 3 9 1 0 . 2 0 . 3 0 . 4 0 enter time : test b 1 2 3 _ k 5 6 7 . 8 9 1 0 . 11 12 1 3 14 15 16 1 7 18 1 9 2 0 . 21 22 23 24 25 26 27 28 2 9 3 0 . 31 32 33 34 35 36 37 38 39 4 0 . enter t iT»e 65 APPENDIX B F o r t r a n IV CCTT Model L O G I C A L A T N C A L L A T N T R P ( A I N I D I M E N S I O N G C 2 5 ) , A L < 2 0 ) , I T ( 5 ) , A I 2 5 ) , P H ( 4 0 ) , P L ( 4 0 ) I N T E G E R G R 1 2 6 ) R E A L A V { 4 ) / 1 . 3 9 , 2 „ 3 0 , 3 . 0 0 , 4 . 6 1 7 I N T E G E R * 2 ANSfc40 ) , R E S , R O N / • R• / , S S / * S• / , S 0 / » O V / , S6/« 6* / I N T E G E R * 2 N S , N U M , S 7 / • 7 « / , S 8 / » 8 » / , S 9 / « 9 » / , S 5 / » 5 • / C ### SET ERROR C O U N T E R S AND T E S T I N D I C A T O R S R E A L * 8 SN NE=0 N E S = 0 N I=0 NL=2 NQ=0 . NEND=0 NM=5 N H - 2 0 NT = 8 LG=0 C ### S T A C K ANY A T T E N T I O N I N T E R R U P T S C A L L A T N T R P ( A T N ) L L = 0 NPW=0 C ### D E T E R M I N E T I M E OF DAY AND D A T E C A L L T I M E ( 6 , 0 , . I T ) GO T O 9 8 9 9 N I = N I + 1 WR ITE ( N R , 9 6 ) N I , I C , G R , R E S 9 6 FORMAT {• I N T E R R U P T » , 2 1 6 , 1 3 0 , 5 X , A 1 ) I F (N'l.GE.NM) GO TO 9 9 9 I F ( N I . E Q . N L ) WR ITE ( 6 , 1 0 4 ) C S E T A T T E N T I O N T R A P TO P R E V E N T E X I T FROM PROGRAM C A L L T R R S T R 98 C A L L T R A P ( I C , G R , 6 9 9 ) 1 0 4 FORMAT ( » P L E A S E FOLLOW I N S T R U C T I O N S •) WR ITE < 6 , 4 ) 4 FORMAT (• P L E A S E E N T E R YOUR S E C T I O N NUMBER { S I N G L E D I G I T ) ' / ) 2 NEND=NEND+1 IF < N E N D . G T . 6 § S T O P 2 6 READ ( 5 , 5 , E N 0 = 2 ) NS 5 FORMAT { A l l . C ### PASSWORDS TO E X I T P R O G R A M / E N T E R MTS OR S T O P T E S T I N G IF ( N S . E Q . R O N ) GO TO 5 3 2 1 C A L L Q U I T IF I N S . E Q . S S ) S T O P 2 2 C ### D E T E R M I N E T E S T I N G ORDER BY S E C T I O N NUMBER IF { ( N S . E Q . S 7 ) : . 0 R . ( N S . E Q . S 5 ) ) NQ=2 IF ( ( N S . E Q . S 9 ) . 0 R . ( N S . E Q . S 6 ) . 0 R . I N S . E Q . S 8 ) ) NQ=1 IF ( N Q . G T . O ) GO TO 7 7 I F < N P W . G E . N L ) GO TO 9 9 9 1001 IF ( N P W . G E . N L ) I GO TO 1 0 0 3 NPW=NPW+1 GO T O 3 77 WR ITE ( 6 , 7 4 ) 7 4 FORMAT I* P L E A S E ENTER YOUR S T U D E N T NUMBER { S E V E N D I G I T S ) ' / ) C ### S A V E INPUT RECORD FOR. R E R E A D 76 C A L L S E T S T A ( 5 * 2 ) READ ( 5 , 7 5 ) SN 75 FORMAT ( A 8 ) READ ( 5 , 8 1 1 ) KF 8 1 1 FORMAT < I 7 ) C ### CHECK FOR F I LB EX I S T A N C E £ P R E V I O U S T E S T I N G , 6 1 0 C A L L C H K F I L ( S N , £ 8 0 9 } G / C A L L S E T L I 0 ( « 8 » , S N , £ 8 0 4 ) C A L L G E T L S T ( N T , L S T ) FI.N0 ( NT * L ST ) C A L L S E T S T A < NT , 2 ) READ ( N T , 3 3 2 2 * E N 0 = 8 7 6 ) 13 3 3 2 2 FORMAT ( I X , I I ) I F ( I 3 . N E . 3 ) GO TO 8 0 7 3 3 2 4 READ (NT , 3 3 2 3 , E N D = 8 7 7 ) I , J , L L , L G , N Q , T , R R 3 3 2 3 FORMAT ( 3 4 X , 5 1 5 , 2 F 1 2 . 6 ) C A L L C L O S F L l ' 8 » ) C A L L T I M E ( 6 , 0 , I T ) WR ITE < N T , 6 0 3 ) N S , S N , I T N L I = 3 6 0 0 0 F I N D ( N Q ' N L I ) READ ( N Q , 1 5 , E N D = 8 8 ) A N S , P H , P L 3 3 9 7 FORMAT (• Q U E S T I O N S W I L L C O N T I N U E FROM S I G N O F F P O I N T * I WR ITE ( 6 , 3 3 9 7 ) . N L I = ( 2 * 1 + 1 ) * 1 0 0 0 + 3 8 0 0 0 F I N D ( N Q ' N L I ) GO T O 101 8 0 7 WR ITE ( 6 , 8 0 6 ) 8 0 6 FORMAT (• RECORDS I N D I C A T E YOU H A V E DONE T H I S P H A S E - I F N O T , C A L L R O N ' ) GO TO 9 9 9 8 0 9 F I N D ( 3 » K F ) READ ( 3 , 2 1 , E N O = 1 0 0 1 ) NUM 21 FORMAT i I X , A 1) IF ( N U M . E Q . N S ) i GO TO 3 4 5 3 2 2 IF ( N E S . G E . N L ) i GO TO 9 9 9 WR ITE ( 6 , 3 4 6 ) 3 4 6 FORMAT (* YOUR S E C T I O N NUMBER D O E S NOT A P P E A R C O R R E C T - P L E A S E R E E N T E R * ) NES=NES+1 READ ( 5 , 5 ) NS IF ( N U M . N E . N S ) i GO TO 3 2 2 C ### C R E A T E S U B J E C T F I L E 3 4 5 C A L L C R E A T E ! S N , 1 , 0 , 0 , £ 8 0 4 , £ 8 0 8 , £ 8 0 8 , 6 8 0 8 , 6 8 2 0 , £ 8 0 8 ) C A L L C L O S F H ' 8 • ) 8 1 0 C A L L S E T L I 0 ( « 8 « , S N , £ 8 0 4 ) 8 8 7 WR ITE ( N T , 6 0 3 ! ) N S , S N , I T 6 0 3 FORMAT ( 1 X , A 1 > A 8 , 3 X , 2 A 4 , I X , 3 A 4 J IF ( L G . E Q . l ) GO TO l i l l GO TO 6 1 8 3 WR ITE ( 6 , 7 2 ) 7 2 FORMAT (• R E S P O N S E IS I N V A L I D - P L E A S E TRY A G A I N ' / / ) NE=NE+1 IF ( N E . G T . N L ) GO TO 9 9 9 IF ( N Q . E Q . O ) GO TO 6 ' 7 9 GO TO 76 C ### D I S P L A Y T E S T I N S T R U C T I O N S 6 1 8 DO 6 5 8 I L = 1 , 3 5 READ ( N Q , 6 6 1 ) AL 6 6 1 FORMAT ( 2 0 A 4 ) WR ITE ( 6 , 6 6 2 > AL 6 6 2 FORMAT ( 1 X , 2 0 A 4 ) IF ( I L . E Q . 1 7 ) R E A D ( 5 , 1 7 ) R E S 6 5 8 C O N T I N U E READ ( 5 , 1 7 ) RES 102 T = 0 . C ### LOAD T E S T ANSWERS AND P R O B A B I L I T I E S 108 READ < N Q , 1 5 , E N D = 8 8 ) A N S , P H , P L 15 FORMAT ( 4 0 A 1 , 4 { / , 2 0 F 4 . 3 ) ) ° ° J = l R R - O . 1=1 101 READ 1 N Q , 8 0 , E N D = 8 8 ) Q , A 8 0 FORMAT ( 3 X , 2 5 A 4 f / , 3 X t 2 5 A 4 ) C MM O I S P L A Y Q U E S T I O N AND P O S S I B L E ANSWERS WR ITE < 6 , 8 2 ) QtA 82 FORMAT { 1 X , / , 1 X , 2 5 A 4 , / , 1 X , 2 5 A 4 J C A L L T I M E ( O ) I T 2 = 0 NE=0 C ### READ S U B J E C T ' S R E S P O N S E 16 READ ( 5 , 1 7 ) RES 1 7 FORMAT ( A l ) C A L L T I M E ( 3 , 0 , I T ) I T 3 = I T C 2 ) - I T 2 I T 2 = I T ( 2 ) IF. ( R E S . E Q . S S ) GO T O 3 3 3 3 IF ( ( R E S . G E . S 0 ) . A N D . ( R E S . L T . S 6 ) )" GO TO 20 18 NE=NE+1 IF ( N E . G T . N H ) GO TO 9 9 9 WR ITE ( 6 , 1 9 ) 19 F O R M A T . ( « YOUR R E S P O N S E WAS NOT A NUMBER BETWEEN 0 AND 5 - T R Y A G A I N * / ) WR ITE ( 6 , 8 2 ) Q , A GO TO 16 2 0 READ ( N Q , 8 0 , E N D = 8 9 ) Q , A WR ITE ( 6 , 8 2 ) Q , A C C A L C U L A T E C L A S S I F I C A T I O N I N F O R M A T I O N 8 9 IF ( R E S . E Q . A N S ( I ) ) GO TO 3 0 22 R = U 1 . - PH( I) )/( 1 . - P L ( l ) l ) NRW=1. GO T O 3 2 3 0 R = P H { I 1 / P L ( I ) RR=RR+; i . N R W = 1 0 0 . 32 T=T + A L O G ( R ) R I= I PR=RR/R I IF ( T . G T . A V ( J ) ) GO TO 4 0 35 B = - 1 . * A V ( J ) IF ( T . L T . B ) GO T O 5 0 36 IF < I . L E . 4 0 ) GO TO 6 9 38 IF ( T . G E . O . ) GO TO 4 0 3 9 IF ( T . L T . O . ) GO TO 5 0 4 0 L L - 9 GO TO 6 9 5 0 L L = 1 C ### RECORD I TEM T E S T I N G I N F O ON S U B J E C T ' S F I L E 6 9 WRITE ( N T , 3 3 ) N S , S N , I , R E S , N R W , P H ( I ) , P L ( I } , R , T , A V < J V , R R , P R , L L , 1 N I , N E , I T U ) , I T ( 2 ) , I T 3 33 FORMAT ( 1 X , A 1 „ A 8 , I 2 , I X , A 1 , 1 4 , 3 F 6 . 3 , F 7 . 2 , F 6 . 2 - F 4 . 0 , F 8 . 4 , 3 1 4 , 3 1 8 ) 1=1 + 1 66 IF ( J . G E . 4 ) GO TO 65 62 IF ( L L . N E . O ) J= J+1 L L = 0 65 IF ( I . L E . 4 0 ) ' GO TO 16 88 IF {LG.EQ. 1) GO TO 888 600 LG=1 C ### ADMINISTER SECOND TEST 69 87 GO TO (601,602),NQ 601 NQ=2 WRITE (6,501) 501 FORMAT I* HERE ARE SOME NUMERICAL ANALOGIES. • ,/, 1* THE PROCEDURE FOR ANSWERING THEM IS THE SAME. •/) GO TO 646 602 NQ=1 WRITE (6,502) 502 FORMAT (' HERE ARE SOME VERBAL ANALOGIES. «,/, IV THE PROCEDURE FOR ANSWERING THEM IS THE SAME. •/) 646 DO 656 IL=1,35 656 READ (NQ,661) AL GO TO 102 1003 WRITE (6,1002)1 1002 FORMAT (• YOUR STUDENT NUMBER IS NOT IN THE ID TABLE - CALL RON*) STOP 1001 804 STOP 804 808 STOP 808 820 STOP 820 876 STOP 876 877 STOP 877 888 CALL TIME(6,0,IT) GO TO 887 999 WRITE (6,3399) 3399 FORMAT (* ERROR ACCUMULATION PREVENTS FURTHUR PROCESSING*) 1111 WRITE 16,1112). 1112 FORMAT { 1* THIS CONCLUDES THE COMPUTER PHASE OF THIS RESEARCH PROJECT.*/ 2' INFORMATION CONCERNING THE DESIGN AND FINDINGS OF THE STUDY*/ 3* WILL BE PROVilDED TO YOU IN CLASS AT A LATER DATE.*,// 4* THE TERMINAL WILL SHUT ITSELF OFF.*,//, 5* PLEASE DO NOT TRY TO SIGN ON AGAIN.*// 6* THANK YOU FOR YOUR PARTICIPATION. SEE YOU IN CLASS!!') C MAINTAIN SCREEN DISPLAY FOR 15 SEC. CALL RTWAIT(4500) STOP C ### DETERMINE AND RECORD FINAL TIMES 3333 CALL t IME< 6, 0,.IT ) 13 = 3 WRITE (NT,3336) 13,SN,IT,I,J,LL,LG,NQ,T,RR 3336 FORMAT!IX,II,A8,3X,2A4,IX,3A4,5I5,2F12.6) STOP 333 5321 CALL CLOSFLl»8 « ) STOP 000 END APPENDIX C P r i n t e d I n s t r u c t i o n s f o r ' s i g n - o n ' t o CCTT O R G A N I Z A T I O N A L B E H A V I O U R - R E S E A R C H P R O J E C T P L A C E : ROOM 2 0 7 , C I V I L E N G I N E E R I N G B U I L D I N G 71 D A T E S : WED. NOVEMBER 7 T O T H U R S . NOVEMBER 1 5 / 7 3 T I M E S : ANY T I M E OF DAY OR N I G H T A T YOUR C O N V E N I E N C E ( S A T U R D A Y AND H O L I D A Y HOURS ARE 8 AM TO 5 PM) T I M E R E Q U I R E D WILL BE A P P R O X I M A T E L Y ONE HOUR M A T E R I A L S : T H I S I N S T R U C T I O N S H E E T , YOUR S E C T I O N NUMBER YOUR S T U D E N T NUMBER I N S T R U C T I O N S : 1 GO TO ROOM 2 0 7 IN T H E C I V I L E N G I N E E R I N G B U I L D I N G . YOU W ILL S E E S E P A R A T E C A R R E L S , E A C H WITH I T S OWN C A T H O D E RAY T E R M I N A L . P O S I T I O N Y O U R S E L F AT A V A C A N T T E R M I N A L WHICH IS D I S P L A Y I N G » M T S ' IN L A R G E G R E E N L E T T E R S . 2 F A M I L I A R I Z E Y O U R S E L F WITH T H E K E Y B O A R D , P A R T I C U L A R L Y N O T I N G THE UPPER ROW WHICH C O N T A I N S T H E NUMBERS 0 THROUGH 9, AND THE BOTTOM ROW WHICH C O N T A I N S T H E S P A C E BAR ( L O N G 6 D A R K ) AND T H E ' E N T E R * K E Y TO I T S I M M E D I A T E R I G H T . 3 NOW T Y P E T H E F O L L O W I N G L E T T E R K E Y S : S I G RONL AND P R E S S T H E ' E N T E R * K E Y . NOW LOOK AT THE S C R E E N . I T SHOULD SAY ' E N T E R USER P A S S W O R D ' . I F IT DOES N O T , R E P E A T I N S T R U C T I O N 3 . 4 NOW T Y P E T H E F O L L O W I N G L E T T E R K E Y : X AND P R E S S T H E ' E N T E R * K E Y . T HE S C R E E N SHOULD NOW SAY ' P L E A S E E N T E R YOUR S E C T I O N N U M B E R * . I F IT DOES N O T , R E P E A T I N S T R U C T I O N 4 . IF A M E S S A G E I S R E T U R N E D I N D I C A T I N G YOU CANNOT S I G N ON R IGHT NOW, T R Y A G A I N L A T E R . 5 NOW T Y P E YOUR S E C T I O N NUMBER ( A S I N G L E D I G I T ) AND P R E S S ' E N T E R * . T H E T E R M I N A L S C R E E N W ILL PRODUCE FURTHUR I N S R U C T I O N S . 6 I F YOU ARE U N A B L E TO GET F U L L Y S I G N E D ON TO A T E R M I N A L OR H A V E TO ABANDON I T , P L E A S E T Y P E * S I G N O F F * AND P R E S S • E N T E R * • 7 I F PROBLEMS P E R S I S T THAT P R E V E N T YOU FROM C O M P L E T I N G T H I S P H A S E OF THE P R O J E C T , P L E A S E C A L L RON L O N G B O T T O M AT 683-8711, L . 3 4 4 8 , B E T W E E N T H E HOURS OF 8 AND 4 : 3 0 ( W E E K D A Y S ) . * P L E A S E O B S E R V E THE F O L L O W I N G R U L E S : T H E Y ARE E S S E N T I A L ! ! DO NOT P R E S S ANY T E R M I N A L K E Y S OTHER T H E N T H O S E M E N T I O N E D . DO THE P R O J E C T ON YOUR OWN (W ITHOUT YOUR F R I E N D S ) . DO T H E T E R M I N A L S E S S I O N ONCE O N L Y . * * * THANKS * * * 72 APPENDIX D Computer V e r b a l and N u m e r i c a l Analogy T e s t s I N S T R U C T I O N S 73 F O L L O W I N G A R E S O M E A N A L O G Y Q U E S T I O N S WE W O U L D L I K E Y O U T O A N S W E R . H E R E I S A N E X A M P L E O F A W O R D A N A L O G Y : F E A T H E R : _ . : S C A L E : F I S H ( 1 ) S W I M 1 2 ) A I R < 3 ) W A T E R ( 4 ) B I R D 1 5 ) F L Y T H I S I S T H E W A Y I T S H O U L D R E A D : " F E A T H E R I S T O B L A N K A S S C A L E I S T O F I S H ? " Y O U A R E T O C H O O S E T H E W O R D W H I C H B E S T C O M P L E T E S T H E A N A L O G Y . I N T H I S E X A M P L E . Y O U M I G H T R E A S O N T H A T A S C A L E G R O W S O N A F I S H W H A T D O E S A F E A T H E R G R O W O N ? L O O K A T T H E F I V E P O S S I B L E A N S W E R S . " B I R D " I S T H E B E S T A N S W E R , S O Y O U W O U L D P R E S S T H E K E Y W I T H T H E N U M B E R " 4 " O N I T A N D T H E N P R E S S T H E " E N T E R " K E Y . G O A H E A D A N D T R Y I T . Y O U W I L L F I N D T H A T T H E R E A S O N I N G R E Q U I R E D F O R E A C H A N A L O G Y Q U E S T I O N I S D I F F E R E N T , A N D T H A T T H E B L A N K M A Y A P P E A R I N A N Y P O S I T I O N . F O R E A C H Q U E S T I O N Y O U M U S T D E T E R M I N E T H E M O S T A P P R O P R I A T E R E A S O N I N G A N D T H E N S E L E C T T H E W O R D W H I C H B E S T C O M P L E T E S T H E A N A L O G Y . F O R E A C H A N A L O G Y Q U E S T I O N , P R E S S T H E K E Y W I T H T H E N U M B E R O F T H E C O R R E C T A N S W E R A N D T H E N P R E S S T H E " E N T E R " K E Y . I F Y O U A R E N O T S U R E W H I C H I S T H E C O R R E C T A N S W E R , P R E S S T H E K E Y W I T H T H E N U M B E R " 0 " ( T O P R O W ) A N D T H E N P R E S S T H E " E N T E R " K E Y . D O N O T P R E S S A N Y K E Y S O T H E R T H A N T H E N U M B E R K E Y S A N D T H E E N T E R K E Y . P L E A S E B E A S S U R E D T H A T Y O U R R E S P O N S E S W I L L B E K E P T I N S T R I C T I S T C O N F I D E N C E . P L E A S E W O R K Q U I C K L Y . Y O U R S T A R T A N D F I N I S H T I M E S W I L L B E R E C O R D E D . P R E S S ' E N T E R * W H E N Y O U A R E R E A D Y T O P R O C E E D . 4 4 3 5 4 5 3 < r 3 5 3 2 1 4 4 2 5 1 4 3 1 5 1 5 1 4 5 1 2 2 4 1 2 4 4 4 2 4 3 4 9 3 8 6 4 6 8 7 5 6 2 5 4 3 8 3 5 4 8 9 6 7 7 1 6 0 4 6 0 4 5 6 3 1 6 7 9 5 8 8 9 6 5 2 1 7 5 0 5 4 2 3 9 6 8 9 6 5 6 3 7 2 9 6 4 6 4 7 9 8 3 3 7 5 0 8 1 3 6 6 7 5 2 1 1 8 8 8 7 5 7 9 2 7 2 9 6 6 7 3 5 4 5 6 3 8 7 5 8 3 3 6 0 4 9 7 3 6 2 2 6 7 6 2 4 3 2 1 6 1 8 9 7 8 4 3 5 1 4 5 9 6 2 2 2 9 7 2 4 3 5 9 5 5 6 8 2 9 7 5 6 8 4 0 5 2 9 7 6 7 6 5 4 1 3 5 1 3 2 4 2 1 6 4 0 5 2 4 3 4 8 6 3 7 8 2 4 3 0 8 1 7 0 3 5 6 8 4 0 5 4 5 9 1 8 9 2 1 6 7 8 4 7 0 3 5 4 1 O i l : H U N T E R : : D O C T O R : L A W Y E R 0 1 2 ( 1 ) E N G I N E E R (2) G E N T L E M A N ( 3 ) D R U N K ( 4 ) F I S H E R M A N ( 5 ) C I T I Z E N 0 2 1 D I V E : S W I M : : D O V E : 0 2 2 ( 1 ) T R E E ( 2 ) C O O ( 3 ) F L Y ( 4 ) S W A M ( 5 ) S W I M M I N G 0 3 1 : C L O W N : : T A L L : D W A R F 0 3 2 ( 1 ) S H O R T ( 2 ) L A U G H ( 3 ) D U L L ( 4 ) B E A R D ( 5 ) C I R C U S 0 4 1 : B A C O N : : B R E A S T : H A M 0 4 2 ( 1 ) S T E A K ( 2 ) P I G ( 3 ) B A B Y ( 4 ) E G G S ( 5 ) W I N G 0 5 1 : M I S S : : H U S B A N D : M R 0 5 2 ( 1 ) H O U S E W I F E ( 2 ) L A D Y ( 3 ) N U R S E ( 4 ) L A S S 1 5 1 Q U E E N 0 6 1 D U L L : : : S H I N Y : S H A R P 0 6 2 1 1 ) D U M B ( 2 ) K E E N ( 3 ) B R I G H T ( < r ) B O R I N G ( 5 ) D U L L 0 7 1 C H E C K : : : C A S H : M O N E Y 0 7 2 ( 1 ) M A R K ( 2 ) G O L D ( 3 ) N O T E ( 4 ) G I A N T ( 5 ) S O O S E 0 8 1 T R E E : M A N : : S A P : 0 8 2 C 1 ) A X E ( 2 ) W O M A N ( 3 ) M A P L E ( 4 ? B L O O D ( 5 ) A R M 0 9 1 P I C K : V I O L I N : : : B A N J O 0 9 2 ( 1 ) P L A Y ( 2 ) S I N G ( 3 ) B O W (4*7 S T R I N G ( 5 ) F I D D L E 1 0 1 : B E H I N D : : D R A G : A H E A D 1 0 2 ( 1 ) B E F O R E (2) B E S I D E ( 3 ) A F T E R ( 4 ) P U L L ( 5 ) P U S H 1 1 1 G A S 0 L I N E : C A R : : : C L O C K 11211) OIL (2) WIND (3) SPRING (4) STEM ( 5 ) SEAR 121WALK : STAND MOW : 1 2 2 ( 1 ) CLIMB ( 2 ) SHAVE (3) GROW (4) RUN (5) LAWN 131 : CANOE :: CANNON : PISTOL 1 3 2 ( l T BATTLESHIP (2) ROWBOAT ( 3 ) R I F L E ( 4 ) OCEAN (5) OAR 141 S I N : SONG :: SON : 1 4 2 ( 1 ) CHORUS ( 2 ) SINGLE ( 3 ) HUM ( 4 ) SING ( 5 ) DAUGHTER 151MARBLE : EARTH : SUN 1 5 2 ( 1 ) F I R E ( 2 ) BRIGHT (3) ROUND ( 4 ) BASKETBALL 15) GAME 161 TON : :: T I N : TINKLE 1 6 2 ( 1 ) POUND ( 2 ) TONGUE ( 3 ) TONIGHT (4) WET ( 5 ) STEEL 171 : COLLEGE :: PRAY : CHURCH 1 7 2 ( 1 ) BOOKS (2) TEACHER ( 3 ) LEARN (4) CLASS 15) STUDY 181ASH : BEECH :: OAK : ;  .182(1) PINE <2) MAPLE ( 3 ) WALNUT ( 4 ) HICKORY ( 5 ) ELM 191TRAIM : AUTO :: TRACK : . 1 9 2 M ) T I E (2) TIRE (3) CAR ( 4 ) ROAD (5) PATH 201 : YESTERDAY :: TOMORROW : TODAY 202?1) TOMORROW ( 2 ) TONIGHT ( 3 ) TODAY (4) NOW 15) THEN 211UMKIND : SWEETHEART :: : GUARD 2 1 2 ( 1 ) CARELESS (2) CRUEL <3) SAFE ( 4 ) ROYAL ( 5 ) HAPPY 221HIRE : CENT :: : SCENT 2 2 2 ( 1 ) F I R E (2) SMELL (3) DENT (4) WORK ( 5 ) HIGHER 231T0MAT0 : CARROT : ORANGE 23211) BEET ( 2 ) CELERY (3) PEACH «4) POTATO ( 5 ) PEAR 241 : FAITH :: VALUABLE : TRUST 2 4 2 ( 1 ) BELIEVE ( 2 ) HONOR ( 3 ) SURE ( 4 ) HEAVEN ( 5 ) DEAR 251GOBBLE : PERSON : TURKEY 25211) TALK (2) THANKSGIVING (3) NOVEMBER (4) EAT (5) FEED 261AC0RN : COLT : HORSE 2 6 2 ( 1 ) NUT (2) MARE (3) SEED ( 4 ) OAK 151 SADDLE 271DEER : DOES :: CATTLE : 2 7 2 ( 1 ) DID (2) HORSES (3) CALF ( 4 ) DOESN'T (5) 'COWS 281BLUE : t : YELLOW : ORANGE 2 8 2 ( 1 ) PURPLE ( 2 ) RED (3) GREEN (4) BROWN (5) GOLD 291D0G : :: MUSIC : HYMN 2 9 2 ( 1 ) BARK ? 2 ) HOUND 13) CHURCH ( 4 ) HOWL (5) CAT 301D0T : CIRCLE :: MARBLE : 3 0 2 ( 1 ) BALL ( 2 ) TIRE (3) SQUARE (4) CIGARETTE ( 5 ) GLASS 311SCH00L : F I S H :: : TREE 3 1 2 ( 1 ) COLLEGE ( 2 ) TURTLE ( 3 ) PLANT ( 4 ) GROVE 15) BOOK 3210DD : PROPER :: QUEER : 3 2 2 ( 1 ) CORRECT ( 2 ) REASON ( 3 ) IDEAL ( 4 ) WISE ( 5 ) WRONG 331WHCD : WOULD WE'VE : 3 3 2 ( 1 ) WENT (2) HAVE (3*) WILL (4) THEY ( 5 ) WOULDN'T 341CL0TH : :: AMIMAL : ELEPHANT 3 4 2 ( 1 ) SHIRT ( 2 ) LEATHER ( 3 ) S A I L ( 4 ) LINEN ( 5 ) FASHION 351NAIL : SCREW :: : NUT 3 5 2 ( 1 ) TACK (2) C L I P ( 3) BOLT ( 4 ) WASHER (5) HAMMER 361SEA : WAVE : BROOK 3 6 2 ( 1 ) WATER ( 2 ) OCEAN (3) SWELL ( 4 ) RIVER ( 5 ) CREEK 371PRUNE : GRAPE :: : PLUM 3 7 2 ( 1 ) ORANGE ( 2 ) R A I S I N ( 3 ) WINE (4) FRUIT ( 5 ) GRAPEFRUIT 381 EAGER : HONOR : SHAME 3 8 2 ( 1 ) L I V E L Y <2) HURT ( 3 ) PITY ( 4 ) DULL (5) S P I R I T 391B0LD : HONEST :: : UNFAIR 3 9 2 ( 1 ) FORWARD ( 2 ) UNKIND ( 3 ) SHY (4) GENTLE ( 5 ) TRUST 401POT : :: POOL : LOOP 4 0 2 ( 1 ) ROUND ( 2 ) PAN (3) ROPE ( 4 ) TOP (5) POND 75 I N S T R U C T I O N S F O L L O W I N G A R E S O M E A N A L O G Y Q U E S T I O N S WE W O U L D L I K E Y O U T O A N S W E R . H E R E I S A N E X A M P L E O F A N U M B E R A N A L O G Y : 746 : 4 :: 839 : (1) 9 (2) 4 (3) 76 (4) 8 15) 3 T H I S I S T H E W A Y I T S H O U L D R E A D : " F I R S T N U M B E R I S T O S E C O N D N U M B E R A S T H I R D N U M B E R I S T O B L A N K ? " Y O U A R E T O C H O O S E T H E N U M B E R W H I C H B E S T C O M P L E T E S T H E A N A L O G Y . I N T H I S E X A M P L E * Y O U M I G H T R E A S O N T H A T T H E N U M B E R "4" I S I N T H E M I D D L E O F N U M B E R 746. W H A T N U M B E R I S I N T H E M I D D L E O F 839? L O O K A T T H E F I V E P O S S I B L E A N S W E R S . S I N C E T H E N U M B E R "3" I S I N T H E M I D D L E O F 839, Y O U W O U L D P R E S S T H E K E Y W I T H T H E N U M B E R "5" O N I T A N D T H E N P R E S S T H E " E N T E R " K E Y . G O A H E A D A N D T R Y I T . Y O U W I L L F I N D T H A T T H E R E A S O N I N G R E Q U I R E D F O R E A C H A N A L O G Y Q U E S T I O N I S D I F F E R E N T , A N D T H A T T H E B L A N K M A Y A P P E A R I N A N Y P O S I T I O N . F O R E A C H Q U E S T I O N Y O U M U S T D E T E R M I N E T H E M O S T A P P R O P R I A T E R E A S O N I N G A N D T H E N S E L E C T T H E W O R D W H I C H B E S T C O M P L E T E S T H E A N A L O G Y . F O R E A C H A N A L O G Y Q U E S T I O N , P R E S S T H E K E Y W I T H T H E N U M B E R O F T H E C O R R E C T A N S W E R A N D T H E N P R E S S T H E " E N T E R " K E Y . I F Y O U A R E N O T S U R E W H I C H I S T H E C O R R E C T A N S W E R , P R E S S T H E K E Y W I T H T H E N U M B E R "0" ( T O P R O W ) A N D T H E N P R E S S T H E " E N T E R " K E Y . D O N O T P R E S S A N Y K E Y S O T H E R T H A N T H E N U M B E R K E Y S A N D T H E E N T E R K E Y . P L E A S E B E A S S U R E D T H A T Y O U R R E S P O N S E S W I L L B E K E P T I N S T R I C T I S T C O N F I D E N C E . P L E A S E W O R K Q U I C K L Y . Y O U R S T A R T A N D F I N I S H T I M E S W I L L B E R E C O R D E D . P R E S S ' E N T E R * W H E N Y O U A R E R E A D Y T O P R O C E E D . 1551145422132525 544243254322222312143114 455 733 068 864 114 955 136 977 6590999 432 977 432 818 432 682 364 477 772 955 841 227 568 568 795 909 159 959 977 2270999 9770999 886 955 9550999 295 463 5610001 829 171 780 073 659 244 878 195 829 268 585 317 415 195 415 63< 805 585 146 171 268 488 805 220 951 902 098 927 927 878 585 829 732 878 311 0112 : 6 :: : 17 01211) 11 (2) 3 (3) 28 (4) 9 (5) 42 0216 : :: 14 : 8143 022(1) 93 (2) 108 (3) 4137 (4) 7 (5) 163 031 : 8 :: 37 : 16 032(1) 4 (2) 13 (3) 26 (4) 419 (5) 3 041 : 827 :: 3 : 1963 042(1) 7 (2) 28 (3) 2 (4) 8 (5) 14 05123 : 67 :: 13 : 052(1) 45 (2) 78 (3) 46 (4) 34 (5) 56 061 : 3147 :: 6 : 367 062(1) 17 (2) 34 (3) 31 (4) 14 (5) 7 07112 : :: 456 : 345 072(1) 431 (2) 13 (3) 64 (4) 235 (5) 67 081405060 : 7 :: 314151 : 082U) 2 ( 2 ) 1 ( 3 ) 5 ( 4 ) 6 ( 5 ) 7 0914 : 1411 :: : 2223 092(1) 2 (2) 3 (3) 4 (4) 23 (5) 1 1011345 : 2 :: 5679 : 102(1) 2 (2) 8 (3) 6 (4) 7 (5) 3 111 : 2478 :: 4 : 1354 112(1) 7 (2) 4 (3) 2 (4) 32 (5) 578 121 : 4120 :: 7 : 470 76 122(1) 2 (2) 24 (3) 12 (4) 1 (5) 4 131 : 3246 :: 3 : 1728 132(1) 4 (2) 1 (3) 3 (4) 6 (5) 2 141 : 3, 7, 11 :: 13, 17, 21 : 15, 19, 23 142(1) 7,9,11 (2) 1,3,5 (3) 2,6,10 (4) 7,11,15 (5) 1,5,9 1511 + 1 : 1 X 1 :: : 3 X 3 152(1) 18 (2) 6 (3) 2 + 2 (4) 3 - 3 (5) 9 161192837 : 4 :: 283746 : 162(1) 4 (2) 6 (3) 9 (4) 8 (5) 5 1711/2, 1/4, 1/8 : :: 16, 8, 4 : 2. 4, 8 172(1) 1/3,1/6,1/12 (2) 0,2,4 (3) 2/4,2/8,2/16 (4) 1/16,1/32,1/64 (5) 1/2,1,2 1817 X 3 : 4 - 1 : (6/3) - 1 182(1) (7/3) - 1 (2) 21 (3) 7 - 4 (4) 4 + 3 (5) (9/3) - 1 191 : 2 + 2 : : l - l : 2 - 2 192(1) 3 + 3 (2) 2 X 2 (3) 1 X 1 (4) 1 + 1 (5) 4 * 4 2017717 : :: 6466 : 4 202(1) 8 (2*7 1 (3) 7 (4) 6 (5) 5 211 : 789 :: 2 : 34 212(1) 4 (2) 23 (3) 78 (4) 6 (5) 5 22113 : 24 :: 24 : 222(1) 47 (2) 13 (3) 35 (4) 15 (5) 36 231283 : :: 46 : 12 232(1) 43 (2) 729 (3) 61 (4) 25 (5) 3818 241525 : 718 : 26 242(1) 191 (2) 8 (3) 3 (4) 7438 (5) 25 2511792 : 3383 :: 6 : 252(1) 1686 (2) 491 (3) 37 (4) 9 (5) 5566 2610 : 10 :: 3 : 262(1) 30 (2) 7 (3) 13 (4) 10 - 3 (5) .3 271414 : :: 629 : 2 272(1) 4 (2) 1 (3) 2 (4) 14 (5) 3 281 : 21 9 : 14 282(1) 7 (2) 8 (3) 9 (4) 5 (5) 3 29123 : 32 :: : 12 292(1) 63 (2) 21 (3) 13 (4) 22 (5) 32 3014197 : 7149 :: : 91 302(1) 9247 (2) 19 (3) 29 (4) 49 (5) 74 311 : 6 98 : 3 312(1) 7 ?2) 36 (3) 198 (4) 104 (5) 6983 321418 : 814 :: 27 : 322(1) 724 (2) 278 (31 72 (4) 87 (5) 74 33191 : :: 431 : 134 332(1) 19 (2) 90 (3) 910 (4) 119 (5) 09 3412 : :: 7 : 471 342(1) 27 (2) 924 (3) 4 (4) 142 (5) 782 3518 X 0 : :: 8 + 0 : 8 + 1 352(1) 8 X 1 ( 2 ) 1 (3) 9 ( 4 ) 9 X 0 ( 5 ) 1 X 0 361641 : 146 :: 717 : 362(1) 171 (2) 614 (3) 177 (4) 717 (5) 617 371987 : 234 :: 876 : 372(1) 769 (2) 454 (3) 345 (4) 235 (5) 798 381557 : : : 28 : 82 382(1) 755 (2) 577 (3) 757 (4) 575 (5) 75 3911 : 6 - 1 :: 2 X 2 : 392(1) 1 2 - 2 (2) 5 (3) 5 + 3 (41 6 - 2 (5) 8 + 1 4012 : 1345 :: : 689 402(1) 5 (2) 8 (3) 4 (4) 7 (5) 1 77 APPENDIX E V a l i d Student Numbers and S e c t i o n P a r t i a l Computer F i l e 1121.73 7 1162.593 8 1227.727 7 1234.567 7 1384.683 8 1384.726 7 1503.739 7 1520.725 7 1523.734 6 1559.715 7 1644.723 5 1661.235 8 1772.722 5 1788.728 7 1803.725 7 1839.729 9 1851.724 6 1855.717 6 1861.715 6 1863.711 7 1867.712 7 1887.66 6 1927.714 7 1966.704 9 2009.71 6 2035.731 5 2040.707 6 2049.716 7 2071.71 7 2074.714 5 2089.704 9 2094.712 7 2095.719 7 2096.717 8 2124.717 5 2130.714 5 2153.716 6 2165.71 6 2207.71 9 2218.709 7 2238.707 6 2244.713 9 2250.71 5 2287.704 7 2292.712 9 2302.719 7 2367.696 5 2373.71 5 2375.715 8 2377.711 7 2431.716 7 2448.71 9 2468.102 7 2473.718 5 2520.716 7 2530.707 6 2642.7 7 2706.703 8 2739.712 5 79 APPENDIX F S u b j e c t ' s T e s t R e c o r d P a r t i a l Computer F i l e so Codes for Subject's Test Record Symbol Interpretation A Student Number B Question I d e n t i f i c a t i o n C Item Response D Correct/Incorrect Indicator (100 - Correct, 1 = Incorrect) E High Group Probability Level F Low Group Pr o b a b i l i t y Level G R^^ Score H A Score I A Level In Ef f e c t J Total Number of Items Right K Total Percent of Items Right L Subject C l a s s i f i c a t i o n Indicator (9 - High, 1 - Low, 0 = None) M Number of Program Interrupts N Number of Terminal Operation Errors 0 Total CPU (Central Processing Unit) Time P To t a l Elapsed Time Q Elapsed Item Response Time A B C D E F G H : I J K L M N 0 P Q 5 3 2 1 6 6 1 14 : 2 7 : 32 NOV 1 1 , L 9 7 3 5 3 2 1 6 6 1 1 4 1 0 0 0 . 9 3 8 0 . 9 7 3 0 . 9 6 4 - 0 . 0 4 1 . 3 9 1 . 1 . 0 0 0 0 0 0 0 4 0 1 2 3 3 6 1 2 3 3 6 5 3 2 1 6 6 1 2 4 1 0 0 0 . 6 4 6 0 . 6 2 2 1 . 0 3 9 0 . 0 0 1 . 3 9 2 . 1 . 0 0 0 0 0 0 0 1 4 4 2 4 4 3 0 1 2 0 9 4 5 3 2 1 6 6 1 3 3 1 0 0 0 . 8 7 5 0 . 6 7 6 1 . 2 9 4 0 . 2 6 1 . 3 9 3 . 1 . 0 0 0 0 0 0 0 2 5 2 3 6 3 8 6 1 1 9 5 6 5 3 2 1 6 6 1 4 5 1 0 0 0 . 6 2 5 0 . 2 4 3 2 . 5 7 2 1 . 2 0 1 . 3 9 4 . 1 . 0 0 0 0 0 0 0 3 7 4 5 6 0 9 3 1 9 7 0 7 5 3 2 1 6 6 1 5 4 1 0 0 0 . 4 3 8 0 . 2 1 6 2 . 0 2 8 1 . 9 1 1 . 3 9 5 . 1 . o o o o 9 0 0 4 7 4 6 7 8 6 6 1 1 7 7 3 5 3 2 1 6 6 1 6 4 1 0 . 3 5 4 0 . 1 8 9 0 . 7 9 7 1 . 6 8 2 . 3 0 5 . 0 . 8 3 3 3 0 0 0 5 7 6 8 3 8 9 3 1 6 0 2 7 5 3 2 1 6 6 1 7 3 1 0 0 0 . 8 9 6 0 . 7 8 4 1 . 1 4 3 1 . 8 2 2 . 3 0 6 . 0 . 8 5 7 1 0 0 0 6 8 0 9 2 4 2 0 8 5 2 7 5 3 2 1 6 6 1 8 4 1 0 0 0 . 7 7 1 0 . 3 5 1 2 . 1 9 7 2 . 6 0 2 . 3 0 7 . 0 . 8 7 5 0 9 0 0 7 8 2 9 8 4 7 3 6 0 5 3 5 3 2 1 6 6 1 9 3 1 0 0 0 . 6 0 4 0 . 4 5 9 1 . 3 1 6 2 . 8 8 3 . 0 0 8 . 0 . 8 8 8 9 0 0 0 8 8 7 1 0 9 4 4 6 1 0 9 7 3 5 3 2 1 6 6 1 1 0 5 1 0 0 0 . 6 0 4 0 . 6 2 2 0 . 9 7 1 2 . 8 5 3 . 0 0 9 . 0 . 9 0 0 0 0 0 0 9 9 3 1 3 5 5 2 6 2 6 0 8 0 5 3 2 1 6 6 1 11 3 1 0 0 0 . 5 6 3 0 . 2 9 7 1 . 8 9 6 3 . 4 9 3 . 0 0 1 0 . 0 . 9 0 9 1 9 0 0 1 1 0 1 1 4 9 2 6 0 1 3 7 3 4 5 3 2 1 6 6 1 1 2 3 1 0 . 1 6 7 0 . 2 4 3 1 . 1 0 0 3 . 5 8 4 . 6 1 1 0 . 0 . 8 3 3 3 0 0 0 1 2 2 3 1 7 4 4 8 6 2 5 2 2 6 5 3 2 1 6 6 1 1 3 1 1 0 0 0 . 9 5 8 0 . 5 9 5 1 . 6 1 0 4 . 0 6 4 . 6 1 1 1 . 0 . 8 4 6 2 0 0 0 1 3 2 3 1 8 1 2 8 3 6 7 9 7 5 3 2 1 6 6 1 1 4 4 1 0 0 0 . 8 9 6 0 . 5 6 8 1 . 5 7 7 4 . 5 2 4 . 6 1 1 2 . 0 . 8 5 7 1 0 0 0 1 4 2 8 1 9 1 0 4 0 9 7 5 7 5 3 2 1 6 6 1 15 4 1 0 0 0 . 5 2 1 0 . 2 9 7 1 . 7 5 4 5 . 0 8 4 . 6 1 1 3 . 0 . 8 6 6 7 9 0 0 1 5 2 9 2 0 1 3 1 3 1 0 2 7 3 5 3 2 1 6 61 16 2 1 0 0 0 . 7 5 0 0 . 5 6 8 1 . 3 2 0 5 . 3 6 4 . 6 1 1 4 . 0 . 8 7 5 0 9 0 0 1 6 3 9 2 2 0 6 9 0 1 9 3 7 7 5 3 2 1 6 6 1 1 7 3 1 0 . 5 4 2 0 . 4 0 5 0 . 7 7 0 5 . 0 9 4 . 6 1 1 4 . 0 . 8 2 3 5 9 0 0 1 7 4 2 2 2 7 1 8 6 6 4 9 6 5 3 2 1 6 6 1 18 2 1 0 . 3 9 6 0 . 2 9 7 0 . 8 5 9 4 . 9 4 4 . 6 1 1 4 . 0 . 7 7 7 8 9 0 0 1 8 4 7 2 3 9 1 6 0 1 1 9 7 4 5 3 2 1 6 6 1 1 9 4 1 0 0 0 . 8 9 6 0 . 6 7 6 1 . 3 2 5 5 . 2 2 4 . 6 1 1 5 . 0 . 7 8 9 5 9 0 0 1 9 5 1 2 4 8 8 0 6 9 6 4 6 5 3 2 1 6 6 1 2 0 3 1 0 0 0 . 7 9 2 0 . 6 2 2 1 . 2 7 3 5 . 4 7 4 . 61 1 6 . 0 . 8 0 0 0 9 0 0 2 0 5 4 2 6 2 6 6 6 1 3 8 6 0 5 3 2 1 6 6 1 2 1 4 1 0 . 5 6 3 0 . 5 4 1 0 . 9 5 2 5 . 4 2 4 . 6 1 1 6 . 0 . 7 6 1 9 9 0 0 2 1 5 6 2 8 6 9 5 3 2 4 2 8 7 5 3 2 1 6 6 1 2 2 5 1 0 0 0 . 7 2 9 0 . 3 5 1 2 . 0 7 7 6 . 15 4 . 6 1 1 7 . 0 . 7 7 2 7 9 0 0 2 2 6 5 2 9 7 3 6 3 1 0 4 1 0 5 3 2 1 6 6 1 2 3 2 1 0 . 6 4 6 0 . 3 2 4 0 . 5 2 4 5 . 5 0 4 . 6 1 1 7 . 0 . 7 3 9 1 9 0 0 2 3 6 3 3 1 8 2 4 3 2 0 8 8 0 5 3 2 1 6 6 1 2 4 1 1 0 . 4 7 9 0 . 2 1 6 0 . 6 6 5 5 . 0 9 4 . 6 1 1 7 . 0 . 7 0 8 3 9 0 0 2 4 6 8 3 4 8 3 9 6 3 0 1 5 3 5 3 2 1 6 6 1 2 5 1 1 0 0 0 . 8 3 3 0 . 4 0 5 2 . 0 5 7 5 . 8 1 4 . 61 1 8 . 0 . 7 2 0 0 9 0 0 2 5 7 2 3 5 7 7 3 6 9 3 4 0 5 3 2 1 6 6 1 2 6 4 1 0 0 0 . 7 5 0 0 . 2 4 3 3 . 0 8 6 6 . 9 4 4 . 6 1 1 9 . 0 . 7 3 0 8 9 0 0 2 6 8 2 3 6 8 6 6 3 1 0 9 2 7 5 3 2 1 6 6 1 2 7 4 1 0 . 8 1 3 0 . 4 8 6 0 . 3 6 4 5 . 9 3 4 . 6 1 1 9 . 0 . 7 0 3 7 9 0 0 2 8 1 2 3 8 9 2 0 3 2 0 5 4 0 5 3 2 1 6 61 2 8 1 1 0 0 0 . 6 6 7 0 . 3 7 8 1 . 7 6 5 6 . 5 0 4 . 6 1 2 0 . 0 . 7 1 4 3 9 0 0 2 9 2 1 4 0 4 5 6 3 1 5 3 6 0 5 3 2 1 6 6 1 2 9 4 1 0 . 5 2 1 0 . 2 4 3 0 . 6 3 3 6 . 0 4 4 . 6 1 2 0 . 0 . 6 8 9 7 9 0 0 3 0 2 3 4 2 2 5 8 6 1 8 0 2 3 5 3 2 1 6 6 1 3 0 1 1 0 . 1 8 8 0 . 0 8 1 0 . 8 8 4 5 . 9 2 4 . 6 1 2 0 . 0 . 6 6 6 7 9 0 0 3 1 2 6 4 3 8 4 7 0 1 5 8 8 4 5 3 2 1 6 6 1 3 1 4 1 0 0 0 . 8 7 5 0 . 7 0 3 1 . 2 4 5 6 . 1 3 4 . 6 1 2 1 . 0 . 6 7 7 4 9 0 0 3 2 0 8 4 4 4 2 9 3 5 8 2 3 5 3 2 1 6 6 1 3 2 1 1 0 0 0 . 7 9 2 0 . 5 6 8 1 . 3 9 4 6 . 4 7 4 . 6 1 2 2 . 0 . 6 8 7 5 9 0 0 3 3 2 2 4 5 4 8 6 0 1 0 5 6 7 5 3 2 1 6 6 1 3 3 2 1 0 0 0 . 7 2 9 0 . 4 0 5 1 . 8 0 0 7 . 0 5 4 . 6 1 2 3 . 0 . 6 9 7 0 9 0 0 3 4 3 0 4 7 6 6 2 6 2 1 7 6 6 5 3 2 1 6 6 1 3 4 1 1 0 . 6 6 7 0 . 4 5 9 0 . 6 1 6 6 . 5 7 4 . 6 1 2 3 . 0 . 6 7 6 5 9 0 0 3 5 1 0 4 8 4 8 1 3 8 1 8 7 5 3 2 1 6 6 1 3 5 4 1 0 0 0 . 3 5 4 0 . 1 8 9 1 . 8 7 3 7 . 2 0 4 . 6 1 2 4 . 0 . 6 8 5 7 9 0 0 3 6 0 2 4 9 7 0 4 0 1 2 2 2 7 5 3 2 1 6 6 1 3 6 3 1 0 . 5 6 3 0 . 2 1 6 0 . 5 5 7 6 . 6 1 4 . 6 1 2 4 . 0 . 6 6 6 7 9 0 0 3 7 0 0 5 0 8 8 4 0 1 1 8 0 0 5 3 2 1 6 6 1 3 7 2 1 0 0 0 . 8 7 5 0 . 7 8 4 1 . 1 1 6 6 . 7 2 4 . 6 1 2 5 . 0 . 6 7 5 7 9 0 0 3 8 0 3 5 1 8 8 1 0 9 9 7 0 5 3 2 1 6 6 1 3 8 2 1 0 . 8 3 3 0 . 7 0 3 0 . 5 6 2 6 . 15 4 . 6 1 2 5 . 0 . 6 5 7 9 9 0 0 3 9 0 1 5 4 0 5 3 3 2 1 7 2 3 5 3 2 1 6 6 1 3 9 2 1 0 . 6 0 4 0 . 5 4 1 0 . 8 6 3 6 . 0 0 4 . 6 1 2 5 . 0 . 6 4 1 0 9 0 0 4 0 0 2 5 5 7 0 1 3 1 6 4 8 0 5 3 2 1 6 6 1 4 0 4 1 0 0 0 . 8 3 3 0 . 5 9 5 1 . 4 0 0 6 . 3 4 4 . 6 1 2 6 . 0 . 6 5 0 0 9 0 0 4 1 0 4 5 6 9 5 7 6 1 2 5 6 3 APPENDIX G R e l a t i v e Cost Comparison o f CCTT and Paper T e s t i n g S3 APPENDIX G RELATIVE COST COMPARISON OF CCTT AND PAPER TESTING The following cost comparison i s based on certain a r b i t r a r y assumptions regarding conventional test administration and should thus be treated with caution. Computing costs quoted are p a r t i c u l a r to the i n s t a l l a t i o n and approximate actual costs. Capital investment, overhead and material costs are ignored. An average group setting of 10 subjects i s assumed. Paper testing time taken (max.) Verbal . - 18 min. Numerical 32 min. Instructions, etc. 6 min. Scoring and recording (10 tests at 2.5 min.) 25 min. 81 min. test administration and scoring costs per hour $5.00 (assumed) Cost per subject = $5.00 x 8 l = $.675 W 10" CCT testing Elapsed time taken Verbal 205.3 sec. Numerical 47&*4_sec. 11.39 minT CPUT:1 time taken Verbal 1.101 sec, Numerical 1.200 sec, Terminal connect cost $3/hr. - |.569 CPU cost at $250/hr. - .161 Cost per subject $.730 2.301 sec, 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            data-media="{[{embed.selectedMedia}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.831.1-0092983/manifest

Comment

Related Items