UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Conversational computer terminals in psychological testing Longbottom, Ronald Arthur 1974

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-UBC_1974_A4_5 L65.pdf [ 4.15MB ]
Metadata
JSON: 831-1.0092983.json
JSON-LD: 831-1.0092983-ld.json
RDF/XML (Pretty): 831-1.0092983-rdf.xml
RDF/JSON: 831-1.0092983-rdf.json
Turtle: 831-1.0092983-turtle.txt
N-Triples: 831-1.0092983-rdf-ntriples.txt
Original Record: 831-1.0092983-source.json
Full Text
831-1.0092983-fulltext.txt
Citation
831-1.0092983.ris

Full Text

CONVERSATIONAL COMPUTER TERMINALS IN PSYCHOLOGICAL TESTING by RONALD ARTHUR LONGBOTTOM B.Com., U n i v e r s i t y of B r i t i s h Columbia,  1969  A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF BUSINESS ADMINISTRATION i n the F a c u l t y of Commerce and Business  Administration  We accept t h i s t h e s i s as conforming to the required  standard  THE UNIVERSITY OF BRITISH COLUMBIA April,  1974  In p r e s e n t i n g an the  advanced degree at Library  I further for  this thesis  shall  the  of  this thesis  written  University  of B r i t i s h  permission  s c h o l a r l y p u r p o s e s may his  f u l f i l m e n t of  make i t f r e e l y a v a i l a b l e  agree t h a t  by  in partial  representatives.  be  for  for f i n a n c i a l gain  permission.  Department  of  The U n i v e r s i t y o f B r i t i s h V a n c o u v e r 8, Canada  the  It i s understood  Columbia  shall  requirements  Columbia,  for reference  extensive  g r a n t e d by  the  that  not  and  copying of Head o f my  be  I agree  that  study.  this  thesis  Department  copying or  for  or  publication  allowed without  my  ABSTRACT The  concept o f CCTT ( c o n v e r s a t i o n a l computer  testing) i s investigatedas  terminal  a p r a c t i c a l and advantageous  s o l u t i o n to the problem of a d m i n i s t r a t i n g a s t a t i s t i c a l l y complex t e s t i n g model, which w i l l minimize the time l o s s e s and  delays  of c o n v e n t i o n a l  psychological testing.  In the process of personnel s e l e c t i o n , l a r g e r  organiz-  a t i o n s o f t e n i n v i t e c e r t a i n of t h e i r a p p l i c a n t s to undergo psychological testing.  This usually involves  inconvenience  for  the a p p l i c a n t , a d d i t i o n a l expense f o r the o r g a n i z a t i o n ,  and  time l o s s e s f o r both.  The t e s t i n g procedure w i l l  typi-  c a l l y be of a f i x e d treatment, group t e s t i n g nature whereas for  many a p p l i c a n t s an adaptive  more a p p r o p r i a t e .  t e s t i n g procedure would be  Although t e s t i n g mechanisms f o r  adaptive  treatments e x i s t , they tend to be awkward and l i m i t e d .  More  s o p h i s t i c a t e d t h e o r e t i c a l models have been developed but tend to l a c k appropriate  means of a d m i n i s t r a t i o n .  In t h i s i n v e s t i g a t i o n a F o r t r a n IV CCTT p r o b a b i l i t y / s e q u e n t i a l model i s developed and comparisons a r e made between CCTT and conventional  t e s t a d m i n i s t r a t i o n o f v e r b a l and  numerical analogy t e s t s t o 180 s u b j e c t s . presents and  The CCTT model  an a p p l i c a n t with a t e s t item, scores the response,  statistically  computes and p r e d i c t s group membership  within pre-definable  accuracy l e v e l s .  The three  possible  c l a s s i f i c a t i o n s and a s s o c i a t e d t r e a t m e n t s a r e : h i g h a d m i n i s t e r next t e s t ; low s c o r e r , ified,  continue present  unclass-  test.  Findings of the investigation significant differences  terminate t e s t i n g ;  scorer,  indicate  i n overall test  two methods o f t e s t a d m i n i s t r a t i o n .  that  t h e r e a r e no  s c o r e s between t h e  The CCTT m o d e l i s a b l e t o  c l a s s i f y i n d i v i d u a l s w i t h a n a c c u r a c y l e v e l e x c e e d i n g $0% on bo verbal  a n d n u m e r i c a l t e s t s a f t e r o n l y 25% o f t h e t e s t  have b e e n a d m i n i s t e r e d , w i t h a r e s u l t a n t t i m e o f 65%.  Additionally,  reduction i n testing  CCTT o f f e r s t h e b e n e f i t s  i n d i v i d u a l i z e d demand t e s t i n g , a u t o m a t i c s c o r i n g complex and t a i l o r e d d e c i s i o n pretation  items  of  and r e c o r d i n g ,  b r a n c h i n g , and immediate i n t e r -  of results.  Suggestions f o rfurther areas are provided.  r e s e a r c h i n t h i s and a s s o c i a t e d  TABLE OF CONTENTS Page vii  LIST OF TABLES CHAPTER I  II  INTRODUCTION The Personnel S e l e c t i o n Process  1  The Problem  2  Proposed  6  Solution  REVIEW OF THE PERTINENT LITERATURE S t a t i s t i c a l D e c i s i o n Theory Application to Psychological Testing  III  POSSIBLE COMPUTERIZED TESTING MODELS Models  9 9 10 14 14  P r o b a b i l i t y / s e q u e n t i a l s i n g l e stage  14  P r o b a b i l i t y multi-stage  15  Branching m u l t i - s t a g e .  15  Complexity/branching  16  Summary IV  1  COMPUTERIZED DEMONSTRATION MODEL  16 17  Purpose  17  Limitations  17  Design  18  C l a s s i f i c a t i o n procedure  18  V  Page Psychological  tests  20  Computer model  21  Discussion V  24  APPLICATION OF THE MODEL  26  Sample  26  Methodology Results Test  VI  .:  and i n t e r p r e t a t i o n order  26 30 30  Learning e f f e c t s  30  Test a d m i n i s t r a t i o n  32  Scores  32  Times  35  Items  35  Accuracy  37  Costs  37  Anxiety  37  CONCLUSIONS AND SUGGESTIONS FOR FURTHER RESEARCH  39  Discussion  39  Advantages  39  Disadvantages  40  Limitations  41  Conclusions  41  Suggestions f o r f u r t h e r r e s e a r c h  42  Summary  45  BIBLIOGRAPHY  4-6  vi APPENDICES  Page  APPENDIX A - M a t e r i a l s used i n Paper Test A d m i n i s t r a t i o n  50  Shortened V e r b a l Form Minnesota Multimodel Analogy Test - 1971 R e v i s i o n  51  Shortened Numerical Form Minnesota Multimodal Analogy Test - 1971  57  Answer Sheet f o r Paper T e s t s  63  APPENDIX B - F o r t r a n IV CCTT Model...  65  APPENDIX C - P r i n t e d I n s t r u c t i o n s f o r 'sign-on'  to CCTT  APPENDIX D - Computer V e r b a l and Numerical Analogy T e s t s APPENDIX E - V a l i d  70 72  Student Numbers and S e c t i o n - P a r t i a l  Computer F i l e  77  APPENDIX F - Subject's Test Record - P a r t i a l Computer File  79  APPENDIX G - R e l a t i v e Cost Comparison of CCTT and Paper Testing  82  L I S T OF TABLES TABLE I. II.  PAGE Test Item P r o b a b i l i t i e s Comparison  - Calibration  Sample....  o f T e s t i n g O r d e r S c o r e s - CCTT a n d  Paper T e s t i n g III.  Comparison  Comparison  33 o f T e s t T i m e s - CCTT a n d P a p e r 34  Testing V.  Time a n d Number o f I t e m R e q u i r e d f o r CCTT t o Classify  VI.  31  o f T e s t S c o r e s - CCTT a n d P a p e r  Testing IV.  28  Subjects  CCTT S u b j e c t C l a s s i f i c a t i o n T a b l e s  36 38  CHAPTER I INTRODUCTION • I THE PERSONNEL SELECTION PROCESS Typically, involves survive  the personnel s e l e c t i o n process o f today  a s e q u e n c e o f e v e n t s w h i c h a n a p p l i c a n t must i f he i s t o be s u c c e s s f u l  icular position.  i n h i sb i d f o ra part-  Although the order,  extent and emphasis  o f t h e s e e v e n t s may v a r y ,  there  considered  o f t h e medium t o l a r g e  representative  organization. interviews,  are several that  These a r e t h e a p p l i c a t i o n b l a n k ,  psychological  medical examination.  t e s t s , reference  c a n be sized personal  checks, and  What e a c h o f t h e s e s e l e c t i o n t o o l s  i s ultimately attempting to a s s i s t i n i s the prediction of an a p p l i c a n t ' s w i t h i n a given these obstacles  performance o r p o t e n t i a l f o r performance  task  range.  F a i l u r e t o p a s s a n y one o f  w i l l u s u a l l y mean e l i m i n a t i o n f r o m t h e  b a l a n c e o f t h e s e l e c t i o n p r o c e s s , p a r t i c u l a r l y where a l a r g e number o f o t h e r a p p l i c a n t s  are available.  Many v a r i a t i o n s o f t h i s t y p i c a l p r o c e s s e x i s t b u t most a r e s e q u e n t i a l w i t h a t each subsequent  stage.  fewer a p p l i c a n t s  being  considered  2  I I THE PROBLEM The and  personnel s e l e c t i o n process i s a time  consuming  c o s t l y procedure both from the point o f view of the  s e l e c t i n g o r g a n i z a t i o n and o f t h e a p p l i c a n t .  When t h e s e  p r o c e s s e s a r e conducted i n a s e q u e n t i a l , step by manner, v a r y i n g each  time l a p s e s w i l l u s u a l l y o c c u r between  step. For  e x a m p l e , when a n a p p l i c a n t  an a p p l i c a t i o n he i s n o t n o r m a l l y i n t e r v i e w but r a t h e r i s required i n v i t a t i o n f o r one.  completes and submits  g i v e n an immediate t o await a possible  Once i n t e r v i e w e d  he may n e x t be  i n v i t e d t o complete a few p s y c h o l o g i c a l often involves a return t r i p session. the  t e s t s but t h i s  f o r a scheduled group t e s t i n g  A f t e r t e s t i n g i t c a n be a n o t h e r d a y o r two  r e s u l t s have been s c o r e d  before  and i n t e r p r e t e d and these i n  t u r n may l e a d t o a f u r t h e r i n t e r v i e w , r e f e r e n c e and  step  u l t i m a t e l y a job o f f e r (usually subject  checking,  to passing  a  medical). E l i m i n a t i o n o f t h e a p p l i c a n t may come a f t e r a n y o f the above s t e p s w i t h an e a r l y r e j e c t i o n f a v o r a b l e parties.  From t h e o r g a n i z a t i o n ' s  a d d i t i o n a l step to the cost  point  l e a d i n g t o an u l t i m a t e  of f i n a l  selection.  o f view,  t o both each  r e j e c t i o n only  Time l o s t b e t w e e n  adds  steps  a l s o tends t o reduce the p r o b a b i l i t y that the a p p l i c a n t will  r e m a i n a v a i l a b l e f o r h i r e , p a r t i c u l a r l y i f he i s  a c t i v e l y seeking  employment e l s e w h e r e .  side, early rejection eliminates consumption of h i s time.  This  From t h e a p p l i c a n t '  f u r t h e r and u n n e c e s s a r y  i s of p a r t i c u l a r importance  when he i s c u r r e n t l y e m p l o y e d a n d must t a k e t i m e o f f f o r i n t e r v i e w s and t e s t s . Of  the a c t u a l steps  i n t h e above s e l e c t i o n p r o c e s s  p s y c h o l o g i c a l t e s t i n g i s p e r h a p s t h e most t i m e and  uncomfortable f o r the job a p p l i c a n t .  blank merely requests  with  some i n f o r m a t i o n  normally  transparent  usually only given been extended.  m i n u t e s , and p r o v i d e s i n return.  exercise. it  the applicant  and m e d i c a l s a r e  o n c e a c o n d i t i o n a l o f f e r o f employment P s y c h o l o g i c a l t e s t s on t h e o t h e r h a n d  probing,  t o f i v e o r s i x h o u r s o f what  pressured,  and even  exhaustive  O f t e n when t e s t r e s u l t s a r e f i n a l l y  becomes e v i d e n t  analyzed  that further t e s t i n g i n a s p e c i f i c  w o u l d have been d e s i r a b l e o r t h a t really required.  extent,  Reference checking i s  to the applicant  may r e q u i r e f r o m o n e - h a l f many c o n s i d e r  fifteen  The i n t e r v i e w c a n v a r y  g e n e r a l l y d u p l i c a t e s a p p l i c a t i o n d a t a t o some  takes f i f t e e n to f i f t y  has  The a p p l i c a t i o n  b i o g r a p h i c a l d a t a and about  minutes of the a p p l i c a n t ' s time. but  consuming  area  c e r t a i n t e s t s were n o t  I t i s n o t uncommon f o r a n a p p l i c a n t t o  4  be e l i m i n a t e d on t h e b a s i s o f s p e c i f i c but because t h e s e t e s t s cannot  t e s t s (eg.  be i m m e d i a t e l y s c o r e d  i n t e r p r e t e d t h e r e m a i n i n g t e s t s must a l s o be Furthermore, be  abilities) and  completed.  many c o n v e n t i o n a l p s y c h o l o g i c a l t e s t s t e n d t o  somewhat i n e f f i c i e n t a s t h e y a r e d e s i g n e d t o m e a s u r e a  g i v e n t r a i t a c r o s s a wide range  of l e v e l s .  This leads to  h i g h e r - s c o r i n g i n d i v i d u a l s wasting time answering i t e m s w h i l e l o w - s c o r e r s may  guess at d i f f i c u l t  Several t h e o r e t i c a l or p a r t i a l h a v e e x i s t e d f o r some t i m e . 'programmed' and  easy  items.  s o l u t i o n s to these  problems  Among them a r e t h e t e c h n i q u e s  'multi-stage' testing.  W i t h programmed  of  tests  t h e s u b j e c t i s g u i d e d t o i t e m s o f an a p p r o p r i a t e d i f f i c u l t y l e v e l through a system  of s e q u e n t i a l branching.  a n i t e m r i g h t he p r o c e e d s to  a n e a s i e r one.  t o a more d i f f i c u l t one  Typically,  the range  successive items normally decreases. a d m i n i s t r a t i o n and by p e r f o r m a n c e  and  of d i f f i c u l t y  content of successive t e s t s i s  on p r e c e d i n g t e s t s .  d i f f i c u l t i e s w i t h t h e s e and  This allows  level.  other solutions  s u g g e s t s a cumbersome p a p e r - a n d - p e n c i l p r o c e d u r e  for  The  *  between  subsequent  Lord  testing.  i f wrong,  determined  been i n the a c t u a l a d m i n i s t r a t i o n of the t e s t s .  two-stage  gets  In multi-stage testing,  t e s t s t o be m a t c h e d t o t h e s u b j e c t ' s a b i l i t y The  I f he  answer sheet f o r the f i r s t  have (1971a)  test i s  5 completed  i n d u p l i c a t e w i t h the o r i g i n a l b e i n g immediately-  c o l l e c t e d f o r l a t e r s c o r i n g as the o f f i c a l r e c o r d of p e r formance.  The  d u p l i c a t e i s s c o r e d by t h e e x a m i n e e a c c o r d i n g  t o i n s t r u c t i o n s p r o v i d e d by t h e e x a m i n e r determines the second-stage Hubbard  (1963),  test  this  score  t o be a d m i n i s t e r e d .  e n c o u n t e r i n g p r o b l e m s i n programmed  t e s t a d m i n i s t r a t i o n developed The  and  a r a t h e r unique  solution.  answer sheet c o n s i s t e d of a s e r i e s o f i n k e d b l o c k s ,  e a c h numbered t o c o r r e s p o n d t o t h e p o s s i b l e a n s w e r s . t h e e x a m i n e e made h i s c h o i c e he  simply erased the  appro-  p r i a t e i n k b l o c k r e v e a l i n g t h e number o f t h e n e x t q u e s t i o n t o be a t t e m p t e d .  Annett  similar solution utilizing a grid bossed onto a metal or p l a s t i c p a s t e d o v e r t h e g r i d and  test  (1964) d e v e l o p e d  matrix.  The  a soft  was  pencil  number  item.  S u c h methods h a v e p r o v i d e d l i m i t e d means o f shortening psychological tests.  tailoring  What i s s t i l l  required  h o w e v e r , i s a f o r m o f demand t e s t i n g n o t r e q u i r i n g services of a f u l l  em-  answer sheet  over h i s chosen answer, thus e x p o s i n g the hidden  and  a  d e v i c e w i t h numbers  the examinee rubbed  d i r e c t i n g him t o the next t e s t  When  t i m e t e s t a d m i n i s t r a t o r and  the  psychologist,  yet capable of i n s t a n t a n e o u s s c o r i n g , t a i l o r i n g ,  interpret-  a t i o n , and f e e d b a c k w h i l e s h o r t e n i n g t h e t e s t s .  Additionally,  such t e s t i n g  s h o u l d be c a p a b l e o f a p p l y i n g t h e more  sophist-  6 i c a t e d s t a t i s t i c a l d e c i s i o n models t h a t a r e a v a i l a b l e today. Ill In the f i e l d put  PROPOSED SOLUTION  o f p s y c h o l o g i c a l t e s t s , computers have been  t o work i n such a r e a s as s c o r i n g , i n t e r p r e t i n g , v a l i d -  a t i n g , and even d e v e l o p i n g  tests.  L i m i t e d use however, has  b e e n made o f t h e c o m p u t e r a s a t e s t a d m i n i s t r a t o r . proposed that  through the use o f c o n v e r s a t i o n a l  t e r m i n a l t e s t i n g (CCTT) many o f t h e p r e c e d i n g c o u l d be  It i s  computer  problems  solved.  CCTT i s a p r o c e s s w h i c h p e r m i t s demand t e s t i n g w i t h t h e f a c i l i t y t o make i m m e d i a t e d e c i s i o n s , t o b r a n c h t o other  t e s t s where r e q u i r e d  and t o p r o v i d e  t e s t i n t e r p r e t a t i o n and feedback. time lapses  instantaneous  CCTT c a n e l i m i n a t e t h e  on e i t h e r s i d e o f p s y c h o l o g i c a l  testing i n the  p e r s o n n e l e n v i r o n m e n t and a p p e a r s t o have t h e c a p a c i t y t o shorten  t e s t s and enhance t h e i r q u a l i t y .  U n d e r t h e c o n c e p t o f CCTT, when a n a p p l i c a n t i s i d e n t i f i e d f o r psychological t e s t i n g (usually during the interview) ions the  he i s s i m p l y  seated  a t a OCT w i t h a f e w i n s t r u c t -  (verbal, printed or displayed computer does t h e r e s t .  how t o o p e r a t e t h e t e r m i n a l  on t h e t e r m i n a l ) a n d  CCTT w i l l  teach  the applicant  i n a matter o f seconds and w i l l  7 then a d m i n i s t e r the a p p r o p r i a t e t e s t s .  The  possible  t e s t i n g models are v i r t u a l l y l i m i t l e s s as i s d i s c u s s e d i n Chapter t h r e e . As the t e r m i n a l a d m i n i s t e r s the t e s t items the computer scores and analyzes each response.  I t c o n s t a n t l y monitors  performance, d e c i d i n g whether f u r t h e r t e s t i n g i s r e q u i r e d if  so, i n what areas.  and  Thus t e s t i n g i s t a i l o r e d f o r the  a p p l i c a n t as he proceeds and  i t can be terminated  as soon as  s u f f i c i e n t d e c i s i o n i n f o r m a t i o n has been c o l l e c t e d . while the a p p l i c a n t i s s t i l l working at the CCT  Even  a complete  t e s t e v a l u a t i o n can be p r i n t e d at a remote t e r m i n a l . I t i s c o n c e i v a b l e t h a t a l o g i c a l e x t e n s i o n of the CCTT concept  would be the development of a CCTS/P ( C o n v e r s a t i o n a l  Computer Terminal Selection/Placement) i n t e g r a t e d and  model.  Such an  comprehensive model could be used to  b i o g r a p h i c a l and  other data normally  a p p l i c a t i o n blank, While attempting  gathered  i n t e r v i e w s , t e s t i n g , and  from the  even  to decide i f an a p p l i c a n t was  collect  pre-medical.  acceptable  f o r employment the model would a l s o be determining  the most  s u i t a b l e areas of placement and whether such vacancies e x i s t e d or were l i k e l y to i n the near f u t u r e . T h i s paper i s , however, c o n f i n e d to the development and  t e s t i n g of a s i n g l e CCTT model.  U t i l i z i n g one  type  of  t e s t i t p r i m a r i l y explores whether s i g n i f i c a n t d i f f e r e n c e s  i n performance occur between paper and computer administ r a t i o n s and whether s i g n i f i c a n t r e d u c t i o n s i n t e s t i n g time can be r e a l i z e d u s i n g CCTT. are a l s o examined.  R e l a t i v e c o s t s o f CCTT  CHAPTER I I REVIEW OF THE  PERTINENT LITERATURE  I S T A T I S T I C A L DECISION S t a t i s t i c a l decision theory,  THEORY  orginated  b y Abraham  i n t h e l a t e 1930's ( G i r s h i c k , 1954), i s s t i l l young a n d i n c o m p l e t e s c i e n c e .  a  Wald  relatively  W a l d (1950) i n v e s t i g a t e d  problems o f i n s p e c t i o n and q u a l i t y c o n t r o l where e i t h e r an a c c e p t a n c e o r r e j e c t i o n d e c i s i o n was r e q u i r e d .  C l e a r l y , the  personnel s e l e c t i o n process o u t l i n e d i n Chapter I r e q u i r e s similar ultimate  d e c i s i o n and one o f t h e mechanisms b y w h i c h  t h a t d e c i s i o n may be r e a c h e d i s p s y c h o l o g i c a l t e s t i n g . a r r i v i n g at t h i s ultimate may  a  utilize a third  d e c i s i o n , each o f t h e above  processes  a l t e r n a t i v e when t h e d e c i s i o n t o a c c e p t  or r e j e c t i s not c l e a r - c u t , that by c o l l e c t i n g more  In  i s , to continue  the process  information.  T y p i c a l l y t h i s i s what h a p p e n s i n t h e o v e r a l l s e q u e n t i a l s e l e c t i o n p r o c e s s b u t n o t i n t h e t e s t i n g segment. p r e - d e t e r m i n e d amount o f t e s t d a t a i s f i r s t analyzed, is  and then an a c c e p t / r e j e c t  inefficient,  Here, a  c o l l e c t e d and  d e c i s i o n i s made.  This  h o w e v e r , a s u s u a l l y more t h a n t h e r e q u i r e d  amount o f i n f o r m a t i o n  to reach a d e c i s i o n i s c o l l e c t e d or  occasionally i t i s discovered,  a f t e r the f a c t , that  sufficient  10 test information sequential  was not gathered.  Thus e x i s t s the case f o r  testing strategies.  A strategy i n any p o s s i b l e  must s t a t e what the d e c i s i o n maker w i l l do contingency ( G i r s h i c k , 1954, p. 4 6 4 )  a n u  -  normally c o n s i s t s of a set of c o n d i t i o n a l p r o b a b i l i t i e s , (Cronbach and G l e s e r ,  1965, p. 19) each between 0 and 1.  Although p r o b a b i l i t i e s may be c l a s s i f i e d as e i t h e r or s u b j e c t i v e  ( L e v i n and K i r k p a t r i c k ,  here i s w i t h the former.  objective  1965), the concern  These are r e f e r r e d to as a s s i g n e d  p r o b a b i l i t i e s that are supported by h i s t o r i c a l evidence and common experience, or are s e l f - e v i d e n t from the p h y s i c a l a t t r i b u t e of the system.  They i n d i c a t e the chance of an  event happening. I I APPLICATION TO PSYCHOLOGICAL TESTING Sequential  testing  One form of s e q u e n t i a l programmed t e s t i n g .  t e s t i n g i s r e f e r r e d to as  This involves a sequential  branching which guides the subject appropriate  difficulty level.  system o f  to t e s t items of an  Cleary,  L i n n , and Rock (1968,  p. 183) i n d i c a t e that programmed t e s t s have the c a p a c i t y to s i g n i f i c a n t l y reduce t e s t i n g time r e q u i r e d  for ability  assessment, to reduce the component of u n r e l i a b l e due to random guessing, and to i n c r e a s e  variance  the accuracy of  11 measurement near c u t - o f f s additional consists  items.  by having s u b j e c t s respond t o  I n i t s simplest form a programmed  of a r o u t i n g  test  s e c t i o n where the necessary branching  takes place to d i r e c t the subject to the a p p r o p r i a t e l e v e l o f t e s t items and a measurement s e c t i o n  which concentrates these  items. Lord (1968) has a p p r o p r i a t e l y l a b e l l e d t h i s form o f sequential testing made t o ' t a i l o r ' 'ability'  the d i f f i c u l t y of the t e s t items t o the  of the i n d i v i d u a l being t e s t e d .  (Lord, 1971 jb,p. in  ' t a i l o r e d - t e s t i n g ' where an attempt i s  k) that  He  indicates  c o n s i d e r a b l e work remains to be done  determining what the i n i t i a l d i f f i c u l t y l e v e l should be,  how much the d i f f i c u l t y l e v e l should change a f t e r any r i g h t or wrong answer, how responses should be scored, and how the effectiveness  of the p o s s i b l e  procedures should be compared.  A ' l e s s - t a i l o r e d ' form o f t a i l o r e d - t e s t i n g two-stage t e s t i n g .  T h i s procedure c a l l s f o r a r o u t i n g  f o l l o w e d by one o f s e v e r a l ional tests.  i s sequential  alternative  second-stage convent-  Cronbach and G l e s e r (1965, Chapter 6) explore  two-stage t e s t i n g as a s e q u e n t i a l d e c i s i o n  theory technique  whereby examinees may be c l a s s i f i e d a f t e r the f i r s t accept, r e j e c t , o r continue t e s t i n g  categories.  test  elimination  o f unnecessary  tests.  into  The main  advantages o f t h i s procedure a r e a savings i n t e s t i n g and  test  time  12 A s e q u e n t i a l model that has been shown t h e o r e t i c a l l y capable of reducing the number of t e s t items by  one-half  i s a p r o b a b i l i t y approach by L i n n , Rock and Cleary (1972). This type of approach has, however, lacked an mechanism f o r i t s a d m i n i s t r a t i o n .  appropriate  CCTT appears to provide  t h i s mechanism and thus a sequential p r o b a b i l i t y model i s explored more f u l l y i n subsequent  chapters.  P a i t i c h (1973) presents a computer program capable of generating automatic p s y c h o l o g i c a l reports f o r a b a t t e r y of measures of i n t e l l i g e n c e and p e r s o n a l i t y .  The various t e s t s  are i n t e r p r e t e d by rather simple procedures that i n v o l v e p r i n t i n g given sentences or short paragraphs that correspond to scores f a l l i n g w i t h i n a s p e c i f i e d range.  Hedl, O'Neil  and Hansen (1971a) have developed an automated a d m i n i s t r a t i o n and scoring program f o r the Slosson I n t e l l i g e n c e Test (Slosson, 1963).  A considerable amount of work has a l s o been done on  computerized i n t e r p r e t a t i o n of the Minnesota M u l t i p h a s i c P e r s o n a l i t y Inventory  (MMPI) (Hansen, Hedl and O'Neil, 1971),  but only r e c e n t l y has an attempt been made to computerize i t s a d m i n i s t r a t i o n (Dunn, Lushene and O'Neil, 1972).  I t was  found that the t e s t could be s u c c e s s f u l l y administered scored i n t h i s manner although the study i t s e l f was concerned w i t h l a t e n c y a n a l y s i s .  and  mainly  Waters (1970) i n a com-  parison of computer-simulated branching and  conventional  13 t e s t s of v a r i e d lengths found branching t e s t s had h i g h e r c o r r e l a t i o n w i t h u n d e r l y i n g a b i l i t y than d i d any of the conventional t e s t s .  Gedye and M i l l e r  (1969, pp. 253-259)  i n d i c a t e d automated t e s t i n g i s probably j u s t i f i e d  on  economic grounds and a l s o found s u b j e c t s of a l l ages f a v o r a l l y disposed to automated t e s t i n g .  Elwood (1972)  found automated a d m i n i s t r a t i o n of the Wechsler A d u l t igence S c a l e (WAIS) was  indeed cheaper  Intell-  than f a c e - t o - f a c e  a d m i n i s t r a t i o n by a f a c t o r of one-half.  Hedl, O'Neil  and  Hansen (1971b) have, however, found that computer t e s t i n g procedures  can l e a d to higher l e v e l s of a n x i e t y .  S e v e r a l of the above f i n d i n g s have r e s u l t e d from  computer-  a s s i s t e d i n s t r u c t i o n (CAI) a p p l i c a t i o n s b u t most have been generated by a need to improve s e l e c t i o n and t e s t i n g Other i d e n t i f i e d h o r i z o n s f o r r e l a t e d CCT  procedures.  a p p l i c a t i o n s are  i n the f i e l d s of c o u n s e l l i n g (Veldman and Menaker,  1968;  E l l i s and Tiedeman, 1970), and m e d i c a l h i s t o r y and d i a g n o s i s (Edwards, 1968;  Mayne, Weksel and S h o l t z , 1968).  CHAPTER I I I POSSIBLE COMPUTERIZED TESTING MODELS In the preceding d i s c u s s i o n o f s t a t i s t i c a l d e c i s i o n theory, three main c l a s s e s of d e c i s i o n s were  presented.  These were s e q u e n t i a l two-stage, p r o b a b i l i t y , and branching decisions.  The present chapter i s intended t o i n d i c a t e  some o f the p o s s i b l e CCTT models t h a t c o u l d r e s u l t from  such  decision patterns. I MODELS Single  stage  The t y p i c a l t e s t i n g model i n the f i e l d s e l e c t i o n today i s one o f f i x e d - t r e a t m e n t .  o f personnel The i n d i v i d u a l  completes a s e r i e s of t e s t s , they a r e analyzed and a d e c i s i o n i s made to h i r e or not h i r e .  As d i s c u s s e d i n Chapter  model i s i n e f f i c i e n t f o r s e v e r a l reasons.  I this  To computerize i t  would be o f minimal b e n e f i t . Sequential multi-stage In s e q u e n t i a l m u l t i - s t a g e t e s t i n g a d e c i s i o n i s p o s s i b l e and normally expected  a f t e r each stage.  Typically  this  d e c i s i o n i s to e i t h e r suspend t e s t i n g or t o continue and i s based on t e s t performance to t h a t p o i n t w i t h fewer i n d i v i d u a l s being t e s t e d a t each stage.  Simple  c u t - o f f scores can e x i s t  15 f o r each t e s t and c u m u l a t i v e s c o r e s a r e a l s o  possible.  A l t h o u g h t h e two s t a g e v e r s i o n o f t h i s m o d e l i s s o m e t i m e s applied i n psychological testing,  i t i s awkward, a n d  s t a g e a p p l i c a t i o n s o f i t a r e v e r y much more s o . such a model i s r e a d i l y c o m p u t e r i z e d  multi-  Although  i t does l a c k  e f f i c i e n c y i n t h a t e n t i r e s t a g e s o f t h e t e s t must  some still  be g i v e n . Probability/sequential  single  T h i s model i s based and  i s virtually  stage  on p r o b a b i l i t y o f g r o u p m e m b e r s h i p  impossible to administer i n a conventional  s e t t i n g , due t o t h e m o d e r a t e l y  complex s t a t i s t i c a l  r e q u i r e d a f t e r each r e s p o n s e .  I t i s on t h e o t h e r hand  r e a d i l y computerized  and t h e o r e t i c a l l y  calculations very  c a p a b l e o f a 50%  r e d u c t i o n i n t h e number o f t e s t i t e m s r e q u i r e d a s r e f e r e n c e d i n the previous chapter. is  The d e c i s i o n r e a c h e d  by t h i s  model  dichotomous (accept or r e j e c t ) .  Probability  mult-stage  This i s i d e n t i c a l to the preceding s i n g l e  stage  model  e x c e p t t h a t a t h i r d a l t e r n a t i v e d e c i s i o n i s now p o s s i b l e t h a t i s t o continue t e s t i n g w i t h the next stage. o f model i s d e v e l o p e d  i n Chapter  This  and  type  IV.  Branching multi-stage The b r a n c h i n g m u l t i - s t a g e m o d e l a d d s two new  decision  16  dimensions  i n t h a t i t determines  be a d m i n i s t e r e d n e x t .  w h i c h t e s t o r item(§) w i l l  Thus i t i s c a p a b l e , o f t a i l o r i n g  w i t h i n and a c r o s s s u b j e c t  tests  areas.  Complexity/branching T h i s t y p e o f model i n v o l v e s b r a n c h i n g t e s t s or treatments  to other  b a s e d on what c o u l d be c o m p l e x  items, statistical  c a l c u l a t i o n s r e s u l t i n g from answer p a t t e r n s , a s s o c i a t e d l a t e n c i e s , p r e v i o u s p e r f o r m a n c e , b i o g r a p h i c a l and t e s t i n t e r v i e w r e s e r v a t i o n s and so o n .  Considerable  data,  developmental  work w i l l  be r e q u i r e d b e f o r e t h i s t y p e o f m o d e l becomes a  reality.  I t b e g i n s t o encompass a n a l y s i s o f d a t a  i n other aspects  of the s e l e c t i o n process  collected  and a t t e m p t s  to  d u p l i c a t e o r make t h e P e r s o n n e l Manager's s e l e c t i o n f o r h i m . I I SUMMARY The a b o v e m o d e l s w e r e b r i e f l y d e s c r i b e d t o i l l u s t r a t e the apparent research w i l l feasibility,  c a p a b i l i t i e s o f CCTT.  C o n s i d e r a b l y more  be r e q u i r e d on t h e c o n c e p t applicability,  and  t o examine i t s  acceptability.  CHAPTER I V COMPUTERIZED DEMONSTRATION I  PURPOSE  A multi-stage probability/sequential m o d e l was d e v e l o p e d f o r s e v e r a l - t o demonstrate that multiple-choice  - to investigate on  overall test  purposes,  psychological  format could  scored, interpreted,  MODEL  computerized  testing  namely: tests with  a  r e a d i l y be a d m i n i s t e r e d ,  a n d r e c o r d e d b y means o f CCTT;  what e f f e c t s i f a n y CCTT w o u l d h a v e performance;  - t o determine i f s i g n i f i c a n t time savings i n t e s t administration  could  be r e a l i z e d t h r o u g h u s e o f a  probability/sequential - to obtain  CCTT m o d e l ;  some r e l a t i v e m e a s u r e o f t e s t i n g  costs  u s i n g CCTT. II As  LIMITATIONS  d i s c u s s e d i n t h e preceding chapter t h e degree o f  c o m p l e x i t y o f CCTT m o d e l c a n be a l m o s t l i m i t l e s s . F o r purposes o f d e m o n s t r a t i o n however, a s i m p l e sequential  decision  m o d e l was c h o s e n .  probability/  O n l y one t y p e o f p s y -  16c h o l o g i c a l t e s t (2 forms) was c r i t e r i o n was  developed.  used and  a simple  To f a c i l i t a t e  c o l l e c t i o n , the t e s t - c u t o f f and  complete data  multi-stage  of the model were not a c t i v a t e d .  pass/fail  decision  options  Thus i n a d d i t i o n to  p r e d i c t i n g f i n a l performance, the model was  used t o a c t u a l l y  determine a s u b j e c t ' s f i n a l performance as w e l l . Ill Classification The  procedure  design  of the p r o b a b i l i t y / s e q u e n t i a l d e c i s i o n  model i n c o r p o r a t e d and  DESIGN  a technique developed by Armitage (1950)  r e c e n t l y applied i n research  on s e q u e n t i a l t e s t i n g f o r  dichotomous d e c i s i o n s ( L i n n , Rock, and  Cleary,  1972).  B a s i c a l l y the technique r e q u i r e s that a c a l i b r a t i o n sample be c o l l e c t e d , scored, groups.  and  s p l i t i n t o h i g h and  low  scoring  I t i s then p o s s i b l e to prepare estimates of each  item's d i f f i c u l t y f o r each group by simply proportion  of s u b j e c t s  i n each group who  c a l c u l a t i n g the  answer that  item  correctly. Letting i n the high  be the p r o p o r t i o n s c o r i n g group who  c o r r e c t l y and i n the low  Pij_  be  answer item i of the t e s t  the p r o p o r t i o n  s c o r i n g group who  of c a l i b r a t i o n s u b j e c t s  of c a l i b r a t i o n s u b j e c t s  answer item i of the t e s t  c o r r e c t l y , a d d i t i o n a l s u b j e c t s may  be t e s t e d and  classified  19 as f o l l o w s : 1. t e s t item i i s presented t o and answered by the subject; 2. t e s t item i i s scored; 3. the subject i s a s s i g n e d t o the h i g h group a f t e r the nth item i f : n f (n) = £  log  A,  i =l and to the low group i f : f  (n) = £  log Ri<-A  i=l where R^ = (P i./ l.:i.) P  n  fc  ^  e  r  e  s  P  o  n  s  e  t  0  t e s t item i  i s c o r r e c t , R^ = ((1-Pj^)/(1-P-^)) i f the response to t e s t item i i s i n c o r r e c t ,  and A= a c o n s t a n t .  I f assignment i s not made then another t e s t item (i+1) i s presented to the s u b j e c t .  T h i s process i s continued  either  u n t i l a s u b j e c t i s c l a s s i f i e d , or the end o f the t e s t i s reached, a t which p o i n t c l a s s i f i c a t i o n i s to the h i g h group i f f(n)^-  0 and  t o the low group i f  f(n)<0.  I f t e s t items were o f equal d i f f i c u l t y then A values of 1.39,  2.30,  3.00,  and 4.61  would correspond to upper  bounds on the p r o b a b i l i t i e s of m i s c l a s s i f i c a t i o n o f .25,  .10,  20 .05, and .01 r e s p e c t i v e l y ( L i n n , et a l . , p.89).  As the t e s t  items used f o r t h i s r e s e a r c h p r o j e c t were not of equal d i f f i c u l t y i t may only be s t a t e d t h a t t h e r e i s an i n v e r s e r e l a t i o n s h i p between the value of A and the e r r o r p r o b a b i l ities.  T h i s i s not a s e r i o u s l i m i t a t i o n however, as the  value of A may be e a s i l y a d j u s t e d to provide the d e s i r e d l e v e l of accuracy i n c l a s s i f i c a t i o n . At any given value of A the accuracy  l e v e l may  found by simply determining what percentage have been c o r r e c t l y c l a s s i f i e d .  be  of subjects  The value o f A may then be  a d j u s t e d a c c o r d i n g l y and the percentage r e c a l c u l a t e d , r e p e a t i n g t h i s process u n t i l an A value i s determined f o r the d e s i r e d l e v e l of accuracy i n c l a s s i f i c a t i o n .  As t h i s  l e v e l i n c r e a s e s however, so should the average number of t e s t items r e q u i r e d f o r c l a s s i f i c a t i o n . Psychological Tests Only two c r i t e r i a were used i n s e l e c t i n g t e s t s f o r demonstrating  the model.  The f i r s t was t h a t the t e s t s  should u t i l i z e m u l t i p l e - c h o i c e format input to the computer.  Secondly,  answers to f a c i l i t a t e  i t was d e s i r a b l e that the  t e s t s should be of a d i f f i c u l t y l e v e l such t h a t there would be s u f f i c i e n t v a r i a n c e i n the observed  test scores.  What  the t e s t s a c t u a l l y measured was not c r i t i c a l to the d e s i g n . As the model was only designed  to p r e d i c t o v e r a l l score on  21 the t e s t s b e i n g a d m i n i s t e r e d , o t h e r c r i t e r i a were not The  tests  required.  chosen f o r the d e m o n s t r a t i o n were the V e r b a l  and N u m e r i c a l f o r m s o f t h e M i n n e s o t a M u l t i m o d a l A n a l o g y T e s t 1971  Revision  (Appendix A ) .  Only the f i r s t  forty  items  from each form were used t o a l l o w f o r c l a s s r o o m a d m i n i s t r a t i o n i n gathering the c a l i b r a t i o n  sample.  Computer model A F o r t r a n IV computer  m o d e l ( A p p e n d i x B) was  developed  to p e r f o r m the f o l l o w i n g f u n c t i o n s from a v i d e o s c r e e n conv e r s a t i o n a l computer  t e r m i n a l ( I B M 3270);  1. d e t e r m i n e t h a t t h e s u b j e c t i s g e n u i n e a n d h a s p r e v i o u s l y undergone  the computerized  not  testing;  2. d e t e r m i n e o r d e r o f t e s t a d m i n i s t r a t i o n a c c o r d i n g to subject's  identification;  3. a d m i n i s t e r t e s t i n s t r u c t i o n s and s a m p l e  questions;  4. a d m i n i s t e r t e s t i t e m s , one a t a t i m e ; 5. p e r f o r m s t a t i s t i c a l a n a l y s i s a f t e r e a c h r e s p o n s e t o d e t e r m i n e i f s u b j e c t c a n be c l a s s i f i e d  (and  test  terminated); 6.  store t e s t i n g data i n a f i l e a f t e r each t e s t  (unique to that  response;  7. p r e s e n t t e r m i n a t i o n message t o  subject.  subject)  An i d e n t i f i c a t i o n f i l e  i s created which c o n s i s t s o f  each s u b j e c t ' s s e c t i o n and student 'signs-on'  number.  A subject  to the c o n v e r s a t i o n a l computer t e r m i n a l u s i n g  p r i n t e d i n s t r u c t i o n s (Appendix C) to operate keyboard. and  The computer then requests  the t e r m i n a l ' s  the s u b j e c t ' s  student  s e c t i o n numbers by d i s p l a y i n g i n s t r u c t i o n s on the t e r -  minal's  video screen.  Next a storage f i l e  i s c r e a t e d by the  computer u s i n g the s u b j e c t ' s i d e n t i f i c a t i o n number as the f i l e name, and the calendar date and time of day are entered on the f i l e .  I f the s u b j e c t ' s number cannot be found i n the  identification f i l e  (which i n d i c a t e s he i s probably  genuine s u b j e c t ) or i f a storage f i l e  not a  f o r that number a l r e a d y  e x i s t s (which means the s u b j e c t has p r e v i o u s l y undergone the computerized t e s t i n g ) an a p p r o p r i a t e message i s r e t u r n e d t o the screen and the s u b j e c t i s a u t o m a t i c a l l y  'signed-off  the t e r m i n a l .  The  v e r b a l and numerical  analogy t e s t s and a s s o c i a t e d  i n s t r u c t i o n s r e s i d e i n separate (Appendix D). is for  computer storage  files  The order i n which the t e s t s are administered  determined by the s u b j e c t ' s s e c t i o n number. . I n s t r u c t i o n s the f i r s t  t e s t s a r e then d i s p l a y e d on the t e r m i n a l  along with a sample q u e s t i o n .  screen  These remain on the screen  u n t i l the subject depresses a key i n d i c a t i n g he i s ready to  • proceed.  23  At t h i s p o i n t a t i m i n g r o u t i n e i s a c t i v a t e d which  computes and  records ( i n m i l l i s e c o n d s ) the time taken by  s u b j e c t to respond to each question, the CPU  the  (central  pro-  c e s s i n g u n i t ) time a s s o c i a t e d with a d m i n i s t r a t i o n and  cal-  c u l a t i o n s f o r each q u e s t i o n , and Next, an analogy (only one  the t o t a l elapsed  q u e s t i o n and f i v e p o s s i b l e answers  of which i s c o r r e c t ) are d i s p l a y e d on the  and remain there u n t i l a response between '0' and entered on the t e r m i n a l keyboard.  response i n d i c a t e s the answer i s not known and ered an i n c o r r e c t response to the  '5' is'  A  '0'  i s consid-  question.  response i s then evaluated as d i s c u s s e d above under  ' c l a s s i f i c a t i o n procedure'  (p. 18).  A l l t e s t i n g data a s s o c i a t e d  with the t e s t item are a u t o m a t i c a l l y p l a c e d on the file.  screen  A l l other responses are  r e j e c t e d and an a p p r o p r i a t e message i s d i s p l a y e d .  The  time.  These i n c l u d e :  subject's  q u e s t i o n i d e n t i f i c a t i o n , item  response,  c o r r e c t / i n c o r r e c t i n d i c a t o r , p r o b a b i l i t y l e v e l s f o r high  and  low groups, c a l c u l a t e d Rj_ and A scores, A l e v e l i n e f f e c t , t o t a l number of items r i g h t , t o t a l percent r i g h t ,  classificat-  i o n i n d i c a t o r ( i e . high, low,  and  or none), i n t e r r u p t  o p e r a t i n g e r r o r counters,  t o t a l CPU  and  time.  elapsed item response  time,  t o t a l elapsed  time,  24 If  classification  i s n o t made t h e n t h e n e x t  is  p r e s e n t e d and t h e p r o c e s s i s r e p e a t e d u n t i l  is  made.  classification  I f t h e s u b j e c t i s c l a s s i f i e d as b e l o n g i n g  low group then t e s t i n g i s t h e o r e t i c a l l y classified  terminated.  t o the h i g h group then t h e next t e s t  t e r e d f o l l o w i n g t h e same p r o c e d u r e test.  question  to the :  If  i s adminis-  as o u t l i n e d f o r the f i r s t  When t e s t i n g i s t e r m i n a t e d , t h e d a t e a n d t i m e a r e  w r i t t e n on t h e e n d o f t h e s u b j e c t ' s f i l e ,  a termination  message i s d i s p l a y e d on t h e v i d e o s c r e e n , a n d t h e s u b j e c t i s automatically  'signed-off'.  F a c i l i t i e s are b u i l t  into the  program t o permit t h e s u b j e c t t o s i g n h i m s e l f o f f i n t h e c a s e o f a n e m e r g e n c y a n d t o s i g n on a g a i n l a t e r , up w h e r e he l e f t file  off.  picking  Any s u c h a c t i o n i s r e c o r d e d  on h i s  i n c l u d i n g p l a c e a n d l e n g t h o f i n t e r r u p t i o n so t h a t t h e  d a t a c a n be e x c l u d e d  f r o m a n a l y s i s where d e s i r e d . I V DISCUSSION  The and  CCTT m o d e l was i n t e n d e d t o s e r v e b o t h  research purposes  demonstration  a n d was t h u s k e p t r a t h e r s i m p l e i n  terms o f d e c i s i o n d e s i g n and t y p e s o f t e s t s used. a t i o n purposes  Demonstr-  r e q u i r e d t h a t t e s t i n g be t e r m i n a t e d o n c e a  s u b j e c t was c l a s s i f i e d w h i l e t h e r e s e a r c h d e s i g n r e q u i r e d a d m i n i s t r a t i o n o f t h e complete  tests i n different  (see Chapter  For these reasons, the  V, M e t h o d o l o g y ) .  sequences  25 d e c i s i o n by t h e c o m p u t e r m o d e l t o c l a s s i f y a s u b j e c t to test and  c o m p l e t i o n was  he was  simply recorded  allowed to f i n i s h the t e s t s ,  valuable v a l i d a t i o n data.  on t h e s u b j e c t ' s  o f CCTT b u t  recorded  This also permitted  to permit  file  thus p r o v i d i n g • collection  of c l a s s i f i c a t i o n data at a v a r i e t y of A values. i n f o r m a t i o n was  prior  relative  cost  Timing estimates  s e v e r a l other uses of t h i s data are a l s o p o s s i b l e  (see Chapter V I ) .  CHAPTER V APPLICATION OF THE MODEL I SAMPLE The  sample was comprised of a t o t a l o f 180 U n i v e r s i t y of  B r i t i s h Columbia students from f o u r s e c t i o n s of an undergraduate i n t r o d u c t o r y o r g a n i z a t i o n a l behavior course.  P a r t i c i p a t i o n was  on a v o l u n t a r y b a s i s and r e q u i r e d approximately two hours o f each subject's  time, one hour o f which was during r e g u l a r l y scheduled  c l a s s e s and the other a t the s u b j e c t ' s time p e r i o d .  convenience w i t h i n a g i v e n  I n d i v i d u a l performance data and o v e r a l l f i n d i n g s  were presented t o the s u b j e c t s upon completion o f the study and bonus grade p o i n t s were awarded.  T e s t i n g wa-s conducted over a  seven week p e r i o d during the f a l l  of 1973.  I I METHODOLOGY Two of the f o u r s e c t i o n s were randomly s e l e c t e d and the paper v e r s i o n o f the t e s t s (Appendix A) was administered One s e c t i o n was g i v e n the v e r b a l t e s t f i r s t , numerical t e s t . the reverse  followed  i n class.  by the  The second s e c t i o n r e c e i v e d the same t e s t s i n  order.  Responses were scored  f o r each t e s t and r e s u l t s  f o r the s e c t i o n s were combined i n t o a 'paper l ' s t ' group. The  median score f o r the group on the v e r b a l t e s t was then  calculated low  (22/40) and used t o d i v i d e the group i n t o high and  scoring categories.  Next, the responses o f the  27 h i g h category f o r each v e r b a l t e s t item were examined to determine what p r o p o r t i o n answered the item r i g h t . p r o p o r t i o n was  This  the e q u i v a l e n t t o the p r o b a b i l i t y of a high  v e r b a l s c o r e r g e t t i n g the item r i g h t .  Similarly,  item  p r o b a b i l i t i e s were c a l c u l a t e d f o r each v e r b a l item when membership was The  i n the low  same procedure was  a b i l i t i e s f o r high and test  category.  (median * 26/40).  low  used to determine item probc a t e g o r i e s on the numerical  These v e r b a l and  numerical  paper  item  p r o b a b i l i t i e s (Table I) were then placed on computer  files  (Appendix D) f o r input i n t o the CCTT model as d e s c r i b e d i n the p r e v i o u s  chapter.  A p e r i o d of approximately  three weeks was  elapse from when the f i r s t two paper v e r s i o n of the t e s t s .  s e c t i o n s had  allowed  completed  the  Next the computer v e r s i o n of  the t e s t s (the CCTT model) was  made a v a i l a b l e to a l l  s u b j e c t s ( f o u r s e c t i o n s ) f o r a p e r i o d of nine days. t h i s p e r i o d each s u b j e c t was  to  permitted  During  to undergo the com-  puter a d m i n i s t r a t i o n of the t e s t s at h i s or her  convenience  using the i n s t r u c t i o n s handed out i n c l a s s (Appendix C) get  'signed-on' to the computer.  video  Once 'signed-on',  screen of the c o n v e r s a t i o n a l computer t e r m i n a l  a l l necessary t e s t s with  the provided  i n s t r u c t i o n s f o r the s u b j e c t to complete  ' s i g n - o f f being handled a u t o m a t i c a l l y by  to  the  the  TABLE I TEST ITEM PROBABILITIES - CALIBRATION SAMPLE  Item  Verbal Pli  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 p  hi  Pli  .973 .622 .676 .243 .216 .189 .784 .351 .459 .622 .297 .243 .595 .568 .297 .568 .405 .297 .676 .622 .541 .351 .324 .216 .405 .243 .486 .378 .243 .081 .703 .568 .405 .459 .189 .216 .734 .703 .541 .595  Numerical Phi  .933 .646 .375 .625 .438 .354 .896 .771 .604 .604 .563 .167 .958 .896 .521 .750 .542 .396 .396 .792 .563 .729 .646 .479 .833 .750 .813 .667 .521 .183 .875 .792 .729 .667 .354 .563 .875 .833 .604 .833  p  l i  .463 .561 .000 .829 .171 .780 .073 .659 .244 .878 .195 .329 .268 .585 .317 .415 .195 .415 .634 .512 .805 .535 .146 .171 .268 .488 .805 .220 .951 .902 .098 .927 .927 .878 .585 .829 .732 .878 .317 .805  p  hi  .455 .733 .068 .864 .114 .955 .136 .977 .659 1.000 .432 .977 .432 .818 .432 .682 .364 .477 .773 .864 .955 .341 .227 .568 .568 .795 .909 .159 .959 .977 .227 1.000 .927 1.000 .386 .955 .955 1.000 .295 1.000  - p r o p o r t i o n of s u b j e c t s i n h i g h s c o r i n g group who answer t e s t item i c o r r e c t l y - p r o p o r t i o n of s u b j e c t s i n low s c o r i n g group who answer t e s t item i c o r r e c t l y  28  29 computer.  D e t a i l s o f how the CCTT model a d m i n i s t e r e d and  scored the t e s t s were provided i n the p r e v i o u s chapter under ' d e s i g n ' ( p . 18). =  The  model was a r t i f i c i a l l y  l i m i t e d t o a maximum o f s i x  s u b j e c t s at any one time to a v o i d t y i n g up too many s a t i o n a l t e r m i n a l s f o r one p r o j e c t . automatically ' s i g n s - o f f  conver-  An abandoned t e r m i n a l  a f t e r a short i n a c t i v e p e r i o d .  Random v i s i t s were made (by this r e s e a r c h e r ) to the t e r m i n a l area t o check f o r c o l l a b o r a t i o n o f s u b j e c t s and to p r o v i d e a s s i s t a n c e i f any d i f f i c u l t i e s were being encountered i n terminal operation.  N e i t h e r were  observed.  For those s u b j e c t s that had a l r e a d y completed the paper v e r s i o n o f the t e s t s , the CCTT model a d m i n i s t e r e d the two t e s t s i n the o p p o s i t e order. did  the v e r b a l t e s t f i r s t  That  i s , those s u b j e c t s that  on the paper v e r s i o n o f the t e s t s  were g i v e n the numerical t e s t f i r s t also resulted i n a different  by the computer.  sample q u e s t i o n being  This  presented  as o n l y one sample q u e s t i o n ( e i t h e r v e r b a l or numerical) was presented f o r a complete t e s t i n g . Once the computer phase o f t e s t i n g was complete, three weeks were allowed t o e l a p s e . of  the t e s t s was administered  another  Then, the paper v e r s i o n  i n c l a s s t o the two s e c t i o n s  that had only completed the computer v e r s i o n .  Again the  order o f t e s t s was r e v e r s e d from that of the p r e v i o u s administration.  30  III Test  RESULTS AND  order Administration  the  INTERPRETATION  o f t h e v e r b a l and n u m e r i c a l  forms o f  a n a l o g y t e s t was c o u n t e r b a l a n c e d t o o f f s e t a n y  ordering  effects.  Table I I i n d i c a t e s that f o r the verbal  t e s t t h e r e was no s i g n i f i c a n t  difference i n verbal  w h e t h e r t h i s t e s t came b e f o r e  or a f t e r the numerical  This  a p p l i e d t o b o t h t h e paper and computer  of the t e s t s .  For the numerical  With paper a d m i n i s t r a t i o n the numerical  test.  administrations.  score  followed  was  test followed  W i t h computer a d m i n i s t r a t i o n however,  performance d e c l i n e d  of  e f f e c t w h i c h was i n  o p p o s i n g d i r e c t i o n s f o r p a p e r and c o m p u t e r  when t h e n u m e r i c a l  test.  administration  t e s t however, o r d e r  a d m i n i s t r a t i o n d i d have a s i g n i f i c a n t  cantly higher  scores  signifi-  the verbal numerical  s i g n i f i c a n t l y when t h e n u m e r i c a l  the verbal t e s t .  The r e a s o n s f o r t h e s e  test  observed  d i f f e r e n c e s are not c l e a r . Learning  effects  A period the  first  of approximately three  weeks was l e f t  between  paper a d m i n i s t r a t i o n o f t h e t e s t s and t h e computer  a d m i n i s t r a t i o n , a n d a n o t h e r t h r e e weeks b e t w e e n t h i s a n d the  s e c o n d p a p e r a d m i n i s t r a t i o n so a s t o m i n i m i z e a n y  learning effects.  Nevertheless,  significant  learning effects  31  TABLE I I COMPARISON OF TESTING ORDER SCORESCCTT AND PAPER TESTING  Group  N  Test  P a p e r l ' s t 45 40  V  45 40  N  26  V  CCTT l ' s t  32 23 33  N  Administration  x = .1 xx - .05 xxx = .01  (2  Standard T Test D e v i a t i o n Value  V/N N/V  22.222  5.464  23.625  5.16*2  V/N N/V  25.639 23.225  2.891 4.933  1.200  2.729 xxx  V/N  22.335  4.913  N/V  23.933  5.325  0.733  V/N N/V  21.733 25.379  6.082 3.577  2.839 xxx  N- N u m e r i c a l V= V e r b a l Significance level  Mean  tail)  32  did  o c c u r on o v e r a l l t e s t  those  subjects completing  t h e r e was t e s t and  a significant  scores (see Table  administration f i r s t  For  the paper a d m i n i s t r a t i o n f i r s t ,  l e a r n i n g e f f e c t on t h e v e r b a l  a p o s i t i v e though not  on t h e n u m e r i c a l t e s t .  III).  significant  For those  learning  completing the  t h e r e were s i g n i f i c a n t  effect  computer  learning  on b o t h t h e v e r b a l a n d n u m e r i c a l t e s t s w i t h t h e  effects  numerical  l e a r n i n g e f f e c t being the s t r o n g e r . Significant overall test  learning  times  e f f e c t s were a l s o e v i d e n t i n  (see Table  the paper t e s t s f i r s t  took  IV).  Those s u b j e c t s c o m p l e t i n g  s i g n i f i c a n t l y l e s s t i m e on  c o m p u t e r v e r s i o n o f b o t h t h e v e r b a l and than d i d the computer f i r s t  group.  l e t i n g the computer t e s t s f i r s t  numerical  significantly  t i m e on t h e p a p e r v e r s i o n s o f t h e v e r b a l and t e s t s than d i d the paper f i r s t  tests  S i m i l a r l y , those  took  the  comp-  less  numerical  group.  Test a d m i n i s t r a t i o n Scores. overall test  No  s i g n i f i c a n t d i f f e r e n c e s were observed  s c o r e s between the paper f i r s t  and  first  a d m i n i s t r a t i o n s of e i t h e r the v e r b a l or  tests  (see Table  t h e two are  III).  computer  numerical  T h i s would tend to i n d i c a t e  m e t h o d s o f t e s t a d m i n i s t r a t i o n , p a p e r and  comparable.  in  that  computer,  33 TABLE I I I  COMPARISON OF TEST SCORES CCTT AND PAPER TESTING T Test Value  Test  Administration  Mean  Standard Deviation  69  V  CCTT Paper  25.725 23.203  5.377 5.293  2.756 x x x  69  N  CCTT Paper  26.058 25.319  2.950 3.367  1.361  48  V  CCTT Paper  23.417 25.438  5.222 4.855  1.943  48  N  CCTT Paper  23.896 26.375  5.203 3.279  2.763 x x x  Paper l ' s t CCTT l ' s t  85 67  V V  Paper CCTT  22.882 23.896  5.348 5.240  1.166  Paper 1'st CCTT l ' s t  85 63  N N  Paper CCTT  24.529 24.333  4.151 5.016  -0.251  Group  Paper 1'st  CCTT l ' s t  N  N= N u m e r i c a l V= V e r b a l Significance level x = .1 x x = .05 x x x = .01  (2 t a i l )  x  34 TABLE IV COMPARISON OF TEST TIMESCCTT AND PAPER TESTING  Test  Administration  Mean  Standard Deviation  T Test Value  Group  N  Paper 1'st CCTT-I'st  70 60  V  CCTT  762.228 920.089  172.698 312.669  3.454 xxxx  Paper I ' s t CCTT I ' s t  70 60  N  CCTT  062.058 1586.636  311.601 665.179  5.55S xxxx  Paper I ' s t CCTT I ' s t  37 68  V  Paper  763.800 633.540  155.340 15 8. 040  4.033 xxxx  Paper I ' s t CCTT I ' s t  37 68  N  Paper  1201.620 990.900  284.760 244.980  3.756 xxxx  Paper 1'st CCS! I ' s t  37 60  V  Paper CCTT  763.800 920.089  155.340 312.669  3.240 xxx  Paper 1'st CCTT I ' s t  37 60  N  Paper CCTT  1201.620 1586.636  284.760 665.179  3.399 xxxx  N= Numerical V= V e r b a l Significance level x = .1 xx = .05 xxx = .01 xxxx = .001  (2 t a i l )  35  Times.  F o r complete t e s t s ,  computer a d m i n i s t r a t i o n  was s i g n i f i c a n t l y slower than paper a d m i n i s t r a t i o n f o r both v e r b a l and numerical t e s t i n g is,  on an item f o r item comparison,  (see Table I V ) .  That  s u b j e c t s took l o n g e r on  the computer a d m i n i s t r a t i o n with an average  time per v e r b a l  item o f 23.0 seconds compared w i t h 19.1 f o r paper  adminis-  t r a t i o n and 39.7 seconds per numerical item compared with 30.0 seconds f o r paper.  Some of t h i s d i f f e r e n c e may be  e x p l a i n e d by the s l i g h t l y d i f f e r e n t i n s t r u c t i o n s  regarding  times f o r each t e s t a d m i n i s t r a t i o n (see Appendices A, D). I t i s a l s o l i k e l y that s u b j e c t s d i d not experience the same pressure i n CCTT as they d i d i n the group t e s t admini s t r a t i o n where t h e i r completion time was observable by their  classmates. The mean time to c l a s s i f y a s u b j e c t u s i n g CCTT was  substantially  l e s s (V* 205 s e c , N * 4 7 8 s e c , Table V)  than to completely t e s t a s u b j e c t u s i n g paper a d m i n i s t r a t i o n (V= 7 6 4 s e c , N * 1202 s e c , T a b l e I V ) .  T h i s demonstrates  that s i g n i f i c a n t time savings may be r e a l i z e d i n u s i n g a s e q u e n t i a l CCTT model f o r c l a s s i f i c a t i o n w i t h t h i s type of test.  ( T o t a l time saving = 65$, V = 7 3 % N = 60%). f  Items.  The number o f items on the paper a d m i n i s t r a -  t i o n of the v e r b a l and numerical t e s t s was f o r t y each (Appendix A ) . average  S u b j e c t s were c l a s s i f i e d by CCTT a f t e r an  o f 9.0 v e r b a l items and 9.9 numerical items  (Table V ) .  36  TABLE V  TIME AND NUMBER OF ITEMS REQUIRED FOR CCTT TO CLASSIFY SUBJECTS (A LEVEL = 1.39) CCTT I ' s t (N.=60) V e r b a l Numerical  Elapsed Time (sec)  mean s t d . dev, minimum maximum per item  CPU Time (sec) Items  205.3 161.2 43 ^37 22.8  1.101 mean s t d . dev. minimum maximum  9.0 5.3 4 23  473.4 374.3 95 2068 47.3  1.200 9.9 7.1 3 33  Paper t e s t i n g I ' s t (N*70) Verbal Numerical  184.0 125.1 49 636 19.4  1.151 9.5 5.3 4 27  317.9 233.3 42 1299 28.4 1.359 11.2 6.4 3 35  37 This represents  an average r e d u c t i o n o f more than 75% i n the  number o f t e s t items r e q u i r e d u s i n g CCTT. Accuracy.  With the s u b s t a n t i a l r e d u c t i o n i n number o f  t e s t items r e q u i r e d , CCTT was s t i l l able t o c l a s s i f y on the numerical  t e s t w i t h an accuracy  v e r b a l t e s t with an accuracy Costs.  subjects  of $1.0% and on the  of 8*5.1% (Table V I ) .  Although s e v e r a l s i m p l i f y i n g assumptions were made  i t would appear that CCTT c o s t s would be competitive t e s t i n g (Appendix G).  In the example presented,  with  paper  a c t u a l CCTT  c o s t s were computed to be $.730 per s u b j e c t compared with hypot h e t i c a l c o s t s o f $.675 per s u b j e c t u s i n g paper t e s t i n g , a d i f f e r e n c e o f 8%. Anxiety. design,  Although not s p e c i f i c a l l y p a r t o f the study  r e a c t i o n to CCTT was i n f o r m a l l y d i s c u s s e d with  each  of the f o u r t e s t groups i n p o s t - t e s t a n a l y s i s s e s s i o n s . Reaction was f o r the most part f a v o r a b l e to CCTT.  Many i n d i c a t e d  enjoyment o f the n o v e l t y o f CCTT and others i n d i c a t e d they f e l t much l e s s tense than i n the group paper a d m i n i s t r a t i o n .  Several  noted f r u s t r a t i o n a t not being able to change answers on the terminal.  Only three s u b j e c t s expressed o b j e c t i o n s t o the con-  cept o f CCTT and two of these i n d i v i d u a l s were a l s o a g a i n s t any form o f p s y c h o l o g i c a l measurement.  A l l s u b j e c t s were i n favour  of the convenient demand t e s t i n g f e a t u r e .  TABLE V I  CCTT SUBJECT C L A S S I F I C A T I O N TABLES (A L E V E L = 1.39)  Numerical Test  Actual  Classified High Low  Total  High  30  2  32  Low  10  21  31  Total  40  23  63  A c c u r a c y = 81.0% Verbal Test  Classified High Low High  Actual  Low Total  Total  39  8  47  2  18  20  41  26  67  Accuracy  = 85. Ifo  CHAPTER  M  CONCLUSIONS AND SUGGESTIONS FOR FURTHER RESEARCH I DISCUSSION Advantages -CCTT appears i d e a l l y s u i t e d tailored  to m u l t i - s t a g e and  testing.  -CCTT f a c i l i t a t e s the a p p l i c a t i o n complex d e c i s i o n probability  of s t a t i s t i c a l l y  and t e s t i n g models such as the  s e q u e n t i a l model presented here which  cannot be a p p l i e d  i n a conventional t e s t i n g  envir-  onment . -CCTT o f f e r s the convenience o f demand, twenty-four hour t e s t i n g t o both examiner and examinee. -CCTT i s s e l f - i n s t r u c t i o n a l and e l i m i n a t e s the need f o r a formal t e s t  administrator.  -CCTT i s capable of s t a n d a r d i z e d t e s t  instruction.  -CCTT can s i g n i f i c a n t l y reduce t e s t i n g time and the number o f t e s t items r e q u i r e d . -CCTT permits adjustment o f t e s t i n g accuracy l e v e l desired. -CCTT a u t o m a t i c a l l y , i n s t a n t a n e o u s l y and o b j e c t i v e l y scores, r e c o r d s , and i n t e r p r e t s viding  test results,  pro-  immediate feedback c a p a b i l i t i e s without the  40 intervention of a psychologist. -CCTT i s capable  of a c c u r a t e l a t e n c y response  recording. -CCTT data c o l l e c t i o n p r o v i d e s a broad, h i g h l y access i b l e data base f o r f u r t h e r r e s e a r c h and a n a l y s i s . -CCTT i s capable o f c o l l e c t i n g s e l f - v a l i d a t i o n -CCTT does not appear t o a f f e c t o v e r a l l t e s t -CCTT does appear to be r e l a t i v e l y cost with c o n v e n t i o n a l  data.  scores.  competitive  testing.  -CCTT does not appear to generate  undue a n x i e t y .  -CCTT i s p o r t a b l e u s i n g a s u i t c a s e type t e r m i n a l that operates  over any c o n v e n t i o n a l  telephone.  -CCTT i s probably a p p l i c a b l e to many other forms of testing. Disadvantages -CCTT may not be r e a d i l y a p p l i c a b l e to c e r t a i n forms o f t e s t i n g , such as those r e q u i r i n g p i c t o r i a l r e p r e s e n t a t i o n . -CCTT may be r e l a t i v e l y expensive  f o r some forms o f t e s t i n g ,  such as those r e q u i r i n g a d m i n i s t r a t i o n of the e n t i r e -CCTT p r e s e n t l y r e q u i r e s c o n s i d e r a b l e development and testing. -CCTT i s dependent upon a r e l i a b l e computer system which i s not prone to frequent or prolonged  failures.  test.  40-  -CCTT may a d v e r s e l y a f f e c t the performance o f some i n d i v i d u a l s such as those who f e e l threatened  by a  a computer. I I LIMITATIONS Only one CCTT model was developed.  I t s emphasis was  c a t e g o r i z a t i o n of s u b j e c t s r a t h e r than continuous measurement and i t p r e d i c t e d t e s t performance r a t h e r than task ( j o b ) performance.  Reactions  and a n x i e t i e s of s u b j e c t s between  the two modes o f t e s t i n g were not s p e c i f i c a l l y  studied.  Formal and d e t a i l e d cost comparisons of the two methods o f t e s t i n g were not attempted. I l l CONCLUSIONS From the present -Various  i n v e s t i g a t i o n i t i s concluded t h a t :  p s y c h o l o g i c a l t e s t s may be administered v i a  c o n v e r s a t i o n a l computer t e r m i n a l s . -A v a r i e t y of CCTT models are p o s s i b l e . -A s e q u e n t i a l p r o b a b i l i t y CCTT model w i l l  significantly  shorten t e s t i n g time. -CCTT i s s e l f - a d m i n i s t e r i n g . -CCTT appears c o s t - c o m p e t i t i v e testing.  with  conventional  42 IV SUGGESTIONS FOR  FURTHER RESEARCH  F u r t h e r r e s e a r c h of the CCTT concept  i s suggested i n  seven main areas; the present model, other models, t e s t s , criteria,  a n x i e t y , c o s t s , and  applications.  With regard to the present model i t i s l i k e l y that the number of items r e q u i r e d f o r c l a s s i f i c a t i o n may i a l l y reduced  be  substant-  by a r r a n g i n g t e s t items so that maximum-  c r i m i n a t o r s are encountered  e a r l y i n the t e s t .  dis-  Such a  procedure would most l i k e l y r e q u i r e c o l l e c t i o n of a f u r t h e r c a l i b r a t i o n sample to determine s i g n i f i c a n t e f f e c t on  i f item o r d e r i n g had  any  responses.  A d d i t i o n a l , more complex models such as those d i s c u s s e d i n Chapter I I I should be examined.  Two  m u l t i - s t a g e models p a r t i c u l a r l y warrant a model that w i l l determine  types of branching investigation.  which t e s t s should be  One i s  administered.  T h i s would l i k e l y operate on the b a s i s of p r e - d e f i n e d d e c i s i o n r u l e s f o r ranges abilities.  of scores on t e s t s such as those measuring  A more complex v e r s i o n of t h i s model would d e a l  w i t h t e s t s i n areas such as p e r s o n a l i t y measurement. second  The  branching m u l t i - s t a g e model i s one which would branch  to  t e s t items of a p p r o p r i a t e d i f f i c u l t y l e v e l s f o r the s u b j e c t . T h i s again could shorten t e s t s , p r o v i d i n g more f i n i t e ment a t the a p p r o p r i a t e l e v e l .  measure-  Another type of model r e q u i r i n g  more s p e c i f i c , problem-oriented r e s e a r c h i s the  complexity/  43 branching model d e s c r i b e d i n Chapter  III.  It i s  of. many p o s s i b l e segments, each of which may i t s own.  F u r t h e r , i t may  comprised  be researched  have to be developed  on  for specific  a p p l i c a t i o n s which are perhaps unique to a g i v e n environment. Other forms and  types of t e s t s should a l s o be  u s i n g CCTT, i n c l u d i n g v a r i o u s response methods of response  modes.  attempted  Alternative  input should be researched f o r t e s t s t h a t  do not lend themselves to m u l t i p l e choice answers. c o n v e r s i o n of t e s t s to convenient  input formats  p l e choice' should a l s o be c o n s i d e r e d .  Possible  such as ' m u l t i -  Various methods of t e s t  p r e s e n t a t i o n should be explored f o r t e s t s that i n v o l v e nonv e r b a l items such as p i c t o r i a l r e p r e s e n t a t i o n s . CCTT should be a p p l i e d to the p r e d i c t i o n of a c t u a l performance c r i t e r i a as w e l l as known t e s t t e s t i n g model was  scores.  used to p r e d i c t a c t u a l t e s t  The  present  scores when i n  f a c t t e s t s are t y p i c a l l y designed to p r e d i c t c e r t a i n performance criteria.  T h i s r e p r e s e n t s a more d i f f i c u l t a p p l i c a t i o n of the  model as t e s t v a l i d i t y becomes a determining  factor i n i t s  effectiveness. Formal measurement of o b j e c t i o n s to CCTT and a n x i e t y l e v e l s should be s t u d i e d . assume t h a t some i n d i v i d u a l s may  resultant  I t appears reasonable  to  e i t h e r o b j e c t to CCTT or  experience a change i n a n x i e t y l e v e l while undergoing  CCTT.  The nature and  For  extent of these should be determined.  44  example, they may be found to e x i s t but not a d v e r s e l y a f f e c t t e s t performance or may only e x i s t w i t h i n c e r t a i n groups o f i n d i v i d u a l s under c e r t a i n circumstances. Comparative cost data should be c o l l e c t e d i n the a c t u a l a p p l i c a t i o n o f CCTT i n an a p p r o p r i a t e environment.  The present  study was l i m i t e d to h y p o t h e t i c a l paper t e s t i n g c o s t s , and comparisons were on the b a s i s o f a short t e s t i n g p e r i o d i n an experimental environment. A d d i t i o n a l a p p l i c a t i o n s and expansion o f the CCTT concept such as o u t l i n e d i n Chapter I should be explored more f u l l y . Specifically,  t h i s e n t a i l s expansion o f the CCTT concept to  i n t e r v i e w , s e l e c t , and u l t i m a t e l y place a p p l i c a n t s .  45; V SUMMARY CCTT has been demonstrated to be a v i a b l e for  at l e a s t one form of p s y c h o l o g i c a l  will  eliminate  testing.  As such i t  two t i m e l y delays i n the personnel s e l e c t i o n  process as d e s c r i b e d administration  substitute  i n Chapter one, namely a w a i t i n g  and t e s t i n t e r p r e t a t i o n w h i l e a t the same time  p r o v i d i n g numerous other advantages.  I t has the added ad-  vantagenof being capable of s u b s t a n t i a l l y reducing time.  test  testing  More-over the CCTT concept appears expandable to the  e n t i r e s e l e c t i o n process thus s o l v i n g f u r t h e r time delay and o b j e c t i v i t y problems.  Further  research,  g a n i z a t i o n a l s e t t i n g w i l l be r e q u i r e d  p r e f e r a b l y i n an org  to explore  t h i s and  other questions r a i s e d . With the c o s t s of computer f a c i l i t i e s c o n s t a n t l y  declining,  hardware becoming more s o p h i s t i c a t e d , and complete computer systems a v a i l a b l e to the small user on a time s h a r i n g i t appears h i g h l y probable that CCTT w i l l conventional  psychological testing.  eventually  basis, replace  BIBLIOGRAPHY Annett, J . 'A low-cost cheat-proof t e a c h i n g system', Programmed Learning, 196/+., 1, pp. 155-8. Armitage, P. S e q u e n t i a l a n a l y s i s with more than two a l t e r n a t i v e hypotheses, and i t s r e l a t i o n to d i s c r i m i n a n t f u n c t i o n a n a l y s i s . J o u r n a l of the Royal S t a t i s t i c a l S o c i e t y , 1950, 12, pp. 137-144. C l e a r y , ~ T . A., L i n n , R. L., & Rock, D. A. E x p l o r a t o r y study of programmed t e s t s . E d u c a t i o n a l and P s y c h o l o g i c a l Measurement, 1968, 28, pp. 345-360. Cronbach, L. J . and G l e s e r . G. C. P s y c h o l o g i c a l t e s t s and personnel d e c i s i o n s . (2nd ed.) Urbana, 111.: U n i v e r s i t y of. I l l i n o i s Press, 1965. Dunn, T. G., Lushene, R. E., & O ' N e i l , H. F. Complete automation of the MMPI and a study of i t s response latencies. J o u r n a l of C o n s u l t i n g and C l i n i c a l Psychology, 1972, V o l . 39, No. 3, pp. 381-387. Edwards, D. A. W. C l i n i c a l a n a l y s i s of dysphagia - steps towards automation and e a r l y d i a g n o s i s . In Symposium on Carcinoma of the Oesophagus, 29th Annul General Meeting of The B r i t i s h S o c i e t y of Gastroenterology, London. 1968. E l l i s , A. B. and Tiedeman, D. V. Can a machine counsel? In Computer-assisted I n s t r u c t i o n , T e s t i n g , and Guidance, ed. Wayne H. Holtzman. New York, Evanston, & London: Harper and Row, P u b l i s h e r s , 1970, pp. 345-373. Elwood, D. L. Test r e t e s t r e l i a b i l i t y and cost analyses of automated and f a c e to f a c e i n t e l l i g e n c e t e s t i n g . Int. J . Man-Machine S t u d i e s , 1972, 4, pp. 1-22. Gedye, J . L. and M i l l e r , E. The automation of p s y c h o l o g i c a l assessment. I n t . J . Man-Machine S t u d i e s , 1969, 1, pp. 258259. G i r s h i c k , M. theory.  A. An elementary survey Rev, educ. Res., 1954.  of s t a t i s t i c a l d e c i s i o n  4 7  Hansen, D. N., Hedl, J . J . , & O'Neil, H. F., J r . A review of automated t e s t i n g . Paper presented at the meeting of the American E d u c a t i o n a l Research A s s o c i a t i o n , New York, February 1971. Hedl, J . , O'Neil, H. F., & Hansen, D. N. The e f f e c t i v e nature of computer-based t e s t i n g procedure. Proceedings of the Annual Convention of the American P s y c h o l o g i c a l A s s o c i a -  t i o n , 1971, V o l . 6, pp. 535-536. (a)  O'Neil, H. F., & Hansen, D. N. Computer-based i n t e l l i g e n c e t e s t i n g . Paper presented at the meeting of the American E d u c a t i o n a l Research A s s o c i a t i o n , New York, February 1971. (b) Hubbard, J . P. 'Programmed t e s t i n g i n the examinations of the N a t i o n a l Board of M e d i c a l Examiners'. In: Anne A n a s t a s i , ed. T e s t i n g Problems i n P e r s p e c t i v e . Washington D. C : American C o u n c i l on Education, W 6 6 , pp. 195-207.  L i n n , R. L.,'Rock, D. A., and C l e a r y , T. A. Sequential t e s t i n g f o r dichotomous d e c i s i o n s . E d u c a t i o n a l and P s y c h o l o g i c a l  Measurement, 1972, 32, pp . 85-95. y  Lord, F. M. A t h e o r e t i c a l study of two-stage t e s t i n g . Psychometrika, V o l . 36, No. 3, September 1971. (a) . 'Some t e s t theory f o r t a i l o r e d t e s t i n g ' . Paper presented at the Conference on Computer-Assisted I n s t r u c t i o n , T e s t i n g and Guidance, A u s t i n , Texas, October 1968. Research B u l l e t i n , 68-38 and ONR T e c h n i c a l Report Contract Nonr. 2752 ( 0 0 ) , P r i n c e t o n , N. J . : Educational Testing Service. . Robbins-monro procedures f o r t a i l o r e d t e s t i n g . E d u c a t i o n a l and P s y c h o l o g i c a l Measurement, 1971, 31,  pp.3-31. (b)  L e v i n , R. I . and K i r k p a t r i c k , C. A. Q u a n t i t a t i v e Approaches to Management. New York: McGraw-Hill Book Company, 1965. Mayne, J.';G., Weksel, W. & S h o l t z , P. N. Toward automating the medical h i s t o r y . Proc. S t a f f Meet. Mayo C l i n . 1968, 43, 1 P a i t i c h , D. Computers In b e h a v i o r a l s c i e n c e : A comprehensive automated p s y c h o l o g i c a l examination and r e p o r t (CAPER). B e h a v i o r a l Science. 1973, V o l . 18 ( 2 ) , pp. 131-136.  48 Slosson, R. L. The S l o s s o n I n t e l l i g e n c e Test f o r C h i l d r e n and A d u l t s . East Aurora, 111.: Slosson E d u c a t i o n a l Publication, 1963. Veldman, D. J . and Menaker, S. >L. Computer a p p l i c a t i o n i n assessment and c o u n s e l l i n g . J o u r n a l o f School Psychology, S p r i n g 1968, V o l . 6, No. 3 . Wald, J . H., J r . The c o u n s e l l i n g assignment problem. Psychometrika, 1950, 2 3 , pp. 55-65. Waters, C. W. Comparison of computer-stimulated conventional and branching t e s t s . U.S. Army BESRL T e c h n i c a l Research Note, 1970 (Mar), No. 216, p. 38.  APPENDICES  APPENDIX A Materials  Used i n P a p e r T e s t  Administration  51  Shortened V e r b a l Form Minnesota Multimodal Analogy 1971  Revision  Test  INSTRUCTIONS  -  .5.2  6n the f o l l o w i n g pages are some analogy questions ve would l i k e you to answer. This t e s t has bean designed to take approximately 35 minutes to complete, and you w i l l have to work quickly t o . f i n i s h i n the time allowed. The questions are arranged i n two groups. The f i r s t 'Uo questions are i n word form; the l a s t UO questions are i n the form of numbers. Here i s an example of a word analogy: Feather (1)  Swim  (2)  : Air  ' (3)  ::  Scale  Water  (U)  :  Fish Bird  (5)  Fly  This i s the way i t should read: "Feather i s to blank as Scale i s to F i s h ? " You are to choose the word which best completes the analogy. In t h i s example, you might reason that a scale grows on a f i s h - what does a feather grow on? Look at the f i v e possible answers. "Bird" i s the best answer, so you would write "h" i n the correct space on the answer sheet. The procedure f o r answering the number analogies i s the same. You w i l l f i n d that the reasoning required f o r each analogy question i s d i f f e r e n t , and that the blank may appear i n any p o s i t i o n . For each question you must determine the most appropriate reasoning and then select the word which best completes the analogy. For each analogy question, write the number of the correct answer on the answer sheet. I f you are not sure which i s the correct answer, write the figure " 0 " i n the space provided on the answer sheet and go on to the next question. Please work i n pen, and do not change any answers. Do not make any marks on the t e s t booklet i t s e l f . Be c a r e f u l to put the answer i n the r i g h t  space.  Please answer every question. Begin with the word analogies and continue through the number analogies u n t i l you have answered a l l 80 questions. 3e c a r e f u l that you do not skip any questions. Please record the times where indicated on the answer sheet and turn i n a l l test materials as soon as you have f i n i s h e d . I f you have any questions, please ask the person i n charge.  Please be assured that your responses w i l l be kept i n s t r i c t e s t confidence^  W O R D  :  i  (1) Engineer  A N A L O G I E S  Hunter  ::  Doctor  (2) Gentleman  :  Lawyer  ( 3 ) Drunk  (4) Fisherman  (5) C i t i z e n Dive (1)  :  Swim  ::  (2) C o o  Tree  : (1)  Short  (1)  Steak  Check (1)  .  :  ::  Bacon  Pig  Pick (1)  Miss  .  :  Violin  .  :  (2) Sing  :  (5) D u l l  Money  (4) Giant  (5) G o o s e  ' (4) Blood :  (3) 3 o w  (2) B e s i d e  (5) Que  Sharp  (4) Boring  (3) Maple  Behind  Mr.  (4) Lass  Cash  (3) Note  (5) W i n g :  :  ::  : (1) Before  Shiny  ::  (2) Woman  Play  ::  (5) C i r c u s  Ham  ( 3 ) Nurse  .  Sap  :  Husband  (3) Bright  ::  Dwarf  (4) Eggs  (2) Keen  Man  :  Breast  ::  (5) Swimming  (4) Beard  (3) Baby  (2) G o l d  Axe  Tall  :  :  Mark  Tree (1)  ::  (3) D u l l  (2)  Dumb  (4) Swam  (2) Laugh  _ J  :  '  (3) F l y  Housewife - (2), Lady  Dull (1)  :  Clown.  : (1)  Dove  ::  (v  ( 3 ) After  Banjo  String  Drag  (5) Arm  :  (5) Fiddle Ahead  (4) P u l l  (5) P u s h  I  54  11.  Gasoline (1)  12.  :  Oil  (2)  Walk. : (1)  Car  ::  (2)  : (1)  (1)  15.  16.  Battleship  Sin  :  Chorus  Fire  Ton  :  (2)  Pound  (2) •  18.  Ash .(1)  19.  Books :  (1)  Beech  Pine  Train Tie  (2)  :  Auto  ::  Tire :  20. (1)  Oak  Maple  Tomorrow  (2)  (5)  :  Pistol  (4)  Earth  Lawn  (4)  Ocean  Sing  (5)  (4) :  :  (5)  Oar  Daughter  Sun  Basketball  (5)  Game  Tinkle  (3).Tonight  Teacher ::  Hum  Tin  College  (2)  (2)  ::  Tongue  Gear  Run  Rifle  (3) R o u n d  _ _ _ _ _ _ _  (5)  .  ::  Bright  Stem  (4)  (3)  (3)  Single  Clock  Cannon  :  .  17. (1)  ::  :  •  (3) G r o w  Son  :  (1)  (1)  (2)  •  Rowboat  ::  (4)  :  Canoe (2)  Song  Marble  Mow  Shave  13.  14.  (3) S p r i n g  Wind  Stand  Climb  ::  ::  Pray  (3)  Learn  (4) : (4)  Wet  (5)  Steely  Church Class  (5)  Study  :  (3) W a l n u t Track  (3) C a r Yesterday Tonight  :  (4)  Hickory  (5)  Elm  ' • (4) ::  Road  (5)  Tomorrow  (3) T o d a y  (4)  Path :  Today  Now  (5)  Then  55  •21.  Unkind (1)  22.  23.  : Fire  25.  (2) Smell  26.  27.  Gobble  (2) C e l e r y  28.  Acorn  Deer  : Did  Blue  30.  :  (1)  Bark  Dot(1)  ( 2 ) Honor  ::  ::  Seed  (2) Horses  (4)  Circle  Ball  (2)  :: Tire  (3) Marble (3)  (4)  Eat  (5) Feed  Horse  (4)  Yellow  (3) G r e e n  (2) Hound  (5) D e a r  Turkey  Oak  (5) Saddle  :  (5) C o w s  Orange  (4)  Church  Square  Doesn't :  Music  :  Heaven  :  (3)- C a l f ::  :  (5) Pear  Trust  ( 3 ) November  Cattle  ( 2 ) Red  Orange  :  (3)  (5) Higher  :  (4)  Colt  (5) H a p p y  Potato  Valuable  ::  Does  :  (4)  (3) Sure  (2) Mare  Royal  Work  ::  :  Dog  (4)  Guard  Scent  Carrot  (2) Thanksgiving  Purple  (4)  ( 3 ) Peach  Faith  Person  :  Nut  (1) 29.  :  Talk  (1)  (3) Dent ::  Believe  (1)  (3) Safe  :  Beet  :  :  :  (1)  '  ::  24. (1)  ::  . (2) C r u e l  Cent  Tomato (1)  Sweetheart  Careless  Hire (1)  :  Brown :  (5) G o l d  Hymn (4)  Howl  (5) C a t  " (4)  Cigarette  (5) G l a s s  56 31.  School  : Fish  (1) College 32.  Odd  33.  Who'd  34.  Cloth  Nail  Sea .: Wave (1) Water  37.  Prune  Eager  39.  Bold  (5) Wouldn't  : Elephant  (4) Linen  (5) Fashion  (3). Bolt  (4) Washer  (5) Hammer  : Brook (3) Swell  ::  (4) River  -  (2) Raisin  (2) Hurt  (3) Wine  (3) Pity  (4) Fruit  (5) Grapefruit  : Shame  (4) Dull  ::  (5) Spirit  : Unfair  (2) Unkind  (3) Shy  :: Pool (2) Pan  (5) Creek  : Plum  :: Honor  Pot : (1) Round  (5) Wrong  : Nut  (2) Ocean  : Honest  (1) Forward 40.  (3) Sail  :  (1) Lively  (4) Wise  (4) They  ::  : Grape  (1) Orange 38.  (3) W i l l  ::  (2) C l i p  (5) Book  :  :: Animal  : Screw  (1) Tack 36.  :: We've  (2) Leather  (4) Grove  : (3) Ideal  :  (1) Shirt 35.  (2) Reason  (2) Have  : Tree  (3) Plant  :: Queer  : Y/ould  (1) Went  .  (2) Turtle  : Proper  (1). Correct  ::  (3) Rope  (4) Gentle  (5) Trust  : Loop (4) Top  (5) Pond  Shortened Numerical Form Minnesota Multimodal Analogy 1971  Revision  Test  58  INSTRUCTIONS  On the following pages are some analogy questions we would l i k e you to answer. This test has been designed t o take approximately 35 minutes to complete, and you w i l l have to work quickly t o f i n i s h i n the time allowed. The questions are arranged i n two groups. The f i r s t UO questions are i n number form; the l a s t kO questions are i n word form. Here i s an example of a number analogy:  (1)  7U6  :  9  (2)  ::  h  1*  839 (3)  : 76  ' (M  8  (5)  3  This i s the way i t should read: " F i r s t number i s to Second number as Third number i s to blank." You are t o choose the number which best completes the analogy. In t h i s example, you might reason that the number "U" i s i n the middle of number 7^6. What number i s i n the middle of 839? Look at the f i v e possible answers. Since the number "3" i s i n the middle of 839» you would write "5" i n the correct space on the answer sheet. The procedure f o r answering the word analogies i s the same. You w i l l f i n d that the reasoning required f o r each analogy question i s d i f f e r e n t , and that the blank may appear i n any p o s i t i o n . For each question you must determine the most appropriate reasoning and then select the word which best completes the analogy. For each analogy question, write the number of the correct answer on the answer sheet. I f you are not sure which i s the correct answer, write the figure "0" i n the space provided on the answer sheet and go on to the next question. . Please work i n pen, and do not change any answers. Do not make any marks on the test booklet i t s e l f . Be c a r e f u l to put the answer i n the r i g h t  space.  Please answer every question. Begin with the number analogies and continue through the word analogies u n t i l you have answered a l l 80 questions. Be c a r e f u l that you do not skip any questions. Please record the times where indicated on the answer sheet and turn i n a l l t e s t materials as soon as you have f i n i s h e d . I f you have any questions, please ask the person i n charge.  Please  be assured  t h a t your r e s p o n s e s w i l l be kept i n s t r i c t e s t  confidence.  .  N U M B E R  1.  :  2  6  93  4.  ::  (2) 108  (3) 4137  4  :  (2)  23  :  (1)  45  (3) 2  67  13  ::  (2) 78 "  6.  8.  (2) 34  (1)  17  12  :  (1)  431  405050  :  (1) 2 9.  4 (1)  10.  : 2  1345 (1) 2  :  2  3147  (2) 8  ::  (3) 31  (5) 14  ::  (3) 5  6  (4)  314151  (5) 56 :  14  456  (5) 7 :  5679 (3) 6  345 (5) 57  :  (4) 6  (5) 7 :  (3) 4  367  (4) 235  :  ::  (5) 3 1963  (4) 34  ::  (2) 3  16  :  (4) 8  (3) 64  (2) 1 1411  3  13  7  :  (5) 163  :  • (2)  (4) 7  ::  (3) 46 :  8143  (4) 419  827  (2) 28  (5) 42 :  37  (3) 26 :  7  14  8 ::  13  17  (4) 9  '  \ (1)  7.  (3) 28  _ _ _ _ _ (1)  5.  :  (2) 3  :  (1)  3.  ::  11  (1)  2.  6  A N A L O G I E S  (4) 23  2223  (5) 1  : (4) 7  (5) 3  '  59  : 7  (1)  (2)  4  2478  (3)  2  : 2  (1)  (2)  24  4  (1)  (2)  1  (1)  :  18  (5) 7 8  ::  7  :  12  (4)  1  (5)  3246  ::  3  :  (3)  3  :  3 , 7, 11  (1)  l x l (2)  192837  (4)  6  470 4 1729  (5)  2  ::  13, 17, 21  4  ::  6  283746  (2)  6  (3)  9  :  (4)  7, 11, 15  '  8  1/64  (5) 5  8  0, 2 / 4  (5)  1/2,  ::  16,8,4  (3)  2/4/2/8,  (2)  21  (3)  2 + 2  2x2  (3)  :  7-4  ::  (2) 1  (3) 7  (4).  (4)  1 - 1  l x l ::  2, 4, 8  2/16  (4)  (5)  (6/3) - 1 4 + 3  :  4 5  (5)  (9/3) - 1  2-2 1 + 1  6466 5  :  1, 2 :  : (2)  (5) 9  .  ' (2)  3x3  (3 - 3  4-1  (7/3) - 1  3 + 3  (4)  :  1/16, 1/32,  7717  :  ::  (4)  .  (4)  4  1 / 3 , 1 / 6 , 1/12  (1)  2 + 2  :  (1)  7x3  :  (3)  1/2,1/4,1/8  (1)  1354  1, 5, 9  1 + 1  (1)  32  :  (2) . 1, - 3, 5 • (3) 2 , 6 , 1 0  (1) .7, 9, 11 (5)  (4)  4  4120 (3)  :  ::  (5)  4 + 4  6 1  21.  22.  23.  : (1)  4  13  :  (1)  47  283 (1)  24.  25. • •. 26.  29.  30.  43  24  :  (4)  35  (1)  (4)  10 30  414  ::  3  ::  3  1  . (1)  7  (2)  23  :  32  (1)  63  4197  (4)  (3)  :  8  : 9247  7149  12  25  (4)  37  9  13  (4)  25  (5)  10-3 629  5566  (4)  14  :  21  ::  9  :  (3)  9  (4)  5  (5)  (3)  13  (5)  :  2  (2) 19  (5)  (4)  (3)  ::  26  __________  (3)  21  3818  _____________  ::  (2)  36  :  7438  :: (2)  5  (5)  46  :  : 4  15  (4)  6  491  7  (5)  :  (2)  (2)  61  (3)  3383  1686  34  .  (2) 8  :  :  (3)  ::  :  6  ::  718  1792  2  .  (3)  ( 2 ) 729  191  (1)  78  :•  (1)  0  ::  ::  (3)  ( 2 ) 13  :  (1) 28.  24  525  (1) 27.  ( 2 ) 23  789  2  (5)  3 14 3  '•  :  12  (4)  22  _ _ _ _ _ _ _ _ (3)  29  (4)  .3  49  (5) :  32 91  (5)  74  (1)  7  (2)  418  :  814  (1)  724  (2)  91  :  (1)  19  2  27  90  (2)  (3)  924  :  146  (1)  171  (2)  987  :  (1)  769  557  :  (2)  1 :: 614  234  454  (4)  717  :  (3)  (2)  577 ::  12 - 2  (2)  1345  ::  (2)  (3)  8  2x2 5  (3)  471  4  (5)  782  8 + 0 : 8  177  9x0  (4)  + 1  (5)  717  (5)  1x0  617  _ _ _ _ _ _ _ _  345  (4) 28  757  235 :  (4)  (5)  793  (5)  75  82 575  : 5 + 3  (4) :  (3)  09  (5)  142  (4)  :  134  119  ::  74  (5)  :  :  ::  6-1  5  4  9  876  (5) 6983  87  (4) 7  (3)  (3)  ::  (2)  755  2  (4)  ::  '  641  :  72  910  (3)  : 8x1  (1)  104  :  (3)  278  (2)  (1)  (1)  27  (4)  431  8x0  (1)  ::  198  :  (1).  1  (3)  36  (4)  7  (5)  6 - 2 689 1  (5)  8  63  Answer Sheet f o r Paper T e s t s  -4-  ANSWER SHEET  vO NAME (PLEASE I.D.  PRINT) SEX  AGE  SECTION  NUMBER  enter time test a 1  2  3_  4.  5.  6.  7.  8.  9.  10.  11  12  1 3 _  14.  15.  16.  17.  18.  19.  20.  21  22  2 3 _  25.  26.  2 7.  28.  29.  30.  31  32_  3 3 _  35.  36.  37  38  39  40  k  5  6  7  34  enter time  :  test b 1  2  11  12  13  14  15  16  21  22  23  24  25  31  32  33  34  35  3_  enter t iT»e  .  8  9  10.  17  18  19  20.  26  27  28  29  30.  36  37  38  39  40.  65  APPENDIX B F o r t r a n I V CCTT M o d e l  LOGICAL ATN CALL ATNTRP(AINI DIMENSION  C  ###  C  ###  C  ###  99 96  GC25),AL<20),IT(5),AI25),PH(40),PL(40)  INTEGER GR126) REAL AV{4)/1.39,2„30,3.00,4.617 INTEGER*2 A N S f c 4 0 ) , R E S , R O N / • R• / , S S / * S• / , S 0 / » O V / , S6/« INTEGER*2 NS,NUM,S7/•7«/,S8/»8»/,S9/«9»/,S5/»5•/ S E T ERROR C O U N T E R S AND T E S T INDICATORS R E A L * 8 SN NE=0 NES=0 NI=0 NL=2 NQ=0 . NEND=0 NM=5 NH-20 NT = 8 LG=0 S T A C K ANY A T T E N T I O N CALL ATNTRP( ATN) LL=0 NPW=0 DETERMINE TIME OF CALL TIME(6,0,.IT ) GO T O 9 8 NI=NI+1 WRITE (NR,96) F O R M A T {•  IF  INTERRUPTS  DAY  AND  DATE  NI,IC,GR,RES INTERRUPT »,2  (N'l.GE.NM) G O T O 9 9 9  16,130,5X,A1)  IF (NI.EQ.NL) WRITE (6,104) SET A T T E N T I O N TRAP TO PREVENT  C 98  104 4 2 6  IF <NEND.GT.6§ STOP R E A D ( 5 , 5 , E N 0 = 2 ) NS  ###  FORMAT  FROM  INSTRUCTIONS YOUR  SECTION  PROGRAM  •) NUMBER  PROGRAM/ENTER  GO T O  MTS  OR  STOP  TESTING  QUIT  ###  SAVE  INPUT  READ  (5,75)  74  DIGIT)'/)  5321  77  ###  {SINGLE  2  IF I N S . E Q . S S ) STOP 22 D E T E R M I N E T E S T I N G O R D E R BY S E C T I O N NUMBER IF { ( N S . E Q . S 7 ) : . 0 R . ( N S . E Q . S 5 ) ) NQ=2 IF ((NS.EQ.S9).0R.(NS.EQ.S6).0R.INS.EQ.S8)) IF ( N Q . G T . O ) GO T O 7 7 IF<NPW.GE.NL) GO TO 9 9 9 IF ( N P W . G E . N L ) I GO TO 1 0 0 3 NPW=NPW+1 GO T O 3 WRITE (6,74)  1001  C  ENTER  TO E X I T  (NS.EQ.RON)  CALL  FOLLOW  EXIT  {All.  PASSWORDS IF  C  TRRSTR TRAP( I C , G R , 6 9 9 )  FORMAT (» PLEASE WRITE <6,4) FORMAT (• P L E A S E NEND=NEND+1  5 C  CALL CALL  6* /  FORMAT  76  CALL  75  FORMAT  I*  PLEASE  RECORD  SETSTA(5*2) (A8)  SN  ENTER  YOUR  FOR. R E R E A D  STUDENT  NUMBER  NQ=1  {SEVEN  DIGITS)'/)  811 C ### 610  3322 3324 3323  3397  807  READ ( 5 , 8 1 1 ) KF FORMAT <I7) CHECK FOR FI LB EX I S T A N C E £ P R E V I O U S T E S T I N G CALL C H K F I L ( S N , £ 8 0 9 } CALL S E T L I 0 ( « 8 »,SN,£804) CALL GETLST(NT,LST) FI.N0 ( NT * L ST ) C A L L S E T S T A < NT , 2 ) READ ( N T , 3 3 2 2 * E N 0 = 8 7 6 ) 13 FORMAT (IX,II) IF(I3.NE.3) GO T O 8 0 7 READ (NT , 3 3 2 3 , E N D = 8 7 7 ) I,J,LL,LG,NQ,T,RR FORMAT ( 3 4 X , 5 1 5 , 2 F 1 2 . 6 ) CALL C L O S F L l ' 8 » ) CALL TIME(6,0,IT) WRITE < N T , 6 0 3 ) NS,SN,IT NLI=36000 FIND (NQ'NLI ) READ ( N Q , 1 5 , E N D = 8 8 ) ANS,PH,PL F O R M A T (• QUESTIONS WILL CONTINUE WRITE (6,3397). NLI=(2*1+1)*1000+38000 FIND (NQ'NLI) GO T O 1 0 1 WRITE (6,806)  806  F O R M A T (• GO T O 9 9 9  809  FIND (3»KF) READ ( 3 , 2 1 , E N O = 1 0 0 1 ) F O R M A T i I X , A 1) IF ( N U M . E Q . N S ) i GO T O IF (NES.GE.NL)i GO TO WRITE (6,346)  21 322 346  FORMAT  (*  RECORDS  YOUR  NES=NES+1 READ ( 5 , 5 ) C  ### 345 810 887 603  3 72  C  79 ### 618 661 662 658  INDICATE  YOU  FROM  HAVE  SIGNOFF  DONE  THIS  , G  POINT*I  PHASE  SECTION  NUMBER  READ ( N Q , 6 6 1 ) AL FORMAT (20A4) WRITE ( 6 , 6 6 2 > AL (1X,20A4)  READ  (5,17)  READ  RES  IF  NOT,  CALL  RON')  345 999 DOES  NOT  APPEAR  CORRECT  NS  IF (IL.EQ.17) CONTINUE  -  NUM  IF ( N U M . N E . N S ) i GO T O 3 2 2 CREATE SUBJECT FILE CALL CREATE!SN,1,0,0,£804,£808,£808,6808,6820,£808) CALL C L O S F H '8 •) CALL S E T L I 0 ( « 8 «,SN,£804) WRITE (NT,603!) NS,SN,IT FORMAT (1X,A1>A8,3X,2A4,IX,3A4J IF (LG.EQ.l) GO TO l i l l GO T O 6 1 8 WRITE (6,72) F O R M A T (• RESPONSE IS INVALID - PLEASE TRY AGAIN'//) NE=NE+1 IF ( N E . G T . N L ) GO TO 9 9 9 IF (NQ.EQ.O) GO TO 6 ' GO T O 7 6 DISPLAY TEST INSTRUCTIONS DO 6 5 8 IL=1,35  FORMAT  /  (5,17)  RES  -  PLEASE  REENTER*)  C  102 ### 108 15  101 80 C MM 82  C  ###  16 17  T=0. LOAD T E S T ANSWERS AND P R O B A B I L I T I E S READ < N Q , 1 5 , E N D = 8 8 ) ANS,PH,PL FORMAT ( 4 0 A 1 , 4 { / , 2 0 F 4 . 3 ) ) J=l RR-O. 1=1 READ 1 N Q , 8 0 , END=88) Q , A FORMAT ( 3 X , 2 5 A 4 / , 3 X t 2 5 A 4 ) O I S P L A Y Q U E S T I O N AND P O S S I B L E ANSWERS WRITE < 6 , 8 2 ) QtA FORMAT { 1 X , / , 1 X , 2 5 A 4 , / , 1 X , 2 5 A 4 J  ° °  f  CALL TIME(O) IT2=0 NE=0 READ  SUBJECT'S  RESPONSE  READ ( 5 , 1 7 ) RES FORMAT (Al) CALL TIME ( 3 , 0 , IT) IT3=ITC2)-IT2  IT2=IT(2) IF. ( R E S . E Q . S S ) GO T O 3 3 3 3 IF ( (RES.GE.S0).AND.(RES.LT.S6) 18  19  NE=NE+1 IF (NE.GT.NH) WRITE (6,19) FORMAT.(« WRITE  GO  YOUR  (6,82)  TO  )" GO  TO  20  999  RESPONSE  WAS  NOT  A NUMBER  BETWEEN  0  AND 5 -  TRY  Q, A  GO T O 1 6 READ ( N Q , 8 0 , E N D = 8 9 ) Q , A WRITE ( 6 , 8 2 ) Q,A C CALCULATE C L A S S I F I C A T I O N INFORMATION 89 IF ( R E S . E Q . A N S ( I ) ) GO T O 3 0 22 R = U 1 . - P H ( I) ) / ( 1 . PL(l)l) NRW=1. GO T O 3 2 30 R=PH{I1/PL(I) RR=RR+;i. NRW=100. 32 T=T + A L O G ( R ) RI=I PR=RR/RI IF (T.GT.AV(J)) GO T O 4 0 35 B=-1.*AV(J) IF (T.LT.B) GO T O 5 0 36 IF <I.LE.40) GO TO 6 9 38 IF ( T . G E . O . ) GO TO 4 0 39 IF ( T . L T . O . ) GO T O 5 0 40 LL-9 GO T O 6 9 50 LL=1 C ### R E C O R D I T E M T E S T I N G I N F O ON S U B J E C T ' S FILE 69 WRITE ( N T , 3 3 ) NS,SN,I,RES,NRW,PH(I),PL(I},R,T,AV<JV,RR,PR,LL, 1NI,NE,ITU),IT(2),IT3 20  33  FORMAT  66  IF  62 65  1=1 + 1  (1X,A1„A8,I2,IX,A  (J.GE.4)  IF  (LL.NE.O)  IF  ( I.LE.40)'  LL=0  GO T O  J=J+1  GO  TO  65  16  1,14,3F6.3,F7.2,F6.2-F4.0,F8.4,314,318)  AGAIN*/)  88 600 C ### 87 601  IF {LG.EQ. 1) GO TO 888 LG=1 ADMINISTER SECOND TEST 69 GO TO (601,602),NQ NQ=2 WRITE (6,501) 501 FORMAT I* HERE ARE SOME NUMERICAL ANALOGIES. • ,/, 1* THE PROCEDURE FOR ANSWERING THEM IS THE SAME. •/) GO TO 646 602 NQ=1 WRITE (6,502) 502 FORMAT (' HERE ARE SOME VERBAL ANALOGIES. «,/, I V THE PROCEDURE FOR ANSWERING THEM IS THE SAME. •/) 646 DO 656 IL=1,35 656 READ (NQ,661) AL GO TO 102 1003 WRITE (6,1002)1 1002 FORMAT (• YOUR STUDENT NUMBER IS NOT IN THE ID TABLE - CALL RON*) STOP 1001 804 STOP 804 808 STOP 808 820 STOP 820 876 STOP 876 877 STOP 877 888 CALL T I M E ( 6 , 0 , I T ) GO TO 887 999 WRITE (6,3399) 3399 FORMAT (* ERROR ACCUMULATION PREVENTS FURTHUR PROCESSING*) 1111 WRITE 16,1112). 1112 FORMAT { 1* THIS CONCLUDES THE COMPUTER PHASE OF THIS RESEARCH PROJECT.*/ 2' INFORMATION CONCERNING THE DESIGN AND FINDINGS OF THE STUDY*/ 3* WILL BE PROVilDED TO YOU IN CLASS AT A LATER DATE.*,// 4* THE TERMINAL WILL SHUT ITSELF OFF.*,//, 5* PLEASE DO NOT TRY TO SIGN ON AGAIN.*// 6* THANK YOU FOR YOUR PARTICIPATION. SEE YOU IN CLASS!!') C MAINTAIN SCREEN DISPLAY FOR 15 SEC. CALL RTWAIT(4500) STOP C ### DETERMINE AND RECORD FINAL TIMES 3333 CALL t IME< 6, 0,.IT ) 13 = 3 WRITE (NT,3336) 13,SN,IT,I,J,LL,LG,NQ,T,RR 3336 FORMAT!IX,II,A8,3X,2A4,IX,3A4,5I5,2F12.6) STOP 333 5321 CALL C L O S F L l » 8 « ) STOP 000 END  APPENDIX C Printed  Instructions  f o r ' s i g n - o n ' t o CCTT  ORGANIZATIONAL  BEHAVIOUR  RESEARCH  PLACE:  ROOM 2 0 7 ,  DATES:  WED.  TIMES:  A N Y T I M E O F D A Y OR N I G H T A T ( S A T U R D A Y AND H O L I D A Y H O U R S  NOVEMBER  TIME MATERIALS:  CIVIL  -  THIS  ENGINEERING  7 TO  REQUIRED  PROJECT  THURS.  WILL  BE  INSTRUCTION  71  BUILDING  NOVEMBER  15/73  YOUR C O N V E N I E N C E A R E 8 AM T O 5 P M )  APPROXIMATELY SHEET,  ONE  HOUR  YOUR  SECTION  NUMBER  YOUR  STUDENT  NUMBER  INSTRUCTIONS: 1  GO T O ROOM 2 0 7 IN T H E C I V I L E N G I N E E R I N G BUILDING. YOU W I L L S E E S E P A R A T E C A R R E L S , EACH WITH ITS OWN C A T H O D E R A Y T E R M I N A L . P O S I T I O N Y O U R S E L F AT A V A C A N T T E R M I N A L WHICH IS DISPLAYING »MTS' IN L A R G E GREEN L E T T E R S .  2  F A M I L I A R I Z E Y O U R S E L F WITH T H E K E Y B O A R D , P A R T I C U L A R L Y N O T I N G T H E U P P E R ROW W H I C H C O N T A I N S T H E N U M B E R S 0 T H R O U G H 9 , AND T H E B O T T O M ROW W H I C H C O N T A I N S T H E S P A C E BAR ( L O N G 6 D A R K ) AND T H E ' E N T E R * K E Y T O I T S IMMEDIATE RIGHT.  3  NOW  TYPE  SIG  RONL  THE  FOLLOWING  AND  PRESS  NOW L O O K A T T H E IF IT D O E S N O T , 4  NOW X  TYPE AND  THE  THE SCREEN IF IT D O E S INDICATING 5  6  7  TYPE  THE  TERMINAL  IF  OF  'ENTER *  KEY.  SCREEN. IT SHOULD REPEAT INSTRUCTION  THE  LETTER  'ENTER*  SAY 3.  YOUR  PROBLEMS  THE  PLEASE DO  SECTION SCREEN  THE  DO  OF  THE  PRESS  THE  THAT  PLEASE  HOURS  OBSERVE NOT  USER  PASSWORD'.  KEY.  NUMBER  WILL  PERSIST  PROJECT,  DO  'ENTER  KEY:  (A  SINGLE  PRODUCE  RON  AND 4 : 3 0  FOLLOWING  ANY  PROJECT  THE  8  PREVENT  CALL  TERMINAL  ON Y O U R  TERMINAL  ***  DIGIT)  FURTHUR  YOU A R E U N A B L E T O G E T F U L L Y S I G N E D ABANDON I T , PLEASE TYPE *SIGNOFF*  BETWEEN *  KEYS:  S H O U L D NOW S A Y ' P L E A S E E N T E R Y O U R S E C T I O N N U M B E R * . NOT, REPEAT INSTRUCTION 4. IF A MESSAGE IS RETURNED Y O U C A N N O T S I G N ON R I G H T N O W , T R Y A G A I N L A T E R .  NOW  IF TO  THE  FOLLOWING  PRESS  LETTER  YOU  ON T O A T E R M I N A L OR AND P R E S S •ENTER*• FROM  COMPLETING THIS  (WEEKDAYS).  KEYS OWN  SESSION  THANKS  :  THEY  OTHER  ONCE  AT  ARE THEN  (WITHOUT  ***  'ENTER*.  INSRUCTIONS.  LONGBOTTOM  RULES  AND P R E S S  683-8711,  PHASE  L.3448,  ESSENTIAL!! THOSE  YOUR  ONLY.  HAVE  MENTIONED.  FRIENDS).  72  APPENDIX D C o m p u t e r V e r b a l and  Numerical  Analogy  Tests  73  I N S T R U C T I O N S FOLLOWING HERE  I S  ARE  AN  FEATHER (1)  SWIM  T H I S  I S  YOU IN  ARE  " B I R D "  I S " 4 "  YOU IS  I T  TO  WILL  THE  I T  F I N D  AND  EACH  Q U E S T I O N  THEN  S E L E C T  FOR  EACH YOU  DO  ARE  WITH NOT  NOT  THE  P R E S S  P L E A S E  BE  P L E A S E  WORK  P R E S S  ANY  WHEN  A  AT  THE  TO  ANSWER.  F L Y  P R E S S  P R E S S I S  THE  (TOP  YOUR ARE  KEY  AND  THAN  THE AND  READY  WITH  THE AND  ANALOGY  TRY  I T .  Q U E S T I O N  P O S I T I O N .  A P P R O P R I A T E  R E A S O N I N G  ANALOGY.  THE  NUMBER  ANSWER,  OF  W I L L  THE  THE  THE  " E N T E R "  KEYS  AND  THE  BE  F I N I S H  P R E S S  P R E S S  NUMBER  TO  F I S H  ANSWERS.  K E Y .  THEN  RESPONSES  A  AHEAD  ANY THE  WITH  CORRECT  ROW)  START  MOST  "ENTER"  KEY GO  EACH  I N  COMPLETES  THE  THE  THE  K E Y .  FOR  THE  ON  P O S S I B L E  P R E S S  A P P E A R  ANALOGY.  GROWS  F I V E  "ENTER"  BEST  THE  S C A L E  WOULD  THE  DETERMINE  WHICH  4 4 3 5 45 3 < r 3 5 3 2 1 4 4 2 5 1 4 3 1 5 15  15)  COMPLETES  THAT  REQUIRED  YOUR  YOU  YOU  BLANK  OTHER  THAT  YOU  F I S H ? "  BEST  MAY  " 0 "  L I K E  B I R D  REASONING  WHICH  Q U I C K L Y .  'ENTER*  SO  MUST  K E Y S  ASSURED  F I S H  THE  THEN  SURE  TO  LOOK  P R E S S  WORD  NUMBER  : ( 4 )  REASON  Q U E S T I O N ,  AND  I S  WHICH  ON?  THEN  YOU  THE  ANALOGY ANSWER  GROW  THE  WOULD  READ:  WORD  THAT  S C A L E  S C A L E  ANSWER,  THAT  WE  ANALOGY:  WATER  MIGHT  AND  AND  IF  THE  BEST  ON  AS  YOU  FOR  KEY  SHOULD  F E A T H E R  D I F F E R E N T ,  CORRECT  QUESTIONS  WORD  <3)  BLANK  CHOOSE A  A  :  E X A M P L E .  NUMBER  OF  A I R  WAY  I S  DOES  ANALOGY  _.  12)  TO  T H I S  WHAT  :  THE  "FEATHER  SOME  EXAMPLE  KEPT  T I M E S  I N  K E Y .  ENTER  K E Y .  S T R I C T I S T  WILL  BE  C O N F I D E N C E .  RECORDED.  PROCEED.  1 4 5 1 2 2 4 1 2 4 4 4 2 4 3 4  938  6 4 6  8 7 5  6 2 5  4 3 8  3 5 4  8 9 6  771  6 0 4  6 0 4  5 6 3  167  9 5 8  8 9 6  5 2 1  7 5 0  5 4 2  3 9 6  563  7 2 9  6 4 6  4 7 9  8 3 3  7 5 0  8 1 3  6 6 7  5 2 1  1 8 8  8 7 5  7 9 2  7 2 9  667  3 5 4  5 6 3  8 7 5  8 3 3  6 0 4  973  6 2 2  6 7 6  2 4 3  2 1 6  1 8 9  7 8 4  3 5 1  4 5 9  6 2 2  297  2 4 3  5 9 5  5 6 8  2 9 7  5 6 8  4 0 5  2 9 7  6 7 6  541  3 5 1  3 2 4  2 1 6  4 0 5  2 4 3  4 8 6  3 7 8  2 4 3  0 8 1  7 0 3  5 6 8  4 0 5  4 5 9  1 8 9  2 1 6  7 8 4  7 0 3  5 4 1  O i l 0 1 2 ( 1 )  : E N G I N E E R  0 2 1 D I V E 0 2 2 ( 1 )  :  (2)  SWIM  TREE  ( 2 )  SHORT  ( 2 )  : S T E A K  ( 2 )  0 5 2 ( 1 )  : HOUSEWIFE  0 6 1 D U L L 0 6 2 1 1 )  0 8 1 T R E E 0 8 2 C 1 ) 0 9 1 P I C K 0 9 2 ( 1 )  ( 3 )  M I S S  ( 2 )  K E E N  : AXE :  ( 2 ) MAN  GOLD ::  ( 2 )  SAP  WOMAN  V I O L I N  P L A Y  ( 2 )  : BEFORE  1 1 1 G A S 0 L I N E  (2) :  CAR  ( 3 ) ( 3 )  F I S H E R M A N  ( 4 ) ( 4 )  :  BRIGHT NOTE  ( 5 )  :  ( 4 )  WING  L A S S  151  QUEEN  SHARP  (<r) B O R I N G  CASH  C I R C U S  MR ( 4 )  :  ( 5 )  HAM  EGGS  NURSE  S H I N Y  SWIMMING  BEARD :  HUSBAND ( 3 )  ( 5 )  ( 5 )  DULL  MONEY G I A N T  ( 5 )  SOOSE  : ( 3 )  MAPLE  ( 4 ?  BLOOD :  ( 3 )  BEHIND B E S I D E ::  ( 4 )  DWARF  BREAST  BABY  LADY  :  :: S I N G  LAWYER  SWAM  DULL  ::  MARK  :  DRUNK  T A L L  :: ::  :  101 1 0 2 ( 1 )  ( 3 )  ::  DUMB  ( 4 )  ::  BACON  ( 2 )  ( 3 )  F L Y  :  071 CHECK 0 7 2 ( 1 )  ( 3 ) CLOWN  P I G  051  DOCTOR  :  LAUGH  041 0 4 2 ( 1 )  DOVE  COO :  ::  GENTLEMAN  ::  031 0 3 2 ( 1 )  HUNTER  BOW :: ( 3 )  (4*7 DRAG A F T E R  ( 5 )  ARM  BANJO  S T R I N G :  ( 5 )  F I D D L E  AHEAD ( 4 ) :  P U L L  CLOCK  ( 5 )  PUSH  ( 5 )  C I T I Z E N  8 9 6  11211) OIL ( 2 ) WIND (3) SPRING ( 4 ) STEM ( 5 ) SEAR 121WALK : STAND MOW : 1 2 2 ( 1 ) CLIMB ( 2 ) SHAVE ( 3 ) GROW ( 4 ) RUN ( 5 ) LAWN 131 : CANOE :: CANNON : PISTOL 1 3 2 ( l T BATTLESHIP ( 2 ) ROWBOAT (3) RIFLE ( 4 ) OCEAN ( 5 ) OAR 141SIN : SONG :: SON : 1 4 2 ( 1 ) CHORUS ( 2 ) SINGLE ( 3 ) HUM ( 4 ) SING ( 5 ) DAUGHTER 151MARBLE : EARTH : SUN 152(1) FIRE ( 2 ) BRIGHT ( 3 ) ROUND ( 4 ) BASKETBALL 1 5 ) GAME 161 TON : :: T I N : T I N K L E 1 6 2 ( 1 ) POUND ( 2 ) TONGUE ( 3 ) TONIGHT ( 4 ) WET ( 5 ) STEEL 171 : COLLEGE :: PRAY : CHURCH 1 7 2 ( 1 ) BOOKS ( 2 ) TEACHER ( 3 ) LEARN (4) CLASS 15) STUDY 181ASH : BEECH :: OAK : .182(1) P I N E <2) MAPLE ( 3 ) WALNUT ( 4 ) HICKORY ( 5 ) ELM 191TRAIM : AUTO :: TRACK : . 192M) TIE (2) TIRE ( 3 ) CAR ( 4 ) ROAD ( 5 ) PATH 201 : YESTERDAY :: TOMORROW : TODAY 2 0 2 ? 1 ) TOMORROW ( 2 ) TONIGHT ( 3 ) TODAY ( 4 ) NOW 1 5 ) THEN 211UMKIND : SWEETHEART :: : GUARD 212(1) CARELESS ( 2 ) CRUEL <3) S A F E ( 4 ) ROYAL ( 5 ) HAPPY 221HIRE : CENT :: : SCENT 222(1) FIRE ( 2 ) SMELL ( 3 ) DENT ( 4 ) WORK ( 5 ) HIGHER 231T0MAT0 : CARROT : ORANGE 2 3 2 1 1 ) BEET ( 2 ) CELERY ( 3 ) PEACH « 4 ) POTATO ( 5 ) PEAR 241 : FAITH :: V A L U A B L E : TRUST 242(1) BELIEVE ( 2 ) HONOR ( 3 ) SURE ( 4 ) HEAVEN ( 5 ) DEAR 251GOBBLE : PERSON : TURKEY 25211) TALK ( 2 ) T H A N K S G I V I N G ( 3 ) NOVEMBER ( 4 ) EAT ( 5 ) FEED 261AC0RN : COLT : HORSE 2 6 2 ( 1 ) NUT ( 2 ) MARE ( 3 ) SEED ( 4 ) OAK 151 SADDLE 271DEER : DOES :: C A T T L E : 2 7 2 ( 1 ) DID ( 2 ) HORSES ( 3 ) CALF ( 4 ) DOESN'T (5)'COWS 281BLUE : t : YELLOW : ORANGE 2 8 2 ( 1 ) PURPLE ( 2 ) RED ( 3 ) GREEN ( 4 ) BROWN ( 5 ) GOLD 291D0G : :: MUSIC : HYMN 2 9 2 ( 1 ) BARK ? 2 ) HOUND 1 3 ) CHURCH ( 4 ) HOWL ( 5 ) CAT 301D0T : CIRCLE :: MARBLE : 3 0 2 ( 1 ) BALL ( 2 ) TIRE ( 3 ) SQUARE ( 4 ) CIGARETTE ( 5 ) GLASS 311SCH00L : FISH :: : TREE 3 1 2 ( 1 ) COLLEGE ( 2 ) TURTLE ( 3 ) PLANT ( 4 ) GROVE 1 5 ) BOOK 3210DD : PROPER :: QUEER : 3 2 2 ( 1 ) CORRECT ( 2 ) REASON ( 3 ) IDEAL ( 4 ) WISE ( 5 ) WRONG 331WHCD : WOULD WE'VE : 3 3 2 ( 1 ) WENT ( 2 ) HAVE (3*) W I L L ( 4 ) THEY ( 5 ) WOULDN'T 341CL0TH : :: AMIMAL : ELEPHANT 3 4 2 ( 1 ) SHIRT ( 2 ) LEATHER (3) SAIL (4) LINEN ( 5 ) FASHION 351NAIL : SCREW :: : NUT 3 5 2 ( 1 ) TACK (2) CLIP ( 3 ) BOLT ( 4 ) WASHER ( 5 ) HAMMER 361SEA : WAVE : BROOK 3 6 2 ( 1 ) WATER ( 2 ) OCEAN ( 3 ) SWELL ( 4 ) RIVER ( 5 ) CREEK 371PRUNE : GRAPE :: : PLUM 3 7 2 ( 1 ) ORANGE (2) RAISIN ( 3 ) WINE ( 4 ) FRUIT ( 5 ) GRAPEFRUIT 3 8 1 EAGER : HONOR : SHAME 382(1) LIVELY <2) HURT (3) PITY ( 4 ) DULL (5) SPIRIT 391B0LD : HONEST :: : UNFAIR 3 9 2 ( 1 ) FORWARD ( 2 ) UNKIND ( 3 ) SHY ( 4 ) GENTLE ( 5 ) TRUST 401POT : :: POOL : LOOP 4 0 2 ( 1 ) ROUND ( 2 ) PAN ( 3 ) ROPE ( 4 ) TOP ( 5 ) POND ;  I N S T R U C T I O N S  75 FOLLOWING HERE  I S  AN  746 9  T H I S  I S  ON YOU IS  TO  T H I S  LOOK  WAY  IT I S  AT  AND  WILL  THEN F I N D  AND  EACH  Q U E S T I O N  AND  THEN  S E L E C T  FOR  EACH  IF KEY DO  YOU  ARE  WITH NOT  NOT  THE  BE  P L E A S E  WORK  P R E S S  SURE  ANY  K E Y S  ASSURED  Q U I C K L Y .  'ENTER*  WHEN  THE  K E Y .  TO  ANSWER.  GO  P R E S S  P R E S S I S  (TOP  YOUR  ARE  BEST  THE  THAN  KEY  CORRECT AND THE  START READY  AND TO  I S  NUMBER  I N  "3"  I S  AND  TRY  I N  MOST  WITH  EACH  I T . Q U E S T I O N  P O S I T I O N .  A P P R O P R I A T E THE  I N "5"  ANALOGY  ANY  THE  839?  OF  AHEAD  R E A S O N I N G  ANALOGY.  THE  NUMBER  OF  THE  K E Y . ANSWER,  THEN  WILL  F I N I S H  P R E S S  THE  P R E S S  THE  "ENTER"  KEYS  AND  THE  NUMBER  RESPONSES  "4"  NUMBER  FOR  "ENTER"  B L A N K ? " ANALOGY.  THE  COMPLETES  THE  THE ROW)  THE  TO  WITH  A P P E A R  DETERMINE  WHICH  I S  MIDDLE  THE  KEY  MAY  YOUR  YOU  YOU  THE  NUMBER  THE  S I N C E  P R E S S  BLANK  OTHER  THAT  THE I N  REQUIRED  WHICH "0"  IS  R E A S O N I N G MUST  THEN  THAT  NUMBER  COMPLETES  THE  Q U E S T I O N ,  AND  L I K E  3  T H I R D  BEST  ANSWERS. "ENTER"  WORD  NUMBER  P R E S S  P L E A S E  YOU  THE  ANALOGY ANSWER  THE  AS  NUMBER  WOULD  THAT  15)  WHICH  REASON  WHAT  THE  8  NUMBER  NUMBER MIGHT  P R E S S  FOR  CORRECT  SECOND  YOU  THAT  D I F F E R E N T ,  READ:  P O S S I B L E  839,  WOULD  :  SHOULD  THE  WE  ANALOGY:  (4)  YOU  F I V E OF  NUMBER  76  746.  NUMBER  THE  A  QUESTIONS  839  TO  E X A M P L E * OF  OF  (3)  CHOOSE  MIDDLE I T  4  NUMBER  MIDDLE THE  THE  ARE  ANALOGY  ::  4  (2)  " F I R S T IN  SOME  EXAMPLE  :  (1)  YOU  ARE  BE  KEPT  T I M E S  I N  WILL  K E Y .  ENTER  S T R I C T I S T BE  K E Y . C O N F I D E N C E .  RECORDED.  PROCEED.  1551145422132525 544243254322222312143114 455 733 068 864 114 955 136 977 6590999 432 977 955 841 227 568 568 795 909 159 959 977 2270999 463 5610001 829 171 780 073 659 244 878 195 829 805 585 146 171 268 488 805 220 951 902 098 927 0112 : 6 :: : 17 01211) 11 (2) 3 (3) 28 (4) 9 ( 5 ) 42 0216 : :: 14 : 8143 022(1) 93 (2) 108 (3) 4137 (4) 7 (5) 163 031 : 8 :: 37 : 16 032(1) 4 (2) 13 (3) 26 ( 4 ) 419 (5) 3 041 : 827 :: 3 : 1963 042(1) 7 (2) 28 (3) 2 (4) 8 (5) 14 05123 : 67 :: 13 : 052(1) 45 (2) 78 (3) 46 (4) 34 (5) 56 061 : 3147 :: 6 : 367 062(1) 17 (2) 34 (3) 31 ( 4 ) 14 (5) 7 07112 : :: 456 : 345 072(1) 431 (2) 13 (3) 64 (4) 235 (5) 67 081405060 : 7 :: 314151 : 082U) 2 ( 2 ) 1 ( 3 ) 5 ( 4 ) 6 ( 5 ) 7 0914 : 1411 :: : 2223 092(1) 2 (2) 3 (3) 4 (4) 23 (5) 1 1011345 : 2 :: 5679 : 102(1) 2 (2) 8 (3) 6 (4) 7 (5) 3 111 : 2478 :: 4 : 1354  432 818 9770999 268 585 927 878  432 682 364 477 772 886 955 9550999 295 317 415 195 415 63< 585 829 732 878 311  112(1) 7 (2) 4 (3) 2 ( 4 ) 32 (5) 578 121 : 4120 :: 7 : 470 122(1) 2 (2) 24 (3) 12 (4) 1 (5) 4 131 : 3246 :: 3 : 1728 132(1) 4 (2) 1 (3) 3 (4) 6 (5) 2 141 : 3, 7, 11 :: 13, 17, 21 : 15, 19, 23 142(1) 7,9,11 (2) 1,3,5 (3) 2,6,10 (4) 7,11,15 ( 5 ) 1,5,9 1511 + 1 : 1 X 1 :: : 3 X 3 152(1) 18 (2) 6 (3) 2 + 2 (4) 3 - 3 (5) 9 161192837 : 4 :: 283746 : 162(1) 4 (2) 6 (3) 9 (4) 8 (5) 5 1711/2, 1/4, 1/8 : :: 16, 8, 4 : 2. 4, 8 172(1) 1/3,1/6,1/12 (2) 0,2,4 (3) 2/4,2/8,2/16 (4) 1/16,1/32,1/64 1817 X 3 : 4 - 1 : (6/3) - 1 182(1) (7/3) - 1 (2) 21 (3) 7 - 4 (4) 4 + 3 (5) (9/3) - 1 191 :2 + 2 : : l - l : 2 - 2 192(1) 3 + 3 (2) 2 X 2 (3) 1 X 1 (4) 1 + 1 (5) 4 * 4 2017717 : :: 6466 : 4 202(1) 8 (2*7 1 (3) 7 (4) 6 (5) 5 211 : 789 :: 2 : 34 212(1) 4 (2) 23 (3) 78 (4) 6 (5) 5 22113 : 24 :: 24 : 222(1) 47 (2) 13 ( 3 ) 35 (4) 15 (5) 36 231283 : :: 46 : 12 232(1) 43 (2) 729 (3) 61 (4) 25 ( 5 ) 3818 241525 : 718 : 26 242(1) 191 (2) 8 (3) 3 ( 4 ) 7438 ( 5 ) 25 2511792 : 3383 :: 6 : 252(1) 1686 (2) 491 (3) 37 (4) 9 (5) 5566 2610 : 10 :: 3 : 262(1) 30 (2) 7 (3) 13 (4) 10 - 3 (5) .3 271414 : :: 629 : 2 272(1) 4 (2) 1 (3) 2 ( 4 ) 14 (5) 3 281 : 21 9 : 14 282(1) 7 (2) 8 (3) 9 (4) 5 (5) 3 29123 : 32 :: : 12 292(1) 63 (2) 21 (3) 13 (4) 22 (5) 32 3014197 : 7149 :: : 91 302(1) 9247 (2) 19 (3) 29 (4) 49 (5) 74 311 : 6 98 : 3 312(1) 7 ?2) 36 (3) 198 (4) 104 (5) 6983 321418 : 814 :: 27 : 322(1) 724 (2) 278 (31 72 ( 4 ) 87 (5) 74 33191 : :: 431 : 134 332(1) 19 (2) 90 (3) 910 ( 4 ) 119 ( 5 ) 09 3412 : :: 7 : 471 342(1) 27 (2) 924 (3) 4 (4) 142 (5) 782 3518 X 0 : :: 8 + 0 : 8 + 1 352(1) 8 X 1 ( 2 ) 1 (3) 9 ( 4 ) 9 X 0 ( 5 ) 1 X 0 361641 : 146 :: 717 : 362(1) 171 ( 2 ) 614 ( 3 ) 177 (4) 717 (5) 617 371987 : 234 :: 876 : 372(1) 769 (2) 454 (3) 345 (4) 235 (5) 798 381557 : : : 28 : 82 382(1) 755 (2) 577 (3) 757 (4) 575 (5) 75 3911 : 6 - 1 :: 2 X 2 : 392(1) 1 2 - 2 (2) 5 (3) 5 + 3 (41 6 - 2 (5) 8 + 1 4012 : 1345 :: : 689 402(1) 5 (2) 8 (3) 4 (4) 7 (5) 1  76  (5) 1/2,1,2  77  APPENDIX E  V a l i d S t u d e n t Numbers a n d S e c t i o n P a r t i a l Computer  File  1121.73 1162.593 1227.727 1234.567 1384.683 1384.726 1503.739 1520.725 1523.734 1559.715 1644.723 1661.235 1772.722 1788.728 1803.725 1839.729 1851.724 1855.717 1861.715 1863.711 1867.712 1887.66 1927.714 1966.704 2009.71 2035.731 2040.707 2049.716 2071.71 2074.714 2089.704 2094.712 2095.719 2096.717 2124.717 2130.714 2153.716 2165.71 2207.71 2218.709 2238.707 2244.713 2250.71 2287.704 2292.712 2302.719 2367.696 2373.71 2375.715 2377.711 2431.716 2448.71 2468.102 2473.718 2520.716 2530.707 2642.7 2706.703 2739.712  7 8 7 7 8 7 7 7 6 7 5 8 5 7 7 9 6 6 6 7 7 6 7 9 6 5 6 7 7 5 9 7 7 8 5 5 6 6 9 7 6 9 5 7 9 7 5 5 8 7 7 9 7 5 7 6 7 8 5  79  APPENDIX F Subject's Test  Record  P a r t i a l Computer  File  so  Codes f o r Subject's Test Record Symbol  Interpretation  A  Student Number  B  Question  C  Item Response  D  C o r r e c t / I n c o r r e c t I n d i c a t o r (100 1 = Incorrect)  E  High Group P r o b a b i l i t y L e v e l  F  Low Group P r o b a b i l i t y L e v e l  G  R^^ Score  H  A Score  I  A L e v e l In E f f e c t  J  T o t a l Number o f Items Right  K  T o t a l Percent  L  Subject C l a s s i f i c a t i o n I n d i c a t o r (9 - High, 1 - Low, 0 = None)  M  Number o f Program I n t e r r u p t s  N  Number of Terminal  0  T o t a l CPU ( C e n t r a l P r o c e s s i n g Unit) Time  P  T o t a l Elapsed  Q  Elapsed  Identification  Correct,  o f Items Right  Operation  Time  Item Response Time  Errors  A  B  5321661 5321661 1 5321661 2 5321661 3 5321661 4 5321661 5 6 5321661 7 5321661 5321661 8 5321661 9 5 3 2 1 6 6 1 10 5 3 2 1 6 6 1 11 5 3 2 1 6 6 1 12 5 3 2 1 6 6 1 13 5 3 2 1 6 6 1 14 5 3 2 1 6 6 1 15 5 3 2 1 6 61 16 5 3 2 1 6 6 1 17 5 3 2 1 6 6 1 18 5 3 2 1 6 6 1 19 5 3 2 1 6 6 1 20 5321661 21 5321661 22 5321661 23 5321661 24 5321661 2 5 5 3 2 1 6 6 1 26 5 3 2 1 6 6 1 27 5 3 2 1 6 61 2 8 5 3 2 1 6 6 1 29 5321661 30 5 3 2 1 6 6 1 31 5 3 2 1 6 6 1 32 5321661 33 5321661 34 5 3 2 1 6 6 1 35 5 3 2 1 6 6 1 36 5 3 2 1 6 6 1 37 5321661 38 5321661 39 5321661 40  C 14 4 4 3 5 4 4 3 4 3 5 3 3 1 4 4 2 3 2 4 3 4 5 2 1 1 4 4 1 4 1 4 1 2 1 4 3 2 2 2 4  D :27: 100 100 100 100 100 1 100 100 100 100 100 1 100 100 100 100 1 1 100 100 1 100 1 1 100 100 1 100 1 1 100 100 100 1 100 1 100 1 1 100  E  F  G  3 2 NOV 1 1 , L 9 7 3 0.938 0.973 0.964 0. 646 0 . 622 1 . 0 3 9 0 . 8 7 5 0 . 6 7 6 1. 2 9 4 0.625 0. 243 2.572 0.438 0.216 2.028 0 . 3 5 4 0 . 189 0 . 7 9 7 0 . 8 9 6 0 . 784 1. 143 0.771 0.351 2.197 0.604 0.459 1.316 0.604 0.622 0.971 0.563 0.297 1.896 0.167 0.243 1.100 0.958 0.595 1.610 0.896 0.568 1.577 0 . 5 2 1 0 . 2 9 7 1. 754 0.750 0.568 1.320 0.542 0.405 0.770 0.396 0.297 0.859 0.896 0.676 1.325 0.792 0.622 1.273 0.563 0.541 0.952 0.729 0.351 2.077 0 . 6 4 6 0 . 3 2 4 0. 524 0.479 0.216 0.665 0. 833 0 . 4 0 5 2 . 0 5 7 0.750 0.243 3.086 0.813 0.486 0.364 0.667 0.378 1.765 0.521 0.243 0.633 0 . 188 0 . 0 8 1 0 . 8 8 4 0.875 0.703 1.245 0.792 0.568 1.394 0.729 0.405 1.800 0.667 0.459 0.616 0.354 0.189 1.873 0.563 0.216 0.557 0.875 0.784 1.116 0.833 0.703 0.562 0.604 0.541 0.863 0.833 0.595 1.400  H -0.04 0.00 0.26 1.20 1.91 1.68 1.82 2.60 2.88 2.85 3.49 3.58 4.06 4.52 5.08 5.36 5.09 4.94 5.22 5.47 5.42 6 . 15 5.50 5.09 5.81 6.94 5.93 6.50 6.04 5.92 6.13 6.47 7.05 6.57 7.20 6.61 6.72 6 . 15 6.00 6.34  :I 1. 1. 1. 1. 1. 2. 2. 2. 3. 3. 3. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4. 4.  39 39 39 39 39 30 30 30 00 00 00 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61  J 1. 2. 3. 4. 5. 5. 6. 7. 8. 9. 10. 10. 11. 12. 13. 14. 14. 14. 15. 16. 16. 17. 17. 17. 18. 19. 19. 2 0. 20. 20. 21. 22. 23. 23. 24. 24. 25. 25. 25. 26.  K 1.0000 1.0000 1.0000 1.0000 1. oooo 0.8333 0.8571 0.8750 0.8889 0.9000 0.9091 0.8333 0.8462 0.8571 0.8667 0.8750 0.8235 0.7778 0.7895 0.8000 0.7619 0.7727 0.7391 0.7083 0.7200 0.7308 0.7037 0.7143 0.6897 0.6667 0.6774 0.6875 0.6970 0.6765 0.6857 0.6667 0.6757 0.6579 0.6410 0.6500  L 0 0 0 0 9 0 0 9 0 0 9 0 0 0 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9  M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  N 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  0 40 144 252 374 474 576 680 782 887 993 1101 1223 1323 1428 1529 1639 1742 1847 1951 2054 2156 2265 2363 2468 2572 2682 2812 2921 3023 3126 3208 3322 3430 3510 3602 3700 3803 3901 4002 4104  P 12336 24430 36386 56093 67866 83893 92420 98473 109446 135526 149260 174486 181283 191040 201313 220690 227186 239160 248806 262666 286953 297363 318243 348396 357736 368663 389203 404563 422586 438470 444293 454860 476626 484813 497040 508840 518810 540533 557013 569576  Q 12336 12094 11956 19707 11773 16027 8527 6053 10973 26080 13734 25226 6797 9757 10273 19377 6496 11974 9646 13860 24287 10410 20880 30153 9340 10927 20540 15360 18023 15884 5823 10567 21766 8187 12227 11800 9970 21723 16480 12563  APPENDIX G R e l a t i v e Cost Comparison CCTT and P a p e r  of  Testing  S3 APPENDIX G RELATIVE COST COMPARISON OF CCTT AND PAPER TESTING The f o l l o w i n g cost comparison assumptions  regarding  i s based on c e r t a i n a r b i t r a r y  conventional  should thus be t r e a t e d with  t e s t a d m i n i s t r a t i o n and  caution.  Computing c o s t s  are p a r t i c u l a r to the i n s t a l l a t i o n and approximate costs. ignored.  quoted  actual  C a p i t a l investment, overhead and m a t e r i a l c o s t s a r e An average group s e t t i n g o f 10 s u b j e c t s i s assumed.  Paper t e s t i n g time taken (max.)  Verbal . - 18 min. Numerical 32 min. Instructions, etc. 6 min. S c o r i n g and r e c o r d i n g (10 t e s t s a t 2.5 min.) 25 min. 81 min. t e s t a d m i n i s t r a t i o n and s c o r i n g costs per hour $5.00 (assumed) Cost  per subject = $5.00 x 8 l = $.675 10"  W  CCT t e s t i n g Elapsed  time taken  CPUT: time taken 1  Verbal Numerical  205.3 sec. 47&*4_sec. 11.39 minT  Verbal Numerical  Terminal connect cost $3/hr. CPU cost at $250/hr. Cost per subject  1.101 sec, 1.200 sec, 2.301 sec, |.569 .161 $.730  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0092983/manifest

Comment

Related Items