{"@context":{"@language":"en","Affiliation":"http:\/\/vivoweb.org\/ontology\/core#departmentOrSchool","AggregatedSourceRepository":"http:\/\/www.europeana.eu\/schemas\/edm\/dataProvider","Campus":"https:\/\/open.library.ubc.ca\/terms#degreeCampus","Creator":"http:\/\/purl.org\/dc\/terms\/creator","DateAvailable":"http:\/\/purl.org\/dc\/terms\/issued","DateIssued":"http:\/\/purl.org\/dc\/terms\/issued","Degree":"http:\/\/vivoweb.org\/ontology\/core#relatedDegree","DegreeGrantor":"https:\/\/open.library.ubc.ca\/terms#degreeGrantor","Description":"http:\/\/purl.org\/dc\/terms\/description","DigitalResourceOriginalRecord":"http:\/\/www.europeana.eu\/schemas\/edm\/aggregatedCHO","FullText":"http:\/\/www.w3.org\/2009\/08\/skos-reference\/skos.html#note","Genre":"http:\/\/www.europeana.eu\/schemas\/edm\/hasType","IsShownAt":"http:\/\/www.europeana.eu\/schemas\/edm\/isShownAt","Language":"http:\/\/purl.org\/dc\/terms\/language","Program":"https:\/\/open.library.ubc.ca\/terms#degreeDiscipline","Provider":"http:\/\/www.europeana.eu\/schemas\/edm\/provider","Publisher":"http:\/\/purl.org\/dc\/terms\/publisher","Rights":"http:\/\/purl.org\/dc\/terms\/rights","ScholarlyLevel":"https:\/\/open.library.ubc.ca\/terms#scholarLevel","Title":"http:\/\/purl.org\/dc\/terms\/title","Type":"http:\/\/purl.org\/dc\/terms\/type","URI":"https:\/\/open.library.ubc.ca\/terms#identifierURI","SortDate":"http:\/\/purl.org\/dc\/terms\/date"},"Affiliation":[{"@value":"Education, Faculty of","@language":"en"}],"AggregatedSourceRepository":[{"@value":"DSpace","@language":"en"}],"Campus":[{"@value":"UBCV","@language":"en"}],"Creator":[{"@value":"MacPherson, Eric Duncan","@language":"en"}],"DateAvailable":[{"@value":"2011-12-15T02:08:03Z","@language":"en"}],"DateIssued":[{"@value":"1960","@language":"en"}],"Degree":[{"@value":"Master of Arts - MA","@language":"en"}],"DegreeGrantor":[{"@value":"University of British Columbia","@language":"en"}],"Description":[{"@value":"It is commonly agreed that subjective evaluation tends to be less reliable than objective evaluation. This study represents an attempt to discover whether or not there are predictable characteristics of subjective evaluation which account for part of this unreliability.\r\nAfter logical analysis, three possible effects were proposed, (1) A contrast effect, in which the difference between a sample and the preceding sample or samples is minimized or exaggerated. (2) An experience effect, in which there is a long term shift in values. (3) An end effect, in which the last few samples in a series are judged according to different standards.\r\nSeveral possible test materials were considered. Finally, handwriting samples were chosen as satisfying the criteria of explicitness and necessity for subjective rather than disguised objective evaluation.\r\nTwo experiments were devised; the first to test for a contrast effect, and the second to test for the experience and end effects.\r\nEvidence significant at the one percent level was presented for rejecting the hypothesis that there is no contrast effect. It was shown that there is a long term increase in the marks given, in which the increase was spread over the whole range of marks.\r\nThere was evidence significant at the five percent level for rejecting the hypothesis that there is no end effect. It was shown that the last three samples of a long series tend to be downgraded.","@language":"en"}],"DigitalResourceOriginalRecord":[{"@value":"https:\/\/circle.library.ubc.ca\/rest\/handle\/2429\/39707?expand=metadata","@language":"en"}],"FullText":[{"@value":"EFFECTS OF ORDER OF JUDGMENT ON SUBJECTIVE EVALUATION by ERIC DUNCAN MacPHERSON B.A., U n i v e r s i t y of B r i t i s h Columbia, 1952 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS i n the Department of Education We accept t h i s t h e s i s as conforming to the re q u i r e d standard THE UNIVERSITY OF BRITISH COLUMBIA September, i 9 6 0 In presenting t h i s t h e s i s i n p a r t i a l f u l f i l m e n t of the requirements f o r an advanced degree at the U n i v e r s i t y of B r i t i s h Columbia, I agree that the L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r reference and study. I f u r t h e r agree that permission f o r extensive copying of t h i s t h e s i s f o r s c h o l a r l y purposes may be granted by the Head of my Department or by h i s r e p r e s e n t a t i v e s . I t i s understood that copying or p u b l i c a t i o n of t h i s t h e s i s f o r f i n a n c i a l g a i n s h a l l not be allowed without my w r i t t e n permission. Department of & o U cfl \u2014 i o N  The U n i v e r s i t y of B r i t i s h Columbia, Vancouver &, Canada. ABSTRACT EFFECTS OF ORDER IN SUBJECTIVE EVALUATION I t i s commonly agreed that s u b j e c t i v e e v a l u a t i o n tends t o be l e s s r e l i a b l e than o b j e c t i v e e v a l u a t i o n . T h i s study r e p r e s e n t s an attempt to d i s c o v e r whether or not there are p r e d i c t a b l e c h a r a c t e r i s t i c s of s u b j e c t i v e e v a l u a t i o n which account f o r p a r t of t h i s u n r e l i a b i l i t y . A f t e r l o g i c a l a n a l y s i s , three p o s s i b l e e f f e c t s were proposed, (1) A c o n t r a s t e f f e c t , i n which the d i f f e r e n c e between a sample and the p r e c e d i n g sample or samples i s minimized or exaggerated. ( 2 ) An experience e f f e c t , i n which there i s a long term s h i f t i n v a l u e s . (3) An end e f f e c t , i n which the l a s t few samples i n a s e r i e s are judged a c c o r d i n g to d i f f e r e n t standards. S e v e r a l p o s s i b l e t e s t m a t e r i a l s were considered.. F i n a l l y , handwriting samples were chosen as s a t i s f y i n g the c r i t e r i a of e x p l l c i t n e s s and n e c e s s i t y f o r s u b j e c t i v e r a t h e r than d i s g u i s e d o b j e c t i v e e v a l u a t i o n . Two experiments were devised; the f i r s t t o t e s t f o r a c o n t r a s t e f f e c t , and the second to t e s t f o r the experience and end e f f e c t s . Evidence s i g n i f i c a n t a t the one percent l e v e l was p r e s e n t e d f o r r e j e c t i n g the hypothesis t h a t there i s no i i i c o n t r a s t e f f e c t . I t was shown t h a t there i s a long term i n c r e a s e i n the marks given, i n which the i n c r e a s e was spread over the whole range of marks, There was evidence s i g n i f i c a n t at the f i v e p ercent l e v e l f o r r e j e c t i n g the hypothesis t h a t there i s no end e f f e c t . I t was shown t h a t the l a s t three samples of a long s e r i e s tend to be downgraded. i v TABLE OP CONTENTS Page CHAPTER I . ANALYSIS OP THE PROBLEM 1 General Statement of the Problem 1 D i s c u s s i o n of the Problem . . . . . 1 Summary of the T h e o r e t i c a l D i s c u s s i o n . . . . . 4 P r e c i s e Statement of the Problem 5 I I . BACKGROUND AND SURVEY OP THE LITERATURE 7 The Need f o r Studies of Subjective Tests . . . 7 Choice of Test M a t e r i a l 9 S e l e c t i o n of Handwriting . . . . . 10 D e f i n i t i o n of Handwriting Q u a l i t y 11 The P o s s i b i l i t y of Other V a r i a b l e s 12 Related Studies and Previous Research . . . . . 12 I I I . EXPERIMENTAL DESIGN 15 General Statement of the Method 15 The Method i n D e t a i l 16 The s e l e c t i o n of handwriting samples . . . . 16 The s e l e c t i o n of judges . , 18 The use of f i l m s t r i p s 18 Experiment A 19 Experiment B 21 The P r o t e c t i o n against Confounding and Other Sources of E r r o r 23 V CHAPTER Page In t e s t i n g f o r the short term contrast e f f e c t 23 In t e s t i n g f o r the long term contrast e f f e c t 23 In t e s t i n g f o r the experience e f f e c t . . . . 24 In t e s t i n g f o r the end e f f e c t . . . . . . . . 24 The l e n g t h of passages 24 IV. COLLECTION OP DATA 25 Choosing Stable Samples . . . 25 Choosing G, A and P samples 26 The Experimental S e r i e s , 26 Experiment A . . , 27 Experiment B 28 The End E f f e c t 30 V. CONCLUSIONS . . . , 31 The Contrast E f f e c t 31 The Experience E f f e c t . 31 The End E f f e c t 32 V I . SUMMARY 33 S t r u c t u r i n g the Problem 33 The Experiments 34 The Conclusions 34 Further Questions . . . . 35 Using other v a r i a b l e s 36 F i n d i n g the fundamental determiners . . . . . 36 v i Page BIBLIOGRAPHY . . \u2022 37 APPENDIX A - Tables , 39 APPENDIX B - M a t e r i a l s 74 v i i LIST OP TABLES TABLE Page I. The Ayres Qategories to which 215 Handwriting Samples Were Assigned by Twelve Judges. . . . 42 I I . The C o r r e l a t i o n of the Evaluations Assigned 215 Handwriting Samples by Twelve Judges w i t h the Average of the Eva l u a t i o n s of These Judges 47 I I I . The D i s t r i b u t i o n of Evaluations of 215 Handwriting Samples by Ten Stable Judges . . 48 IV. A L i s t of Good, Average and Poor Samples. . . . 54 V. Experiment A - The Experimental S e r i e s . . . . 55 V I . Experiment B - The Experimental S e r i e s . . . . 56 V I I . The Eva l u a t i o n s i n Experiment A Runs 1 and 2 . 57 V I I I . The S t a b i l i t y of Judges - Experiment A . . . . 66 IX. The Ev a l u a t i o n s i n Experiment B Runs 1 and 2 . 67 X. The S t a b i l i t y of Judges - Experiment B . . . . 73 ACKNOWLEDGMENT The author would l i k e to render g r a t i t u d e a l l those who have been of a s s i s t a n c e i n t h i s research and p a r t i c u l a r thanks to Dr. R.N. Smith f o r h i s patience and he l p . CHAPTER I ANALYSIS OP THE PROBLEM General Statement of the Problem This study i s designed to d i s c o v e r whether or not there are c o n d i t i o n s under which a s u b j e c t i v e e v a l u a t i o n may be a f f e c t e d to some extent by preceding e v a l u a t i o n s of the same k i n d . For example, the mark given a handwriting sample of average q u a l i t y might depend t o some extent upon the q u a l i t y of the preceding sample. I t i s a l s o p o s s i b l e t h a t judges are conservative at f i r s t , r e s t r i c t i n g t h e i r marks to a narrow range, and become more l i b e r a l w i t h both h i g h and low marks i n succeeding judgments. D i s c u s s i o n of the Problem I d e a l l y , a r e g r e s s i o n equation should be w r i t t e n g i v i n g the t o t a l e f f e c t of a l l preceding samples upon the mark of the sample being evaluated. I t would be d i f f i -c u l t t o w r i t e such an equation at t h i s stage of r e s e a r c h . A simpler procedure has been adopted. The problem has been subjected t o l o g i c a l a n a l y s i s to determine the general ways i n which the e v a l u a t i o n of one sample might i n f l u e n c e f o l l o w i n g e v a l u a t i o n s . T h i s d i s c u s s i o n focuses upon these general p o s s i b i l i t i e s . The f i r s t few samples of the s e r i e s may be the most important. In t h i s case a l l of the succeeding samples are In some way evaluated against a f i x e d c r i t e r i o n , f o r once the f i r s t few samples are given they are f i x e d , This i s known as anchoring. Although s t u d i e s of anchoring might he of i n t e r e s t i n education, i t i s not considered t o be of prime importance i n t h i s study. The l a s t few samples graded before the sample i n question might be the most important. In t h i s case each sample i s evaluated p a r t l y In the l i g h t of other recent e v a l u a t i o n s . Depending upon the way In which the e f f e c t of any one sample dropped o f f w i t h succeeding samples, the e v a l u a t i o n of any one sample might depend upon, besides of course i t s own q u a l i t y , e i t h e r the s i n g l e preceding sample or the cumulative e f f e c t of s e v e r a l preceding samples. The p o s s i b i l i t y considered above i s one of those which w i l l be s t u d i e d i n these experiments. At t h i s stage of research, i t has been considered s u f f i c i e n t t o discover whether or not there i s such an e f f e c t and, i f there i s , whether i t i s s u b s t a n t i a l l y short term, - s u b s t a n t i a l l y long term or somewhere between the two. The f u l l l i s t of preceding samples might c o n t r i b u t e to the present e v a l u a t i o n i n a roughly even way. For a s e r i e s of any considerable l e n g t h , and where there were no r a d i c a l changes i n the samples being graded, t h i s e f f e c t would be i n d i s t i n g u i s h a b l e from that of simple anchoring. The l a s t few samples of a long s e r i e s would not change the average c h a r a c t e r i s t i c s of that s e r i e s to any considerable extent. 3 Any p o s s i b l e way i n which the marks assigned previous samples may i n f l u e n c e the present e v a l u a t i o n may then be i n t e r p r e t e d as e i t h e r a form of anchoring or a form of the c o n t r a s t e f f e c t . The number of preceding samples might be an important f a c t o r . There may be some change i n the judge 1s frame of reference such that these standards s h i f t as a s e r i e s of samples progresses. This second p o s s i b i l i t y i s a l s o s t u d i e d i n these experiments. I t w i l l be r e f e r r e d to as a p o s s i b l e experience e f f e c t . There seems to be but one other p o s s i b l e p r e d i c t a b l e way i n which the standards by which the samples i n a s e r i e s are evaluated might a l t e r . Perhaps there are c e r t a i n samples along the way which i n v a r i a b l y r e c e i v e higher or lower marks. Perhaps the f o u r t h , f i f t h and s i x t h samples are always judged more h a r s h l y than are a l l others. The p o s s i b i l i t y of there being such s p e c i a l samples seems r a t h e r remote. There are two places i n any s e r i e s where such a s i t u a t i o n i s most l i k e l y t o a r i s e . I t i s pos-s i b l e that e i t h e r the f i r s t few or the l a s t few samples of any long s e r i e s may r e c e i v e s p e c i a l treatment. The f i r s t few samples of any long s e r i e s belong t o the area of anchoring. For t h i s reason only the l a s t few samples of a long s e r i e s have been s e l e c t e d f o r s p e c i a l study. In these experiments a p o s s i b l e e f f e c t i n which the l a s t few samples of a s e r i e s r e c e i v e marks d i f f e r e n t from what they would r e c e i v e elsewhere i n the s e r i e s w i l l be r e f e r r e d to as the end e f f e c t . I f a judge were not exposed to the f u l l range of q u a l i t y he was t o meet i n a s e r i e s u n t i l (say) somewhere past the halfway p o i n t i n the s e r i e s , then he would s t i l l have problems of anchoring when the novel m a t e r i a l was f i r s t pre-sented. At t h a t time the judge's o r i g i n a l standards might a l s o be t e m p o r a r i l y upset. In t h i s study no s e r i e s w i l l be used which does not i n i t i a l l y expose a judge to the f u l l range of q u a l i t y l a t e r to be met or does not continue to give experience w i t h the f u l l range of q u a l i t y of the s e r i e s . Summary of the T h e o r e t i c a l D i s c u s s i o n This d i s c u s s i o n has pointed t o three ways i n which previous samples may i n f l u e n c e the present e v a l u a t i o n . A. A Contrast E f f e c t . A judge may minimize or exaggerate the d i f f e r e n c e between a sample and the preceding sample - or samples - so tha t the sample under c o n s i d e r a t i o n i s graded too high or too low. I t has been t r a d i t i o n a l l y assumed that the e r r o r w i l l be i n a d i r e c t i o n opposite to t h a t of the preceding sample. -i John M. Stalnaker, \"The Problem of the E n g l i s h Exam-i n a t i o n , \" Educational Record, September, 1931, pp. 35-48. p C.C. Ross and J.C. Stanley., Measurement i n Today ls Schools. (New York: P r e n t i c e - H a l l Inc., 19WT, P. 194. ^J.W. Wrightstone, J . Justman, and I . Robbins. E v a l u a t i o n i n Modern Education (New York: American Book Company\",\"'1956\"), p. 19W7~ 5 That i s , an average sample f o l l o w i n g a good sample would be judged t o be poorer than i f i t had followed another average or a poor sample. I f the contrast e f f e c t does operate i t may be short term, dependent only upon the immediately preceding sample, long term, dependent upon the cumulative e f f e c t of s e v e r a l preceding samples, or somewhere between the two. B, An Experience E f f e c t . As a s e r i e s of samples to be judged progresses, the judge may g r a d u a l l y r e v i s e h i s standards. The concern here i s not w i t h the a l t e r n a t e moods of e l a t i o n and despair which markers are supposed t o experience and which might be caused by any number of e x t e r n a l v a r i a b l e s . The question i s r a t h e r whether or not there i s any c o n s i s t e n t long term s h i f t i n values common to judges. Among such p o s s i b l e trends i s the one already a l l u d e d t o , that judges may s t a r t by a s s i g n i n g marks over a r e s t r i c t e d range. Cv An End E f f e c t . A marker may react i n a unique way to the l a s t few samples of a s e r i e s . For t h i s reason these samples should not be used i n an experiment t e s t i n g f o r any d i f f e r e n t e f f e c t . A P r e c i s e Statement of the Problem A number of conclusions reached regarding the judges to be used, the most appropriate s u b j e c t i v e l y judged m a t e r i a l , the range of scores t o be c a l l e d f o r and the experiments to be performed are a n t i c i p a t e d so that the problem may be 6 p r e c i s e l y s t a t e d here. Students i n the College of Education at the U n i v e r s i t y of B r i t i s h Columbia w i l l be the judges. Handwriting samples w i l l be graded on a twenty-five p o i n t s c a l e . Two experiments w i l l be performed. The f i r s t w i l l t e s t f o r the two p o s s i b l e forms of the c o n t r a s t e f f e c t . The second w i l l t e s t f o r the experience e f f e c t and the end e f f e c t . Under these circumstances, i t i s p o s s i b l e t o prepare three n u l l hypotheses. A. There i s no short or long term c o n t r a s t e f f e c t . B. There i s no experience e f f e c t . C. There i s no end e f f e c t . CHAPTER I I BACKGROUND AND SURVEY OP THE LITERATURE The Need f o r Studies of Subjective Tests We l i v e i n a century of enthusiasm f o r o b j e c t i v e t e s t s . At the height of the wave i n the 1920*s i t seemed but a matter of time u n t i l a l l t e s t s would be completely o b j e c t i v e . ^ Even essay questions, where i t was admitted that they had something t o commend them, were to be marked 7 according t o complex but s t r i c t l y o b j e c t i v e standards.' Throughout t h i s p e r i o d a l a r g e number of teachers continued to give s u b j e c t i v e type t e s t s and t o mark them on a l a r g e l y Q s u b j e c t i v e b a s i s . In recent years a more reasonable outlook has pre-v a i l e d . I t i s g e n e r a l l y accepted t h a t t e s t s which may be marked o b j e c t i v e l y have advantages i n r e l i a b i l i t y . ^ 5D,A. Worcester, \"The V a l i d i t y of Testing,\" School Review, v o l . 42 (September, 1934), pp. 527-531, 6 A.R. G i l l i l a n d and R.H. Jordan. The Classroom Teacher (New Yorkj The Century Company, 1'9'2'4Q , ~~p. ~2\"0~'e'tj seq. 7 I b i d , g Wright stone, Justman and Robbins, ap_. c i t . , p. 101.. q R.L. Thorndyke and E. Ha gen, Measurement a_nd E v a l u a t i o n i n Psychology and Education (New York: J.. Wiley and Sons, 19557* passimj I t i s a l s o widely accepted that there are r e s u l t s of teach-i n g which at the present time can best be evaluated sub-j e c t i v e l y . ' 1 ' 0 Apart from t h i s , many argue that the mere w r i t i n g of s u b j e c t i v e type t e s t s , and by t h i s essay t e s t s are u s u a l l y meant, i s of value t o the student. I t i s s t a t e d that the student thereby gains s k i l l i n o r g a n i z i n g m a t e r i a l and pre-senting i t i n a l o g i c a l f a s h i o n . Teachers and p r o f e s s o r s often complain that students enter h i g h schools and u n i v e r -s i t i e s i l l equipped to express themselves i n t h e i r language. Whether these arguments are v a l i d or not, they continue to be used and f o r these or other reasons many teachers con-t i n u e t o use s u b j e c t i v e e v a l u a t i o n . The numerous e a r l y s t u d i e s which demonstrated the u n r e l i a b i l i t y of marking systems which have s u b j e c t i v e 11 12 13 elements i n them, ' ' u are now accepted more as i n d i c a -t i o n s of needed research than as i n d i c a t i o n s t h a t e v a l u a t i o n 14 ought to be on no account s u b j e c t i v e . A glance at \"^Thorndyke and Hagen, Measurement and E v a l u a t i o n i H Psychology and Education, i b i d . \"^J.B.. F a l l s , \"Research i n Secondary Education,\" Kentucky School J o u r n a l , vol.. 6 (March, 1928), pp. 42-46. 12 Starch and E l l i o t t , \" R e l i a b i l i t y of Grading Work i n Mathematics,\" School Review, v o l . 21 ( A p r i l , 1913), pp. 254-259. E.W. Tiegs, Educational Diagnosis, E d u c a t i o n a l B u l l e t i n Number 18, C a l i f o r n i a Test Bureau, 1952. 14 \/ \\ L i n d q u i s t (ed.), Educational Measurement, American C o u n c i l on Education, Washington, p. 528. 9 any modern textbook i n ev a l u a t i o n w i l l confirm that there has developed i n recent years a renewed respect f o r the su b j e c t i v e i n e v a l u a t i o n , but there i s as yet almost no research on many 15 fundamental and cogent questions. y In \"educational w r i t i n g s i t has g e n e r a l l y been assumed tha t the u n r e l i a b i l i t y of s u b j e c t i v e e v a l u a t i o n i s due e n t i r e l y t o random e r r o r s of judgment. Any study which could e s t a b l i s h sources of e r r o r i n s u b j e c t i v e judgments which are not random might w e l l stimu-l a t e f u r t h e r research i n t o t h i s and r e l a t e d questions. 17 Stalnaker, i n a widely quoted a r t i c l e , c a l l s f o r such research i n t o s u b j e c t i v e e v a l u a t i o n and continues: I t may \"well be a b a s i c t e s t form which, p r o p e r l y \u2022controlled, can measure Important outcomes of l e a r n i n g not yet otherwise measured. Choice of Test M a t e r i a l I t i s c e r t a i n that any s u b j e c t i v e judgment, such as regarding a person's honesty or the q u a l i t y of a w r i t t e n paragraph, i s a f f e c t e d by many d i f f e r e n t f a c t o r s . Since d i f f e r e n t judges are l i k e l y t o examine the d i f f e r e n t f a c t o r s \" ^ L l n d q u i s t (ed.) E d u c a t i o n a l Measurement, i b i d . \"^Robert R. Ashburn, \"An Experiment i n the Essay Type Question,\" J o u r n a l of Experimental Education, v o l . 7 (September, 1938)7 PP-* 1-3. 17 'John M. Stalnaker, The Essay Type Examination i n L i n d q u i s t (ed.), Educational Measurement, op. c i t . , p. 53\u00b0\u00ab 10 i n d i f f e r e n t orders, i f indeed they examine more than one of them, even la r g e contrast e f f e c t s - a r e not l i k e l y to he observed. For example, one judge might be mainly i n f l u e n c e d by handwriting, another by grammar, and a t h i r d by s t y l e . A l l three might show lar g e i n d i v i d u a l c o n trast e f f e c t s , yet the b l u r r e d r e s u l t s due to averaging might not y i e l d any s i g n i f i c a n t r e s u l t s . I t should be c l e a r that i f e f f e c t s due to order i n making s u b j e c t i v e judgments are to be found, the experiment, at l e a s t during the f i r s t stages of research, must be con-f i n e d to one v a r i a b l e . I t i s necessary t o choose a v a r i a b l e which can be considered atomic i n the sense that by some means of other a group of judges may be l e d t o consider the same one f a c t o r . At the same time i t must be of d i r e c t s i g n i f i c a n c e i n education. S e l e c t i o n of Handwriting I t would be d i f f i c u l t to have a group of judges agree on what i s meant by s t y l e or o r i g i n a l i t y . Such f a c t o r s , i f they are f a c t o r s at a l l , would be very d i f f i c u l t to d e f i n e . Such f a c t o r s as grammar and s p e l l i n g might be consid-ered d e s i r a b l e as s a t i s f y i n g the c r i t e r i o n of e x p l i c i t n e s s . They are p r o s c r i b e d by an unfortunate d i s c r e t e n e s s . The judge might e a s i l y enumerate the number of e r r o r s and base h i s e v a l -u a t i o n upon t h i s o b j e c t i v e t a l l y . Handwriting seems to stand w i t h grammar and s p e l l i n g i n being capable of e x p l i c i t d e f i n i t i o n . At the same time there i s no simple way i n which the judge may convert the 11 t a s k of e v a l u a t i o n i n t o one of enumeration. While other f a c t o r s might be found which s a t i s f y these c r i t e r i a , handwriting thus seems to be at l e a s t one sample of behaviour w e l l s u i t e d f o r study at t h i s stage of research Into s u b j e c t i v e e v a l u a t i o n . D e f i n i t i o n of Handwriting Q u a l i t y I t would be a mistake t o assume that an unselected group of judges would immediately use a common c r i t e r i o n i f asked t o judge the q u a l i t y of a set of handwriting samples. One of the f i r s t subjects used i n these experiments i n q u i r e d at some l e n g t h e x a c t l y what i t was he was supposed t o be l o o k i n g f o r j slope, evenness w i t h the l i n e , shape of l e t t e r s or what. I t would be p o i n t l e s s t o s i n g l e out one of these f o r i n v e s t i g a t i o n . The whole process would immediately s t a r t again and i n the end the r e s u l t s would be h o p e l e s s l y remote from the problem under study. The approach sketched below demonstrates the p o s s i b i l i t y of working w i t h an o p e r a t i o n a l d e f i n i t i o n of handwriting q u a l i t y which more or l e s s i n s u r e s that the judges w i l l be of one mind i n t h i s matter. Before the experiments proper were begun, a group of judges evaluated a l a r g e set of handwriting samples. On the b a s i s of these e v a l u a t i o n s , samples were designated as good, average or poor. The marks assigned to these samples by.the judges used i n the experiments proper were then examined. Judges who d i d 12 not s u b s t a n t i a l l y agree w i t h t h i s p r i o r l a b e l l i n g were r e j e c t e d . This device does not p r e j u d i c e the r e s u l t s of the experiments. Any judge, whether he agrees w i t h other judges or not as to what c o n s t i t u t e s good handwriting, may or may not show a c e r t a i n type of marking behavior. The P o s s i b i l i t y of Other V a r i a b l e s This study deals w i t h handwriting, but the f i n a l i n t e r e s t i s of course i n s u b j e c t i v e e v a l u a t i o n . I t turns out t h a t handwriting i s the most s u i t a b l e v a r i a b l e f o r t h i s p r e l i m i n a r y study, but any conclusions reached here w i l l have t o be r e t e s t e d u s i n g the l e s s e a s i l y managed v a r i a b l e s before any g e n e r a l i z a t i o n s may be considered. Related Studies and Previous Research Where s u b j e c t i v e e v a l u a t i o n i n education i s concerned, o l d proverbs have p r e v a i l e d by d e f a u l t . Elementary t e x t -books on educational measurement r e f e r t o i t , u s u a l l y i n , 18 p a s s i n g . E f f e c t s analogous to the contrast and experience e f f e c t s were i n v e s t i g a t e d i n the f i e l d of general psychology some years ago. Sources of e r r o r analogous t o the c o n t r a s t e f f e c t were studied In the judging of weights, musical p i t c h , C.C.. Ross and J.C. Stanley, op, c i t . , p. 194. 19 20 21 i n t e n s i t y of sound, and such thi n g s as a e s t h e t i c value. ' ' In most of these experiments a con t r a s t e f f e c t was found i n which the d i f f e r e n c e between two succeeding samples was exaggerated. There I s no evidence a v a i l a b l e t o the w r i t e r of p e r t i n -ent p s y c h o l o g i c a l s t u d i e s of the judging of handwriting samples. Sources of e r r o r somewhat analogous to what has been c a l l e d the experience e f f e c t were a l s o s t u d i e d i n general psychology. What p s y c h o l o g i s t s c a l l experiments i n anchoring were concerned w i t h d i s c o v e r i n g how r a p i d l y a judge would b u i l d f o r himself a c o n s i s t e n t scale when faced w i t h the task of grading some novel m a t e r i a l . Here again the p a r a l l e l i s not too c l o s e . Most people are w e l l acquainted w i t h hand-w r i t i n g and probably have some strong notions as to i t s q u a l i t y . Furthermore the second experiment i n t h i s study assumes th a t anchoring takes p l a c e . I t then seeks to deter-mine whether or not there i s a c o n s i s t e n t s h i f t i n standards. I t i s s c a r c e l y necessary at t h i s p o i n t to r e f e r to the p a u c i t y of e a r l i e r e d u cational research i n t o t h i s ques-t i o n . \"^S. Truman and E.. Weber, \"Judgment of P i t c h as a Function of the S e r i e s , \" U n i v e r s i t y qf_ C a l i f o r n i a Pub, i n Psychology, 1 9 2 8 , pp. 215-223. 20 H. Wpodrow,\"Behavior w i t h Respect t o Short Stimulus Forms,\" J o u r n a l of Experimental Psychology, v o l . 11 , 1928, pp. 167-193. 21W..A. Hunt, \"Anchoring E f f e c t s i n Judgment,\" American J o u r n a l of Psychology, v o l . 54, 1 9 4 l , pp. 395-^03. 14 One author a s s e r t s that marks given s u b j e c t i v e l y tend 2 2 t o r i s e as the s e r i e s being judged progresses. No research was quoted t o s u b s t a n t i a t e the view. In summary, i t seems that respected a u t h o r i t i e s are agreed that there i s l i t t l e fundamental research i n t o sub-j e c t i v e e v a l u a t i o n and that such research may w e l l be an important c o n t r i b u t i o n to the f i e l d of evaluation,. C.C. Ross and J.C. Stanley, op_. c i t . , p. 194. CHAPTER I I I THE EXPERIMENTAL DESIGN A_ General Statement of the Method Altogether three groups of judges were used. The f i r s t group was used t o s e l e c t those handwriting samples which were t o be used i n two f o l l o w i n g experiments. Each of the remaining groups of judges was used i n one of the experiments. The contrast e f f e c t was studied i n one e x p e r i -ment and the experience e f f e c t and the end e f f e c t were studied i n the other. A procedure was e s t a b l i s h e d f o r the s e l e c t i o n of s t a b l e samples and judges. In the f i r s t of the experiments each member of a group of judges evaluated an ordered set of handwriting samples. Handwriting samples which were known to be of average q u a l i t y were placed immediately f o l l o w i n g one or four good or poor handwriting samples. At a l a t e r date each judge re-evaluated the set of samples he f i r s t evaluated, but they had been interchanged so that average samples which had at f i r s t f o l l o w e d good samples now fol l o w e d the same number of poor samples and v i c e v ersa. In the second experiment each member of a group of judges a l s o evaluated an ordered set of handwriting samples. I d e n t i c a l p a t t e r n s of good, average and poor samples appeared near the beginning and near the end of the long s e r i e s . At a l a t e r date each judge re-evaluated the set of samples he f i r s t evaluated, but the samples making up the two i d e n t i c a l p a t t e r n s had been interchanged. A set of samples arranged i n the same p a t t e r n appeared near the middle of the s e r i e s In each case, but the purpose of t h i s set of samples being so placed i s not of immediate con-cern. The second experiment was a l s o used t o t e s t f o r the end e f f e c t . In the f i r s t run, one set of samples known to be aver-age was t o be found at the end of the long s e r i e s and another near the middle. In the second run these two sets of average samples had been interchanged. The Method i n D e t a i l The s e l e c t i o n of handwriting samples. I n i t i a l l y 215 handwriting samples were prepared. Twelve judges were s e l e c t e d from the po o l of v o l u n t e e r s . Since these judges were t o be used only once during the experiments and since a number of judges was not l i k e l y t o be a v a i l a b l e at a l a t e r date, there were some r e s t r i c t i o n s on the choice made. Each such judge was asked t o evaluate the e n t i r e p o ol of samples u s i n g the Ayres scale of handwriting.. As a r e s u l t of t h i s grading, each sample r e c e i v e d twelve scores on the Ayres handwriting s c a l e . This scale has a range of scores from two t o nine. The average of these scores was obtained f o r each sample, and each 'sample was 17 t e m p o r a r i l y assigned to the Ayres category c l o s e s t to t h i s average, The c o r r e l a t i o n was then obtained between the scores assigned by any one judge and the scores obtained by the average procedure. Where the c o r r e l a t i o n was found to be l e s s than .75* the Judge was considered e r r a t i c . A l l of the scores assigned by e r r a t i c judges were then s t r u c k from the r e c o r d . The i n c l u s i o n of the scores assigned by e r r a t i c judges would not have d i s t u r b e d the r e s u l t s of e i t h e r experiment,; they would simply .have produced an u n n e c e s s a r i l y high rate of a t t r i t i o n among the samples. Stable samples were then s e l e c t e d by choosing those whose assigned values were w i t h i n p r e v i o u s l y assigned l i m i t s . Good samples (henceforth G) were designated as those whose assigned values did.not drop below seven. In f a c t , the scores of most of the samples s e l e c t e d d i d not f a l l below e i g h t . Poor samples (henceforth P) were designated as those whose assigned values d i d not r i s e above f o u r . The scores of most of the samples d i d not r i s e above three. Average samples (henceforth A) were designated as those whose assigned values were f i v e or s i x . Two deviant e v a l u a t i o n s of one d i v i s i o n on the Ayres scale or one deviant e v a l u a t i o n of two d i v i s i o n s was allowed, Apart from the e l i m i n a t i o n of those who were w i l d l y e r r a t i c , the judges doing t h i s e v a l u a t i o n were unselected, and there i s the p o s s i b i l i t y that strong c o n t r a s t e f f e c t s of the k i n d being 18 studied i n these experiments were operating. Under these c i r -cumstances, the two u n i t d e v i a t i o n s permitted seems s t r i c t enough. A number of samples were then chosen from a l l p o i n t s on the e ight p o i n t Ayres scale w i t h a preponderance of samples from the four and seven p o i n t s . These were designated as C samples. Their f u n c t i o n was mainly t o guard against confound-in g of e f f e c t s i n the experiments. I t should be noted that the equivalence of the samples l a b e l l e d G, A or P was at no time assumed. I t was simply r e q u i r e d that a s t a b l e judge would consider samples which were l a b e l l e d G to be b e t t e r than samples which were l a b e l l e d A, and samples which were l a b e l l e d A to be b e t t e r than those which were l a b e l l e d P. The s e l e c t i o n of judges. In the experiments, the c r i t i c a l r e l a t i o n s h i p was i n v a r i a b l y the d i f f e r e n c e between an A sample and the preceding G or P sample. For each judge, the means of a l l the G samples, a l l the A samples and a l l the P samples Involved i n such c r i t i c a l p a i r s were found. Where the d i f f e r e n c e between the means of these G and A samples and A and P samples was s i g n i f i c a n t at the two percent l e v e l of confidence the judge was deemed s t a b l e . This c r i t e r i o n e l i m i n -ated two prospective judges from experiment B, The use of_ f i l m s t r i p s . . The time of exposure of each sample, the p o s s i b i l i t y of a judge r e f e r r i n g back to an e a r l i e r sample, and the p o s s i b i l i t y of samples being damaged by handling were c o n t r o l l e d by having photographs of the p e r t i n e n t samples place d i n c o r r e c t sequence on a f l l m s t r i p f o r both experiments A and B. I t was p o s s i b l e that samples which had been judged s t a b l e when viewed at f i r s t hand might become unstable when viewed on a screenw The p o s s i b i l i t y of t h i s i n s t a b i l i t y a r i s i n g was remote and i n f a c t d i d not occur. A random s e l e c t i o n of samples from the experimental runs were t e s t e d and found to be s l i g h t l y more st a b l e than they had been when viewed f i r s t hand. Experiment A. Each judge was presented w i t h a s e r i e s of 1 6 8 handwriting samples. In the s e r i e s , d i s t r i b u t e d evenly, were eig h t A samples each f o l l o w i n g one G sample. There were the same number of A samples each f o l l o w i n g one P sample. There were a l s o eight A samples each f o l l o w i n g four G samples and the same number of A samples each f o l l o w i n g f o u r P samples,. The f o l l o w i n g represents the p a t t e r n of G, A, P and C samples from the f i r s t run of Experiment A. A A C C C C G G G G A C C P P P P A C C G A P A C C G G G G A C C P P P P A C C G A P A C C G G G G A C C P P P P A C C C A P A C C G G G G A C C P P P P A C C G A P A C C G G G G A C C P P P P A C C G A P A G C G G G G A C C P P P P A C C G A P A G C G G G G A C C P P P P A C C G A P A C C G G G G A C C P P P P A C C G A P A C C C C 20 The f i r s t s i x samples were shown once to each judge to f i x i n h i s mind the range of q u a l i t y to be expected and were a l s o the f i r s t s i x to be evaluated. This was to provide f o r anchoring. Each judge was asked to evaluate each sample on a twenty-five p o i n t s c a l e , twenty-five being the best and one the poorest:. The exact d i r e c t i o n s to the judges are given i n Appendix B. The f i n a l C samples were used to p r o t e c t the e x p e r i -mental samples from the end e f f e c t . The f i r s t four C samples were chosen from the four to seven p o i n t s on the scale to provide the judge w i t h experience w i t h that p a r t of the range to be encountered. The sets of four G samples followed by an A sample and four P samples fo l l o w e d by .an A sample occur i n p a i r s , as do the sets of one G sample fo l l o w e d by an A sample and one P sample f o l l o w e d by an A sample. Because there was a shortage of s t a b l e P samples, some of the f i r s t P samples i n the PPPPA sets are r e p e t i t i o n s of P samples i n e a r l i e r s e t s . Even i f these were recognized the scores would not i n f l u e n c e the experimental evaluations be.cause only the scores given A samples are used i n t e s t i n g f o r the co n t r a s t e f f e c t . Two C samples were interposed at every opportunity to prevent any p o s s i b l e p a t t e r n from developing i n the mind of the judge. The f i n a l C sample before the experimental G sample i n the GAPA sets i s i n a l l cases average or l e s s . This 21 was done to p r o t e c t the short term contrast e f f e c t from the long term contrast e f f e c t . Each judge was presented w i t h t h i s s e r i e s by means of successive frames on a f i l m s t r i p . A maximum of ten seconds was allowed f o r the e v a l u a t i o n of each frame.. I f the judge f i n i s h e d before that time had elapsed, the experimenter immediately presented the f o l l o w i n g frame so that the judge would not be tempted to re-evaluate the frame. The judge wrote h i s evalua-t i o n on a sheet provided f o r that purpose. An ev a l u a t i o n sheet i s given i n Appendix B. I f any experience e f f e c t was operating i t was c l e a r l y necessary t o leave a considerable time between the f i r s t and second runs. For t h i s reason, and f o r ease of t i m e t a b l i n g , each judge was asked to r e t u r n one- week a f t e r the f i r s t run. On t h i s occasion he evaluated the same samples as before, but the A samples i n the GGGGA - PPPPA sets and the A samples i n the GAPA sets had been interchanged. In the combined f i r s t and second runs, there was then a set of s i x t e e n A samples each of which at one time had been evaluated f o l l o w i n g four G samples and at another time had been evaluated f o l l o w i n g four P samples. In the combined runs there was a l s o a set of s i x t e e n A samples each of which had at one time been evaluated f o l l o w -one G sample and at another time had been evaluated f o l l o w i n g one P sample. ^ ^ ^ i 2 1 ? _ S i To f a c i l i t a t e the d i s c u s s i o n of t h i s experiment i t i s d e s i r a b l e to define a short s e r i e s and a long 22 s e r i e s . The long s e r i e s i s the complete set of samples to be evaluated at any one s i t t i n g . A model s e r i e s of samples - say GGPAPAA - which i s used at var i o u s p o i n t s i n the long s e r i e s , but p o s s i b l y constructed from d i f f e r e n t samples each time i t appears, i s c a l l e d a short s e r i e s . In t h i s experiment the short s e r i e s AAAAAGAPPGAAGGAPPAG was used. A l l of the short s e r i e s i n t h i s experiment are of t h i s c o n f i g u r a t i o n . The three short s e r i e s used are designated SS 1, SS 2 and SS^. The long s e r i e s at the f i r s t s i t t i n g was c c c c c s s 1 c c c c c s s 2 c c c c c A A A A C C C C C SS 3 C C C A A A A. As i n experiment A, each judge evaluated each sample i n the long s e r i e s on a twenty-five p o i n t s c a l e . In the second run one week l a t e r the f i r s t and t h i r d short s e r i e s had been interchanged. The f i r s t f i v e A samples i n each short s e r i e s were to p r o t e c t the f o l l o w i n g samples from c o n t r a s t e f f e c t s . I t w i l l be remembered that the equivalence of the samples i n any one category G, A or P i s not to be assumed. I f ther.e i s an experience e f f e c t which i s caused by the samples themselves, then SS-L might very w e l l do t h i s i n a d i f f e r e n t way from SS^. SS 2 makes i t p o s s i b l e to dis c o v e r whether or not t h i s i s so. 23 The sets of four A samples are to t e s t f o r the end e f f e c t . Only the l a s t three are Involved i n the t e s t . The preceding A and C samples, \"being placed as they are, prevent any con-t r a s t e f f e c t from operating on the experimental A samples. The P r o t e c t i o n against Confounding and Other Sources of E r r o r By confounding Is meant the operation of one e f f e c t to obscure the t e s t f o r some other e f f e c t . Since confounding of e f f e c t s would i n v a l i d a t e any r e s u l t s , care was taken t o elim-i n a t e them. In t e s t i n g f o r the short term c o n t r a s t e f f e c t . The long term c o n t r a s t e f f e c t was guarded against by the appropriate choice of samples to precede each GAPA set. The f o u r t h pre-ceding sample was i n v a r i a b l y P. The t h i r d was i n v a r i a b l y A. The f i r s t of the two C samples was chosen to be somewhat above average and the second somewhat below average. I t would be u n l i k e l y f o r any long term c o n t r a s t e f f e c t to become operative under these circumstances. Obviously the end e f f e c t cannot be o p e r a t i v e . In t e s t i n g f o r the long term contrast e f f e c t . I t i s not p o s s i b l e t o prevent the short term contrast e f f e c t from ope r a t i n g . The a l t e r n a t i v e procedure of assuming that i t does operate and t e s t i n g f o r any f u r t h e r e f f e c t due to the long term e f f e c t was adopted. Once more, the end e f f e c t cannot become op e r a t i v e . 24 In t e s t i n g f o r the experience e f f e c t . A l l of the short s e r i e s are of the same design. Therefore i t i s assumed th a t any contrast e f f e c t s operating w i t h i n the short s e r i e s may be s a f e l y ignored. The beginning of each short s e r i e s i s care-f u l l y p r o t e c t e d against any con t r a s t e f f e c t s by average samples. As i n the t e s t s f o r the contrast e f f e c t s , the l a s t few samples from the long s e r i e s were not used. The end e f f e c t cannot then be confounded w i t h the t e s t f o r the experience e f f e c t . In t e s t i n g f o r the end e f f e c t . The beginning of each set of samples used to t e s t f o r the end e f f e c t i s pro t e c t e d by C and A samples i n such a way that n e i t h e r the short nor long term c o n t r a s t e f f e c t s can operate. The l e n g t h of passage. The handwriting samples could have been prepared w i t h the selfsame passage w r i t t e n out to completion or e l s e w i t h i n the same p h y s i c a l space, the passage coming t o an end when the space provided was exhausted. I t was probably not too important which was done. I t i s p o s s i b l e that the f i x a t i o n about l e n g t h which some teachers have ( w r i t e one page on ...) might i n f l u e n c e t h e i r judgments and f o r that reason a l l of the samples have the same p h y s i c a l l e n g t h . The makers of the Ayres scale adopted the same procedure. CHAPTER IV COLLECTION OF THE DATA This chapter d e t a i l s the s p e c i f i c steps i n the c o l -l e c t i o n of the data from the choosing of s t a b l e handwriting samples to the f i n a l s t a t i s t i c a l a n a l y s i s of the r e s u l t s . Choosing Stable Samples Each of the twelve judges s e l e c t e d from the pool eva-l u a t e d the f u l l 215 samples. The r e s u l t s of t h i s e v a l u a t i o n are given i n Appendix A, Table I . I f a l l of the scores assigned t o a c e r t a i n sample were t i g h t l y grouped about some p o i n t on the Ayres s c a l e , then the sample could immediately be considered s t a b l e . I f the scores assigned were somewhat more widespread then i t was p o s s i b l e that the sample was t o be considered unstable. I t was a l s o p o s s i b l e that one or two e r r a t i c judges were c o n t r i b u t i n g t o the unnecessary r e j e c t i o n of samples which were i n f a c t s t a b l e . To guard against t h i s , each sample was given a mean value based upon these f i r s t e v a l u a t i o n s . This mean value i s also-shown i n Appendix A, Table I I . Any judge who demonstrated a c o r r e l a t i o n of l e s s than .75 was considered unstable. By t h i s c r i t e r i o n judge 6, w i t h a c o r r e l a t i o n of l e s s than .748, and judge b, w i t h a c o r r e l a t i o n of .448, were considered unstable and t h e i r scores were e l i m i n a t e d . 26 The correc t e d scores obtained by s t r i k i n g out the ev a l u a t i o n s of judges 6 and b are given i n Appendix A, Table I I I . Choosing G, A, and P_ samples The c o n f i g u r a t i o n of scores f o r each sample i n Table I I I was then examined. Each sample was placed i n a category of G, A or P i f i t s a t i s f i e d the c r i t e r i o n given on page 17. I f t h i s was not p o s s i b l e i t was returned to the pool from which C samples would l a t e r be drawn. The samples which were c l a s s i f i e d as G, A, and P are given i n Table IV. Samples were then chosen t o make up the experimental s e r i e s . C samples were chosen from the pool to complete the experimental s e r i e s , each s a t i s f y i n g the c r i t e r i o n f o r that i n d i v i d u a l sample. For example, the C sample which immed-i a t e l y precedes the G sample i n each GAPA set must be aver-age or l e s s . The samples f o r any one run of any one experiment were placed i n c o r r e c t order and photographed. I t was noted e a r l i e r t h a t a shortage of P samples r e q u i r e d the r e p e t i t i o n of c e r t a i n n o n - c r i t i c a l P samples. The Experimental S e r i e s The experimental s e r i e s f o r experiments A and B are given i n Appendix A, Tables V and VI.. In both cases only the s e r i e s f o r the f i r s t run i s given. The s e r i e s f o r the second run i s i d e n t i c a l , save that the c r i t i c a l A samples are interchanged. 2 7 Experiment A A group of ten judges was then chosen from the pool of v o l u n t e e r s . A l l of the judges were Education students at the U n i v e r s i t y of B r i t i s h Columbia. Each judge graded each sample as i t appeared on the screen i n f r o n t of him and wrote h i s e v a l u a t i o n on the sheet provided. The evaluations are ta b u l a t e d i n Appendix A, Table V I I . The e v a l u a t i o n s assigned by each judge i n the f i r s t run of the experiment were used to measure the s t a b i l i t y of the judge. Where the d i f f e r e n c e s of the means of the G, A and P samples were not s i g n i f i c a n t at the two percent l e v e l the v judge was e l i m i n a t e d . The measures of the s t a b i l i t y of the judges i n Experiment A are given i n Table V I I I . A l l of the pr o s p e c t i v e judges i n t h i s experiment surpassed t h i s c r i t e r i o n . The scores assigned i n both runs were then examined f o r evidence of the short term contrast e f f e c t and the long term contrast e f f e c t . The d i f f e r e n c e between means used to t e s t f o r the short term c o n t r a s t e f f e c t was s i g n i f i c a n t at the one percent l e v e l , w i t h the judge tending to minimize the d i f f e r e n c e between an average sample and the preceding extreme sample. The mean score of the A samples f o l l o w i n g one G sample was 12.7 and the mean score of the A samples f o l l o w i n g one P sample was 11.8. The standard e r r o r of the d i f f e r e n c e was .31. The d i f f e r e n c e of means used t o t e s t f o r the long term c o n t r a s t e f f e c t was, as would be expected, a l s o s i g n i f i c a n t at 28 the one percent l e v e l , but the d i f f e r e n c e of means was not s i g n i f i c a n t l y g r e a t e r than the d i f f e r e n c e of means used t o t e s t f o r the short term c o n t r a s t e f f e c t . The mean score of the A samples f o l l o w i n g f o u r G samples was 13.5 and the mean score of the A samples f o l l o w i n g f o u r P samples was 12 .1 . The standard e r r o r of the d i f f e r e n c e was . 2 6 . Experiment B The remaining group of ten judges was used f o r t h i s experiment. They were a l s o students i n E d u c a t i o n at the U n i v e r s i t y of B r i t i s h Columbia. Once more, each judge graded the samples on a twenty-f i v e p o i n t s c a l e as p i c t u r e s of the samples appeared on the scre e n . These e v a l u a t i o n s are given i n Appendix A, Table IX. The f i r s t run e v a l u a t i o n s were again used t o measure the s t a b i l i t y of the judges. Where the d i f f e r e n c e s of the means of the G, A and P samples were not s i g n i f i c a n t at the two pe r c e n t l e v e l , the judge was deemed u n s t a b l e . The measures of the s t a b i l i t y of the judges i n Experiment B are given i n Table X. The e v a l u a t i o n s of two judges were e l i m i n a t e d by t h i s c r i t e r i o n . F o l l o w i n g the second run, the scores a s s i g n e d i n the second short s e r i e s f o r both runs were examined t o determine whether or not the experiment was i n v a l i d . The means of the G, A and P samples from the f i r s t and second runs were com-pared. A d i f f e r e n c e of means f o r any one k i n d of sample s i g n i f i c a n t a t the ten percent l e v e l was co n s i d e r e d grounds f o r r e j e c t i n g the experiment. There was no such s i g n i f i c a n t d i f f e r e n c e of means. For G samples the mean scores were 17.4 and 1 7 . 6 . For A samples they were 12.8 and 12.4 and f o r P samples they were 11.0 and 11 .1 . The means of the G, A and P samples from the combined f i r s t and second runs were then examined to discover whether or not an experience e f f e c t was operating. While the means of a l l three kinds of samples from the p o s i t i o n near the end of the long s e r i e s were somewhat higher than the means from the p o s i t i o n near the beginning of the long s e r i e s , none of the d i f f e r e n c e s were s i g n i f i -cant . The G, A and P samples were then considered as c o n s t i -t u t i n g one group. The mean of the group from near the end of the long s e r i e s was found to be s i g n i f i c a n t l y greater than the mean of the group from near the beginning of the long s e r i e s . The d i f f e r e n c e was s i g n i f i c a n t at the one percent l e v e l of confidence. The mean score f o r the former group was 12.9 while that f o r the l a t t e r group was 1 3 . 3 . Because the samples i n v o l v e d i n t h i s t e s t were spread over a much wider range than formerly the c o r r e l a t i o n between the two sets of scores was higher than i n previous t e s t s . Consequently the standard e r r o r of the d i f f e r e n c e was only .143 and the above d i f f e r e n c e was found t o be s i g n i f i c a n t . 30 The End E f f e c t The data used i n the t e s t f o r the end e f f e c t was taken from the experimental s e r i e s In Experiment B. Each judge graded three A samples i n an intermediate p o s i t i o n i n the long s e r i e s and three other A samples at the end of the long s e r i e s . Before the second run, these two sets of A samples had been interchanged. \u2022 The mean of the set from the intermediate p o s i t i o n was then compared w i t h the mean of the same set of samples from the end p o s i t i o n . The mean of the evaluations of the samples from the end p o s i t i o n was 13.1 while the mean of the evalua-t i o n s of the samples from the intermediate p o s i t i o n was 1 3 . 9 . This d i f f e r e n c e was found to be s i g n i f i c a n t at the f i v e percent l e v e l . CHAPTER V CONCLUSIONS The Contrast E f f e c t I t i s concluded that under the c o n d i t i o n s of t h i s experiment, we may r e j e c t the hypothesis that there i s no short term contrast e f f e c t i n the e v a l u a t i o n of handwriting samples. I t operates i n a d i r e c t i o n opposite to that In which i t has t r a d i t i o n a l l y been assumed i t might operate i n that a judge appears to minimize the d i f f e r e n c e between an extreme sample and the f o l l o w i n g average sample. There I s a l s o a contrast e f f e c t when an average sample f o l l o w s f o u r good or poor samples, but the e f f e c t i s no more than can be accounted f o r by the e f f e c t of the l a s t of the four preceding samples. I t i s th e r e f o r e concluded that we may accept the hypothesis that there i s no long term con-t r a s t e f f e c t . The Experience E f f e c t I t i s concluded that we may r e j e c t the hypothesis that there i s no experience e f f e c t . Under the circumstances of t h i s experiment, e v a l u a t i o n s of handwriting samples tend to r i s e as the s e r i e s progresses. This r i s e i s not confined to any one p a r t of the range of values i n v o l v e d . This experiment provides evidence f o r no other form of experience e f f e c t . 32 The End E f f e c t I t i s concluded t h a t we may t e n t a t i v e l y r e j e c t the hypothesis t h a t there i s no end e f f e c t . Under the circum-stances of t h i s experiment, the l a s t three samples of a s e r i e s of h a ndwriting samples r e c e i v e lower marks than do s i m i l a r samples from an i n t e r m e d i a t e p o s i t i o n i n the s e r i e s . CHAPTER VI SUMMARY S t r u c t u r i n g the Problem I t was f i r s t of a l l asked whether or not there are circumstances of s i g n i f i c a n c e i n Education i n which the p o s i t i o n of a sample i n a s e r i e s might p r e j u d i c e the way i n which i t i s evaluated.. Where the s e r i e s of samples to be judged i s r e g u l a r , i t was decided t h a t there are three and o n l y three essen-t i a l l y d i f f e r e n t ways i n which the e v a l u a t i o n of one sample could be i n f l u e n c e d . There c o u l d be an experience e f f e c t i n which, f o r reasons not s p e c i f i e d , a judge might r e v i s e h i s standards as the s e r i e s p r o g r e s s e d . There c o u l d be a c o n t r a s t e f f e c t i n which the judge minimized or exaggerated the d i f f e r e n c e between a sample and the succeeding sample. Such an e f f e c t , i f i t e x i s t e d , c o u l d be due to e i t h e r the immediately p r e c e d i n g sample or to the cumulative e f f e c t of s e v e r a l p r e c e d i n g samples. There c o u l d be an end e f f e c t , i n which the judge a l t e r e d h i s standards f o r the l a s t few samples of the s e r i e s . S e v e r a l components of essay w r i t i n g were c o n s i d e r e d i n choosing t e s t m a t e r i a l , i n c l u d i n g grammar, s t y l e and hand-w r i t i n g . I t was decided t h a t handwriting was the best v a r i a b l e f o r a p r e l i m i n a r y study. 34 Teachers from B.C. schools and students i n Education at the U n i v e r s i t y of B r i t i s h Columbia were accepted as volunteers i n the experiments. The Experiments Two experiments of the s i n g l e group t e s t - r e t e s t v a r i e t y were designed to t e s t f o r the three hypothesized e f f e c t s . In t e s t i n g f o r the short term c o n t r a s t e f f e c t a set of average samples was evaluated while f o l l o w i n g one good sample and while f o l l o w i n g one poor sample. The same s o r t of t e s t was used f o r the long term con-t r a s t e f f e c t , except that the average samples i n t h i s case each f o l l o w e d f o u r good or four poor samples. The experimental s e r i e s used i n t e s t i n g f o r an exper-ience e f f e c t contained three short s e r i e s of samples, i d e n t i c a l i n c o n f i g u r a t i o n of good, average and poor samples. They were evaluated t w i c e . In the second run the f i r s t and t h i r d short s e r i e s had been interchanged. The same experimental s e r i e s was used t o t e s t f o r the end e f f e c t . A set of three average samples appeared i n an intermediate p o s i t i o n and' another set of three average samples appeared at the end of the long s e r i e s . These two sets of average samples were interchanged between runs. 35 The Conclusions I t was concluded that there i s a short term contrast e f f e c t i n the e v a l u a t i o n of handwriting samples.. I t operates i n a d i r e c t i o n opposite to that i n which i t has t r a d i t i o n a l l y been assumed i t might operate. There i s a l s o a contrast e f f e c t when an average sample f o l l o w s four good or poor samples, but the e f f e c t i s no more than can be accounted f o r by the e f f e c t of the l a s t of the four extreme samples. The hypothesis that there Is no long term c o n t r a s t e f f e c t was accepted. I t was concluded that e v a l u a t i o n s of handwriting samples tend to r i s e as the s e r i e s progresses. This r i s e i s not confined t o any one part of the range of values i n v o l v e d , but seems to be d i s t r i b u t e d evenly over that range. I t was concluded t h a t the l a s t three samples of a s e r i e s of handwriting samples may r e c e i v e lower marks than do s i m i l a r samples from an intermediate p o s i t i o n i n the s e r i e s . I t may then be concluded t h a t there are circumstances of s i g n i f i c a n c e i n Education i n which the contrast e f f e c t , the experience e f f e c t and the end e f f e c t may be found to operate. Further Questions I t seems as though f u r t h e r research should develop along two main l i n e s . Using other v a r i a b l e s . S i m i l a r s t u d i e s using other v a r i a b l e s ought to e s t a b l i s h whether the e f f e c t s here noted are general f a c t o r s to be contended w i t h i n most s u b j e c t i v e e v a l u a t i o n or are p e c u l i a r i t i e s of handwriting or the experimental.controls of these experiments. F i n d i n g the fundamental determiners. I f i t Is once supposed that these e f f e c t s do g e n e r a l l y operate i n the way concluded here, i t immediately becomes a matter of concern what i t i s which d i s t i n g u i s h e s t h i s s i t u a t i o n from the one i n v e s t i g a t e d by the general p s y c h o l o g i s t s . Using novel m a t e r i a l and sensory s t i m u l i they found contrast e f f e c t s which operate i n a contrary manner to that observed i n Experiment A. I t might be p o s s i b l e to discover those b a s i c elements of a s i t u a t i o n which determine whether or not there s h a l l be a c o n t r a s t e f f e c t and the d i r e c t i o n i n which i t s h a l l operate. B I B L I O G R A P H Y BIBLIOGRAPHY Ashburn, Robert R. \"An Experiment i n the Essay Type Question,\" J o u r n a l of Experimental Education, v o l . 7 (September, 1938)* pp. 1-3. P a l l s , J.B. \"Research i n Secondary E d u c a t i o n , \" Kentucky  School J o u r n a l , v o l . 6 (March, 1928), pp. 42-46. G i l l i l a n d , A.R. The Classroom Teacher. New York: The Century Company, 1924. Hunt, W.A. \"Anchoring E f f e c t s i n Judgment,\" American J o u r n a l  of Psychology, v o l . 54, 1941, pp. 395-403. L i n d q u i s t (ed.) E d u c a t i o n a l Measurement. American C o u n c i l on Education, Washington, 1956. Ross, C C . and Stanley, J.C. Measurement i n Today's Schools. New York: P r e n t i c e - H a l l Inc., 1954. S t a l n a k e r , John M. \"The Problem of the E n g l i s h Examination,\" E d u c a t i o n a l Record (September, 1937)* PP. 35-48. S t a r c h and E l l i o t t . \" R e l i a b i l i t y of Grading Work i n Mathematics,\" School Review, v o l . 21 ( A p r i l , 1913)* pp. 254-259. T e i g s , E.W. E d u c a t i o n a l D i a g n o s i s . E d u c a t i o n a l B u l l e t i n Number 18, C a l i f o r n i a Test Bureau, 1952. Thorndyke, R.L. and Hagen, E. Measurement and E v a l u a t i o n  i n Psychology and E d u c a t i o n . New York: J . Wiley and Sons, 1955. Truman, S . v and Weber, E. \"Judgment of P i t c h as a F u n c t i o n of the S e r i e s , \" U n i v e r s i t y of C a l i f o r n i a P u b l i c a t i o n s i n  Psychology, 1 9 2 8 , pp. 215-223. Woodrow, H. \"Behavior w i t h Respect to Short Stimulus Forms,\" J o u r n a l of Experimental Psychology, v o l . 11, 1928, pp. 167-193. Worcester, D.A. \"The V a l i d i t y of T e s t i n g , \" School Review (September, 1934). ' Wrightstone, J.W., Justman, J . , and Robbins, I. E v a l u a t i o n  i n Modern E d u c a t i o n . New York: American Book Company, 195^: A P P E N D I C E S APPENDIX A TABLES 41 TABLE I THE AYRES CATEGORIES TO WHICH 215 HANDWRITING SAMPLES WERE ASSIGNED BY TWELVE JUDGES (The numbers i n the columns.refer t o the twelve judges. Judges 10, 11, and 12 are l a b e l l e d a, b, and c to reduce the p o s s i b i l i t y of confusion.) Sample Number Ayres Category 2 3 4 5 6 7 8 9 1 b 3,7, a 1245689 c 2 c 489a 12567b 3 3 2 139c 4578ab 6 4 29 13c 4567 8a b 5 23b 1467a 589c 6 38 269 147c b 5a 7 3 48 67 59ac 12 b 8 4b 35 1279 68ac 9 b 245789 0 n 16 3 10 b CL U 4 13589c 267a 11 5 1239a 48c 67b 12 5b 1 347c 2698a 13 4a 5c 128b 369 7 14 2 13469 78 5 b 15 4 & U 12569 378 b 16 3a a, C 126c 4 9 . 57 8 b 17 b 23 l49ac 8 56 7 18 3b ac 129 4567 8 19 34a 19 26b 57 8c 20 1 348c 26 57 9a b 21 3 12469a 78c 5b 22 4 1 2359 7a 468bc 23 68c 29 1357 a b 24 78 15a 24c 369 b 25 6 4 l c 235789a b 26 16b 2348c 579a 27 1234578a 69 u c 28 b 45a 368c 19 27 29 4 67 239bc 158a 30 3a l 4 b c 78 2569 31 349 25ac 178 6b 32 a l46bc 5 378 29 33 4 378 1569ac 2b 34 l b 54 2378c 69a TABLE I (continued) 42 Sample Ayres Category 3 35 36 37 38 39 40 4 l 3 42 43 44 45 46 7 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 38c 70 71 72 73 34a 48c 3c 1234 34 4 5 6 7 4 239 57a 4ab 3489c 12567a 236789 r> 15 12c 8b 79 56 5 b b 1 67 157a 26b 9 6b 1239ac 458 7 6 b 245a 6b48 8 1459 37a 1579 2a 5a 6 4 b 46b 358 4 56b b 56 6b 79 6 9a 48 b 15a 23c 1289c 7a I678c4 4 4 168c b 6b 23489c 167a 2458a 39c 2c 6b b 24 369 1459< 5b 35 5689a 359 12678ac 179c 2a 1478c 239a 1478a 239c 15a 2348c 468 123579ac 135689 7ac 7 67 2b 1237c 12589ac 37 4567 12389 2b 8b 6b b 1489 43 568 2 37b 56ac 34 12689 57 b ac 23 l c 679a 45 8b 48ac b 45 123679 5 b 347c 12a 56 8 9b b 124689 37 5 ac 27a 149 56 b 3 b 24 l b 7b c 134a 269 58 b 24 l689ac 357 ac 1279ac b 289 1367ac 157c 689a 23456 79a TABLE I (continued) Sample Number Ayres Category 2 3 4 5 6 7 8 9 74 34 8 129a 67c 5b .75 3 46 12578ac 9 b 76 b 4 23689a 157c 77 3 6c 457b 128a 9 78 b 126 378ac 459 79 4 c 2378a 1569b 80 ' 389c 2457ab 16 81 4b 6ac 23579 18 6 82 3b 1269c 45a 78 83 4 29 7a 1358b 6c 84 3b 1679 2458ac 85 6 348 17ac 29 5 b 86 3 b 2456a 178c 9 87 4b 6a 3579c 128 88 3b 7c 1245689a 89 46 1389 27 5 abc 90 3b 245789 16c 91 3 d 6c 145789a 2b 92 b 2a 13457c 689 93 4 b 136 57a 289c 94 4 b 3 159ac 2678 95 c 23479< a 168 5 b 96 2 134567 b 89ac 97 be 2349 16a 5 7 8 98 3ac 4689 127 5b 99 3a 249 15b 7c 68 100 4a 12c 359 7b 68 101 4 6 27 1358abc 9 102 b8 3479 156 2 103: ct U 1 24678c 359a b 104 3 9 12458abc 7 6 105 3a 468c 129 5 7 b 106 9b 134ac 2568 7 107 49 6a 378c 2 15b 108 4 136ac 258b 7 9 109 b 239 58ac 1467 110 34 24 59ac 137 68b 111 89c 12567a b 112 b89 37a 1246a 5 113 4 3 2bc 157 69 8a 44 TABLE I (Continued) Sample Number Ayres Category 2 3 4 5 6 7 8 9 114 289c 36b 147 5a 115 49 236ac 178 5 b 116 3 b 27 4689a 15c 117 3478c 126a 9 5b 118 3 48 1269ac 7 5b 119 4 12569 37ab 8c 120 8 1369a 4c 257 b 121 3b 12 479a c 568 122 b 349ac 127 568 123 289b 1345 67ac 124 46 7 ab 18c 39 25 125 48 239ac 17b 5 6 126 7c 3469 28a 15 b 127 4b 3c 1259 68a 7 128 349 l a 278c 56 b 129 249 1358c 7a 6b 130 3 24b 69c 158a 7 131 b 4 1 23589c 67a 132 b 2 46 1356789 133 4 b l c 23578 ac 69a 134 34 b 26 7 134589ac 135 29 678ac 1 5b 136 8b 79 145a 6c 23 137 4 12678a 59c 3b 138 5b 1348c 2679a 139 23589a 147c 6b 140 6 4b 12357 8a 141 9c 3a I89 457 2 6c b 142 b 6 4 12358 79ac 143 b 13456 a 2 144 b8 789c 2347 1569 145 23479ac d. C 1568 b 147 148 46 2589ac 137 123467 8 9a c 123467 58a 9c 5 b b b TABLE I (continued) Sample Number Ayres Category 149 b 150 151 152 379a 153 154 155 156 157 2 158 b 159 160 161 1 162 163 164 165 166 167 168 2358 169 170 171 172 173 174 175 176 177 178 179 180 181 8 182 183 184 185 8 186 3 4 789ac 1234 3 12689 349 67abc 12468 5 c b 4c 24c 23 b 138a 139a 56 45a 128 l478ac 37 39 569 578 9 24578 ac 6 7c 5 b 4 569 459c 1268c 7 6 248 139 7 8 b 35a 18 7 b 3ab 126789c b 2a 4a 157 6b 248b 379 56ac b 45 38c 79a 28 134579 6 b ac 1267 39 48 5ac b 8 57 1249ac 3 12b 489 367a 5c 258 134679ac b l479ac 6 b 45b b 234 78a 69c 23789 n 16a 46bc 1279 34568ac b 2a 17 359 8 2b 89 1467c 3a 5 48 5b 2349c 23c 57 l69ab 1 5ac 24389 7 6b 1389 27c 45a 6b 3 2b l457ac 689 49 1238ac 7 5b 3 7 2b l 6 a c 459 179a 2356c 4b l b 25 347ac 689 148 4 2b 15789ac 3 267ac 39 5b 1379a 246c 5 b 4 1257c 689a 3b 6 5b 612 6b 15 1678a 6 TABLE I (continued) 46 Sample Number Ayres Category 187 188 8b 189 190 191 192 193 194 23489a 195 7c 196 197 198 199 8 200 201 202 203 49 204 205 8 206 207 208 1239 209 210 211 3489 212 213 214 67 3b 3b 18 l c b 167c 12489 a 3b 9 23479 ac 37b 4b 12568 c 13479 ac b 468 48 a 167a 279c 279c 9 4 5 6 2a 9 14678 3 12349 5 ac 126 4ac 58 79 14689 257 ac 237c 4569ab 249 3568a 7 b 1239 468ac 57 5b 35 6 ' b 234 1789a 5c 6b 249 578a 16c 245 13678b ac 1 56 b 12456 89ac 369 2ac 157 8 34ab 29c 78 16 7a 3b 24a 138c 79 5 25 b 6 349 268ac 17 5b 1 23489a 567c 57a c b 1379ac 26 b 5 2 l c 3458 7 25c b 13468a 5 b 13468a 5 b 134a 2567c 8b 7 5bc 5 6b 69b TABLE I I THE CORRELATION OF THE EVALUATIONS ASSIGNED 215 HANDWRITING SAMPLES BY TWELVE JUDGES WITH THE AVERAGE EVALUATIONS OF THE SAME GROUP OF JUDGES Judge D e c i s i o n 1 .884 accept. 2 .847 accept. 3 .829 accept. 4 .796 accept. 5 .763 accept. 6. .748 r e j e c t . 7 .858 accept. 8 .857 accept. 9 .824 accept. a .836 accept. b .448 r e j e c t . c .895 accept. TABLE I I I THE DISTRIBUTION OP EVALUATIONS OF 215 HANDWRITING SAMPLES BY TEN STABLE JUDGES Sample Number Ayres Category 2 3 4 5 6 7 8 9 1 xxx xxxxxx X xxxx X 3 X xxxx X X X X X 4 xx xxx xxx X X 5 X X xxxx xxxx 6 X X X X xxxx X X 7 X X X X xxxx X X 8 X X X xxxx xxx 9 xxxxxx X X X X 10 X xxxxxx xxx 11 X xxxxx xxx X 12 X X xxxx xxxx 13 X X X X xxx X X X 14 X xxxxx X X X 15 x xxxxx xxx 16 xx xxx X X X X X 17 X X xxxxx X X X 18 X X X xxx xxx X 19 xxx X X X X X X X 20 x xxxx X X X X X 21 X xxxxx xxx X 22 X xxxx X X xxx 23 X X X X X xxxx X 24 X X xxx xxx X X 25 X X X xxxxxx 26 X xxxxx xxxx 27 X X X X X X X X X X 28 xxx xxx X X X X 29 X X xxxx xxxx 30 X X xxx X X xxx 31 xxx xxxx xxx 32 X xxx X xxx X X 33 x xxx xxxxx X 34 X X X xxxxx X X 35 X xxx xxx xxx 36 X X X X X xxxxx 37 X X xxxxxx X X 38 xxx xxx X X X X 49 TABLE I I I (continued) Sample Number Ayres Category 2 3 4 5 6 7 8 9 39 X xxxxxx xxx 40 X X xxxxx xxx 4 l x xxx X X X X X X 42 xxxxxx xxx X 43 xxxx xxxxx X 44 X xxxx xxx X X 45 xx X X xxxx X X 46 x xxxxx X X X X 47 X xxx xxxxxx 48 X xxx xxxx X X 49 X xxxxx xxxx 50 X X xxxxx xxxx 51 X X xxx xxxxx 52 X X xxxxxxxx 53 X X xxxxx xxx 54 xxx xxx X X X X 55 xxx xxxxx X X 56 X X xxxxx X X X 57 X X xxxxx X X X 58 X xxxx xxxxx 59 X xxxxxx X X 60 xxx xxxxxxx 61 X X X X xxxxxx 62 X X X X X X X xxx 63 xx X X X X X X X X 64 xx X X xxx X X X 65 X X xxx xxxx 66 xxxx X X X X X X 67 xxxx X X X X X X 68 xxxxx X X X X X 69 xxx xxx X X X X 70 X X X xxxx xxx 71 X xxxxxx xxx 72 X X X X X X X X X X 73 X X xxxxx xxx 74 X X X xxxx X X X 75 X X xxxxx X 76 X X X xxxxx xxxx 77 X X xxx xxxx X 78 X X xxxxx xxx 50 TABLE I I I (continued) Sample Number Ayres Categ ;ovy 2 3 4 5 6 7 8 9 79 X X x x x x x x x x 80 X X X X x x x x x X 81 X X X x x x x x X X X 82 X x x x x x x x X X 83 X X X X X x x x x X 84 X x x x x x x x x x 85 x x x x x x x X X X 86 X x x x x x x x x X 87 X X x x x x x x x x 88 X X X x x x x x x x 89 X x x x x x x X X X 90 X x x x x x X X X X 91 X X x x x x x X X X 92 X X x x x x x x X X 93 X X X x x x x x x x 94 X X x x x x x x x x 95 x x x x x x x X X X 96 X x x x x x x x x x 97 X X X X X X X X X X 98 x x x x x x x x x X 99 X X x x x X X X X 100 X X x x x x x x X X 101 X X X x x x x x x X . 102 x x x x x x x X X X 103 X X x x x x x x x x x 104 X X x x x x x X X X 105 X X X X X x x x X X 106 X x x x x x x x x X X 107 X X X X X X X X X X 108 X x x x x x x x X X 109 X x x x x x x X X X 110 x x x x x x x x x x i l l X X x x x x x x x X 112 X X x x x x x x x x 113 x x x x x x x x x X 114 X X X X x x x X X X 115 x x x x X x x x X X 116 X X x x x X 117 X X X X X X X x x x TABLE .III (continued) 51 Sample Number Ayres Category 2 3 4 5 6 7 8 118 x x x x x x x x X X 119 X X X x x x x x X X 120 X x x x x x x x 121 X x x x x X X x x x 122 X X X x x x x x X X 123 x x x x x x x x X X 124 x x x X X X X X X X 125 X X X x x x X X X X 126 X X x x x x x X X X 127 X X x x x x x x X X 128 X X X x x x x X X X 129 x x x X X x x x x X 130 x x x x x x x x X X 131 X X X X X x x x x X 132 X X x x x x x x 133 X X 134 X X X x x x x x 135 X X 136 X X X X x x x x X X 137 X X X x x x x X X X 138 X x x x x x x x x X 139 X x x x x x 140 x x x x x x x x x x 141 X x x x x x 142 X X X X x x x x x x X X 143 X x x x x x 144 x x x x x X X 145 x x x X X X X X X X x x x 146 x x x x x x x x 147 X X X x x x x x x x x x 148 x x x x x X X x x x 149 x x x x x x x x X X 150 x x x x x x x x x X 151 X x x x x x x x X X 152 x x x x x x x x x X 153 x x x x x x x x x X 154 X x x x x x x x x x 155 X X x x x x x x X X 156 X X x x x x X X X X 157 X X x x x x X X X X X x x x x x x x x X X x x x x x x x x X 52 TABLE I I I (continued) Sample Number Ayres Cate gory 2 3 4 5 6 7 8 9 158 X X X xxxx X X X 159 xxx xxxx xxx 160 X xxx X X X X X X 161 xxxxx xxx X X 162 X xxx xxx xxx 163 X X xxx X X X X X 164 X X xxxxxxxx 165 xxx X X X X xxx 166 X X X xxxxxx X 167 X X xxx xxx xx 168 xxx xxxxxxx 169 xxxx xxxxxx 170 xxx xxx X X X X 171 xxxxxx X X X X 172 xxxx xxxxxx 173 X X X X X X xxx X 174 X X X xxxx X X X 175 X xxxxx xxxx 176 xxx X X X X xxx 177 X xxx xxxxx X 178 xxxx xxx xxx 179 X X xxxxxx X X 180 X X xxxxxx X X 181 X X X xxx X X X X 182 X xxxx xxxx X 183 X X X X X xxxxx X X 184 X X xxxxx X 185 X X xxx xxxx X X X 186 xxxxx xxx X 187 X xxxxx xxx X 188 xxx X xxxx X X X 189 X X xxxxx X X X 190 X X X xxx X X X X 191 X xxxxxx xxx 192 X X xxxx xxxx 193 X X xxx xxxx X 194 xxxx xxxx X X 195 xxxxxx xxx X 196 X X xxxxxx X X 197 xxx xxxxx X X 53 TABLE I I I (continued) Sample Number Ayres Category 4 7 8 198 199 200 x 201 202 203 204 x x 205 206 x 207 208 209 x x x x 210 211 212 x x x x 213 214 x 215 x x x x x x x x x x x x x x x x x x x x x x XX XX XX X x x x x x x x x x x x x x x x X x x x x x x X x x x x x x x x X XX x x x x x x x x XX x x x XX x x x XX x x x X x x x x x x x x x X x x x x x x x x X x x x x x x x x x x X x x x x x x x XX XX XXXX XX x x x x x x x x x X X XX x x x x X XX X X X X X X x x x x X TABLE IV A LIST OP GOOD, AVERAGE AND POOR SAMPLES GOOD 8, 9, 10, 11, 12, 22, 34, 39, 40, 43, 47, 48, 49, 50, 51, 52, 53, 58, 59, 6o, 6 l , 65, 70, 71, 79, 87, 92, 93, 94, 120, 132, 133, 134, 135, 139, 141, 143, 154, 163, 175. AVERAGE 3, 5, 14, 15, 21, 26, 36, 37, 42, 68, 73, 75, 76, 80, 84, 88, 89, 90, 91, 96, 103, 104, 109, 112, 138, 140, 144, 147, 148, 149, 155, 161, 164, 168, 177, 183, I 8 7 , 191, 208. POOR 2, 27, 46, 63, 66, 69, 95, 102, 113, 118, 124, 145, 146, 150, 153, 169, 171, 182, 186, 189, 195, 196, 200, 201, 204, 206, 209, 212, 213, 2 l 4 . 55 TABLE V EXPERIMENT A THE EXPERIMENTAL SERIES - RUN ONE Sample Sample Sample Sample Number Kind Number Kind Number Kind Number Kind 144 A 200 P 23 C 94 G 90 A 88 P 55 C 9 G 129 C 210 C 133 G 154 G 25 C 119 C 52 G 12 G 136 C 86 G 79 G 89 A 6 C 139 G 135 G 54 C 47 G 120 G 26 A 72 C 132 G 61 G 181 C 95 P 53 G 149 A 125 C 150 P 71 G 38 C 2 P 66 P 112 A 111 C 145 P 63 P 188 C 213 P 46 P 148 A 29 C 146 P 206 P 28 C 118 P 124 P 191 A 197 C 69 P 195 P 83 C 39 G 189 P 80 A 64 . C 68 A 113 P 122 C 40 G 204 P 37 A 56 C 36 A 109 A 35 C 65 G 212 P 97 C 107 C 208 A 103 A 127 C 43 G 27 P 180 C 11 G 96 A 75 A 126 C 51 G 186 P 101 G 93 G 141 G 187 A 117 C 10 G 143 G 85 C 8 G 87 G 14 A 205 C 58 G 70 G 77 C 49 G 34 G 42 A 20 C 50 G 59 G 78 C 124 P 22 G 155 A 142 C 195 P 48 G 31 C 189 p 213 P 15 A 7 C 113 p 146 P 18 C 201 P 118 p 21 A 17 C 33 P 69 p 114 C 66 P 196 P 91 A 152 C 63 P 214 P 104 C 60 G 95 P 5 A 115 C 147 A 150 P 57 C 134 G 200 P 183 A 16 C 84 A 88 A 184 C 92 G 182 P 137 C 207 C 73 A 76 A 44 C 60 G 102 P 98 C 202 C 147 A 3 A 105 C 116 C 56 TABLE VI EXPERIMENT B THE EXPERIMENTAL SERIES Short s e r i e s Short s e r i e s Short s e r i e s The experimental number one number two number three s e r i e s \u2014 run ' one Sample Sample Sample Sample Number Kind Number Kind Number Kind Number Kind 147 A 127 A 80 A 137 C 68 A 26 A 149 A 44 C 21 A 91 A 75 A 202 C 88 A 36 A 208 A 116 C 14 A 103 A 5 A 114 C 60 G 39 G 134 G S S l 109 A 42 A 112 A 77 C 200 P 204 P 182 P 20 C 124 P 150 P 113 P 152 C 143 G 12 G 70 G 28 C 89 A 191 A 15 A 97 C 148 SS2 A 73 A 183 A 90 A 141 G 154 G 87 G 21 A 51 G 9 G 10 G 88 A 195 P 95 P 189 P 14 A 84 A 3 A 96 A 64 C 146 P 63 P 69 P 78 C 213 P 66 P 118 P 142 C 76 A 155 A 37 A 104 C 11 G 94 G 40 G 115 C s s 3 83 C 126 C 180 C 144 A 91 A 36 A 103 A 57 TABLE V I I THE EVALUATIONS IN EXPERIMENT A Run 1 Sample Judge Number 1 2 3 4 5 6 7 8 9 10 1 11 14 15 10 15 17 15 10 12 10 2 8 19 15 10 13 15 5 8 14', 11 3 15 14 16. 10 16 17 20 13 14 13 4 14 20 18 10 17 20 10 21 11 13 5 10 11 17 9 14 16 10 10 9 10 6 12 10 19 12 13 16 20 8 10 12 7 18 19 22 15 19 17 22 18 20 16 8 17 21 21 14 18 21 10 20 18 13 9 12 14 23 13 15 17 12 16 19 15 10 18 13 23 18 18 18 10 22 18 15 11 14 12 22 15 14 16 8 20 16 11 12 15 9 19 14 13 16 10 15 14 12 13 17 10 19 13 15 17 10 15 15 13 14 7 7 12 6 6 13 5 7 5 8 15 . 8 7 13 8 10 13 5 7 7 7 16 9 9 13 9 13 13 15 10 7 8 17 10 5 14 4 10 12 3 9 8 7 18 12 12 16 13 14 15 15 12 11 11 19 19 13 19 16 18 15 15 23 21 17 20 13 10 22 15 17 15 15 22 19 16 21 14 12 22 14 19 16 20 19 18 19 22 9 10 18 11 10 15 8 16 16 10 23 6 9 12 8 8 14 20 4 16 9 24 11 10 14 11 12 14 10 8 17 10 25 17 8 17 11 10 16 10 10 18 13 26 10 9 14 10 13 15 8 10 16 10 27 14 14 21 12 17 19 15 18 19 15 28 15 18 23 13 16 18 17 24 19 19 29 16 12 20 13 15 18 20 18 18 11 30 13 14 20 14 17 17 10 22 18 15 31 12 10 19 12 16 16 15 13 17 10 32 18 12 18 12 12 15 15 14 17 11 33 11 11 17 10\" 14 15 15 10 14 9 34 9 8 14 8 10 14 10 12 15 9 35 12 9 13 7 8 14 15 12 10 8 36 8 8 13 8 10 14 8 13 11 8 37 9 10 14 10 13 15 5 15 8 6 38 9 15 15 12 15 15 8 18 10 11 39 8 10 14 10 13 15 10 13 9 11 40 12 17 18 16 18 17 8 20 15 11 5 8 TABLE VII (continued) Sample Judge Number 1 2 3 4 5 6 7 8 9 10 4 1 19 16 1 7 10 14 18 15 1 5 15 1 3 4 2 13 18 1 7 9 14 1 7 1 5 8 16 11 43 7 6 11 7 12 1 4 6 8 12 8 44 1 4 1 4 13 9 14 1 4 17 1 4 13 11 45 9 12 14 8 11 1 5 2 0 12 10 10 46 12 10 13 9 11 15 22 18 11 10 47 1 4 12 19 1 3 1 4 1 7 2 0 1 7 1 8 12 48 1 5 2 0 2 0 16 2 0 1 9 8 2 3 22 14 49 1 4 18 20 15 18 1 7 22 2 4 19 1 2 5 0 16 1 7 20 13 16 1 7 2 5 22 19 11 51 11 1 5 1 5 1 3 12 16 10 12 1 3 8 5 2 10 10 1 4 11 11 15 12 12 13 9 53 13 13 1 7 1 5 1 9 18 5 18 16 10 54 9 11 11 8 10 15 6 \u2022 12 13 8 55 7 7 10 7 5 13 4 8 11 7 8 5 6 8 6 11 6 7 12 3 5 10 57 8 7 11 5 6 12 3 5 9 7 58 13 16 16 9 1 5 18 12 1 7 19 12 59 11 12 14 12 12 16 1 7 16 13 11 6 0 12 11 1 5 10 12 1 6 15 18 15 12 6 1 1 7 2 0 1 9 1 5 16 1 8 22 2 2 0 17 6 2 7 1 4 14 1 3 10 18 5 2 0 12 10 63 5 5 1 0 4 6 12 2 \u2022 2 5 6 64 12 10 1 4 10 12 13 10 2 0 12 8 6 5 11 11 16 13 13 1 5 12 1 7 14 10 66 11 12 14 12 12 ' 16 8 13 16 10 67 17 14 18 12 14 1 9 2 0 19 15 11 68 1 5 1 7 1 9 12 13 2 0 2 0 19 1 7 11 6 9 14 17 2 0 13 18 21 2 5 1 5 1 9 10 7 0 13 1 6 18 12 15 1 9 2 5 1 7 1 7 12 7 1 11 10 14 10 10 16 10 11 13 8 7 2 12 1 5 2 3 12 15 1 9 10 1 8 16 9 73 13 1 3 21 10 12 1 9 18 1 6 16 10 74 7 9 1 5 8 10 1 6 10 9 12 8 75 8 8 14 10 10 1 5 10 9 11 6 7 6 7 6 1 2 7 7 14 5 5 7 5 77 6 4 11 8 6 12 5 3 6 4 7 8 10 13 1 5 12 13 ' 15 8 6 14 10 79 12 11 2 0 11 13 15 12 13 14 11 8 0 1 5 12 22 12 13 16 2 0 18 15 10 81 16 13 22 10 15 16 10 1 7 1 6 9 59 TABLE VII (continued) Sample Number 1 2 3 Judge 4 5 6 7 8 9 10 82 8 6 14 7 7 12 12 7 14 7 83 7 6 12 6 4 12 8 6 10 5 84 8 13 15 11 10 16 15 13 19 6 85 9 14 15 12 13 16 17 17 16 10 86 8 20 16 14 15 18 8 16 16 10 87 14 21 21 13 17 19 20 23 16 11 88 16 21 22 14 15 20 20 22 16 11 89 13 17 21 11 12 19 15 19 17 10 90 13 18 22 13 14 20 15 22 16 11 91 9 9 16 10 13 16 10 12 16 7 92 11 13 18 12 17 16 8 13 16 9 93 14 11 19 10 14 16 12 20 17 8 94 7 10 15 8 12 16 15 15 15 7 95 8 8 14 8 10 15 8 12 15 6 96 7 6 13 7 10 14 2 8 15 5 97 6 9 12 7 10 14 12 8 16 6 98 9 11 15 10 13 14 8 9 14 10 99 12 19 16 14 14 16 17 15 18 11 100 11 16 17 13 15 16 7 14 16 10 101 14 13 18 12 13 17 15 19 17 12 102 11 13 18 13 14 17 20 13 17 10 103 8 9 12 6 10 13 4 7 8 5 104 10 10 13 6 12 15 6 6 9 5 105 10 14 14 10 14 18 6 13 12 6 106 8 11 15 10 11 16 20 12 12 7 107 13 19 20 14 14 17 12 21 16 6 108 14 18 19 12 17 17 15 24 17 12 109 16 20 20 12 16 18 20 23 18 15 n o 13 20 21 14 15 18 10 19 21 19 i l l 11 14 18 11 13 17 20 11 16 12 112 12 12 16 9 10 15 25 9 13 8 113 10 10 14 6 10 15 15 14 15 10 114 7 7 12 4 8 13 5 7 5 7 115 8 9 13 10 10 14 4 6 8 6 116 9 10 14 10 12 15 8 12 11 6 117 8 7 14 5 10 13 7 8 13 5 118 11 11 16 8 11 15 15 13 13 10 119 13 13 18 9 14 16 10 20 16 11 120 10 12 14 10 12 15 4 11 14 7 8 121 12 14 21 12 15 16 17 20 15 122 9 12 15 9 13 17 12 15 14 7 60 TABLE VII (continued) Sample Judge Number 1 2 3 4 5 6 7 8 9 10 123 8 10 13 10 10 15 8 11 15 5 124 10 8 15 9 10 16 12 9 17 7 125 10 12 14 11 12 16 10 11 16 8 126 9 12 15 10 12 16 17 10 14 6 127 12 20 21 12 15 19 4 21 17 10 128 12 16 20 13 14 20 8 17 '15 8 129 13 17 21 12 15 19 10 23 17 9 130 15 19 22 11 17 21 4 15 16 12 131 9 11 15 7 11 15 20 7 18 7 132 11 13 17 9 13 17 15 11 18 8 133 10 12 14 11 13 16 6 10 13 6 134 9 9 13 7 10 15 20 8 13 6 135 10 9 13 6 9 13 12 8 10 5 136 9 10 13 8 7 13 12 9 12 6 137 11 11 14 9 13 16 6 12 11 4 138 11 13 18 10 14 16 17 21 14 7 139 12 14 18 11 13 15 14 17 16 6 140 15 15 18 11 17 -15 9 14 12 6 141 16 19 20 14 16 20 10 24 18 7 142 12 13 17 10 9 17 7 12 12 7 143 9 5 14 4 7 12 7 8 10 5 144 11 14 14 9 12 15 17 10 12 6 145 10 14 15 10 12 16 5 11 10 6 146 11 12 17 12 12 16 6 14 11 7 8 147 14 16 18 '12 14 19 10 17 15 148 15 19 18 12 15 21 12 20 18 12 149 14 14 16 10 14 18 17 19 18 13 150 15 19 19 11 15 19 15 23 18 15 151 9 12 17 10 13 15 20 11 13 10 152 12 12 16 8 13 15 10 19 17 9 153 8 12 16 8 10 15 12 15 15 9 154 8 9 14 6 11 14 10 11 15 7 155 7 5 12 5 8 13 5 7 12 5 156 7 6 12 7 8 12 6 7 6 5 157 6 5 11 5 9 12 5 5 4 4 158 11 10 14 11 11 16 15 14 15 12 159 10 9 14. 8 10 15 12 10 16 10 160 10 11 13 10 13 16 8 17 17 11 161 14 13 19 11 14 19 15 19 18 15 162 12 15 17 10 13 16 12 13 8 18 9 163 6 9 12 10 10 14 8 13 7 164 13 12 15 10 13 15 10 15 15 10 I65 12 19 17 10 15 17 5 24 17 8 6 1 TABLE VII (continued) Sample Judge Number 1 2 3 4 5 6 7 8 9 1 0 1 6 6 1 2 1 6 1 8 8 1 4 1 8 9 2 0 1 9 8 1 6 7 6 7 1 3 6 1 0 1 2 1 1 5 1 0 6 1 6 8 8 9 1 3 8 8 1 6 5 9 9 7 6 2 TABLE VII (continued) Run 2 Sample Number 1 2 3 4 5 6 7 8 9 1 0 1 9 1 2 1 6 1 8 1 5 1 7 1 5 1 3 1 0 9 2 8 1 3 1 3 8 1 2 1 6 1 2 9 1 2 9 3 1 0 1 2 1 4 1 0 1 3 1 5 1 5 1 8 1 1 1 0 4 2 0 1 6 1 7 8 18. 1 6 1 0 2 1 1 1 1 0 5 9 1 1 1 6 1 0 1 5 1 5 1 2 1 6 1 0 8 6 1 2 1 3 1 7 8 1 3 1 5 1 7 1 0 1 0 9 7 1 8 1 8 2 1 1 2 1 7 1 7 2 2 2 0 1 5 1 2 8 1 5 1 9 1 7 1 0 1 8 18 1 0 2 4 1 2 13 9 1 7 1 9 1 8 14 1 4 1 9 8 1 9 14 1 5 1 0 1 4 1 7 1 9 1 4 1 7 1 6 1 2 2 4 1 2 1 4 1 1 1 0 1 5 1 6 1 2 14 16 1 0 1 6 1 0 1 3 1 2 1 2 1 4 1 6 14 1 6 1 6 1 0 1 7 1 1 1 0 1 3 1 9 1 3 1 9 1 5 18 1 8 1 2 2 0 1 1 1 1 1 4 8 7 1 1 8 5 1 3 5 5 4 7 1 5 9 9 1 2 1 0 7 1 4 5 9 6 6 16 9 8 1 3 8 1 2 1 3 8 1 1 8 6 1 7 8 6 1 2 5 5 1 2 5 7 6 5 1 8 1 0 9 1 7 1 0 1 7 1 6 1 2 2 0 1 0 6 1 9 1 7 1 5 1 8 1 7 1 7 1 7 1 5 2 0 1 5 1 7 2 0 14 1 2 2 0 14 1 9 1 7 1 0 2 2 1 1 1 2 2 1 1 1 1 4 1 9 1 8 1 9 1 8 2 0 1 8 1 1 1 1 2 2 9 1 0 1 3 1 2 9 14 1 0 1 2 9 8 2 3 8 9 1 1 1 0 1 1 1 3 1 0 7 1 0 9 24 1 0 9 1 3 1 0 1 3 14 1 2 1 0 n 6 2 5 1 5 1 1 1 5 14 18 1 5 1 0 n 1 3 1 0 2 6 9 1 0 1 3 1 0 1 0 1 4 1 7 1 5 1 0 7 2 7 1 5 1 3 2 0 1 5 2 2 1 9 2 0 2 0 14 1 1 2 8 1 7 14 2 1 1 7 2 0 18 2 0 24 1 8 1 7 2 9 1 2 1 3 1 7 1 3 1 5 18 . 2 2 1 5 1 7 1 2 3 0 1 4 1 3 1 7 1 5 2 0 1 9 1 2 1 9 1 7 1 3 3 1 1 2 14 14 1 6 1 7 1 7 2 2 1 7 1 1 1 1 3 2 1 4 1 2 1 5 1 2 1 5 1 6 2 5 1 8 1 2 1 0 3 3 1 0 1 2 1 5 1 8 1 2 1 5 1 0 1 5 9 6 3 4 9 8 1 4 8 1 1 1 5 5 1 1 1 0 6 3 5 8 7 1 4 8 1 0 14 1 0 8 8 5 3 6 8 8 1 1 1 0 1 0 1 5 1 7 7 9 5 3 7 9 1 0 1 3 1 2 1 2 1 5 5 1 2 9 4 3 8 1 6 1 2 1 7 1 5 1 5 1 6 1 2 1 7 1 1 6 3 9 9 9 1 5 1 4 1 3 1 5 1 7 1 1 1 0 6 4 0 1 5 1 5 1 7 1 6 18 1 6 1 5 1 9 1 4 1 0 4 1 2 0 1 3 1 8 1 6 14 1 8 2 5 2 0 1 5 1 2 6 3 TABLE VII (continued) Sample Number 1 2 3 Jud\u00a3 4 5 6 7 8 9 1 0 4 2 1 2 1 2 1 6 1 5 1 6 1 5 2 2 1 5 1 1 1 2 4 3 9 8 1 1 1 2 9 1 3 5 8 7 6 4 4 9 1 8 14 1 3 14 1 7 1 2 2 0 1 2 8 4 5 1 0 1 1 1 3 1 2 1 0 1 6 1 5 1 7 1 1 1 2 46 1 9 9 1 3 1 5 1 3 1 5 1 2 1 9 1 0 1 1 4 7 2 0 1 6 1 9 1 6 1 3 1 8 1 7 2 0 1 5 1 5 48 1 8 1 8 1 9 1 9 2 2 2 1 1 2 2 4 2 2 1 5 4 9 1 2 1 9 2 0 2 1 1 7 1 9 2 0 2 3 2 0 1 2 5 0 1 7 2 0 1 9 1 7 1 6 1 8 2 0 2 3 1 5 1 4 5 1 1 6 1 6 1 8 14 1 5 1 8 15 1 9 1 6 1 1 5 2 1 0 1 2 1 5 1 1 1 0 1 6 1 7 1 8 1 0 8 5 3 1 1 \u2022 1 7 1 7 1 3 1 7 1 9 1 0 1 6 1 7 7 5 4 8 9 1 3 1 0 1 1 1 5 5 1 2 1 3 5 5 5 7 7 1 1 5 7 14 4 9 9 5 5 6 8 6 1 1 5 8 1 3 8 7 5 4 5 7 6 5 1 0 4 5 1 2 5 5 4 5 5 8 1 3 \u2022\u20229 1 3 6 1 0 1 5 1 0 6 6 7 5 9 1 2 7 1 4 1 0 1 2 1 5 1 2 1 3 9 9 6 0 1 1 1 0 14 1 2 1 3 1 6 1 2 1 5 1 1 1 0 6 1 2 0 2 1 1 8 1 5 1 8 1 7 1 7 2 1 14 1 6 6 2 1 2 1 2 1 6 15 1 2 1 6 1 5 1 7 1 0 1 1 6 3 6 5 8 3 3 1 2 3 5 4 4 64 1 0 1 0 1 2 8 14 1 5 5 1 0 6 5 6 5 14 1 2 1 6 9 1 7 1 5 1 0 1 3 1 2 6 6 6 8 1 1 14 9 14 1 5 2 0 1 7 1 4 7 6 7 1 6 14 1 8 14 1 8 2 0 2 0 2 2 1 5 8 6 8 6 9 1 7 1 3 18 1 3 1 4 1 9 1 7 2 4 1 5 9 1 7 1 6 1 7 2 0 2 0 18 1 7 2 1 1 7 9 7 0 1 2 1 5 1 7 1 7 1 7 1 7 2 0 1 7 1 7 8 7 1 1 0 1 3 1 7 1 8 1 3 1 6 2 0 1 9 1 7 7 7 2 1 3 1 6 1 9 1 7 1 9 1 6 2 2 2 2 1 6 8 7 3 14 1 3 1 8 1 6 14 1 6 2 5 1 7 1 9 1 0 7 4 6 1 1 1 1 1 0 9 14 2 2 1 0 14 7 7 5 9 1 0 1 3 1 2 7 1 5 2 0 8 1 2 6 7 6 7 6 1 2 8 5 1 5 1 0 6 9 5 7 7 6 . 5 1 0 6 5 1 2 5 7 8 4 7 8 1 0 8 1 3 8 9 1 4 8 1 0 1 0 5 7 9 1 2 1 0 14 9 1 5 1 5 1 7 1 7 14 8 8 0 1 7 1 3 18 1 2 14 1 6 2 0 1 9 1 4 1 0 81 14 1 1 1 7 1 2 1 7 1 5 1 2 2 0 1 3 8 8 2 1 0 1 0 1 3 1 3 1 1 1 5 1 0 1 7 1 9 8 6 8 3 9 6 1 2 5 4 1 3 8 1 4 6 84 8 8 1 3 6 3 1 3 1 5 7 1 2 5 64 TABLE VII (continued) Sample Number 1 2 3 Judge 4 5 6 7 8 9 1 0 8 5 1 1 1 2 1 5 1 3 1 2 1 5 1 7 1 9 1 2 9 8 6 14 1 4 1 6 1 5 1 6 1 6 1 7 2 1 1 3 1 0 8 7 1 8 1 7 2 0 1 4 2 0 1 9 2 0 2 4 1 5 1 2 8 8 1 7 2 1 2 0 1 7 2 2 1 8 2 0 2 4 1 6 1 5 8 9 1 5 1 5 1 9 14 1 7 1 7 2 0 2 3 1 8 1 2 9 0 2 0 1 4 2 1 1 3 1 9 2 0 2 0 2 3 1 7 1 5 9 1 1 0 1 0 1 4 1 2 1 1 1 4 1 0 1 8 1 1 1 0 9 2 1 1 1 3 1 5 1 5 1 7 1 6 1 5 2 1 1 3 9 9 3 14 1 0 1 8 1 3 1 6 1 6 1 7 8 2 4 2 0 1 1 9 4 1 0 1 0 14 1 3 8 1 5 14 1 4 7 9 5 1 1 9 1 3 1 2 8 1 4 8 1 2 1 0 6 9 6 9 8 1 2 5 9 1 4 4 8 1 0 5 9 7 8 1 0 1 2 5 7 14 1 0 1 0 1 3 7 9 8 1 1 8 1 8 1 0 1 3 1 5 1 2 1 3 1 2 8 9 9 14 1 8 1 8 1 4 1 5 1 5 1 5 17 1 8 9 1 0 0 1 2 14 1 8 1 1 1 5 1 6 1 7 1 8 1 3 9 1 0 1 15 1 5 1 9 1 3 13 1 6 1 7 1 8 1 5 1 1 1 0 2 1 0 9 1 3 8 1 0 1 5 1 2 1 4 1 0 8 1 0 3 7 6 1 1 6 7 1 3 5 8 9 5 104 1 3 1 0 1 8 14 1 3 1 6 2 0 2 0 1 5 6 1 0 5 1 1 1 2 14 1 0 1 8 1 8 5 1 8 1 5 4 1 0 6 1 0 1 0 1 4 1 1 9 1 5 2 0 1 5 1 3 6 1 0 7 14 1 3 2 2 1 7 1 6 2 0 2 0 1 9 1 4 6 1 0 8 1 5 1 5 1 6 2 0 18 2 0 2 0 2 0 1 5 1 4 1 0 9 1 6 1 6 1 8 1 9 1 8 1 9 2 0 2 3 1 6 1 6 1 1 0 14 1 5 1 5 1 8 2 0 2 1 2 2 2 2 1 8 1 5 1 1 1 9 1 0 1 4 1 2 1 1 1 6 2 5 1 9 1 3 1 1 1 1 2 8 1 1 14 1 1 1 0 1 6 1 7 1 6 1 2 8 1 1 3 1 0 1 0 1 5 7 1 3 15 2 0 1 0 1 3 6 1 1 4 7 7 1 3 5 7 1 3 5 8 8 5 1 1 5 7 8 1 3 8 7 1 4 1 0 9 9 4 1 1 6 8 1 2 1 3 1 1 1 1 1 5 1 5 1 0 1 0 5 1 1 7 8 9 1 3 6 5 1 4 8 6 1 0 4 1 1 8 1 2 1 0 1 5 1 1 14 1 5 1 0 8 1 2 7 1 1 9 1 3 l i 1 5 1 2 1 3 1 6 1 7 1 0 1 2 9 1 2 0 9 9 1 3 9 1 0 1 5 1 5 1 1 1 2 7 1 2 1 1 3 1 4 2 1 1 2 1 7 1 7 1 7 . 1 7 1 5 7 1 2 2 9 1 2 1 5 7 1 1 1 5 2 0 1 5 1 6 8 1 2 3 6 9 1 2 1 1 13 14 8 1 1 1 1 6 124 9 9 1 5 1 0 1 0 1 6 1 0 1 5 1 4 7 1 2 5 1 0 1 0 14 1 2 1 3 1 5 1 5 1 6 1 2 7 1 2 6 9 1 0 1 4 9 14 15 1 7 1 7 13 6 6 5 TABLE VII (concluded) Sample Number 1 2 3 Judge 4 5 6 7 8 9 1 0 1 2 7 15 1 6 18 14 1 6 2 0 1 2 2 2 1 6 1 0 1 2 8 14 1 7 1 9 1 0 1 8 1 9 1 5 2 0 1 7 8 1 2 9 1 6 1 3 1 9 1 3 1 6 19 1 7 2 3 1 7 1 1 1 3 0 17 1 6 2 1 1 0 2 0 2 0 1 5 2 0 1 8 1 0 1 3 1 1 1 1 3 1 5 1 2 1 7 1 6 15 1 8 1 4 9 1 3 2 1 3 1 3 1 5 8 1 3 1 6 2 0 1 7 14 \u20229 1 3 3 1 0 1 1 1 4 9 1 1 1 5 1 0 1 7 1 2 ' 6 1 3 4 9 1 0 1 3 7 8 1 5 1 0 1 5 1 5 5 135 8 1 0 14 6 5 1 4 1 7 1 3 1 0 5 1 3 6 7 9 1 3 5 9 1 4 1 0 1 2 1 1 6 137 9 1 0 1 5 1 0 14 1 5 \u2022 ' 5 1 5 1 2 5 1 3 8 8 9 14 6 7 1 5 1 7 1 0 14 6 1 3 9 1 0 1 2 1 7 8 1 4 1 7 1 5 1 3 1 2 9 l 4 o 1 4 14 1 9 1 0 1 7 1 6 1 2 1 7 1 5 8 1 4 1 1 5 1 7 2 0 1 3 1 7 1 9 2 0 2 2 1 6 1 0 1 4 2 1 0 1 2 1 6 1 1 1 3 1 6 1 7 1 8 14 9 143 9 1 0 14 5 7 1 2 5 1 2 1 1 5 144 8 9 1 6 1 1 7 1 5 8 1 7 1 6 7 145 9 1 3 1 5 1 0 1 2 15 1 2 1 8 1 2 6 146 1 0 1 2 1 5 9 1 3 1 5 1 2 1 8 1 2 6 147 1 3 1 6 1 7 1 0 14 1 6 2 0 1 9 1 5 7 148 1 6 18 1 8 1 2 1 5 . 1 6 2 0 2 3 1 7 1 2 149 1 7 1 7 1 8 1 2 1 3 1 6 2 0 2 0 1 7 1 1 1 5 0 1 5 1 8 2 0 1 2 1 5 1 8 2 0 2 2 1 7 1 1 1 5 1 9 1 3 1 7 1 0 1 1 1 6 1 7 1 7 1 7 1 0 1 5 2 14 1 3 1 6 1 0 1 0 1 6 1 2 1 9 1 8 9 1 5 3 8 1 0 1 6 8 1 0 14 1 2 1 3 14 8 1 5 4 7 9 1 5 6 10 14 1 0 1 0 1 0 7 1 5 5 6 7 1 2 5 7 1 3 8 7 1 1 5 1 5 6 6 7 1 3 7 7 1 2 8 6 9 5 1 5 7 5 7 1 2 5 5 1 3 8 5 7 4 1 5 8 1 0 1 0 1 5 9 1 1 1 5 1 5 8 1 1 5 1 5 9 9 8 1 5 9 1 0 1 5 1 2 1 3 1 4 8 1 6 0 1 0 1 2 1 5 9 14 1 5 \u2022 1 2 1 8 1 5 1 0 1 6 1 18 1 3 1 8 1 0 14 1 8 1 2 1 9 1 7 1 2 1 6 2 1 3 1 2 1 7 9 1 7 1 6 2 0 1 7 8 1 4 9 1 6 3 6 7 1 2 8 1 0 14 8 1 1 7 164 1 2 1 6 1 7 1 2 1 2 1 5 1 0 1 7 1 6 7 I 6 5 1 3 14 1 8 1 3 1 6 1 9 1 5 2 3 1 7 6 166 1 3 1 2 1 7 9 1 4 1 8 1 5 2 3 1 8 6 1 6 7 1 1 8 13 1 1 9 1 2 5 1 0 6 5 1 6 8 8 8 13 1 0 7 1 2 1 2 1 2 8 5 6 6 TABLE V I I I THE STABILITY OF JUDGES EXPERIMENT A Judge Mean 1 G 0 1 E> Mean A \u00b0a Mean P 0 P t ga ^ap conclusion 1 1 5 . 7 2 5 2 . 2 8 1 0 . 6 6 1 . 9 8 7 . 7 5 1 . 3 2 1 0 . 1 7 . 1 accept 2 1 6 . 7 5 2 . 8 1 1 2 . 0 6 I . 6 9 7 . 7 5 I . . 8 5 8 . 6 9 . 2 accept 3 2 0 . 1 5 1 . 6 5 1 5 . 7 5 I . 9 2 1 2 . 6 5 1 . 2 8 1 0 . 2 7 - 7 accept 4 1 4 . 5 3 3 . 0 1 1 1 . 0 9 2 . 9 8 8 . 1 2 2 . 7 1 4 . 8 4 . 4 accept 5 1 5 . 5 5 1 .84 1 2 . 2 8 1 . 9 4 9 . 1 3 2 . 1 9 7 . 2 6 . 5 accept 6 1 8 . 2 5 1 M 1 5 . 3 8 1 . 0 2 1 3 . 7 0 1 . 0 0 9 . 9 7 . 0 accept 7 1 5 . 3 0 5 . 5 2 1 2 . 5 3 4 . 1 7 7 , 7 8 . 4 . 4 4 2 . 5 4 . 0 accept 8 2 0 . 0 5 2 . 1 6 1 2 . 6 9 4 . 1 4 8.48 3 . 0 8 9 . 1 4 . 8 accept 9 1 7 . 5 8 1 .64 1 4 . 4 0 2 . 2 5 1 0 . 5 0 3.40 5 . 3 5 . 9 accept 1 0 1 2 . 3 5 3 . 1 6 9 . 0 6 1 . 9 6 6 . 3 5 1.46 5 . 4 6 . 5 accept TABLE IX THE EVALUATIONS IN EXPERIMENT B Run 1 Sample Number 1 2 3 Judge 4 5 7 9 1 0 1 2 1 1 5 1 2 2 1 1 5 1 5 1 5 1 9 2 1 8 1 3 1 3 2 1 1 3 1 6 1 2 1 4 3 1 4 1 2 9 1 8 8 1 0 1 5 1 6 4 1 1 - 1 3 1 0 1 2 1 4 1 0 1 0 7 5 1 1 1 4 1 2 1 4 14 2 0 1 2 8 6 1 2 1 3 1 6 2 0 1 2 1 5 1 0 1 4 7 1 2 14 1 0 1 3 14 1 3 1 2 9 8 1 2 1 3 1 0 1 2 1 5 1 3 1 5 1 2 9 1 2 1 4 1 2 1 8 1 6 1 3 1 5 1 2 1 0 1 0 1 6 9 1 4 1 1 1 5 1 4 1 0 1 1 1 5 1 3 1 1 2 0 1 7 2 0 1 6 1 5 1 2 1 0 1 5 1 1 1 5 1 5 1 3 1 3 1 1 1 3 9 1 1 8 1 1 1 1 1 0 1 0 8 1 4 8 1 0 7 \u2022 8 1 0 8 1 0 6 1 5 1 9 1 3 1 5 2 1 1 5 1 8 1 5 1 8 1 6 1 0 1 2 1 1 1 6 1 2 1 5 1 3 1 0 1 7 1 0 1 4 1 0 1 2 1 2 1 5 1 2 1 1 1 8 1 7 1 5 1 1 1 4 1 7 1 8 14 1 3 1 9 2 0 1 6 1 6 2 0 1 9 2 0 1 4 1 9 2 0 6 1 2 7 8 8 8 8 6 2 1 7 1 3 9 1 6 1 0 1 3 1 2 9 2 2 5 1 1 2 8 9 8 1 0 7 2 3 8 1 5 3 1 8 1 1 1 3 1 2 8 2 4 1 0 1 3 6 1 6 1 3 , 1 5 1 3 1 0 2 5 1 4 1 5 14 2 1 1 4 2 0 1 8 1 4 2 6 1 1 1 3 1 0 1 8 1 5 2 2 1 5 1 2 2 7 1 0 1 2 9 1 4 1 3 1 0 1 4 9 2 8 9 1 2 9 1 2 1 3 1 . 5 14 9 2 9 1 3 1 6 1 4 2 2 16 1 5 2 0 1 7 3 0 1 1 1 7 1 2 1 9 1 5 1 3 1 8 1 5 3 1 1 2 1 5 9 1 5 14 1 3 1 8 1 3 3 2 1 4 1 4 1 0 1 2 1 3 2 0 1 5 1 2 3 3 1 4 1 4 1 1 1 8 14 1 8 1 3 1 1 3 4 14 1 5 1 6 1 7 1 7 2 2 1 8 1 4 3 5 8 1 1 8 2 0 1 2 1 8 1 3 1 2 3 6 1 5 1 7 1 8 2 1 1 6 2 4 1 7 1 6 6 8 TABLE IX (continued) Sample Number 1 2 Judge 3 4 5 7 9 1 0 3 7 1 3 1 5 1 0 1 5 14 1 5 14 1 1 3 8 9 1 1 8 8 1 1 8 1 1 8 3 9 9 1 5 1 2 1 3 1 4 1 3 14 1 2 4 0 1 5 1 4 1 3 1 8 1 8 2 2 1 4 1 6 4 1 1 2 1 3 1 2 1 2 1 2 1 8 1 5 1 0 4 2 1 2 1 2 7 1 0 1 1 1 5 1 2 9 4 3 1 6 1 5 1 4 1 8 1 8 1 8 1 6 1 6 4 4 1 4 1 8 1 6 1 8 1 8 2 2 1 9 1 6 4 5 1 2 1 2 7 1 5 1 1 1 0 1 3 1 1 46 1 0 1 5 1 7 2 0 1 3 1 5 1 5 1 2 4 7 8 1 3 9 16 1 0 1 0 1 3 1 0 48 1 0 1 2 7 1 1 l l 1 3 1 2 9 4 9 1 2 1 0 1 1 1 1 1 0 1 5 14 1 1 5 0 1 5 1 8 1 6 2 1 1 9 2 4 1 6 1 9 5 1 9 1 4 1 0 1 5 . 1 3 1 5 14 1 0 5 2 1 0 14 8 1 0 1 2 1 3 1 5 1 0 5 3 1 2 1 3 1 1 1 9 1 7 2 0 1 4 1 2 5 4 1 0 1 3 1 0 1 4 1 2 1 3 1 2 1 0 5 5 1 3 1 3 1 1 1 8 1 5 1 5 1 3 1 2 5 6 1 1 1 2 1 2 2 0 1 0 1 5 1 1 1 4 5 7 1 3 1 3 1 0 1 6 1 4 1 8 1 6 1 1 5 8 1 2 1 2 1 3 1 6 1 6 1 8 1 4 1 2 5 9 1 0 14 1 0 1 5 1 3 1 5 1 3 1 0 6 0 1 1 1 . 2 1 2 1 6 1 3 1 3 13 1 1 6 l 8 1 2 1 3 15 1 2 1 5 1 2 1 0 6 2 1 1 1 3 9 1 6 1 2 2 0 1 4 9 6 3 1 5 1 5 1 1 14 1 5 1 8 1 5 1 2 64 8 1 1 5 14 1 1 1 0 1 3 1 1 6 5 1 6 14 1 2 1 9 1 6 2 4 1 5 1 5 6 6 1 5 1 3 1 0 1 5 1 3 1 5 1 2 1 1 6 7 1 3 14 9 1 5 1 5 1 3 14 1 0 6 8 8 14 1 0 16 9 1 3 1 2 9 6 9 1 2 14 14 1 7 1 3 1 5 1 2 1 0 7 0 15 1 6 14 1 7 1 6 2 0 1 5 1 2 7 1 1 1 1 5 1 1 1 2 1 5 1 5 1 3 1 0 7 2 9 1 2 8 1 0 1 0 1 0 1 1 9 7 3 7 1 3 8 1 3 9 1 3 1 2 9 7 4 1 3 14 1 7 2 2 1 9 2 4 1 7 2 0 7 5 1 2 1 2 1 3 2 0 1 5 1 5 1 4 1 2 7 6 9 1 3 1 5 1 9 1 6 18 1 6 \u2022 1 3 7 7 1 8 1 7 1 6 2 0 1 8 2 4 1 5 1 5 7 8 1 9 1 6 1 4 1 9 1 8 2 4 1 5 1 6 7 9 1 3 1 2 6 1 5 1 2 1 5 1 2 1 1 8 0 1 3 1 2 1 0 1 6 1 2 18 1 4 1 2 6 9 TABLE IX (continued) Sample Number , 1 2 3 Judge 4 5 7 9 1 0 8 1 9 1 2 7 1 0 1 1 1 3 1 2 1 0 8 2 8 1 2 7 9 9 1 0 1 0 7 8 3 1 0 1 5 1 2 14 1 6 2 0 1 4 1 2 84 1 6 1 5 14 1 2 1 4 2 2 1 5 1 3 8 5 1 3 15 1 3 1 8 1 5 2 0 1 4 1 6 8 6 14 1 8 1 1 1 8 14 2 2 1 7 1 6 8 7 1 0 1 7 1 4 1 6 1 3 1 8 1 3 1 4 8 8 1 1 1 4 1 0 9 1 3 1 5 1 4 1 2 8 9 1 0 14 1 0 1 1 1 4 1 8 1 3 1 1 9 0 1 4 1 5 14 1 5 1 5 2 0 1 5 1 3 9 1 1 0 1 1 7 1 9 1 1 1 5 1 3 1 0 7 0 TABLE IX THE RAW EVALUATIONS PROM EXPERIMENT B Run 2 Sample Judge Number 1 2 3 4 5 7 9 1 0 1 2 1 14 1 3 1 9 1 5 2 5 14 1 6 2 1 7 1 3 1 2 2 0 1 3 2 3 1 2 1 3 3 1 2 1 2 1 0 1 6 1 0 1 5 1 0 9 4 1 3 1 1 9 14 14 1 3 1 2 1 0 5 1 4 \" 1 4 1 0 1 6 1 7 . 18 1 3 1 2 6 1 5 1 4 1 5 1 8 1 6 2 0 14 1 5 7 1 1 1 3 1 1 1 3 1 3 1 3 1 1 1 2 8 1 1 1 3 1 0 1 6 1 5 2 0 1 2 1 1 9 1 1 1 5 1 0 1 6 9 1 0 1 1 1 0 1 0 1 0 14 1 1 1 2 1 5 1 5 1 3 1 2 1 1 1 5 1 6 1 3 1 9 1 6 2 4 1 4 1 4 1 2 1 3 1 3 1 2 1 5 1 4 1 3 1 4 1 1 1 3 9 1 3 8 9 1 1 1 0 1 0 1 0 1 4 8 1 2 8 9 1 0 1 0 1 4 9 1 5 1 0 1 6 1 6 2 0 1 7 2 4 1 6 1 9 1 6 9 1 4 1 2 1 7 1 5 1 5 1 6 1 6 1 7 9 1 6 1 4 2 0 1 3 1 5 1 3 14 18 1 5 1 5 1 6 2 0 1 8 24 1 6 1 8 1 9 18 1 5 1 6 1 8 1 6 2 3 1 6 2 0 2 0 8 1 2 1 0 1 2 1 2 1 0 1 2 1 2 2 1 1 1 1 2 1 2 1 6 1 2 1 5 1 4 1 2 2 2 9 1 1 9 8 1 1 1 3 1 2 1 0 2 3 8 1 1 7 7 9 . 8 1 0 7 2 4 1 2 1 4 1 2 1 6 1 5 1 5 1 5 1 1 2 5 14 1 4 14 1 9 13 2 3 1 5 1 2 2 6 1 3 1 2 1 3 1 8 1 4 2 0 1 3 1 0 2 7 9 1 2 1 0 1 6 1 3 1 5 1 3 9 2 8 9 1 1 1 0 1 5 1 1 1 5 1 3 7 2 9 1 3 1 6 1 6 2 2 1 6 2 0 1 6 1 9 3 0 1 0 1 5 1 4 1 8 1 2 1 3 14 14 3 1 1 0 1 5 l l 1 6 1 2 1 3 14 1 2 3 2 9 1 3 n 1 3 1 4 1 5 1 3 1 0 3 3 14 1 4 1 3 1 1 1 2 1 8 1 3 9 3 4 1 7 14 1 4 1 5 1 7 2 0 1 5 1 5 3 5 1 0 1 3 8 2 0 1 2 1 5 1 3 9 3 6 1 8 1 6 1 6 2 0 1 8 2 4 1 6 1 7 7 1 TABLE IX (continued) Sample Judge Number 1 2 3 4 5 7 9 1 0 3 7 1 3 1 4 1 2 14 1 3 1 5 1 3 1 0 3 8 9 1 0 8 9 1 0 1 0 1 1 7 3 9 1 1 1 5 1 3 2 0 1 2 1 5 1 4 1 0 4 0 2 0 1 5 1 4 2 1 1 6 2 3 14 1 7 4 1 1 2 14 1 1 1 2 1 1 1 5 1 3 9 4 2 1 1 1 2 7 1 0 1 1 1 5 1 2 8 4 3 2 1 1 4 1 5 1 2 1 5 2 3 1 5 1 4 4 4 1 8 1 6 1 8 2 0 1 6 2 3 1 8 1 5 45 1 1 1 2 8 1 2 1 1 1 0 1 2 8 46 1 0 1 3 1 5 2 2 1 3 2 0 1 4 1 2 4 7 1 0 1 2 1 0 1 5 1 1 1 3 1 2 9 48 1 0 1 2 8 1 2 1 3 1 0 1 3 6 4 9 8 14 1 4 9 1 1 1 3 1 3 9 5 0 1 9 1 6 1 5 . 2 1 1 8 2 0 1 8 1 7 5 1 1 0 1 3 1 1 1 2 1 2 1 5 1 3 1 1 5 2 1 1 1 3 1 0 1 6 1 1 1 5 1 2 1 0 5 3 1 6 1 3 1 5 1 9 14 2 0 1 4 1 3 5 4 1 0 1 2 1 1 1 6 1 2 1 3 14 1 1 5 5 2 1 1 5 1 3 1 9 13 1 8 1 6 1 2 5 6 2 2 1 2 1 2 1 9 1 1 1 5 1 3 1 2 5 7 1 7 14 1 2 \u2022 1 7 1 1 1 5 1 3 1 0 5 8 2 4 1 5 1 4 1 9 1 5 2 0 1 6 1 3 5 9 1 1 1 2 8 1 8 1 3 1 3 1 3 1 2 6 0 1 0 1 3 1 1 1 5 1 2 1 3 1 3 1 1 6 1 8 1 2 1 0 1 5 1 1 1 5 1 3 1 0 6 2 1 0 1 3 7 1 5 1 1 1 8 1 4 1 0 6 3 1 7 1 5 1 4 1 6 1 4 1 5 1 5 1 1 64 1 0 1 2 1 0 1 2 1 1 1 0 1 2 1 1 6 5 14 14 1 6 2 0 1 3 2 0 1 4 1 4 6 6 1 2 14 1 0 1 6 1 2 1 5 1 5 9 6 7 1 0 1 3 1 1 1 7 1 3 1 5 1 5 1 0 6 8 6 9 1 9 1 2 1 3 1 3 1 3 1 5 14 1 2 1 1 14 1 0 1 3 1 2 1 5 1 2 1 0 7 0 2 2 1 4 1 5 1 8 1 5 2 3 14 1 4 7 1 1 9 1 3 1 1 1 2 1 2 1 5 1 6 1 0 7 2 14 1 1 6 1 0 1 0 8 1 2 7 7 3 1 0 1 2 4 8 9 8 1 1 6 7 4 2 3 1 5 1 6 1 9 1 6 2 4 1 6 1 7 8 7 5 1 3 1 3 1 0 1 9 1 1 1 5 1 3 7 6 1 1 1 3 1 1 1 9 1 4 1 3 14 9 7 7 2 0 14 1 2 1 9 1 6 2 0 1 4 1 4 7 8 1 8 1 5 14 2 1 1 8 2 3 1 6 1 6 7 9 7 1 0 5 8 9 8 1 0 6 8 0 9 1 2 1 1 1 0 1 4 1 0 1 3 9 TABLE IX (concluded) Sample Number 1 2 3 Judg^e 5 7 9 1 0 8 1 8 1 0 4 6 1 0 8 1 1 6 8 2 1 0 1 4 9 1 0 1 1 1 5 1 3 7 8 3 1 2 1 2 9 1 7 1 2 1 5 1 4 1 0 84 1 4 1 6 1 3 2 0 1 6 2 0 1 6 1 5 8 5 1 7 1 7 1 4 2 1 1 6 1 8 1 5 1 8 8 6 1 1 1 8 1 2 2 1 1 3 2 0 1 7 1 5 8 7 1 2 1 4 1 4 2 0 1 2 1 5 1 5 1 4 8 8 1 0 1 3 1 1 1 2 1 3 1 3 1 3 1 1 8 9 1 2 1 3 1 1 1 2 1 4 1 5 1 5 1 0 9 0 9 1 4 1 1 14 1 4 1 8 1 4 1 2 9 1 1 2 1 3 1 1 1 4 1 2 1 8 1 3 1 1 7 3 TABLE X THE .STABILITY OF JUDGES EXPERIMENT B df = 14 Judge Mean G Og Mean A Oa Mean P Op t t c o n c l u s i o n ga ap 1 1 6 . 1 3 1 . 6 3 1 0 . 9 3 1 . 8 9 8 . 6 7 1 . 9 5 9 . 1 3 . 5 accept 2 1 5 . 1 3 . 8 0 1 3 . 2 6 1 . 0 8 1 1 . 8 0 1 . 3 7 5 . 2 3 . 3 accept 3 1 4 . 6 0 1 . 9 2 1 1 . 0 0 2 . 6 6 7 . 0 7 2 . 0 8 4 . 1 4 . 4 accept : - 4 \u2022 1 8 . 8 0 2 . 6 6 1 4 . 9 3 3 . 0 4 I I . 6 7 3 . 4 2 3 . 6 2 . 7 accept 5 1 7 . 0 7 I . 6 5 1 3 . 0 0 1 . 9 0 1 0 . 4 7 1.41 6 . 1 4 . 0 accept 6 1 8 . 8 7 2 . 3 0 1 3 . 8 0 1 . 5 3 1 2 . 9 3 1 . 9 3 6 . 8 1 . 3 r e j e c t 7 2 1 . 3 3 2 . 3 1 1 5 . 6 7 I . 8 9 1 0 . 8 0 2 . 2 6 7 . 0 6 . 3 accept 8 1 5 . 8 7 3 . 7 5 1 4 . 4 7 4 . 5 1 1 3 . 4 7 3 . 8 0 . 9 . 7 r e j e c t 9 1 5 . 7 3 1 . 6 1 1 3 . 6 0 1 . 1 4 1 1 . 3 3 1 . 4 9 4 . 1 4 . 5 accept 1 0 1 5 . 8 7 2 . 2 7 1 0 . 8 7 1.14 8 . 7 3 1 . 7 7 7 . 3 3 . 8 accept 74 APPENDIX B MATERIALS On the f o l l o w i n g pages are t o be found the f o l l o w -i n g examples of m a t e r i a l s used i n these experiments. 1) An example of the handwriting samples used. 2) A statement of the i n s t r u c t i o n s t o the judges i n \u2022the s e l e c t i o n of s t a b l e samples. 3 ) A sample of the e v a l u a t i o n form f o r Experiment A. 4) A sample of the e v a l u a t i o n f o r f o r Experiment B. 5) A statement of\" the i n s t r u c t i o n s t o the judges f o r Experiment A. 6) A statement of the i n s t r u c t i o n s t o the judges f o r Experiment B. 76 The i n s t r u c t i o n s t o the judges i n the choice of s t a b l e samples. \"This i s a copy of the Ayres s c a l e of q u a l i t y of handwriting. See, there are e i g h t c a t e g o r i e s , running from 2 t o 9 at the top of the sheet.\" \"Here are a number of handwriting samples. I would l i k e you t o compare each sample w i t h the samples of w r i t i n g given on the Ayres s c a l e . Would you p l e a s e p l a c e each sample i n one of e i g h t p l a c e s here on the t a b l e a c c o r d i n g to which Ayres category you t h i n k i t belongs i n . \" \"Are there any q u e s t i o n s ? \" 77 The I n s t r u c t i o n s to the judges i n Experiment A. \"Would you s i t here p l e a s e ? \" \" I am going t o show you a f i l m s t r i p on which there are a number of handwriting samples. I am going to ask you to judge these samples f o r q u a l i t y of w r i t i n g . Please put your scores on t h i s sheet here, working down the columns.\" \"The p e r m i t t e d range of scores i s from one t o twenty-f i v e , t w e n t y - f i v e b e i n g the best and one the p o o r e s t . You w i l l have a maximum of ten seconds to judge each frame, but i f I see you w r i t i n g down a score before t h a t time has p a s t , I w i l l move on to the next frame immediately. I f I should miss see i n g you w r i t e down a score, p l e a s e say 'O.K.1 and I w i l l move on. I f you should not have a score w r i t t e n down when ten seconds have pas t I w i l l say l0.K.\u00bb and move on t o the next frame. You'should w r i t e down a score always.\" \"Use your own standards of handwriting q u a l i t y . Your f i r s t impression i s what counts. Never change any s c o r e . \" \"Are there any q u e s t i o n s ? \" \"I am now going t o show you a few of the frames i n t h i s f i l m s t r i p t o give you an idea of the range of q u a l i t y you may expect. Do not eva l u a t e them now, you w i l l get a chance to do th a t l a t e r . \" \" A l l r i g h t , p l e a s e s t a r t now.\" 78 I n s t r u c t i o n s t o the Judges of experiment B. E x a c t l y as f o r experiment A, w i t h t h i s a d d i t i o n , immediately before questions are p e r m i t t e d . \"Have a l o o k a t the answer sheet. N o t i c e t h a t you o n l y go as f a r as the f i f t h column. When we reach here ( i n d i c a t i n g seventh space i n f i f t h column) we w i l l be s t o p p i n g . \" ","@language":"en"}],"Genre":[{"@value":"Thesis\/Dissertation","@language":"en"}],"IsShownAt":[{"@value":"10.14288\/1.0105983","@language":"en"}],"Language":[{"@value":"eng","@language":"en"}],"Program":[{"@value":"Education","@language":"en"}],"Provider":[{"@value":"Vancouver : University of British Columbia Library","@language":"en"}],"Publisher":[{"@value":"University of British Columbia","@language":"en"}],"Rights":[{"@value":"For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https:\/\/open.library.ubc.ca\/terms_of_use.","@language":"en"}],"ScholarlyLevel":[{"@value":"Graduate","@language":"en"}],"Title":[{"@value":"Effects of order of judgment on subjective evaluation","@language":"en"}],"Type":[{"@value":"Text","@language":"en"}],"URI":[{"@value":"http:\/\/hdl.handle.net\/2429\/39707","@language":"en"}],"SortDate":[{"@value":"1960-12-31 AD","@language":"en"}],"@id":"doi:10.14288\/1.0105983"}