UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

The development and application of a methodology for program evaluation Gleadow, Norman E. 1976

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.

Item Metadata

Download

Media
831-UBC_1976_A8 G59_4.pdf [ 7.89MB ]
Metadata
JSON: 831-1.0093721.json
JSON-LD: 831-1.0093721-ld.json
RDF/XML (Pretty): 831-1.0093721-rdf.xml
RDF/JSON: 831-1.0093721-rdf.json
Turtle: 831-1.0093721-turtle.txt
N-Triples: 831-1.0093721-rdf-ntriples.txt
Original Record: 831-1.0093721-source.json
Full Text
831-1.0093721-fulltext.txt
Citation
831-1.0093721.ris

Full Text

THE DEVELOPMENT AND APPLICATION OF A METHODOLOGY FOR PROGRAM EVALUATION by Norman E. Gleadow, B.Sc, University of B.C. (1967) A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS in the Department of Science Education We accept this thesis as conforming to the required standard THE UNIVERSITY OF BRITISH COLUMBIA May, 1976. In presenting th i s thes is in pa r t i a l fu l f i lment of the requirements for an advanced degree at the Un ivers i ty of B r i t i s h Columbia, I agree that the L ibrary sha l l make it f ree l y ava i l ab le for reference and study. I fur ther agree that permission for extensive copying of this thes is for scho lar ly purposes may be granted by the Head of my Department or by his representat ives. It is understood that copying or pub l i ca t i on of th i s thes i s f o r f i nanc i a l gain sha l l not be allowed without my wr i t ten permission. Department of The Univers i ty of B r i t i s h Columbia 2075 Wesbrook Place Vancouver, Canada V6T 1W5 Date {If^l / f / ^ A b s t r a c t The purpose o f t h i s s t u d y was t o d e v e l o p a p r a c t i c a l methodology o f program e v a l u a t i o n from c o n s i d e r a t i o n s o f what t h e e v a l u a t i v e a c t r e q u i r e d on l o g i c a l g r o u n d s . The methodology d e v e l o p e d was based on P.W. T a y l o r ' s (1961) a n a l y t i c a l t r e a t m e n t o f e v a l u a t i o n and M. S c r i v e n ' s e x t e n s i v e w r i t i n g s on t h e s u b j e c t . The s t u d y i n v o l v e d two broad p hases. The f i r s t phase was a d i s c u s s i o n o f T a y l o r ' s and S c r i v e n ' s t h o u g h t s on e v a l u a t i o n showing where they a r e c o m p a t i b l e and o v e r l a p . The second phase showed how t h e i d e a s d e v e l o p e d i n the f i r s t phase were o p e r a t i o n a l i z e d and a d a p t e d t o the e v a l u a t i o n o f a t e a c h e r t r a i n i n g program a t t h e U n i v e r s i t y o f B r i t i s h C olumbia. A g e n e r a l c o n c l u s i o n o f t h i s s t u d y was t h a t T a y l o r ' s a n a l y s i s o f the method and p r o d u c t o f e v a l u a t i o n p r o v i d e d a s u i t a b l e framework f o r the e v a l u a t i o n o f an e d u c a t i o n a l program. More s p e c i f i c a l l y when T a y l o r ' s a n a l y s i s was combined w i t h S c r i v e n ' s e x t e n s i v e work, a f e a s i b l e model o f e v a l u a t i o n r e s u l t e d w hich r e a d i l y produced a d e f e n s i b l e , o v e r a l l e s t i m a t i o n o f w orth f o r an e d u c a t i o n a l p r o d u c t . A l t h o u g h t h e methodology o f e v a l u a t i o n d e v e l o p e d i n t h i s s t u d y was o n l y a p p l i e d t o a t e a c h e r t r a i n i n g program, i t c o u l d e a s i l y be e x t e n d e d to t h e e v a l u a t i o n o f o t h e r e d u c a t i o n a l programs o r p r o d u c t s . TABLE OF CONTENTS Page LIST OF TABLES . . . . . . . . v i LIST OF FIGURES v i i ACKNOWLEDGEMENTS . . v i i i • C h a p t e r I THE INTRODUCTION .: . . 1 1.00 Purpose o f the' Study 1 1.10 S t a t e m e n t o f the G e n e r a l Problem. ...... I. . 1 1.20 D e f i n i t i o n o f Terms • • • 3 « ' 1.21 E d u c a t i o n a l E v a l u a t i o n ." 3 1.22 Summative E v a l u a t i o n . 3 1.23 F o r m a t i v e E v a l u a t i o n , 3 1.30 S t a t e m e n t o f t h e S p e c i f i c P r o blem 3 1.40 Overview o f t h e Study . 4 1.50 D e l i m i t a t i o n o f the Study 5 II THE THEORETICAL FRAMEWORK FOR AN EVALUATION MODEL . . 7 2.00 I n t r o d u c t i o n 7 2.10 P r e s e n t D i f f i c u l t i e s i n E v a l u a t i o n 7 2.11 • T y l e r ' s Approach t o E v a l u a t i o n 10 2.12 The A c c r e d i t a t i o n Model 11 2.13 S t u f f l e b e a m ' s C o n t e x t , I n p u t , P r o c e s s and P r o d u c t (CIPP) Model 12 2.14 S t a k e ' s Model . 15 2.15 Summary 16 2.20 E v a l u a t i o n : The T h e o r e t i c a l Framework . • • • • 17 2.30 The P r e c o n d i t i o n s . .' . . . . . . •.' . . . . 19 2.31 The Evaluatum . 19 2.32 P o i n t o f View . . ' . . . . 19 2.33 G r a d i n g o r Ranking 21 2.40 L e v e l 1 o f S - G r a d i n g : A d o p t i o n o f S t a n d a r d s . . 28 2.41 I n t r i n s i c V a l u e 28 2.42 E x t r i n s i c V a l u e 28 2.50 L e v e l II o f S - G r a d i n g : O p e r a t i o n a l C l a r i f i c a t i o n o f the S t a n d a r d s i n G r a d i n g 31 2.60 L e v e l I I I : S p e c i f i c a t i o n o f the C l a s s o f Comparison i n G r a d i n g 32 i i . . T a b l e o f C o n t e n t s ( C o n t . ) Page 2.70 L e v e l IV: D e t e r m i n a t i o n o f the Good and Bad C h a r a c t e r i s t i c s o f t h e Evalu a t i o n i n S - G r a d i n g 33 2.71 G a t h e r i n g Data on the C h a r a c t e r i s t i c s . . . . 33 2.72 D e t e r m i n i n g i f the C h a r a c t e r i s t i c s a r e Good o r Bad 34 2.80 L e v e l V: The O v e r a l l Judgement o f Worth . . . . 36 I I I APPLICATION OF THE MODEL TO THE DESIGN OF AN ACTUAL EVALUATION STUDY:. . 38, 3.00 I n t r o d u c t i o n 38 3.10 E s t a b l i s h i n g t h e P r e c o n d i t i o n s 38 3.11 The Evaluatum 38 3.111 L e t t e r t o P o t e n t i a l S t u d e n t s f o r Program B 39 3.112 Program D e s c r i p t i o n 39 3.113 D e s c r i p t i o n o f t h e O f f e r e d C o u r s e s i n Program B 40 3.12 The P o i n t o f View 41 3.13 G r a d i n g 43 3.20 A d o p t i o n o f S t a n d a r d s 44 3.21 S t a n d a r d 1: The Worth o f t h e Program G o a l s . 45 3.22 S t a n d a r d 2: Goal Achievement 48 3.30 O p e r a t i o n a l C l a r i f i c a t i o n o f S t a n d a r d s . . . . 49 3.31 S t a n d a r d 1: The Worth o f Program B's G o a l s . 49 3.32 S t a n d a r d 2: The Degree o f Goal A c h i e v e m e n t . 52 3.33 The C h e c k l i s t f o r D e t e r m i n i n g Good •;' '• -Achievement 54 3.331 C h e c k p o i n t 1: Market 54 3.332 C h e c k p o i n t 2: T r u e F i e l d T r i a l s 55 3.333 C h e c k p o i n t 3: T r u e Consumer 57 3.334 C h e c k p o i n t 4: Long Term 59 3.335 C h e c k p o i n t 5: S i d e E f f e c t s ......... 59 3.336 C h e c k p o i n t 6: P r o c e s s 63 3.337 C h e c k p o i n t 7: C a u s a t i o n 64 3.338 C h e c k p o i n t 8: S t a t i s t i c a l S i g n i f i c a n c e . . 68 3.40 S p e c i f i c a t i o n o f the C l a s s o f Comparison i n G r a d i n g 70 3.50 D e t e r m i n i n g t h e Good and Bad C h a r a c t e r i s t i c s o f t h e E v a l u a tum i n S - G r a d i n g 71 3.51 G a t h e r i n g Data on the Goal Achievement C h a r a c t e r i s t i c s 71 3.52 D e t e r m i n i n g i f the C h a r a c t e r i s t i c s o f t h e Program a r e Good o r Bad 75 3.521 R a t i n g the Worth o f the G o a l s 75 i i i . T a b l e o f C o n t e n t s (Cont.) Page 3.522 Goal Achievement and i t s R a t i n g 78 3.60 O v e r a l l D e t e r m i n a t i o n o f Worth . 81 3.70 E x t e n s i o n o f the Model 85 3.71 I n t r o d u c t i o n 85 3.72 The S t a n d a r d s t o be Used i n Ranking 86 3.721 C o s t s . . . 87 3.722 Extended S u p p o r t 88 3.73 The Precedence o f t h e S t a n d a r d s i n Ranking . 89 3.74 Ranking the E v a l u a t a 92 IV DATA COLLECTION TECHNIQUES IN THE APPLICATION OF THE EVALUATION MODEL 93 4.00 I n t r o d u c t i o n . 93 4.10 I n i t i a l S t a g e o f Data C o l l e c t i o n 93 4.20 Second S t a g e o f Data C o l l e c t i o n . 96 4.21 E v a l u a t i o n o f U.B.C I n s t r u c t o r s ' , T e a c h i n g Performance . . . . . 97 4.22 Q u e s t i o n n a i r e t o S t u d e n t T e a c h e r s o r t h e i r A l t e r n a t e Placement 98 4.23 I n t e r v i e w o f S t u d e n t T e a c h e r s . . . . . . . . 98 4.24 F i n a l Q u e s t i o n n a i r e f o r the S t u d e n t T e a c h e r s 98 4.25 O b s e r v a t i o n o f S t u d e n t T e a c h i n g . 99 4.26 S i d e E f f e c t s o f - P r o g r a m B . . . 100 4.27 I n t e r v i e w w i t h Program B's D i r e c t o r 101 4.28 Q u e s t i o n n a i r e f o r S c h o o l P u p i l s 101 4.29 I n t e r v i e w w i t h U.B.C. A d m i n i s t r a t o r s . . . . 101: 4.210 O t h e r I n f o r m a t i o n S o u r c e s 102 4.30 P r a c t i c a l P r o c e d u r e s f o r the O r g a n i z a t i o n o f Data 105 4.40 C o n f i d e n t i a l i t y o f Data 108 V SUMMARY AND CONCLUSIONS 109 5.00 Summary . . . 109 5.10 C o n c l u s i o n s 113 REFERENCES l l f t o . - * , APPENDICES , . n 9 A. Q u e s t i o n n a i r e f o r t h e Program D e v e l o p e r s , S u b j e c t : E v a l u a t i o n P r i o r i t i e s 1 1 9 B. I n t e r v i e w o f S c h o o l A d m i n i s t r a t i o n 1 2 5 i v . Table of Contents (Cont.) Page C. Likert Attitude Scale (to measure general attitudes toward the teacher training process of the student teachers) 127 D. Reaction Form 130 E. Evaluation of U.B.C. Instructors' Teaching Performance: Form A 132 F. Questionnaire to Student Teachers on Their Alternate Placement . . . . . . . !35 G. Evaluation of U.B.C. Instructors' Teaching Performance: Form B 139 H. Interview of Student Teachers ^2 I. Final Questionnaire for Student Teachers . . . . . . ^45 J. Discipline Techniques K. 'Side Effects of Program B (Teacher's Questionnaire) . *50 L. Interview with Program B's Director "^ 54 M. Questionnaire for School Pupils -^->6 N. Interview with U.B.C. Administrators . . . . . . . . 0. Student Teacher's Questionnaire (Side Effects) . . . l o 2 •P. 'U.B.C. Staff Questionnaire on Courses -.167/ Q. Student Teachers' Questionnaire on the Evaluation _ of U.B.C. Instructors Teaching Performance . . *-<° R. Teachers' Questionnaire S. Interview with Director of Elementary Education at School Board Offices 1 8 7 T. -Proposal for the Evaluation of Student Teachers Teaching Effectiveness 1 8 ° U. Summary of Evaluation Steps in Ranking 2 0 ^ v. LIST OF TABLES T a b l e Page 1 Taxonomy o f E v a l u a t i o n Types 25 2 C a l e n d a r o f Data C o l l e c t i n g E v e n t s . 72-74 3 Data S o u r c e s f o r the Goal Achievement C h e c k p o i n t s . . 103 4 Comparison o f E v a l u a t i o n P r o c e d u r e s 113-114 v i . LIST OF FIGURES F i g u r e Page 1 D e v e l o p i n g E v a l u a t i o n D e s i g n s 14^ 2 An O u t l i n e o f t h e L o g i c a l P r o c e s s 27 3 The S t a n d a r d s and t h e i r R a t i n g s 69 4 D e t e r m i n i n g a R a t i n g f o r Goal Achievement 79 5 P r o d u c t E v a l u a t i o n P r o f i l e f o r Program B 80 6 Comparing Goal Worth and Goal Achievement R a t i n g . . . . 82 7 E x p a n s i o n o f F i g u r e 6 84 8 Combining the Goal Achievement and C o s t R a t i n g s . . . . 91 9 Format f o r O r g a n i z a t i o n o f Data 105 10 Example o f Format f o r O r g a n i z a t i o n o f Data 107 11 S t e p s i n S - G r a d i n g i n F o r m a t i v e E v a l u a t i o n . 112 v i i . ACKNOWLEDGEMENTS I wi s h t o e x p r e s s my s i n c e r e t hanks t o P r o f e s s o r W a l t e r B. B o l d t , who i n t r o d u c e d me t o t h e f i e l d o f e d u c a t i o n a l e v a l u a t i o n , and whose s c h o l a r s h i p and i n t e l l e c t u a l i n t e g r i t y were a c o n s t a n t i n s p i r a t i o n . v i i i . DEDICATION t h e memory o f my f a t h e r Frank R. Gleadow CHAPTER I The I n t r o d u c t i o n 1.00 Purpose o f t h e Study Many p r e s e n t e v a l u a t i o n models a r e u n s a t i s f a c t o r y on t h e o r e t -i c a l grounds because they l a c k c l a r i t y as t o t h e n a t u r e o f t h e e v a l u a t i v e a c t . The purpose o f t h e p r e s e n t s t u d y i s t o shed some l i g h t on t h i s problem by d e v e l o p i n g a p r a c t i c a l model o f e v a l u a t i o n based on an a n a l y s i s o f t h e o r e t i c a l i d e a s about e v a l u a t i o n p r o -c e d u r e s and t o d e m o n s t r a t e how t h e r e s u l t i n g model c o u l d be used i n a p a r t i c u l a r c a s e . The problem i s i m p o r t a n t because much o f what i s c u r r e n t l y c a l l e d e v a l u a t i o n f a i l s t o f u l f i l l t h e e v a l u a t i o n f u n c t i o n s t o be s e r v e d , on t h e o r e t i c a l g r o u n d s . 1.10 S t a t e m e n t ; o f the_.Gener.a1 P r o b l em y.- -,: ' "One can be a g a i n s t e v a l u a t i o n o n l y i f one can show t h a t i t is.improper, t o seek an answer t o q u e s t i o n s a b o u t t h e m e r i t o f e d u c a t i o n a l i n s t r u m e n t s , which would i n v o l v e showing t h a t t h e r e a r e no l e g i t i m a t e a c t i v i t i e s ( r o l e s ) i n which t h e s e q u e s t i o n s can be r a i s e d . " ( S c r i v e n , 1967, p. 41) The g e n e r a l problem t o be i n v e s t i g a t e d i n t h e s t u d y i s t o d e t e r m i n e t h e n a t u r e o f t h e p r o c e d u r e s o f e v a l u a t i o n on t h e o r e t i c a l grounds and t o a t t e m p t t o s p e c i f y t h e e v a l u a t i v e f u n c t i o n s t o be s e r v e d i n p r a c t i c a l terms t h r o u g h i l l u s t r a t i o n and a p p l i c a t i o n t o the e v a l u a t i o n o f a t e a c h e r - t r a i n i n g program. E v a l u a t i o n as p r a c t i s e d f o r t h e p a s t few y e a r s i s not 1 ,2 s u c c e e d i n g i n i t s p e r c e i v e d f u n c t i o n . Evans (1974, pp. 7, 11, 12) i n d i c a t e s t h a t though e v a l u a t i o n i s ". . . now a l l t h e r a g e ..." i t f a c e s many new problems. The problems r e v o l v e around d i f f i c u l -t i e s o f c o l l e c t i n g d a t a , u n r e a l i s t i c e x p e c t a t i o n s on t h e p a r t o f p o l i c y m a k e r s , u n w i l l i n g n e s s , o f p e o p l e t o be e v a l u a t e d and so on. Guba (1969) i n d i c t s e v a l u a t i o n f o r i t s f a i l u r e t o f i n d any d i f f e r -ences among programs, even when, t h e r e a r e o b v i o u s d i f f e r e n c e s , and s u g g e s t s t h a t t h e whole t e c h n i q u e o f e v a l u a t i o n s h o u l d be q u e s t i o n e d and examined. Bloom (1969, p. 46) who equates e v a l u a t i o n w i t h t e s t i n g and m e a s u r i n g , w r i t e s o f t h e p o t e n t i a l l y d e s t r u c t i v e a s p e c t s o f e v a l u a t i o n when t e a c h e r s and s t u d e n t s b e l i e v e t h e p r o c e d u r e s t o be u n f a i r (p. 4 5 ) . In a r e c e n t p a p e r , G l a s s (1975) c r i t i c i s e s e v a l u a t i o n as t a k i n g weakly d e f i n e d s t a n c e s when t h r e a t e n e d and t h e n r e a c t i n g t o t h i s i n h e r e n t weakness w i t h e i t h e r a h u m a n i s t i c a p p r o a c h o f g e n t l e s e l f - e v a l u a t i o n ; o r by r e g a r d i n g " r e s i s t a n c e s " t o e v a l u a t i o n as mere t e c h n i c a l problems overcome by t e c h n i c a l means; o r by r e t r e a t i n g i n t o a v a s t g r a y a r e a between. These a r e some o f t h e c r i t i c i s m s which have been l e v e l e d a t e v a l u a t i o n . They r a i s e two fundamental q u e s t i o n s : what i s e v a l u a -t i o n ; and how can e v a l u a t i o n be c a r r i e d o u t ? Many e v a l u a t i o n models d e a l w i t h t h e f i r s t q u e s t i o n by p o s i n g a n o t h e r which i s g e n e r a l l y o f t h e form; "what i s i t t h a t we a r e d o i n g t o d a y t h a t we c a l l e v a l u a t i o n ? " From t h e answer t o t h i s q u e s t i o n i s d i s t i l l e d a model which i s then used f o r e v a l u a t i o n . In o t h e r words t h e model i s a c l a r i f i c a t i o n o f p r e s e n t p r a c t i s e , and not n e c e s s a r i l y e v a l u a t i o n . The second p a r t o f t h e q u e s t i o n i s u s u a l l y answered 3 i n a s i m i l a r way and e m b r o i l s r e s e a r c h e r s i n t h e e v a l u a t i o n v s . r e s e a r c h i m b r o g l i a ( P r o v u s , 1970: Hemphill 1969, p. 190; Welch 1969, p. 440). T h e r e f o r e t o work from p r e s e n t p r a c t i s e i s not a s a t i s f a c t o r y method o f d e f i n i n g e v a l u a t i o n and w i l l not be used i n t h i s t h e s i s e x c e p t where p r e s e n t p r a c t i s e i s t h e same as t h e t h e o r e t i c a l p o s i t i o n d e v e l o p e d i n C h a p t e r 2. 1.20 D e f i n i t i o n o f Terms 1.21 E d u c a t i o n a l E v a l u a t i o n : E d u c a t i o n a l E v a l u a t i o n i s t h e p r o c e d u r e o f j u s t i f y i n g a v a l u e c l a i m about t h e m e r i t o r worth o f an e d u c a t i o n a l p r o d u c t ( c o u r s e , c u r r i c u l u m , e t c . ) The r e s u l t o f t h i s p r o c e d u r e i s a c o n c l u s i o n o f t h e worth o f t h e p r o d u c t . 1.22 Summative E v a l u a t i o n : T h i s i s an o v e r a l l e v a l u a t i o n o f a completed e d u c a t i o n a l p r o d u c t and s e r v e s t h e needs o f t h e c l i e n t , and more i m p o r t a n t l y , t h e u l t i m a t e consumer(s) o f t h e p r o d u c t . 1.23 F o r m a t i v e E v a l u a t i o n : T h i s t y p e o f e v a l u a t i o n t a k e s p l a c e a t i n t e r m e d i a t e p o i n t s i n the development o f a p r o d u c t and s e r v e s t h e needs o f t h e d e v e l o p e r s f o r m o d i f y i n g and i m p r o v i n g t h e p r o d u c t . 1.30 Statement o f t h e S p e c i f i c Problem The s p e c i f i c problem t o be d e a l t w i t h i n t h i s t h e s i s i s t o 4 d e v e l o p a p r a c t i c a l methodology o f e v a l u a t i o n based on t h e l o g i c a l c o n s i d e r a t i o n s p r e s e n t e d by T a y l o r (1961) and S c r i v e n (1967) and t h e e v a l u a t i v e f u n c t i o n s t o be s e r v e d ^ i n t h e form o f t h e c h e c k l i s t f o r t h e e v a l u a t i o n o f P r o d u c t s , p r o d u c e r s and p r o p o s a l s p r o p o s e d by S c r i v e n (1974b); and t o a p p l y t h e r e s u l t s t o t h e e v a l u a t i o n o f a t e a c h e r - t r a i n i n g program a t t h e U n i v e r s i t y o f B.C. The s p e c i f i c p r o b l e m can be f u r t h e r broken down t o a number o f sub-problems f o r t h e p u r p o s e s o f t h e s t u d y : a) An a n a l y s i s o f t h e e v a l u a t i o n p r o c e s s based on T a y l o r ' s p h i l o s o p h i c a l p o s i t i o n ( C h a p t e r I I ) . b) E s t a b l i s h i n g t h e e v a l u a t i o n f u n c t i o n s t o be s e r v e d and t h e v a l i d i t y o f S c r i v e n ' s C h e c k l i s t . c) A d a p t i o n o f S c r i v e n ' s C h e c k l i s t t o t h e e v a l u a t i o n o f a t e a c h e r - t r a i n i n g program ( C h a p t e r I I I ) . d) Development o f i n s t r u m e n t s and t e c h n i q u e s f o r c o l l e c t i n g t h e n e c e s s a r y e v a l u a t i o n data» ( C h a p t e r s I I I and I V ) . e) Making a f i n a l e v a l u a t i v e c l a i m about t h e t e a c h e r - t r a i n i n g s program (C h a p t e r I I I ) . f ) C r i t i q u e o f t h e e f f e c t i v e n e s s o f t h e c h e c k l i s t f o r t h i s p u r p o s e ( C h a p t e r V ) . 1.40 Overview o f t h e Study The t h e s i s c o m p r i s e s t h e development and i l l u s t r a t i o n o f a methodology o f e v a l u a t i o n based on P.W. T a y l o r ' s (1961) a n a l y t i c a l t r e a t m e n t o f e v a l u a t i o n and M. S c r i v e n ' s (1967, 1974b) e x t e n s i v e 5 o p e r a t i o n a l i z a t i o n o f a c o m p a t i b l e model o f e v a l u a t i o n . T h e r e a r e t h r e e broad phases i n t h i s p r o c e s s . The f i r s t phase i n v o l v e s a d i s c u s s i o n o f P.W. T a y l o r ' s and S c r i v e n ' s t h o u g h t s on e v a l u a t i o n showing where t h e y a r e c o m p a t i b l e and o v e r l a p . T h i s i s done i n C h a p t e r I I . The second phase shows t h e method by which t h e i d e a s d e v e l o p e d i n C h a p t e r II were o p e r a t i o n a l i z e d and a d a p t e d t o a p a r t i c u l a r e v a l u a t i o n p r o j e c t ; t h a t i s t h e e v a l u a t i o n o f a program i n t e a c h e r t r a i n i n g . T h i s was done i n C h a p t e r I I I . C h a p t e r I I I a l s o i l l u s t r a t e s how t h e d e v e l o p e d model can be e x t e n d e d . The t h i r d phase i s a d i s c u s s i o n o f t h e d a t a c o l l e c t i o n t e c h -n i q u e s used i n a p p l y i n g t h e e v a l u a t i o n model d e v e l o p e d . C h a p t e r IV i l l u s t r a t e s t h o s e t e c h n i q u e s . : C h a p t e r V summarizes t h e p r e v i o u s f o u r c h a p t e r s and makes f i n a l recommendations on t h e o v e r a l l use-f u l n e s s o f t h e scheme. 1.50 D e l i m i t a t i o n o f t h e Study The E v a l u a t i o n t e c h n i q u e i s d e v e l o p e d around t h e e v a l u a t i o n o f an e d u c a t i o n a l p r o d u c t . T h e r e f o r e t h e e x t e n s i o n o f t h e scheme t o o t h e r a r e a s , such as t e a c h i n g e f f e c t i v e n e s s , has not been under-t a k e n i n the p r e s e n t s t u d y . The e v a l u a t i o n a p p l i e d was a f o r m a t i v e e v a l u a t i o n . The o v e r a l l s t a t e m e n t o f worth a r r i v e d a t i n C h a p t e r I I I , f o r t h e p a r t i c u l a r program b e i n g e v a l u a t e d , r e p r e s e n t s an i n t e r i m c o n c l u s i o n and not a f i n a l c o n c l u s i o n on the worth o f t h e program. Data were c o l l e c t e d 6 representing good performance and bad performance. To present a balanced picture both types of data should be revealed; however, one of the functions of a formative evaluation is to provide information, in a summarized and judged (as to va l id i ty and worth) format so that program changes can be made to better f u l f i l l the program goals. To reveal only the good performance data and not the bad would be misleading; but to also present the bad would not contribute to the evaluation model being developed and could violate confidential i ty. Therefore the actual data have not been included in this study. F inal ly , the instruments developed for the evaluation of the program being evaluated are not generalizable. Since this is discussed in more detail in Chapter IV, i t wil l only be br ief ly noted here that the questionnaires are specif ic for the particular program. CHAPTER II The T h e o r e t i c a l Framework f o r an E v a l u a t i o n Model 2.00 I n t r o d u c t i o n C h a p t e r Two e x p l i c a t e s t h e t h e o r e t i c a l framework f o r e v a l u a t i o n . The f i r s t p a r t o f t h e c h a p t e r b r i e f l y examines a number o f commonly used a p p r o a c h e s t o e d u c a t i o n a l e v a l u a t i o n , and p o i n t s o u t t h e s h o r t -comings o f t h o s e a p p r o a c h e s . The second p a r t o f t h e C h a p t e r combines T a y l o r ' s (1961) l o g i c a l e x a m i n a t i o n o f e v a l u a t i v e p r o c e d u r e s w i t h S c r i v e n ' s v e r y p r a c t i c a l d e s c r i p t i o n s o f e v a l u a t i o n . The r e s u l t i n g amalgam p r o v i d e s , t h e s t r u c t u r e o f e v a l u a t i o n u n d e r l y i n g t h i s s t u d y . 2.10 P r e s e n t P i f f i c u l t i e s J n E v a l u a t i o n Many o f t h e . p r e s e n t l y used models o f e v a l u a t i o n have been d e v e l o p e d from an e x p e r i e n c e base. T a y l o r and M a g u i r e (1966, p. 12) wrote: "In many ways, t h e e f f i c i e n c y o f e v a l u a t i o n has been hampered by a l a c k o f c l e a r g u i d i n g p r i n c i p l e s i n d i c a t i n g what e v a l u a t i o n i s p o t e n t i a l l y a b l e t o a c c o m p l i s h . • What p r i n c i p l e s t h e r e a r e , e x i s t m a i n l y as e x p e r i e n t i a l l y i n d u c e d p r o c e d u r e s and t e c h n i q u e s h e l d by t h e few who work i n t h e a r e a . " The danger i n d e v e l o p i n g e v a l u a t i o n models from e x p e r i e n c e i s t h a t t h e meaning o f " e v a l u a t i o n " i s never examined. The r e s u l t i s a broad r a n g e o f models which p u r p o r t t o be d o i n g t h e same t h i n g , i . e . e v a l u a t i n g , b u t when examined a r e r e a l l y d o i n g something v e r y d i f f e r e n t such as d e s c r i b i n g , o r d a t a g a t h e r i n g , o r m easuring o r 7 8 s i m p l y r u b b e r s t a m p i n g . These may a l l be p r o c e d u r e s i n e v a l u a t i o n , but not one o f them can s i n g l y be l a b e l l e d as e v a l u a t i o n . The weakness o f e x p e r i e n t i a l l y based e v a l u a t i o n models, and t h e l a c k o f u n d e r s t a n d i n g o f t h e meaning o f e v a l u a t i o n has r e s u l t e d i n a g r e a t d e a l o f c r i t i c i s m o f " e v a l u a t i o n " as p r e s e n t l y a p p l i e d . On t h i s p o i n t Guba (1969, p. 29) noted t h a t : "The t r a d i t i o n a l methods o f e v a l u a t i o n have f a i l e d e d u c a t o r s i n t h e i r a t t e m p t s t o a s s e s s t h e impact o f i n n o v a t i o n s i n o p e r a t i n g systems. Indeed, t h e e v i d e n c e produced . . . has c o n t r a d i c t e d t h e e x p e r i e n t i a l e v i d e n c e o f t h e p r a c t i t i o n e r . I n n o v a t i o n s have p e r s i s t e d i n e d u c a t i o n n o t because o f the s u p p o r t i n g e v i d e n c e o f e v a l u a t i o n but d e s p i t e i t . " T h i s i n d i c t m e n t i s due t o t h e narrowness w i t h which some p r a c t i t i o n e r s d e f i n e e v a l u a t i o n . Guba i s s u g g e s t i n g t h a t many e v a l u a t i o n s a r e not complete o r comprehensive, and t h e y t e n d t o be l i m i t e d t o v a r i a b l e s which can be " s c i e n t i f i c a l l y " measured and s t a t i s t i c a l l y t r e a t e d . A n o t h e r c r i t i c i s m i s ". . . one o f t h e p r i m a r y l a t e n t f u n c t i o n s o f many e v a l u a t i o n s t u d i e s [ i s ] a l e g i t i m i z a t i o n p r o c e s s f o r p r e d e t e r -mined p o l i c i e s . " (House, 1973, p. 8 0 ) . House's c r i t i c i s m p o i n t s o u t one o f t h e most s e r i o u s o m i s s i o n s i n many " e v a l u a t i o n s " ; t h a t i s , f a i l i n g t o d e t e r m i n e whether t h e g o a l s o f t h e program a r e w o r t h w h i l e . The a c c e p t a n c e o f program g o a l s as b e i n g prima f a c i e w o r t h w h i l e g o a l s i s a major weakness i n many e v a l u a t i o n s . I n s t e a d o f l e g i t i m i z i n g p r e d e t e r m i n e d p o l i c i e s , t h e e v a l u a t o r s h o u l d be d e t e r m i n i n g i f t h e p r e d e t e r m i n e d p o l i c i e s a r e l e g i t i m a t e . 9 The u n c l e a r n a t u r e o f e v a l u a t i o n p r o c e d u r e s , and t h e l i n k i n g o f e v a l u a t i o n w i t h c r i t i c i s m has r e s u l t e d i n d e f e n s i v e p o s t u r i n g by t h o s e b e i n g e v a l u a t e d and by t h e e v a l u a t o r s . S t a k e (1967, p. 524) wrote t h a t t h e e d u c a t o r ' s d i s d a i n o f e v a l u a t i o n was due t o h i s s e n s i t i v i t y t o c r i t i c i s m . The e d u c a t o r uses smoke s c r e e n s s u c h as " n a t i o n a l norm c o m p a r i s o n s " and "academic freedom" t o a v o i d e x p o s u r e t h r o u g h e v a l u a t i o n . Provus (1970, p. 54) s u g g e s t e d t h a t t h e e d u c a t i o n a l a d m i n i s t r a t i o n has c o n t r i b u t e d t o t h e f a i l u r e o f e d u c a t i o n a l e v a l u a t i o n by, among o t h e r t h i n g s , not e x p l i c a t i n g h i s v a l u e a s s u m p t i o n s , o r by n o t s e t t i n g p e r f o r m a n c e s t a n d a r d s ; but goes on t o say t h a t p a r t o f t h e r e a s o n i s t h e u n c l e a r n a t u r e o f a n a l y s i s , r e s e a r c h and e v a l u a t i o n i n t h e problem s o l v i n g p r o c e s s . G l a s s (1975, p. 10) s t r o n g l y c r i t i c i z e s t h e e v a l u a t o r f o r r e t r e a t i n g t o t h r e e " s t r a w men" t e n e t s when under a t t a c k . The t e n e t s a r e i d e n t i f i e d as b e i n g t h e s a n c t i t y o f s c i e n c e , t h e p u b l i c ' s r i g h t t o know and a l l f e e d b a c k i s b e n e f i c i a l . In s p i t e o f h i s c r i t i c i s m , G l a s s says ( i b i d ) ". . . e v a l u a t i o n can f i n d a s t r o n g e r f a i t h , even i f t h e s cope o f i t s a p p l i c a t i o n i s somewhat r e d u c e d . " The c o n c e p t o f e v a l u a t i o n has been used to descrifee-many ;, d i f f e r e n t p r o c e d u r e s . Most o f t h e p r e s e n t models o f e v a l u a t i o n a r e d e r i v e d from c o n c e p t s o f e d u c a t i o n a l measurement. They a v o i d p u t t i n g v a l u e judgements on d a t a o r p r o d u c t s by u s i n g r i g o r o u s r e s e a r c h d e s i g n w hich s u p p o s e d l y i s " v a l u e f r e e " , o r by b u n d l i n g - u p t h e e v i d e n c e and g i v i n g i t t o someone e l s e t o j u d g e . B r i e f d e s c r i p -t i o n s o f t h e most common models o f e v a l u a t i o n f o l l o w i n s ubsequent 10 s e c t i o n s o f t h i s c h a p t e r . 2.11 T y l e r ' s Approach t o E v a l u a t i o n T y l e r f i r s t p r o p o s e d hisrrmodel f o r t h e e v a l u a t i o n o f l e a r n i n g e x p e r i e n c e s d u r i n g t h e 1930's. I t e v o l v e d f r o m h i s p a r t i c i -p a t i o n i n t h e , " E i g h t - y e a r Study" ( A i k e n , 1942) and was o r i g i n a l l y p u b l i s h e d i n 1949. H i s model i n v o l v e s t h e f o l l o w i n g s i x s t e p s ( T y l e r , 1942): 1) E s t a b l i s h t h e broad g o a l s o r o b j e c t i v e s o f t h e program. 2) C l a s s i f y t h e o b j e c t i v e s . 3) D e f i n e the o b j e c t i v e s i n b e h a v i o r a l t e r m s . 4) S uggest s i t u a t i o n s i n which a c h i e v e m e n t o f t h e o b j e c t i v e s can be shown; 5) D evelop or s e l e c t measurement t e c h n i q u e s , and, 6) G a t her s t u d e n t p e r f o r m a n c e d a t a and compare i t w i t h t h e b e h a v i o r a l l y - s t a t e d o b j e c t i v e s . Approaches s i m i l a r t o T y l e r ' s a r e f o u n d i n M e t f e s s e l ' s and M i c h a e l ' s (1967), "A Paradigm I n v o l v i n g M u l t i p l e C r i t e r i o n Measures f o r E v a l u a t i n g t h e E f f e c t i v e n e s s o f S c h o o l Programs", and W. James Popham's (1972) An E v a l u a t i o n Guidebook. A l l o f t h e s e methods would have the e v a l u a t o r l o o k i n g a l m o s t e x c l u s i v e l y a t t h e o b j e c t i v e s o f t h e program. The bases o f t h e o b j e c t i v e s a r e n o t t h e f o c a l p o i n t i n t h e e v a l u a t i o n . Popham (1972), i n h i s g u i d e b o o k , does m e n t i o n a needs assessment, but i t r e a l l y i s m a i n l y a r e s o u r c e s i n v e n t o r y and c l a r i f i c a t i o n o f p r e d e t e r m i n e d needs, 11 g o a l s o r o b j e c t i v e s . Welch used t h e T y l e r i a n a p p r o a c h when he e v a l u a t e d a c o u r s e c a l l e d e - ' T h y s i c a l S c i e n c e f o r t h e n o n - s c i e n c e S t u d e n t . " In h i s e v a l u a t i o n he was a t t e m p t i n g t o d e t e r m i n e i f the o b j e c t i v e s o f t h e c o u r s e had been a c h i e v e d . He c a l l s t h i s a summative e v a l u a t i o n (Welch, p. 140, 1972). But he omits d e t e r m i n i n g whether t h e c o u r s e o b j e c t i v e s a r e w o r t h w h i l e g o a l s , and hence, whether t h e program has m e r i t . 2.12 The A c c r e d i t a t i o n Model The f o l l o w i n g quote i s a t y p i c a l d e f i n i t i o n o f a c c r e d i t a t i o n : " A c c r e d i t a t i o n , as a p p l i e d i n e d u c a t i o n , i s t h e r e c o g n i t i o n a c c o r d e d t o an i n s t i t u t i o n t h a t meets t h e s t a n d a r d s o r c r i t e r i a e s t a b l i s h e d by a competent agency o r a s s o c i a t i o n . I t s g e n e r a l p u r p o s e i s t o promote and i n s u r e h i g h q u a l i t y i n e d u c a t i o n a l programs." (U.S. Dept. o f H e a l t h , E d u c a t i o n and W e l f a r e , 1959, p. 3) The a r t i c l e (1959, p. 4) goes on t o o u t l i n e f o u r s t e p s i n f u l l y d e v e l o p i n g a c c r e d i t i n g p r o c e d u r e s . They a r e : 1) E s t a b l i s h m e n t o f s t a n d a r d s o r c r i t e r i a . 2) I n s p e c t i o n o f i n s t i t u t i o n s by competent a u t h o r i t i e s t o d e t e r m i n e whether t h e y meet t h e e s t a b l i s h e d s t a n d a r d s o r c r i t e r i a . 3) P u b l i c a t i o n o f a l i s t o f i n s t i t u t i o n s t h a t meet t h e s t a n d a r d s o r c r i t e r i a . 4) P e r i o d i c r e v i e w s t o a s c e r t a i n whether a c c r e d i t e d i n s t i t u -t i o n s c o n t i n u e t o meet t h e s t a n d a r d s o r c r i t e r i a . . A c c r e d i t a t i o n a p p e a r s t o be v e r y s i m i l a r t o e v a l u a t i o n , 12 p a r t i c u l a r l y t h e f i r s t two s t e p s above. Howeverj t h e i n v a l i d a s s u m p t i o n made i n a c c r e d i t a t i o n i s t h a t t h e f u l f i l l m e n t o f t h e s t a n d a r d s and c r i t e r i a i n S tep 1 n e c e s s a r i l y b e n e f i t t h e l e a r n e r i n t h e i n s t i t u t i o n . As G l a s s r i g h t l y p o i n t s o u t ; "The a c c r e d i t a t i o n c r i t e r i a r e f l e c t t h e i n t e r e s t s o f a d m i n i s t r a t o r s ; a t t e n t i o n i s g i v e n t o the p r o c e s s e s o r means o f e d u c a t i o n and i s opposed t o i t s consequences on l e a r n e r s . " ( G l a s s , 1969, p. 20) 2.13 S t u f f l e b e a m ' s C o n t e x t , I n p u t , P r o c e s s and P r o d u c t (CIPP) Model, In t h i s model, e v a l u a t i o n i s d e f i n e d a s , ". . . t h e p r o v i s i o n o f i n f o r m a t i o n t h r o u g h f o r m a l means, such as c r i t e r i a , measurement, and s t a t i s t i c s , t o s e r v e as r a t i o n a l bases f o r making judgements i n d e c i s i o n s i t u a t i o n s . " ( S t u f f l e b e a m , 1958, p. 6) In o t h e r words, t h e e v a l u a t o r i s e s s e n t i a l l y an appendage t o t h e d e c i s i o n - m a k e r . T h i s d e f i n i t i o n i s o p e r a t i o n a l i z e d t h r o u g h t h e CIPP model, about which worthen (1968, p. 3) w r i t e s : " D e c e p t i v e l y s i m p l e when viewed i n o u t l i n e f o r m , t h e s t r u c t u r e c o n s i s t s o f a l i s t o f no l e s s t h a n twenty-two d e c i s i o n s i t u a t i o n s which a r e common to most e v a l u a t i o n d e s i g n s . Upon even c u r s o r y i n s p e c t i o n , however, i t i s a p p a r e n t t h a t t h e s t r u c t u r e i m p l i e s much more than a s i m p l e l i s t o f twenty-two items which e v a l u a t o r s must keep i n mind, f o r a t each o f t h e d e c i s i o n p o i n t s i d e n t i f i e d i n t h e s t r u c t u r e , t h e e v a l u a t o r needs t o make a c h o i c e among t h e a v a i l a b l e a l t e r n a t i v e s . " S t u f f l e b e a m i d e n t i f i e s f o u r c l a s s e s o f d e c i s i o n s : p l a n n i n g , programming, i m p l e m e n t i n g , and r e - c y c l i n g ; and s u g g e s t s f o u r 13 e v a l u a t i o n s t r a t e g i e s f o r t h o s e d e c i s i o n s . The s t r a t e g i e s a r e : a) C o n t e x t e v a l u a t i o n ^ : w hich i d e n t i f i e s t h e g o a l s and needs to be f u l f i l l e d by t h e program, and t h e environment where t h e program w i l l o p e r a t e . b) I n p u t e v a l u a t i o n : which i d e n t i f i e s and a s s e s s e s a l t e r n a t i v e p r o c e d u r e s and d e s i g n s f o r a t t a i n i n g program o b j e c t i v e s . c) P r o c e s s e v a l u a t i o n s which m o n i t o r s t h e on - g o i n g program f o r problems o r d e v i a t i o n s f r o m program d e s i g n . d) Product, e v a l u a t i o n ^ which i s summative e v a l u a t i o n , and d e t e r m i n e s t h e e x t e n t t o which t h e g o a l s and o b j e c t i v e s have been met. Each s t r a t e g y i s th e n s u b - d i v i d e d i n t o twenty-two c a t e g o r i e s - -t h e same f o r each s t r a t e g y — t o g i v e t h e e v a l u a t i o n d e s i g n . T h i s i s i l l u s t r a t e d i n F i g u r e 1 (Worthen 1968, p. 4 ) . T h e r e a r e two o b j e c t i o n s t o t h i s model. The f i r s t i s a p r a c t i c a l c o n s i d e r a t i o n . T h e r e c o u l d be as many as 88 d e c i s i o n s s i t u a t i o n s ( f o u r c l a s s e s o f d e c i s i o n s w i t h twenty-two d e c i s i o n s i t u a t i o n s i n each c l a s s ) w h i c h s e e m i n g l y would t a k e an army o f d a t a g a t h e r e r s t o f u l f i l l . The second o b j e c t i o n i s t h a t use o f t h e CIPP model would f a i l t o p r o v i d e an o v e r a l l e v a l u a t i o n . The e v a l u a t o r u s i n g t h e CIPP model would p r o v i d e i n f o r m a t i o n t o d e c i s i o n makers, b u t would make no d e c i s i o n on t h e worth o f t h a t i n f o r m a t i o n o r t h e worth o f t h e p r o d u c t b e i n g e v a l u a t e d . As G l a s s (1969, p. 36) wrote: "B e i n g o f a s s i s t a n c e t o t h e program p e r s o n n e l — s o t h e y may b e t t e r c o n d u c t t h e i r b u s i n e s s — i s a p r o x i m a t e aim o f e v a l u a t i o n ; t h e u l t i m a t e a im o f an e v a l u a t i o n i s t o d e c i d e q u e s t i o n s o f w o r t h . " 14 F i g u r e 1 D e v e l o p i n g E v a l u a t i o n D e s i g n s The l o g i c a l s t r u c t u r e o f e v a l u a t i o n d e s i g n i s t h e same f o r a l l t y p e s o f e v a l u a t i o n , whether c o n t e x t , i n p u t , p r o c e s s o r p r o d u c t e v a l u a t i o n . The p a r t s , b r i e f l y , a r e as f o l l o w s : A. F o c u s i n g t h e E v a l u a t i o n 1. I d e n t i f y t h e major l e v e l ( s ) o f d e c i s i o n - m a k i n g t o be s e r v e d , e.g., l o c a l , s t a t e , o r n a t i o n a l . 2. F o r each l e v e l o f d e c i s i o n - m a k i n g , p r o j e c t t h e d e c i s i o n ; s i t u a t i o n s t o be s e r v e d and d e s c r i b e each one i n terms o f i t s l o c u s , ? f o c u s , t i m i n g , and c o m p o s i t i o n o f a l t e r n a t i v e s . 3. D e f i n e c r i t e r i a f o r each d e c i s i o n s i t u a t i o n by s p e c i f y i n g v a r i a b l e s f o r measurement and s t a n d a r d s f o r use i n t h e judgment o f a l t e r n a t i v e s . 4. D e f i n e p o l i c i e s w i t h i n w hich t h e e v a l u a t i o n must o p e r a t e . B. C o l l e c t i o n o f I n f o r m a t i o n 1. S p e c i f y t h e s o u r c e o f t h e i n f o r m a t i o n t o be c o l l e c t e d . 2. S p e c i f y t h e i n s t r u m e n t s and methods f o r c o l l e c t i n g t h e needed i n f o r m a t i o n . 3. S p e c i f y t h e sam p l i n g p r o c e d u r e t o be employed. 4. S p e c i f y t h e c o n d i t i o n s and s c h e d u l e f o r i n f o r m a t i o n c o l l e c t i o n . C. O r g a n i z a t i o n o f I n f o r m a t i o n 1. S p e c i f y a f o r m a t f o r t h e i n f o r m a t i o n which i s t o be c o l l e c t e d . 2. S p e c i f y a means f o r c o d i n g , o r g a n i z i n g , s t o r i n g , and r e t r i e v i n g i n f o r m a t i o n . D. A n a l y s i s o f I n f o r m a t i o n 1. S p e c i f y t h e a n a l y t i c a l p r o c e d u r e s t o be employed. 2. S p e c i f y a means f o r p e r f o r m i n g t h e a n a l y s i s . E. R e p o r t i n g o f I n f o r m a t i o n 1. D e f i n e t h e a u d i e n c e s f o r t h e e v a l u a t i o n r e p o r t s . 2. S p e c i f y means f o r p r o v i d i n g i n f o r m a t i o n ^ t o i ; t h e . a u d i e n c e s . 3. S p e c i f y * t h e f o r m a t " f o r e v a l u a t i o n r e p o r t s and/or r e p o r t i n g s e s s i o n s . 4. S c h e d u l e t h e r e p o r t i n g o f i n f o r m a t i o n . F. A d m i n i s t r a t i o n o f t h e e v a l u a t i o n 1. Summarize t h e e v a l u a t i o n s c h e d u l e . 2. D e f i n e s t a f f and r e s o u r c e r e q u i r e m e n t s and p l a n s f o r m e e t i n g t h e s e r e q u i r e m e n t s . 3. S p e c i f y means f o r meeting p o l i c y r e q u i r e m e n t s f o r c o n d u c t o f t h e e v a l u a t i o n . 4. E v a l u a t e t h e p o t e n t i a l o f t h e i n f o r m a t i o n d e s i g n f o r p r o v i d i n g i n f o r m a t i o n which i s v a l i d , r e l i a b l e , c r e d i b l e , t i m e l y , and p e r v a s i v e . .5. S p e c i f y and s c h e d u l e means f o r p e r i o d i c u p d a t i n g o f t h e e v a l -u a t i o n d e s i g n . 6. P r o v i d e a budget f o r t h e t o t a l e v a l u a t i o n program. 15 I t i s p r e c i s e l y t h e q u e s t i o n o f worth t h a t e v a l u a t o r s u s i n g t h e CIPP model a v o i d . Provus (1969) and A l k i n ' s (UCLA, 1968) models both c l o s e l y r e s e m b l e S t u f f l e b e a m ' s CIPP model. S c r i v e n (1972) s t r o n g l y y c r i t i c i z e d t h e CIPP model. He s t a t e d t h a t what i s c a l l e d e v a l u a t i o n i n t h e CIPP model i s not e v a l u a t i o n a t a l l . C o n t e x t e v a l u a t i o n i s e s s e n t i a l l y a m a r k e t - s u r v e y , i n p u t e v a l u a t i o n i s a s u r v e y o f r e s o u r c e o p t i o n s , p r o c e s s e v a l u a t i o n i s u s u a l l y s o c i a l m o n i t o r i n g and b o o k k e e p i n g , and p r o d u c t e v a l u a t i o n appears t o be a d i l u t e m i x t u r e o f summative and f o r m a t i v e e v a l u a t i o n - - c l a i m i n g t o do b o t h , but a c t u a l l y , not d o i n g e i t h e r ( I b i d , p. 134-135). S c r i v e n ' s most g e n e r a l c r i t i c i s m o f t h e CIPP model i s ( I b i d , p. 36); " [ t h e CIPP model i s ] about t h e most c o m p l i c a t e d and c o n f u s i n g way o f a n a l y s i n g t h e p r a c t i c a l p r o c e d u r e s o f e v a l u a t i o n t h a t I can imagine, and i t ' s c e r t a i n l y t h e most c o m p l i c a t e d one t h a t I've e v e r seen." 2.14 S t a k e ' s Model In 1967 Sta k e p r o p o s e d h i s e v a l u a t i o n model ( S t a k e , 1967). The model f o c u s s e d on t h e d e s c r i p t i o n and judgement o f e d u c a t i o n a l programs. The d e s c r i p t i v e p a r t i s d i v i d e d i n t o " i n t e n t s " ( t h e g o a l s and/or o b j e c t i v e s ) and o b s e r v a t i o n s (what t h e e v a l u a t o r l e a r n s t h r o u g h e m p i r i c a l m e a s u r e s ) . The judgement m a t r i x i s a l s o d i v i d e d i n t o two p a r t s — s t a n d a r d s and judgements. The s t a n d a r d s can e i t h e r be a b s o l u t e ( i . e . c r i t e r i o n l e v e l s ) o r r e l a t i v e ( i . e . compared t o o t h e r , s i m i l a r p r o g r a m s ) , but must p e r t a i n t o t h e t h i n g b e i n g 16 e v a l u a t e d . The judgements a r e t h e a s s i g n i n g o f w e i g h t s t o t h e s t a n d a r d s and j u d g i n g t h e m e r i t o f t h e p r o d u c t under c o n s i d e r a t i o n . S t a k e a l s o says t h a t t h e program s h o u l d have a r a t i o n a l e ; but i t i s not c l e a r from h i s paper (The Countenance o f E d u c a t i o n a l E v a l u a t i o n , S t a k e , 1967), whether t h e e v a l u a t o r makes any judgements o f t h e r a t i o n a l e . He says "The e v a l u a t o r asks h i m s e l f o r o t h e r j u d g e s whether t h e p l a n d e v e l o p e d by t h e e d u c a t o r c o n s t i t u t e s a l o g i c a l s t e p i n t h e i m p l e m e n t a t i o n o f the b a s i c p u r p o s e s . " ( S t a k e , 1967, p. 1 3 ) . However, t h i s i s n o t a judgement o f t h e worth o f t h e program g o a l s as t h e y have been o u t l i n e d i n t h e program's r a t i o n a l e - -r a t h e r , i t i s a judgement o f t h e l o g i c o f t h e i m p l e m e n t a t i o n p l a n s . In an a p p l i c a t i o n o f h i s model Stake (1971, p. 4) w r i t e s : " E v a l u a t o r s have an o b l i g a t i o n t o r a i s e t h e q u e s t i o n s , 'Were the r i g h t g o a l s p u r s u e d ? ' D i f f e r e n t p e o p l e have d i f f e r e n t i d e a s , o f c o u r s e , as t o what t h e r i g h t g o a l s a r e . S t i l l , g o a l s and p r i o r i t i e s s h o u l d be e v a l u a t e d . " He goes on t o s a y , about t h e p a r t i c u l a r p r o j e c t he i s e v a l u a t i n g "In t h e eyes o f t h e e v a l u a t o r s (my e m p h a s i s ) , t h e T c i t y g o a l s a r e worthy g o a l s , s u i t a b l y d i s c u s s e d and r e a s o n a b l y o p e r a t i o n a l i z e d . " ( S t a k e , 1971, p. 4) However, t h e l o g i c a l c o n n e c t i o n between t h e g o a l s and worth o f t h e g o a l s has s t i l l n ot been e x p l i c a t e d , f o r one can s t i l l a s k , "The g o a l s a r e w o r t h w h i l e i n h i s e y e s , but a r e t h e y r e a l l y w o r t h w h i l e ? " 2.15 Summary The problem o f d e c i d i n g what one s h o u l d do when one \ 17 e v a l u a t e s s p r i n g s from an i m p r e c i s e i d e a o f what t h e word " e v a l u a t i o n " means. T h i s has caused many o f t h e s h o r t c o m i n g s o f t h e models d e s c r i b e d above. U n f o r t u n a t e l y , i n none o f the r e c e n t c o m p i l a t i o n s o f e v a l u a t i o n t h e o r y (Popham; 1974, wbrthen & S a n d e r s , 1973; T y l e r , 1969) i s t h e r e a s e r i o u s e f f o r t t o d e a l w i t h t h i s p roblem. T h e r e f o r e the r e s t o f t h i s C h a p t e r w i l l be an a t t e m p t t o p r o v i d e a l o g i c a l base f o r t h e meaning and use o f t h e word " e v a l u a t i o n " . 2.20 E v a l u a t i o n : The T h e o r e t i c a l Framework "The p r o c e s s o f e v a l u a t i o n c o n s i s t s i n t r y i n g t o d e t e r m i n e t h e v a l u e o f s o m e t h i n g . As a p r o d u c t o r outcome o f t h a t p r o c e s s , an e v a l u a t i o n i s a s e t t l e d o p i n i o n t h a t something has a c e r t a i n v a l u e . " ( T a y l o r , 1961, p. 3) T a y l o r ' s views a r e p a r a l l e l e d by S c r i v e n (1967), whose c o n t r i b u -t i o n i s t o p r o v i d e a m e t h o d o l o g i c a l framework f o r t h e a n a l y t i c a l t r e a t m e n t . S c r i v e n c o n s i d e r s t h a t the goal o f e v a l u a t i o n i s t o d e t e r m i n e the m e r i t o r worth o f some e n t e r p r i s e , and s u g g e s t s we can a t t a i n t h a t goal by g a t h e r i n g and c o m bining needs r e l a t e d p e r f o r m a n c e d a t a , w e i g h i n g and c o m bining t h a t - d a t a and j u s t i f y i n g o u r c o n c l u s i o n o f worth by showing t h a t o u r d a t a - g a t h e r i n g i n s t r u m e n t s and s e l c t i o n o f c r i t e r i a a r e v a l i d ( S c r i v e n , 1967, p. 40-41). T a y l o r (1961, p. 9, 10) has i d e n t i f i e d f i v e s t e p s i n e v a l u a t i o n . To t h e s e f i v e s t e p s one c o u l d add a p r e l i m i n a r y s e t o f p r e c o n d i t i o n s t o c l a r i f y t h e f i v e s t e p s which f o l l o w . T h e r e f o r e an 18 e v a l u a t i o n on l o g i c a l grounds r e q u i r e s : 1) The P r e c o n d i t i o n s : a) I d e n t i f y t h e evaluatum : (what i t i s t h a t i s b e i n g e v a l u a t e d ) . _ "\ • . ) b) D e c i d e on whether you a r e g o i n g t o : ( i ) Grade a c c o r d i n g t o s t a n d a r d s ( S - G r a d i n g ) ( t h a t i s e v a l u a t i n g i n terms o f some a b s o l u t e s t a n d a r d s ) ; ( i i ) Rank ( t h a t i s S-Grade and t h e n compare and rank o r d e r t h e evaluatum i n a comparison group o f s i m i l a r e v a l u a t a which have a l s o been S-Graded); ( i i i | G r a d e a c c o r d i n g t o r u l e s ( R - G r a d i n g ) ( t h a t i s g r a d e t h e evaluatum as o b e y i n g o r not o b e y i n g a s e t o f a p p l i c a b l e r u l e s ) c) Determine t h e P o i n t o f View o r P o i n t s o f View. 2) A d o p t i o n o f a s t a n d a r d o r r u l e , o r s e t o f s t a n d a r d s o r r u l e s , f o r e v a l u a t i n g t h e e v a l u a t u m . 3) O p e r a t i o n a l c l a r i f i c a t i o n o f t h e s t a n d a r d s o r r u l e s . 4) S p e c i f i c a t i o n o f the c l a s s o f c o m p a r i s o n . 5) D e t e r m i n i n g t h e good o r bad c h a r a c t e r i s t i c s o f t h e e v a l u a t u m . 6) D e d u c i n g , from (3) and ( 5 ) , t h e degree t o which t h e evaluatum on t h e whole f u l f i l l s o r f a i l s t o f u l f i l l t h e s t a n d a r d s ; o r , i n t h e c a s e o f r u l e s , t h e r e l a t i v e i m p o r t a n c e o f t h e r u l e s t h e e v a l u a t u p p a s s e s o r f a i l s (A r u l e can e i t h e r be obeyed o r v i o l a t e d . 19 T h e r e f o r e t h e r e a r e no d e g r e e s o f ' c o m p l i a n c e o r v i o l a t i o n . ) These s i x s t e p s w i l l not;be d i s c u s s e d i n more d e t a i l . 2.30 The P r e c o n d i t i o n s t o an E v a l u a t i o n I f an e v a l u a t o r i s t o p e r f o r m an e v a l u a t i o n , t h e n t h r e e p r e -c o n d i t i o n s must be s a t i s f i e d . The f i r s t o f t h e s e i s t h a t t h e r e must be something t o e v a l u a t e ( i . e . , an e v a l u a t u m ) ; t h e s e c o n d i s t h a t t h e e v a l u a t o r must adopt a p o i n t o f view, and t h e t h i r d i s t h a t t h e e v a l u a t o r must d e c i d e i f he i s g o i n g t o grade o r r a n k a c c o r d i n g t o r u l e s o r s t a n d a r d s . 2.31 The Evaluatum T h a t which i s t o be e v a l u a t e d has been l a b e l l e d by T a y l o r as t h e evaluatum ( T a y l o r , p. 4, p. 23) and i s a n y t h i n g which can be r a n k e d o r g r a d e d ( d i s c u s s e d l a t e r i n t h i s c h a p t e r ) , o r which has bad and/or good-making c h a r a c t e r i s t i c s , t o be d e t e r m i n e d by t h e e v a l u a t o r . For example, t h e e v a l u a t u m f o r t h i s t h e s i s i s a t e a c h e r -t r a i n i n g program a t U.B.C. 2.32 P o i n t o f View T a y l o r says t h a t : " T a k i n g a c e r t a i n p o i n t o f view i s n o t h i n g but a d o p t i n g c e r t a i n canons o f r e a s o n i n g as t h e framework w i t h i n which v a l u e judgements a r e t o be j u s t i f i e d ; t h e canons o f r e a s o n i n g d e f i n e t h e p o i n t o f view." ( T a y l o r , 1961, p. 109) and 20 " T h i s means p r e c i s e l y t h a t t h e j u d g e o r e v a l u a t o r has a d o p t e d a s e t o f r u l e s o f r e l e v a n c e and v a l i d i n f e r e n c e t h a t r e c o g -n i z e o n l y c e r t a i n r e a s o n s as r e l e v a n t and good." ( T a y l o r , 1961, p. 109) The above e x p l a n a t i o n s were e x e m p l i f i e d by T a y l o r as f o l l o w s : "We c a n n o t e v a l u a t e a c a r as good o r bad, o r as b e t t e r o r worse t h a n a n o t h e r c a r , u n l e s s we know what p o i n t o f view we.-are t o t a k e ... Is i t t o be e v a l u a t e d as a p i e c e o f m e c h a n i c a l e n g i n e e r i n g ? As an o b j e c t o f a e s t h e t i c c o n t e m p l a t i o n ? . . . Each p o i n t o f view we t a k e makes a d i f f e r e n c e , s i n c e d i f f e r e n t s t a n d a r d s (norms) a r e a p p r o p r i a t e t o d i f f e r e n t p o i n t s o f view." ( T a y l o r , 1961, p. 5, 6) I t would be unusual t o be a b l e t o d e c i d e t h e o v e r a l l worth o f any program by e v a l u a t i n g i t from o n l y one p o i n t o f view. Even i n T a y l o r ' s example above, a c a r e f u l s hopper would not buy an a u t o m o b i l e o n l y on t h e b a s i s o f m e c h a n i c a l e n g i n e e r i n g , o r o n l y on t h e b a s i s o f a e s t h e t i c c o n t e m p l a t i o n . Nor i s i t r e a s o n a b l e t o compare t e a c h e r -t r a i n i n g programs o n l y on t h e b a s i s o f t h e i r o r g a n i z a t i o n , o r o n l y on t h e b a s i s o f t h e i r r e l a t i v e c o s t s . A l l r e a s o n a b l e p o i n t s o f view must be t a k e n i n t o c o n s i d e r a t i o n . T a y l o r has i d e n t i f i e d e i g h t b a s i c p o i n t s o f view i n a c i v i l i z e d c u l t u r e ; t h e y a r e : t h e m o r a l , t h e a e s t h e t i c , t h e s c i e n t i f i c , t h e r e l i g i o u s , t h e economic, t h e p o l i t i c a l , t h e l e g a l , and t h e p o i n t o f view o f e t i q u e t t e o r custom. He d e r i v e s t h e s e p o i n t s o f view from t h e e i g h t c o r r e s p o n d i n g major i n s t i t u t i o n s and a c t i v i t i e s p l u s t h e e d u c a t i o n a l i n s t i t u t i o n s ; but as he p o i n t s o u t : "There i s no s i n g l e p o i n t o f view c o r r e s p o n d i n g t o t h e e d u c a t i o n a l i n s t i t u t i o n s o f a s o c i e t y , s i n c e e d u c a t i o n i s a p r o c e s s which may t a k e p l a c e w i t h i n any p o i n t o f view. Thus t h e r e i s moral e d u c a t i o n , a e s t h e t i c e d u c a t i o n , i n t e l l e c t u a l e d u c a t i o n , r e l i g i o u s e d u c a t i o n , and so on." (1961, p. 300) 21 T a y l o r ' s r e a s o n i n g does not sound t h e d e a t h k n e l l f o r t h e e v a l u a t i o n o f e d u c a t i o n a l i n s t i t u t i o n s . I t s i m p l y a l e r t s t h e e v a l u a t o r t o t h e m u l t i p l i c i t y o f p o i n t s o f view which must be a t l e a s t c o n s i d e r e d i n t h e e v a l u a t i o n . F o r example, i s t h e t e a c h e r -t r a i n i n g program immoral (moral p o i n t o f v i e w ) , c r a s s ( a e s t h e t i c p o i n t o f v i e w ) , i l l e g a l " ( l e g a l p o i n t o f v i e w ) , e x p e n s i v e (economic p o i n t o f v i e w ) , and so on. 2.33 G r a d i n g o r Ranking The t h i r d p r e - c o n d i t i o n i s t o d e c i d e what form t h e e v a l u a t i o n w i l l t a k e . T h e r e a r e t h r e e p o s s i b i l i t i e s : g r a d i n g a c c o r d i n g t o s t a n d a r d s , g r a d i n g a c c o r d i n g t o r u l e s , and r a n k i n g „. a c c o r d i n g t o s t a n d a r d s . D a n i e l s (1971, p. 6) g i v e s a p a r t i c u l a r l y l u c i d a c c o u n t o f t h e s e t h r e e f o r m s , so h i s d e s c r i p t i o n i s r e p r o d u c e d i n i t s e n t i r e t y below: "Form 1 G r a d i n g a c c o r d i n g t o s t a n d a r d s . ( L e t us c a l l t h i s S - g r a d i n g . ) Form 2 G r a d i n g a c c o r d i n g t o r u l e s . ( L e t us c a l l t h i s R - g r a d i n g . ) Form 3 R a n k i n g . B e f o r e we see how t h e s e t h r e e f o r m s , S - g r a d i n g , R - g r a d i n g and R a n k i n g , c o u l d be used t o e v a l u a t e c u r r i c u l a , l e t us l o o k a t each i n a somewhat s i m p l e r c o n t e x t . S - g r a d i n g : Suppose we a r e j u d g i n g c a r s f o r t h e i r c o m f o r t . I t makes se n s e t o . s a y . t h a t Car 1 i s u n c o m f o r t a b l e , Car 2 i s f a i r l y c o m f o r t a b l e , Car 3 i s c o m f o r t a b l e , Car 4 i s v e r y c o m f o r t a b l e and Car 5 i s e x t r e m e l y c o m f o r t a b l e . This-shows us t h e two main f e a t u r e s o f S - g r a d i n g : a) The c r i t e r i o n we use as t h e b a s i s f o r o u r e v a l u a t i o n i s a s t a n d a r d , b) T h i n g s can f u l f i l l s t a n d a r d s i n d i f f e r i n g d e g r e e s . Thus, we can e v a l u a t e t h i n g s on a s i m p l e t w o - p a r t s c a l e ( e . g . , c o m f o r t a b l e o r u n c o m f o r t a b l e ) o r on a m u l t i p l e - l e v e l s c a l e (as i n o u r example o f t h e f i v e c a r s ) . R - g r a d i n g : Suppose, however, t h a t we a r e c o n c e r n e d t o d i s c o v e r whether o r n o t t h e h e a d l i g h t s on o u r c a r s a r e c o r r e c t l y a d j u s t e d as s e t down by a law. Now we a r e e v a l u a t i n g a c c o r d i n g t o r u l e s and t h e n o t i o n o f d e g r e e s i s not a p p l i c a l b e ; e i t h e r t h e h e a d l i g h t s conform w i t h t h e r u l e s o r t h e y do n o t ; t h e y a r e r i g h t o r wrong, c o r r e c t o r i n c o r r e c t . 22 Ranking: In o r d e r t o rank t h i n g s we must go t h r o u g h two s t e p s : a) we f i r s t o f a l l S-grade t h e t h i n g s t o be r a n k e d , as we d i d f o r i n s t a n c e w i t h t h e f i v e c a r s , b) We then compare the degree t o which the t h i n g s f u l f i l l t h e s t a n d a r d ( R - g r a d i n g c a n n o t be used f o r R a n k i n g ) , and rank them. Thus, i n r a n k i n g o u r c a r s from t h e p o i n t o f view o f c o m f o r t , we might say something l i k e t h e f o l l o w i n g ; Car 1 - w o r s t , Car 3 - a v e r a g e , Car 5 - b e s t . " I t i s n o t u n r e a s o n a b l e t o s u g g e s t t h a t any e d u c a t i o n a l p r o d u c t can undergo a l l t h r e e forms, R - g r a d i n g , S - g r a d i n g and R a n k i n g , i n i t s e v o l u t i o n from an i n i t i a l i n s p i r a t i o n t o the f i n i s h e d p r o d u c t . For example, suppose a group o f p r o f e s s o r s w i s h e d t o i n i t i a t e a t e a c h e r - t r a i n i n g program based on a p a r t i c u l a r model o f t e a c h e r t r a i n i n g . F i r s t l y , t h e y would have t o d e t e r m i n e , among o t h e r t h i n g s , i f t h e i r p r o p o s e d program ( t h e evaluatum) was f e a s i b l e i n terms o f the u n i v e r s i t y . In o t h e r words, would t h e s t u d e n t s be t a u g h t the r e q u i r e d number o f c o u r s e s ( s p e c i f i e d number o f u n i t s ) ; do t h e p r o -s p e c t i v e s t u d e n t s have t h e r e q u i r e d minimum e n t r a n c e q u a l i f i c a t i o n s ( f o r t h i s p a r t i c u l a r y e a r o f u n i v e r s i t y s t u d y ) ; w i l l t h e p r o f e s s o r s ' t e a c h i n g r e s p o n s i b i l i t i e s be a t l e a s t t h e minimum r e q u i r e d ? These a r e r u l e s which a r e e i t h e r f u l f i l l e d o r n o t f u l f i l l e d . In the example g i v e n , t h i s f i r s t s t e p , though u s u a l l y i n f o r m a l l y done, c o u l d be i n t e r p r e t e d as R-GRADING. Once t h e program has been t h r o u g h i n f o r m a l R - g r a d i n g , i t i s o p e r a t i o n a l i z e d and i s no l o n g e r a p r o p o s a l . The program i s i n a d e v e l o p m e n t a l p e r i o d o r f o r m a t i v e s t a g e . The main c o n c e r n o f the program p r o p o n e n t s i s whether o r n o t the s t a n d a r d s t h e y s e t f o r the program a r e b e i n g met. They ask: " I s the o p e r a t i o n a l program the same as t h a t program we o r i g i n a l l y c o n c e p t u a l i z e d ? " The s t a n d a r d s t o be used i n t h e e v a l u a t i o n come from t h i s v e r s i o n . V a r i o u s c h a r a c t e r -i s t i e s o f t h e program a r e t h e n graded a c c o r d i n g t o t h o s e s t a n d a r d s . S t a k e (1967) has pr o p o s e d t h i s method when he d i s c u s s e s j u d g i n g t h e c h a r a c t e r i s t i c s o f a program w i t h r e s p e c t t o a b s o l u t e s t a n d a r d s . He sa y s t h a t each s e t o f a b s o l u t e s t a n d a r d s , when f o r m a l i z e d , would i n -d i c a t e a c c e p t a b l e and m e r i t o r i o u s l e v e l s o f pe r f o r m a n c e f o r each c h a r a c t e r i s t i c . T h i s f o r m a l i z a t i o n i s i l l u s t r a t e d i n C h a p t e r I I I o f t h i s t h e s i s . The p r o c e s s o f d e t e r m i n i n g whether a program i s good o r bad w i t h r e s p e c t t o a b s o l u t e s t a n d a r d s i s S-GRADING. The f i n a l s t a g e i n the program's e v a l u a t i o n , i s t o d e t e r m i n e how i t compares t o t h e o t h e r , s i m i l a r t e a c h e r - t r a i n i n g programs a v a i l a b l e ( i . e . t o the members o f the c l a s s o f c o m p a r i s o n ) . T h e r e a r e two c r i t e r i a which must be t a k e n i n t o a c c o u n t t o d e c i d e whether t he evaluatum i s b e t t e r o r worse than something e l s e , o r i s t h e b e s t o r w o r s t i n the c l a s s o f c o m p a r i s o n s . They a r e : " . . . whether t he evaluatum i s good on the whole, o r bad on t he whole, when graded a c c o r d i n g t o t h e g i v e n s t a n d a r d s , and whether i t s o v e r a l l goodness o r badness outweighs t he o v e r a l l goodness o r badness o f each member o f the c l a s s o f c o m p a r i s o n . " ( T a y l o r , 1961, p. 15, 16) Of c o u r s e , each member o f t h e c l a s s o f comparison has a l s o been g r a d e d a c c o r d i n g t o t h e same s e t o f s t a n d a r d s . T h e r e f o r e , an ev a l u a t u m can be graded as "bad" on the whole, y e t be b e t t e r than any o f the members o f t h e c l a s s o f c o m p a r i s o n s , o r t h e b e s t o f a bad l o t . Or, s i m i l a r l y , i t c o u l d be gr a d e d as good, and ranked as a v e r a g e , e t c . The p r o c e s s o f d e t e r m i n i n g t he c o m p a r a t i v e v a l u e o f a program i s c a l l e d RANKING. 24 S c r i v e n has argued t h a t we s h o u l d d e t e r m i n e t h e s u p e r i o r i t y o r i n f e r i o r i t y o f the e v a l u a t u m t o t h e c o m p e t i t i o n b e f o r e we have com-p l e t e l y e v a l u a t e d t h e evaluatum. ( S c r i v e n , 1967) In o t h e r words we must have gone th r o u g h a p r o c e s s o f RANKING. In a l a t e r a r t i c l e he p r e s e n t e d a taxonomy o f d i f f e r e n t e v a l u a t i v e c o n c l u s i o n s , ( S c r i v e n , 1974a, p. 146) p r e s e n t e d here as T a b l e 1. In p r a c t i s e , a p a r t i c u l a r e v a l u a t i v e c o n c l u s i o n i s aimed f o r ( o r a r r i v e d a t ) somewhere between a " P r e e v a l u a t i v e , " and a "Best-Buy" e v a l u a t i v e c o n c l u s i o n . A n y t h i n g l e s s t h a n a "Best-Buy e v a l u a t i v e c o n c l u s i o n " , c o n s t i t u t e s l e s s than a c omplete e v a l u a t i o n ; t h e r e f o r e i t i s i m p o r t a n t t h a t the c l i e n t f o r the e v a l u a t i o n knows e x a c t l y what s o r t o f e v a l u a t i o n he i s g e t t i n g , and how much i t f a l l s s h o r t o f a c o mplete e v a l u a t i o n ( i f , i n d e e d , i t d o e s ) . An e v a l u a t i v e c o n c l u s i o n o f "Commendatory" i n S c r i v e n ' s t a x -onomy, i s t h e h i g h e s t l e v e l p o s s i b l e u s i n g S - g r a d i n g . The c o n t r o l group i m p l i e d i n t h e , " u s u a l l y adequate p r e m i s e s " o f T a b l e 1, i s the i d e a l f o r m o f the evaluatum. I t i s a g a i n s t the s t a n d a r d s o f t h a t i d e a l t h a t t h e evaluatum i s g r a d e d . The l a s t t h r e e e v a l u a t i v e c o n c l u s i o n s : L a u d a t o r y , I d e a l , and B est-Buy, i n v o l v e r a n k i n g t h e e v a l u a t a , o r making o v e r a l l c o m p a r a t i v e s t a t e m e n t s . T h e r e f o r e , t h e s e t h r e e c o n c l u s i o n s i l l u s t r a t e i n c r e a s -i n g l y comprehensive RANKING. T h i s p a r t i t i o n i n g o f S c r i v e n ' s taxonomy does not e x c l u d e t h e f i r s t f o u r e v a l u a t i v e c o n c l u s i o n s ( P r e e v a l u a t i v e , Minimal e v a l u a t i v e , O v e r a l l e v a l u a t i v e and Commendatory) from t h e p r o c e s s o f RANKING. I f TABLE 1 25 Taxonomy o f E v a l u a t i o n Types E v a l u a t i o n Type P r e e v a l u a t i v e (Goal o r c r i t e r i o n a c h i e v e m e n t ) Minimal e v a l u a t i v e O v e r a l l e v a l u a t i v e Commendatory L a u d a t o r y I d e a l Best-Buy Usual V e r b a l E x p r e s s i o n "The t r e a t m e n t X had t h e e f f e c t Y on t h e p o p u l a -t i o n o f s t u d e n t s , S, i n c o n d i t i o n s C; and Y was th e g o a l o r shows t h a t .the'.goal was a c h i e v e d . " "X had a good e f f e c t (on S i n C ) . " "X had an o v e r a l l good e f f e c t . " "X was t h e b e s t p o s s i b l e t r e a t m e n t . 1 "X was a Best-Buy." U s u a l l y A dequate P r e m i s e s 1. X was t h e t r e a t m e n t . 2. X caused Y. 3. Y i m p l i e s t h a t t h e g o a l - , was a c h i e v e d . "X was worth d o i n g . " T h i s i s a l m o s t a sub-c a s e o f o v e r a l l e v a l -u a t i o n , i f c o s t s a r e tak e n as a harmful e f f e c t . "X was t h e b e s t c h o i c e . " 1 1 1. X caused Y. 2. S o r non-Ss ( d e s i r e d , e n j o y e d , were bene-f i t t e d by) Y. 1. As f o r m i n i m a l , p l u s , 2. Y had no harmful e f -f e c t s dnSSs; o r ndn-Ss, o r , 3. Y had much l e s s s i g n i -f i c a n t harmful e f f e c t s on S s . 1. As f o r o v e r a l l , p l u s , 2. The c o s t o f X was manageable. 3. Y was worth t he c o s t . As f o r commendatory, p l u s , 2. No o t h e r t r e a t m e n t , on d a t a which was a v a i l -a b l e , appeared as c o s t -e f f e c t i v e . 1. As f o r commendatory, p l u s , 2. No o t h e r t r e a t m e n t was i n f a c t as c o s t - e f f e c -t i v e . 1 As f o r commendatory, p l u s , 2. X i s a member o f a group which o f f e r s t h e b e s t o r a l m o s t t h e b e s t p e r f o r m a n c e f o r s i g n i -f i c a n t l y l e s s c o s t t h a n t h e i r p e r f o r m a n c e p e e r s . These p r e m i s e s g i v e prima f a c i e s u p p o r t f o r e v a l u a t i v e c o n c l u s i o n s , n o t d e d u c t i v e s u p p o r t , b u t e v a l u a t i o n - - l i k e * s c i e n c e - - o n l y needs prima f a c i e i n f e r e n c e . 26 t h e e v a l u a t o r i s o n l y RANKING, then he would base h i s rank o r d e r o f t h e v a r i o u s e v a l u a t a on an i n c r e a s i n g l y comprehensive l i s t o f s t a n d a r d s , as he goes from a p r e e v a l u a t i v e t o a Best-Buy c o n c l u s i o n . An i n t e r n a l , f o r m a t i v e e v a l u a t o r ( i . e . t h e e a r l y d e v e l o p m e n t a l s t a g e o f t h e p r o j e c t i s b e i n g e v a l u a t e d by someone i n v o l v e d i n , and committed to the p r o j e c t ( S c r i v e n , 1967, p. 45) i s u s u a l l y c o n c e r n e d w i t h t h e e a r l y s t a g e s o f S-GRADING. In o t h e r words, he i s g r a d i n g t h e evaluatum on t h e b a s i s o f a s e r i e s o f s t a n d a r d s d e r i v e d from some i d e a l , o r b e s t p o s s i b l e , f o r m o f t h e evaluatum. On t h e o t h e r hand, i f t h e e v a l u a t o r i s an e x t e r n a l , summa.tive e v a l u a t o r ( i . e . t h e f i n a l marketed v e r s i o n i s b e i n g e v a l u a t e d by someone e x t e r n a l t o the p r o j e c t ) , t h e n he i s r a n k i n g t h e evaluatum i n the c l a s s o f c o m p a r i s o n . T h e r e i s an i n t e r m e d i a t e s t e p near t h e end o f t h e f o r m a t i v e s t a g e o f the e v a l u a t i o n . A t t h i s t i m e , an e x t e r n a l f o r m a t i v e e y a l u a t o r c o u l d be engaged t o pe r f o r m t h e l a s t s t a g e s o f S - g r a d i n g and e a r l y s t a g e s o f r a n k i n g b e f o r e t h e evaluatum went t o t h e summative e v a l u a t i o n . The p a r t i c u l a r problem examined by t h i s t h e s i s was an e v a l u a t i o n o f a t e a c h e r - t r a i n i n g program. T h i s program was i n t h e d e v e l o p m e n t a l s t a g e , and t h e a u t h o r o f t h i s t h e s i s was p r i m a r i l y an i n t e r n a l , f o r m a t i v e e v a l u a t o r . T h e r e f o r e , t h e form o f t h e e v a l u a t i o n was S-GRADING, and aimed a t an o v e r a l l e v a l u a t i v e c o n c l u s i o n ( s e e T a b l e 1 ) . S i n c e t h i s was t h e c a s e , t h e n t h e r e s t o f T a y l o r ' s l o g i c a l p r o c e s s o f e v a l u a t i o n w i l l be examined as i t a p p l i e s t o S-GRADING. F i g u r e 2 has been p r e p a r e d t o a i d i n t h e f o l l o w i n g d e s c r i p t i o n o f T a y l o r ' s p h i l o s o p h i c a l p o s i t i o n . . T h i s f i g u r e s h o u l d not be r i g i d l y F i g - 2 An O u t l i n e o f the L o g i c a l P r o c e s s o f E v a l u a t i o n The Evaluatum P r e c o n d i t i o n s I The P o i n t o f View L e v e l 1 L e v e l 2 L e v e l 3 L e v e l 4 L e v e l 5 G r a d i n g or Ranking? A d o p t i o n o f S t a n d a r d s O p e r a t i o n a l C l a r i f i c a t i o r o f S t a n d a r d s I n t r i n s i c V a l u e \ E x t r i n s i c V a l u e S p e c i f i c a t i o n o f t h e C l a s s o f Comparisons D e t e r m i n a t i o n o f t h e Good and Bad C h a r a c t e r i s t i c s o f t h e E v a l u a t i o n O v e r a l l D e t e r m i n a t i o n o f Worth o f the Evaluatum G a t h e r i n g Data b h t h e C h a r a c t e r i s t i c s D e t e r m i n i n g i f t h e C h a r a c t e r i s t i c s a r e .Good o r Bad 28 i n t e r p r e t e d as a f l o w c h a r t d e p i c t i n g a temporal sequence o f e v e n t s . The p r o c e s s o f e v a l u a t i o n does b e g i n w i t h an evaluatum* and end w i t h an o v e r a l l e s t i m a t i o n o f goodness o r badness; however, t h e i n t e r v e n i n g s t e p s may o c c u r i n any o r d e r and r e p e t i t i v e l y , b e f o r e a r r i v i n g a t t h e f i n a l e v a l u a t i v e c o n c l u s i o n . N e v e r t h e l e s s , i t i s a n e c e s s a r y c o n d i -t i o n i n T a y l o r ' s vciew o f e v a l u a t i o n , t h a t a l l t h e s t e p s i n F i g u r e 1 must be completed b e f o r e an e v a l u a t i o n can be l o g i c a l l y v e r i f i e d . 2.40 L e v e l 1 of, S - G r a d i n g : A d o p t i o n o f S t a n d a r d s (See F i g . 1) T a y l o r i d e n t i f i e s two t y p e s o f s t a n d a r d s . They a r e s t a n d a r d s o f i n t r i n s i c v a l u e and s t a n d a r d s o f e x t r i n s i c v a l u e . ( T a y l o r 1961, pp. 20-25). 2.41 I n t r i n s i c V a l u e S t a n d a r d s o f i n t r i n s i c v a l u e a r e used t o e v a l u a t e t h i n g s which a r e v a l u a b l e i n and f o r t h e m s e l v e s ( D a n i e l s , 1971, p. 9) and t h e r e b y s a t i s f y i n g t h e f o l l o w i n g two c o n d i t i o n s . F i r s t l y t h e y must be used o n l y i n e v a l u a t i n g i m m e d i a t e l y f e l t o r p e r c e i v e d q u a l i t i e s o f o u r own e x p e r i e n c e ; and s e c o n d l y , t h e y must be s t a n d a r d s o f n o n d e r i v a t i v e v a l u e ( T a y l o r , 1961, p. 23). I f an evaluatum produced f e e l i n g s o f p l e a s a n t n e s s , then t h o s e f e e l i n g s would have i n t r i n s i c v a l u e . 2.42 E x t r i n s i c V a l u e Whereas s t a n d a r d s o f i n t r i n s i c v a l u e a r e o f n o n d e r i v a t i v e 29 v a l u e , s t a n d a r d s o f e x t r i n s i c v a l u e a r e o f d e r i v a t i v e v a l u e , t h a t i s , dependant on t h e v a l u e o f o t h e r t h i n g s . T a y l o r has s u b - d i v i d e d e x t r i n s i c v a l u e i n t o i n h e r e n t v a l u e , i n s t r u m e n t a l v a l u e and c o n t r i b u t i v e v a l u e . ( T a y l o r , 1961, pp. 26-32, 310-311) I f an evaluatum i s j u d g e d i n terms o f s t a n d a r d s o f i n h e r e n t v a l u e , t h e n t h e e v a l u a t o r would l o o k f o r e v i d e n c e t h a t i t p r o d u c e s e x p e r i e n c e s which a r e j u d g e d as i n t r i n s i c a l l y good. I n h e r e n t v a l u e d i f f e r s f r o m i n t r i n s i c v a l u e i n t h a t t h e f o r m e r must be o t h e r t h a n a q u a l i t y o f o u r own e x p e r i e n c e . "Works o f a r t and t h i n g s o f n a t u r a l beauty may a l s o be s a i d t o be good on the ground t h a t one who l o o k s at. them n o r m a l l y has a good o r r e w a r d i n g e x p e r i e n c e . Then, we may say t h a t t h e y have i n h e r e n t goodness". ( F r a n k e n a , 1973, p. 81) In o t h e r words an o b j e c t ' s i n h e r e n t v a l u e depends on i t s c a p a c i t y t o produce i n t r i n s i c v a l u e o f someone's e x p e r i e n c e i n r e s p o n s e to i t . ( T a y l o r , 1961, p. 28) To have i n s t r u m e n t a l v a l u e , t h e e v a l u a t u m must be e f f e c t i v e i n b r i n g i n g about a g i v e n end which i s i n t r i n s i c a l l y o r e x t r i n s i c a l l y v a l u a b l e . As D a n i e l s s a i d , (1971, p. 9 ) , " A - t h i n g has i n s t r u m e n t a l v a l u e i f i t can be used t o o b t a i n some t h i n g o r c o n d i t i o n o t h e r t h a n i t s e l f which i s , f o r some r e a s o n o r o t h e r , c o n s i d e r e d t o be o f v a l u e . " In the b r o a d e s t sense o f i n s t r u m e n t a l v a l u e an e v a l u a t o r would grade o r rank a t e a c h e r - t r a i n i n g program a c c o r d i n g t o i t s e f f e c t i v e n e s s i n e n a b l i n g s t u d e n t s t o become good t e a c h e r s . In a narrower sense 30 he c o u l d examine methods c o u r s e s ( o r p a r t s o f t h o s e methods c o u r s e s ) f o r t h e i r s u c c e s s i n t e a c h i n g t h e s t u d e n t some p a r t i c u l a r t e c h n i q u e o f v a l u e . To f u l f i l l a s t a n d a r d o f c o n t r i b u t i v e v a l u e , a " p a r t " must c o n t r i b u t e t o a good whole. The more n e c e s s a r y i t i s t o t h e whole, t h e more c o n t r i b u t i v e v a l u e i t has. For example, p r o f e s s o r s o f e d u c a t i o n s h o u l d c o n t r i b u t e g r e a t l y t o t h e p r o d u c t i o n o f good t e a c h e r s , t h e r e f o r e t h e y s h o u l d have h i g h c o n t r i b u t i v e v a l u e i n a t e a c h e r - t r a i n i n g program and would be j u d g e d i n terms o f s t a n d a r d s o f " c o n t r i b u t i o n " . As T a y l o r i n d i c a t e s , the above t h r e e k i n d s o f s t a n d a r d s o f e x t r i n s i c v a l u e a r e dependent upon i n t r i n s i c v a l u e . He says (1961, p. 3 2 ) : " F i r s t , a t h i n g may have i n s t r u m e n t a l v a l u e i n v i r t u e o f the f a c t t h a t i t i s a means t o an end which has c o n t r i b u t i v e v a l u e as p a r t o f a whole. T h i s whole i n t u r n e i t h e r may have i n h e r e n t v a l u e i n i t s e l f o r may have i n s t r u m e n t a l v a l u e as a means t o an end which has i n h e r e n t o r i n t r i n s i c v a l u e . " In o t h e r words, t h e f i n a l a p p e a l f o r t h e r a i s o n d ' e t r e o f a n y t h i n g i s t h a t some one o r some group o f p e o p l e d e r i v e s i n t r i n s i c v a l u e from i t . "Thus we a r r i v e f i n a l l y a t t h i n g s which we j u d g e t o be d e s i r a b l e i n t h e m s e l v e s . T h e i r v a l u e i s i n t r i n s i c t o them." ( T a y l o r , 1961, p. 32) Or as Frankena (1973, p. 81-82) wr o t e : "We a l s o sometimes say t h a t t h i n g s a r e good; d e s i r a b l e , o r w o r t h w h i l e i n t h e m s e l v e s , as ends, i n t r i n s i c a l l y . When someone a s k s 'What i s good f o r ; t h e answer may be g i v e n by t r y i n g t o 31 e x h i b i t i t s u s e f u l n e s s , e x t r i n s i c v a l u e , o r i n h e r e n t goodness; but one may a l s o t r y t o show (and h e r e t h e f i n a l a p p e a l must be 'Try i t and see') t h a t i t i s e n j o y a b l e or o t h e r w i s e good i n i t s e l f ... In f a c t , i t i s hard t o see how money, c a r s , and o t h e r m a t e r i a l p o s s e s s i o n s , even p a i n t i n g s , can have any goodness o r v a l u e a t a l l , e x t r i n s i c o r i n h e r e n t , i f t h e e x p e r i e n c e s t h e y make p o s s i b l e a r e not i n some way e n j o y a b l e o r good i n t h e m s e l v e s . " The above s t a t e m e n t s a b o u t i n t r i n s i c v a l u e do not mean t h a t an e d u c a t i o n a l p r o d u c t i s not w o r t h w h i l e i f i t d o e s n ' t have i n t r i n s i c v a l u e . However, i f t h e program does not have i n t r i n s i c v a l u e , t h e n i t s h o u l d be shown t h a t a c h i e v i n g t h e g o a l s o f t h e program i s i n s t r u m e n t a l i n f u l f i l l i n g some a s p e c t o f l i v i n g which w i l l have i n t r i n s i c v a l u e . I f t h e r e i s no a p p a r e n t c o n n e c t i o n between t h e program g o a l s and f u t u r e a t t a i n m e n t o f i n t r i n s i c v a l u e , t h e n t h e w o r t h w h i l e n e s s o f t h e program g o a l s s h o u l d be s e r i o u s l y q u e s t i o n e d . 2.50 L e v e l II o f S-GRADING (See F i g . 1 ) : O p e r a t i o n a l C l a r i f i c a t i o n o f  t h e S t a n d a r d s i n G r a d i n g " T h i s c o n s i s t s i n a s e t o f s t a t e m e n t s t o t h e e f f e c t t h a t i f an o b j e c t 0 has c h a r a c t e r i s t i c s C, i t f u l f i l l s a c e r t a i n s t a n d a r d S to a c e r t a i n d e g r e e D; t h a t i f 0 has c h a r a c t e r i s t i c s C , i t f u l f i l l s S t o a g r e a t e r ( l e s s e r ) 'degree D'; e t c . " ( T a y l o r , 1961, p. 10) Once a s t a n d a r d has been g e n e r a l l y i d e n t i f i e d as a s t a n d a r d o f i n t r i n s i c o r e x t r i n s i c v a l u e , then i t must be. e l a b o r a t e d i n such terms t h a t p e r m i t t h e e v a l u a t o r t o say: " c h a r a c t e r i s t i c C f u l f i l l s s t a n d a r d S t o d e g r e e D." T h i s i n t e r m e d i a t e s t e p i n t h e p r o c e s s o f e v a l u a t i o n has been c a l l e d " c o n c e p t u a l i z i n g " by S c r i v e n ( S c r i v e n , 1974a, p. 142). He sees t h i s s t e p i n e v a l u a t i o n as a p r o c e s s o f 32 b o t h " d e v i s i n g an a p p r o p r i a t e taxonomy and mea s u r i n g t h e s p e c i f i c ' p e r f o r m a n c e i n terms o f t h e d i m e n s i o n s o f t h e taxonomy." ( S c r i v e n , 1974a, p. 137). In C h a p t e r I I I i t w i l l be shown how each s t a n d a r d c hosen i n t h e e v a l u a t i o n o f a t e a c h e r - t r a i n i n g program i s e l a b o r a t e d so as t o a l l o w some r a t i n g o f t h e de g r e e to which t h e e v a l u a t u m , met t h a t s t a n d a r d . 2.60 L e v e l I l l — S p e c i f i c a t i o n o f t h e . C l a s s o f Comparison i n G r a d i n g In S-GRADING, t h e c l a s s o f comparison i s some i d e a l form o f t h e evaluatu m . T a y l o r ' s example (1961, p. 9) o f t h e S-GRADING o f p a i n t i n g s ended ". . . i s [ t h e p a i n t i n g ] good as compared w i t h what a p a i n t i n g o f t h i s t y p e s h o u l d be?" In the i n s t a n c e o f t h e S-GRADING o f a t e a c h e r - t r a i n i n g program t he p a r a l l e l q u e s t i o n becomes: " I s t h e program good as compared w i t h what a t e a c h e r - t r a i n i n g program a t t h i s l e v e l s h o u l d be?" S - g r a d i n g , as argued e a r l i e r , i s o n l y an i n t e r m e d i a t e s t e p i n t h e complete e v a l u a t i o n o f a p r o d u c t . The s t r o n g e s t e v a l u a t i v e c o n c l u s i o n which can be r e a l i z e d i s "Commendatory" (See T a b l e 1 ) . The c h o i c e o f S-GRADING o r RANKING i s an o p t i o n i n t h e f i r s t f o u r e v a l u a t i v e c o n c l u s i o n s o f T a b l e 1, and i s p r i m a r i l y based on c i r c u m s t a n c e . I f t h e r e i s a r e a d i l y i d e n t i f i e d c o m p a r i s o n g r o u p ( s ) p r e s e n t , and i f th e e v a l u a t o r has a c c e s s t o t h e com p a r i s o n g r o u p ( s ) e a r l y enough t o p l a n a p a r a l l e l e v a l u a t i o n scheme f o r t h e e v a l u a t a , t h e n he s h o u l d p r o b a b l y use RANKING. S c r i v e n (1967, p. 64) a r g u e s : 33 " C o m p a r a t i v e e v a l u a t i o n s a r e o f t e n v e r y much e a s i e r t h a n n o n c o m p a r a t i v e e v a l u a t i o n s , because we can o f t e n use t e s t s w hich y i e l d d i f f e r e n c e s i n s t e a d o f h a v i n g t o f i n d an a b s o l u t e s c a l e and then e v e n t u a l l y compare t h e a b s o l u t e s c o r e s . " I f , on t h e o t h e r hand, t h e co m p a r i s o n g r o u p ( s ) i s not e a s i l y i d e n t i f i a b l e and/or t he e v a l u a t o r cannot g a i n a c c e s s t o i t e a r l y enough t o p e r f o r m e v a l u a t i o n s on i t th e n S-GRADING i s t h e b e s t a l t e r n a t i v e i n t h e e a r l y s t a g e s . However, t h e e v a l u a t o r s h o u l d be c o n s i d e r i n g methods o f u s i n g , o r c r e a t i n g ( S c r i v e n , 1967, p. 6 9 ) , comp a r i s o n groups i f a n y t h i n g l i k e a complete e v a l u a t i o n i s to be a c c o m p l i s h e d . 2.70 L e v e l _IV--Determi n a t i o n o f t h e Good and Bad C h a r a c t e r i s t i c s o f t h e Eva 1 uatuhrrj• n"SSGRADING? T h i s p a r t i c u l a r s t e p i n a l o g i c a l e v a l u a t i o n can be broken down i n t o two s u b p r o c e s s e s ; t h e y a r e : a) G a t h e r i n g d a t a on t h e c h a r a c t e r i s t i c s . b) D e t e r m i n i n g i f t h e c h a r a c t e r i s t i c s a r e good o r bad. 2.71 G a t h e r i n g Data on t h e C h a r a c t e r i s t i c s T h i s s u b - p r o c e s s i n v o l v e s u s i n g v a r i o u s p s y c h o m e t r i c t e c h n i q u e s to g a t h e r d a t a . In t h i s p a r t i c u l a r s t u d y , t h e g e n e r a l t e c h n i q u e s used were: a L i k e r t a t t i t u d e s c a l e , q u e s t i o n n a i r e s , o b s e r v a t i o n , f o r m a l and i n f o r m a l i n t e r v i e w s , e x a m i n a t i o n s o f r e c o r d s o f m e e t i n g s , and e x a m i n a t i o n s o f o t h e r i n t e r n a l e v a l u a t i o n s ( o v e r a l l and o f s t u d e n t t e a c h i n g ) . M e t f e s s e l and M i c h a e l (1967. p. 937-943) 34 p r o v i d e d a v e r y c omprehensive l i s t o f p o s s i b l e t e c h n i q u e s f o r t h e e v a l u a t i o n o f s c h o o l programs. The t e c h n i q u e s used i n t h i s s t u d y w i l l be e l a b o r a t e d i n C h a p t e r I I I , S e c t i o n 3.51. 2.72 D e t e r m i n i n g i f the C h a r a c t e r i s t i c s a r e Good o r Bad Each c h a r a c t e r i s t i c , on which d a t a has been c o l l e c t e d , i s j u d g e d as good, bad o r n e u t r a l ( e q u a l amounts o f goodness and b a d n e s s ) . T h i s judgement depends not o n l y on t h e p o i n t o f view b e i n g t a k e n , but a l s o t h e v a l u e system i n t h e p a r t i c u l a r p o i n t . o f v iew. Thus w h i l e a c h a r a c t e r i s t i c w hich i s j u d g e d as good fro m t h e i n t e l l e c t u a l p o i n t o f view, may be bad f r o m t h e economic p o i n t o f view, i t i s a l s o t h e c a s e t h a t a c o u r s e i n home economics which d e a l t w i t h t h e p r e p a r a t i o n o f pork f o r t h e t a b l e may be r e l i g i o u s l y q u i t e a c c e p t a b l e f o r an A n g l i c a n , but r e l i g i o u s l y r e p u g n a n t t o an o r t h o d o x Jew. The problems o f e v a l u a t o r and c l i e n t w o r k i n g a t c r o s s - p u r p o s e s due t o c o n f l i c t i n g v a l u e systems o r p o i n t s o f view a r e m a g n i f i e d i n i n t e r n a t i o n a l e d u c a t i o n . The f o l l o w i n g quote i s i n c l u d e d as a s c a l e d -up, but p a r t i c u l a r l y good, example o f t h e s e c o n c e r n s : "There a r e f r e q u e n t l y s e v e r a l i n t e r e s t s i n a s i n g l e d e v e l o p -ment p r o j e c t , t h e i n t e r e s t o f t h e r e s p o n s i b l e o r g a n i z a t i o n (government agency, b u s i n e s s e n t e r p r i s e , p r i v a t e i n s t i t u t i o n , e t c . ) b e i n g , i n t h e o r y , paramount and u s u a l l y a l s o i n p r a c t i c e . I f i n t e r g o v e r n m e n t a l o r g a n i z a t i o n s o r f o r e i g n governments o r p r i v a t e f o u n d a t i o n s a l s o p a r t -i c i p a t e i n t h e p r o j e c t , t h e y may be assumed t o have t h e i r o b j e c t i v e s t o a c h i e v e , and t h e s e may o r may not be i d e n t i c a l w i t h t h o s e o f t h e r e s p o n s i b l e o r g a n i z a t i o n . Moreover, i n d i v i d u a l e x p e r t s , a l t h o u g h n o m i n a l l y , and u s u a l l y g overned by t h e g o a l s o f e m p l oying o r g a n i z a t i o n s , sometimes have t h e i r own and d i f f e r e n t o b j e c t i v e s ( s u c h as t e s t i n g o u t a p e t t h e o r y , o r d e m o n s t r a t i n g t h e s u p e r i o r i t y o f a p a r t i c u l a r 35 method o r p r o d u c t ) . The p e o p l e l i k e l y t o be a f f e c t e d by a development p r o j e c t a l s o have t h e i r own g o a l s , which may o r may not be r e f l e c t e d i n t h e p o l i c i e s o f t h e i r government. A government may change, t o o , o r a new o f f i c i a l be a p p o i n t e d d u r i n g t h e l i f e o f a p r o j e c t , and as a r e s u l t d i f f e r e n t g o a l s may be e s t a b l i s h e d . " (Hayes, 1969, pp. 29-30) In a s i m i l a r v e i n , S c r i v e n (1974a, p. 138) wrote: "The d e v e l o p e r o r t e a c h e r always has one c o n c e p t u a l i z a t i o n o f t h e d a t a i n mind: I f he ( o r she) f e e l s he has been s u c c e s s f u l , he o r she n a t u r a l l y sees t h e d a t a as ' d e m o n s t r a t i n g s u c c e s s i n a c h i e v i n g such and such g o a l s , ' and hence ( s i n c e t h o s e g o a l s would not have been adopted had i t not been f e l t t h e y had m e r i t ) t h e p r o j e c t i s j u d g e d m e r i t o r i o u s . But t h e r e a r e many o t h e r ways t o s e e most p r o j e c t s . I t i s j u s t as i m p o r t a n t f o r t h e e v a l u a t o r , as opposed t o t h e d e v e l o p e r , t o r e t a i n an open mind ab o u t t h e l e g i t i m a c y o f r a d i c a l l y d i f f e r e n t i n t e r p r e t a t i o n s o f d a t a , as i t i s f o r t h e s c i e n t i s t r e a d i n g a r e s e a r c h paper i n which t h e a u t h o r p r o p o s e s t h a t c e r t a i n e x p e r i m e n t a l r e s u l t s s u p p o r t h i s t h e o r y . " I t i s f o r t h e s e r e a s o n s t h a t t h e e v a l u a t o r must be s u r e t h a t t h e c l i e n t f o r t h e e v a l u a t i o n u n d e r s t a n d s the p o i n t o f view b e i n g t a k e n i n t h e e v a l u a t i o n ; and t h a t t h e v a l u e systems o f t h e c l i e n t and e v a l u a t o r a r e f a i r l y c o n g r u e n t . When''the e v a l u a t o r makes a judgement t h a t a p a r t i c u l a r ^ c h a r a c t e r i s t i c , "C", i s "good" a c c o r d i n g t o s t a n d a r d "S", t h e n he must have arguments which w i l l s u p p o r t h i s c o n t e n t i o n ; he must be a b l e t o p o i n t a t v a r i o u s e v i d e n c e s f o r "C" and, t h r o u g h them, v e r i f y h i s v a l u e judgement. ( T a y l o r , 1961, p. 70) In o t h e r words, the e v a l u a t o r must have c o m p l e t e r and v a l i d e v i d e n c e a b o u t a p a r t i c u l a r c h a r a c t e r i s t i c and he must have good r e a s o n ( g i v e n t h e e v i d e n c e ) f o r s a y i n g t h e c h a r a c t e r i s t i c i s good, bad o r n e u t r a l . I f t h e e v a l u a t o r i s c h a l l e n g e d on h i s judgement, he would f i r s t show t h a t h i s e v i d e n c e was complete and v a l i d 36 ( v e r i f i c a t i o n ) . I f he was s t i l l c h a l l e n g e d , he would t h e n show t h a t h i s r e a s o n s f o r j u d g i n g t h a t t h a t d a t a on t h a t c h a r a c t e r i s t i c meant t h e c h a r a c t e r i s t i c was good, bad o r n e u t r a l , were good r e a s o n s ( v a l i d a t i o n ) . S c r i v e n (1974a, p. 139) has c a l l e d t h i s p r o c e s s , which l e a d s from t h e c o n c e p t u a l i z e d i n t e r m e d i a t e c o n c l u s i o n s t o t h e e v e n t u a l c o n c l u s i o n s o f w orth o r m e r i t , c r e d e n t i a l i n g . 2.80 L e v e l V-- The O v e r a l l Judgement o f Worth T h e r e a r e two f a c t o r s t h a t d e t e r m i n e whether an evaluatum i s t o be g r a d e d as good o r bad on t h e whole. The f i r s t f a c t o r , i s t h e degree t o which t h e evaluatum f u l f i l l s a l l t h e s t a n d a r d s a p p l i e d t o i t . As T a y l o r (1961, p. 8)*wntes'; "That depends on what s t a n d a r d s one i s a p p e a l i n g t o , . . . how c l e a r l y t h o s e s t a n d a r d s a r e d e f i n e d , t o what e x t e n t t h e d e g r e e s t o w h i c h t h e y can be f u l f i l l e d a r e m e a s u r a b l e , and how " d i s t a n t " from r e a l i t y i s one's i d e a l . " The second f a c t o r i s t h e r e l a t i v e p r e c e d e n c e o f t h e s t a n d a r d s . When t h e evaluatum i s e v a l u a t e d a c c o r d i n g t o a s e t o f s t a n d a r d s , t h o s e s t a n d a r d s may be r a n k e d i n o r d e r o f d e c r e a s i n g i m p o r t a n c e ( i . e . S-j, S^, S^, . . ., S n where i s t h e most i m p o r t a n t s t a n d a r d and S n i s t h e l e a s t i m p o r t a n t s t a n d a r d ) . T h a t i s t o s a y , when we a r e making an o v e r a l l judgement o f v a l u e , g r e a t e r w e i g h t i s g i v e n t o t h e f a c t t h a t t h e evaluatum f u l f i l l s S^ t h a i t t o t h e f a c t t h a t i t f u l f i l l s S,,. I f the evaluatum f u l f i l l s S-, and S 9 t o t h e same d e g r e e , i t s f u l f i l l m e n t 37 o f S-j adds more t o i t s o v e r a l l goodness t h a n i t s f u l f i l l m e n t o f S 2 ( T a y l o r , 1961, p. 1 3 ) . CHAPTER I I I A p p l i c a t i o n o f t h e Model t o The D e s i g n o f an A c t u a l E v a l u a t i o n , Study 3.00 I n t r o d u c t i o n C h a p t e r T;hree a p p l i e s t h e model o f e v a l u a t i o n d e v e l o p e d i n C h a p t e r Two t o t h e d e s i g n o f an e v a l u a t i o n s t u d y o f an u n d e r g r a d u a t e t e a c h e r t r a i n i n g program. The c h a p t e r i l l u s t r a t e s t h e c h o i c e o f s t a n d a r d s , d a t a c o l l e c t i o n on the s t a n d a r d s , and how an e v a l u a t i v e c o n c l u s i o n can be r e a c h e d . Each s e c t i o n o f t h e C h a p t e r i l l u s t r a t e s a p r a c t i c a l a p p l i c a t i o n o f the s i m i l a r l y headed, but t h e o r e t i c a l s e c t i o n s i n C h a p t e r Two.. 3.10 E s t a b l i s h i n g t h e P r e c o n d i t i o n s 3.11 The Evaluatum The evaluatum i s an u n d e r g r a d u a t e t e a c h e r t r a i n i n g program s u b s e q u e n t l y r e f e r r e d t o as Program B. A d e s c r i p t i o n o f t h e Program does not n e c e s s a r i l y p o r t r a y t h e Program as i t a c t u a l l y o p e r a t e d but r a t h e r as i t was i n t e n d e d t o o p e r a t e , i n c l u d i n g a t i m e s c a l e o f o p e r a t i o n , t h e g e o g r a p h i c a l l o c a t i o n , t h e p e r s o n n e l r e q u i r e m e n t s , e t c . D e t e r m i n i n g t h e d e g r e e to which the i n t e n d e d program and t h e a c t u a l program were c o n g r u e n t i s a f u n c t i o n o f t h e e v a l u a t o r . The f o l l o w i n g i s a d e s c r i p t i o n o f t h e evaluatum based on t h e m a t e r i a l s p r e p a r e d f o r t h e Program. 38 39 3.111 L e t t e r t o P o t e n t i a l S t u d e n t s f o r Program B. Dear S t u d e n t : The F a c u l t y o f E d u c a t i o n has i n i t i a t e d s e v e r a l new programs f o r th e p r o f e s s i o n a l t r a i n i n g o f t e a c h e r s . One o f t h e s e new programs, t i t l e d "Program B" i s d e s c r i b e d i n t h e accompanying l e t t e r . B r i e f l y , what we i n t e n d t o do i s have a group o f 25 s t u d e n t s spend t h e e n t i r e y e a r a t s c h o o l X, a Vancouver e l e m e n t a r y s c h o o l . A l l u n i v e r s i t y c o u r s e s and a l l s t u d e n t t e a c h i n g w i l l be u n d e r t a k e n t h e r e . The a d m i n i s t r a t i o n and s t a f f o f School X a n f f a c u l t y members from U.B.C. have c o o p e r a t -i v e l y p l a n n e d t h e program and a r e v e r y e n t h u s i a s t i c about t h i s new app r o a c h t o t e a c h e r e d u c a t i o n . As a 3 r d y e a r t r a n s f e r s t u d e n t , you a r e e l i g i b l e t o be a p a r t o f t h i s new program. The program i s l i m i t e d to 25 s t u d e n t s and we hope t o f i l l t h e s e c t i o n b e f o r e t h e end o f t he Vancouver s c h o o l y e a r . I f you a r e i n t e r e s t e d p l e a s e c o n t a c t me i m m e d i a t e l y so a p r e l i m i n a r y meeting o r two may be a r r a n g e d . S i n c e r e l y , Program B D i r e c t o r , F a c u l t y o f E d u c a t i o n U n i v e r s i t y o f B.C. 3.112 Program D e s c r i p t i o n Program B w i l l be c o n d u c t e d e n t i r e l y i n a p a r t i c i p a t i n g e l e m e n t a r y s c h o o l , b ut s t u d e n t s w i l l be a b l e t o make u s e o f t h e U n i v e r -s i t y l i b r a r i e s . I t has been agreed t h a t i t w i l l c o n t a i n : 1. a heavy component o f a c t u a l c l a s s r o o m e x p e r i e n c e -( E d u c a t i o n 397) 2. p r o f e s s i o n a l methods c o u r s e s 3. a p s y c h o l o g i c a l component 4. a f o u n d a t i o n s component 5. a g e n e r a l e l e m e n t a r y e d u c a t i o n c o u r s e The form which t h e c l a s s r o o m component w i l l t a k e w i l l depend on t h e s c h o o l s t a f f and t h e i r recommendations, but a f l e x i b l e , v a r i e d , t e a c h i n g program t h a t i n c l u d e s e x p e r i e n c e s a t s e v e r a l l e y e l s o f s c h o o l l i f e i s e n v i s i o n e d . The methods c o u r s e s w i l l be i n t e r e s t e d i n t h e s p e c i f i c r e q u i r e m e n t s o f each a r e a , t he m a t e r i a l s a v a i l a b l e , and c u r r i c u l u m development. D u r i n g t h e t i m e d e v o t e d t o p r o f e s s i o n a l 40 methods the following courses wil l be covered: Ed. 304 Language Arts 3 units Professor A Ed. 321 Science Education 1 -1/2 units Professor B Ed. 323 Art Education 1 unit Professor C Ed. 325 Physical Education 1 unit Professor D Ed. 370 Mathematics 1 -1/2 units Professor E Ed. 371 Mathematics.Education 1 -1/2 units Professor E The psychology component wil l be designed to increase the sensit iv i ty of the pre-service teachers to the needs and requirements of children not as consumers of science, mathematics, or any other content area, but rather as human beings. This component is also designed to help the pre-service teachers understand some of the problems associated with learning and evaluation. To this end the theoretical material associated with Education 310 and 311 wil l be covered. The foundations component wil l be designed to accomplish three major tasks: 1. To enable pre-service teachers to make clear, well-founded decisions as to what goals they, as teachers, ought to pursue. 2. To enable them to promote, in an ethical ly responsible manner, the moral development of their students. 3. To give them a clear understanding of the nature of the task of developing the ab i l i ty to think. To this end the theoretical material associated with Ed. 400 wil l be covered. The general elementary education component (Education 449) wil l contain many of the common elements of the methods courses. It wil l also deal with much of what is currently done in student teaching seminars at the University. Further i t wil l deal with such topics as teaching strategies, classroom organization and curricular planning. 3.113 Description of the offered courses in Program B (Taken from the Calendar of the University of B.C. 1974/75). Ed. 304: Curriculum and Instruction in the Language Arts. - - A study of (a) the curriculum organization in the language arts particularly in the intermediate grades; (b) techniques of instruction in these subjects and grades. Ed. 321: Curriculum and Instruction in Elementary Science. -- A study of (a) the curriculum organization in science for the elementary grades; (b) techniques and strategies of instruction in science for these grades. 41 Ed. 323: C u r r i c u l u m and I n s t r u c t i o n i n A r t I . Ed. 325: C u r r i c u l u m and I n s t r u c t i o n i n P h y s i c a l E d u c a t i o n . -- A s t u d y o f (a) the c u r r i c u l u m o r g a n i z a t i o n i n p h y s i c a l e d u c a t i o n f o r t h e e l e m e n t a r y g r a d e s ; (b) t e c h n i q u e s o f i n s t r u c t i o n i n p h y s i c a l e d u c a t i o n f o r t h e s e g r a d e s . Ed. 370: Mathematics f o r E l e m e n t a r y T e a c h e r s . -- A s t u d y o f t h e method and s t r u c t u r e o f e l e m e n t a r y s c h o o l mathematics as a d i s c i p l i n e . Ed. 371: Methods o f t e a c h i n g E l e m e n t a r y School M a t h e m a t i c s . --A s t u d y o f m a t e r i a l s , s k i l l s , s t r a t e g i e s and c u r r i c u l u m o r g a n i z a t i o n . Ed. 310: Growth and Development. -- R e s e a r c h as i t a p p l i e s t o t h e e l e m e n t a r y s c h o o l c h i l d . Not open t o s t u d e n t s who have t a k e n E d u c a t i o n 331. Ed. 311: The N a t u r e and Measurement o f L e a r n i n g . — A s t u d y o f l e a r n i n g and t h e t e c h n i q u e s o f e v a l u a t i o n as t h e y a p p l y to t h e elemen-t a r y s c h o o l c h i l d . Not open t o s t u d e n t s who have t a k e n E d u c a t i o n 331. Ed. 400: P h i l o s o p h y o f E d u c a t i o n . -- An i n t r o d u c t o r y c o u r s e i n which c o n s i d e r a t i o n i s g i v e n t o t h e p h i l o s o p h i c a l f o u n d a t i o n s o f educa-t i o n and t o t h e p r a c t i c a l b e a r i n g s o f t h e o r y upon c u r r i c u l u m c o n t e n t and c l a s s r o o m p r a c t i c e i n our s c h o o l s . 3.12 The P o i n t o f View In C h a p t e r I I , S e c t i o n 2.32, t h e e i g h t b a s i c p o i n t s o f view as o u t l i n e d by T a y l o r (1961; p. 300) were g i v e n . They a r e t h e m o r a l , t h e a e s t h e t i c , t h e i n t e l l e c t u a l , t h e r e l i g i o u s , t h e economic, t h e p o l i -t i c a l , t h e l e g a l and t h e p o i n t o f view o f e t i q u e t t e o r custom. T a y l o r c o n s i d e r s t h e s e p o i n t s o f view t o be b a s i c because: :;" ". . . t h e c o n d u c t o f any g i v e n i n d i v i d u a l i n a c u l t u r e i s always s u b j e c t t o a v a l u e system b e l o n g i n g t o a t l e a s t one o f them and i s u s u a l l y s u b j e c t t o v a l u e systems b e l o n g i n g t o more th a n one o f them. Second, t h e y a r e t h e dominant p o i n t s o f view i n a c u l t u r e , i n the sense t h a t t h e y s e t t h e v a l u e s o f the major s o c i a l i n s t i t u t i o n s and a c t i v i t i e s which c a r r y on t h e c i v i l i z a t i o n o f t h e c u l t u r e . " ( T a y l o r , 1961, p. 300) Moreover, i n a d d i t i o n to the e i g h t b a s i c p o i n t s o f view t h e r e a r e many n o n b a s i c p o i n t s o f view c o r r e s p o n d i n g t o p a r t i c u l a r group i n t e r e s t s . 42 " . . . i t i s a p p r o p r i a t e t o j u d g e t h e o r g a n i z a t i o n i n terms o f i t s group i n t e r e s t , t h a t i s , i n terms o f t h e purposes o f the group as a whole (as d i s t i n c t f r o m t h e s e l f - i n t e r e s t o f each member o f t h e g r o u p ) . " ( T a y l o r , 1961, pp. 301-302) ' A f a c u l t y o f e d u c a t i o n e x i s t s f o r t h e c l e a r l y d e f i n e d p u r p o s e o f p r o d u c i n g good t e a c h e r s , ( u s i n g t h e b e s t t e c h n i q u e s p o s s i b l e ) , and can l e g i t i m a t e l y be j u d g e d a c c o r d i n g t o i t s a b i l i t y t o p e r f o r m t h a t f u n c t i o n . S i m i l a r l y , so can any p a r t i c u l a r program o f t e a c h e r educa-t i o n i n t h e f a c u l t y . T h e r e f o r e , t h e e v a l u a t o r can e v a l u a t e a program o f t e a c h e r e d u c a t i o n from the i n s t i t u t i o n a l p o i n t o f view, t h a t i s , from t h e p o i n t o f view o f a f a c u l t y o f e d u c a t i o n . T h i s p o i n t o f view would encompass t h e l o g i s t i c s o f t e a c h e r e d u c a t i o n , such as o p p o r -t u n i t i e s f o r p r a c t i s e t e a c h i n g , a v a i l a b l e methods c o u r s e , p r o f e s s o r s a v a i l a b l e t o t e a c h t h e s t u d e n t s , f i n a n c i n g o f programs and so on. In o t h e r words, t h e e v a l u a t o r would use t h e s e c r i t e r i a f o r j u d g i n g t h e a c t i v i t i e s and p o l i c i e s o f a program o p e r a t i n g under t h e a u s p i c e s o f a f a c u l t y o f e d u c a t i o n . I t i s a l s o n e c e s s a r y f o r t h e e y a l u a t o r t o j u d g e a t e a c h e r e d u c a t i o n program from t h e e d u c a t i o n a l p o i n t o f view. I f we a r e educa- t i n g p e o p l e t o be t e a c h e r s , r a t h e r t h a n s i m p l y t r a i n i n g them t o be t e a c h e r s , t h e n t h e p r a c t i s e s and p o l i c i e s o f a program o f t e a c h e r e d u c a t i o n must be j u d g e d i n terms o f t h e i r e d u c a t i o n a l w o r t h . I t i s n e c e s s a r y t h a t a l l programs be e v a l u a t e d from t h e moral p o i n t o f view and t h e l e g a l p o i n t o f view "... s i n c e [ t h e Program's] purposes a r e t o a c h i e v e c e r t a i n g o a l s o r t o p u r s u e c e r t a i n ends w i t h o u t v i o l a t i n g t h e - m o r a l code o f t h e s o c i e t y and w i t h o u t b r e a k i n g S o c i e t y ' s l aws." ( T a y l o r , 1961, p. 300) 43 T h e r e f o r e , Program B w i l l be e v a l u a t e d from t h e p o i n t o f view o f t h e f a c u l t y o f e d u c a t i o n ( t h a t i s , t he i n s t i t u t i o n a l p o i n t o f v i e w ) , from t h e e d u c a t i o n a l p o i n t o f view, and from t h e moral and l e g a l p o i n t s o f view. Some o t h e r b a s i c p o i n t s o f vi e w w h i c h c o u l d be c o n s i d e r e d i n t h e e v a l u a t i o n a r e t h e economic, t h e p o l i t i c a l , t h e i n t e l l e c t u a l , and the p o i n t o f view o f e t i q u e t t e o r custom. However, Program c o s t s were not a c o n s i d e r a t i o n i n t h i s e v a l u a t i o n so no e v a l u a t i o n was done from t h e economic p o i n t o f view; t h e i n t e l l e c t u a l p o i n t o f view was i n c l u d e d i n the e d u c a t i o n a l p o i n t o f view; and t h e p o l i t i c a l and e t i q u e t t e o r custom p o i n t s o f view were beyond t h e scope o f t h i s t h e s i s . 3.13 G r a d i n g The method o f e v a l u a t i o n d e s c r i b e d by t h i s t h e s i s i s S-GRADING. (see C h a p t e r I I , S e c t i o n 2.33). T h a t i s , Program B i s not d i r e c t l y compared t o o t h e r e x i s t i n g programs i n terms o f i t s g o a l s and p e r f o r m a n c e . R a t h e r , Program B i s gr a d e d a c c o r d i n g t o the d e g r e e to which i t f u l f i l l s a s e t o f s t a n d a r d s which a r e c o n s i d e r e d by the e v a l u a t o r t o be a p p r o p r i a t e to t h e Program. S-GRADING.is r e a s o n a b l e i n f o r m a t i v e e v a l u a t i o n , where t h e e v a l u a t o r must make judgements as t o t h e worth o f t h e program f o r t h e purposes o f i m p r o v i n g i t and where comparison w i t h o t h e r Programs i s not a major c o n c e r n . R e f e r r i n g t o T a b l e I i n C h a p t e r I I , i t was shown how t h e most comprehensive e v a l u a t i v e c o n c l u s i o n which c o u l d be r e a c h e d i n S-GRADING 44 (when c o s t s have not been t a k e n i n t o a c c o u n t ) , i s t h e " O v e r a l l E v a l u a t i v e C o n c l u s i o n . " T h e r e f o r e , t h e e v a l u a t i o n d e s c r i b e d by t h i s t h e s i s a t t e mpted t o answer t h e f o l l o w i n g q u e s t i o n : "Did t r e a t m e n t X p r o v i d e d by Program B, have t h e e f f e c t o f p r o d u c i n g u n d e r g r a d u a t e s t u d e n t t e a c h e r s ( S ) , competent i n t h e c l a s s r o o m (Y) ( f o r t h i s l e v e l o f u n i v e r s i t y t r a i n i n g ) ; w i t h t h e p r e m i s e s t h a t X was the t r e a t m e n t and i t caused Y; t h e S's e n j o y e d , d e s i r e d and b e n e f i t t e d by Y, and Y had no harmful e f f e c t s on S's ( o r non S ' s ) ? " 3.20 A d o p t i o n o f S t a n d a r d s I t i s u s e f u l t o l o o k a t a g e n e r a l l y a g r e e d upon d e s c r i p t i o n o f e d u c a t i o n i n o r d e r t o e s t a b l i s h t h e broad s t a n d a r d s t o be used i n t h i s e v a l u a t i o n . P e t e r s (1966, p. 3) p r o v i d e d t h i s d e f i n i t i o n when he w r o t e : " [ E d u c a t i o n ] i m p l i e s t h a t something w o r t h - w h i l e i s b e i n g o r has been i n t e n t i o n a l l y t r a n s m i t t e d i n a m o r a l l y a c c e p t a b l e manner." Thus, t h e r e a r e two c r i t e r i a f o r e v a l u a t i n g e d u c a t i o n a l p r o c e s s e s . The f i r s t i s t h e worth o f t h e p r o c e s s ( o r p r o d u c t ) g o a l s and t h e second i s t h e " t r a n s m i s s i o n i n a m o r a l l y a c c e p t a b l e manner." " . . . t o e d u c a t e someone i m p l i e s not o n l y some s o r t o f a c h i e v e m e n t , but a l s o one t h a t i s w o r t h - w h i l e . I t a l s o i m p l i e s t h a t the manner o f d o i n g t h i s s h o u l d not be m o r a l l y o b j e c t i o n a b l e . " ( P e t e r s , 1966, p. 4) I t i s t h i s e s s e n t i a l d u a l i t y o f end p r o d u c t and p r o c e s s which has been o f t e n o v e r l o o k e d i n e d u c a t i o n a l e v a l u a t i o n . S e r i y e n (1967, p. 51) w r o t e , 45 "One o f t h e r e a c t i o n s t o t h e t h r e a t o f e v a l u a t i o n . . . was the extreme r e l a t i v i z a t i o n o f e v a l u a t i o n r e s e a r c h . The s l o g a n became: How w e l l does t h e c o u r s e a c h i e v e i t s g o a l s ? i n s t e a d o f How good i s t h e c o u r s e ? But i t i s o b v i o u s t h a t i f t h e g o a l s a r e n ' t worth a c h i e v i n g t h e n i t i s u n i n t e r e s t i n g how w e l l t h e y a r e a c h i e v e d . . . Thus e v a l u a -t i o n p r o p e r must i n c l u d e as an equal p a r t n e r w i t h t h e measuring o f p e r -formance a g a i n s t g o a l s , p r o c e d u r e s f o r t h e e v a l u a t i o n o f t h e g o a l s . " T h e r e f o r e , f o r t h e e v a l u a t i o n model d e v e l o p e d i n t h i s t h e s i s , two s t a n d a r d s w i l l be used. The f i r s t s t a n d a r d i s goal worth. The d e g r e e t o which P r o g r a n r B f u l f i l l s o r f a i l s t o f u l f i l l t h i s s t a n d a r d i s dependent on the worth o f t h e Program's g o a l s . The second s t a n d a r d i s goal a c h i e v e m e n t . The degree t o which Program B f u l f i l l s o r f a i l s t o f u l f i l l t h i s s t a n d a r d i s dependent on whether o r not t h e o p e r a t i o n o f t h e Program r e s u l t s i n t h e g o a l s b e i n g a c h i e v e d . T h e r e i s one r e s t r i c t i o n p l a c e d on t h e s t a n d a r d o f g o a l a c h i e v e - , ment. T h a t i s , i t s d e g r e e o f f u l f i l l m e n t s h o u l d o n l y be d e t e r m i n e d i f the g o a l s have been shown t o be w o r t h w h i l e . 3.21 S t a n d a r d 1: The Worth o f t h e Program G o a l s One a p p r o a c h t o e s t a b l i s h i n g t h e worth o f a s e t o f g o a l s i s to d e t e r m i n e t h e e x t e n t t o w h i c h t h e y a r e based on d e f e n s i b l e needs and/or wants. F o r some, d e f e n s i b l e needs a r e seen as a r i s i n g o u t o f the d i s c r e p a n c y between a d e s i r e d s t a t e o f a f f a i r s i n e d u c a t i o n and t h e c u r r e n t s t a t e o f a f f a i r s . Both Popham (1969, p. 23) and Provus (1.972, p. 37) have a d v o c a t e d t h i s d i s c r e p a n c y as a measure o f e d u c a t i o n a l need. Popham wrote t h a t d e s i r e d l e a r n e r outcome minus c u r r e n t l e a r n e r s t a t u s e q u a l s an e d u c a t i o n a l need. P r o v u s , s i m i l a r l y , 46 wrote t h a t t h e d i s c r e p a n c y between i d e a l i z e d performance, o f c i t i z e n s and a c t u a l p e r f o r m a n c e e q u a l s need. However, t h i s does not n e c e s s a r i l y d e t e r m i n e need i n e d u c a t i o n . Suppose t h e " d e s i r e d l e a r n e r outcomes" o r " i d e a l i z e d p e r f o r m a n c e o f c i t i z e n s " have not been shown ( i n t h e c o n t e x t o f a p a r t i c u l a r e d u c a t i o n a l e n v i r o n m e n t ) t o be w o r t h -w h i l e g o a l s ; o r , suppose t h e y a r e i n f a c t w o r t h l e s s g o a l s ; i n t h a t c a s e , t h e d i s c r e p a n c y between c u r r e n t l e a r n e r s t a t u s and t h o s e g o a l s c o u l d not be d e s c r i b e d as an e d u c a t i o n a l need. T h i s argument c o u l d be c o u n t e r e d , o f c o u r s e , by s a y i n g t h a t d e s i r e d l e a r n e r outcomes would never be d e s i r e d u n l e s s t h e y were w o r t h w h i l e . T h i s can be shown t o be f a l l a c i o u s by u s i n g t h e " o p e n - q u e s t i o n argument." Frankena (1973, ! p. 99) wrote about t h i s t y p e o f argument: "Suppose t h a t a d e f i n i s t h o l d s t h a t "good" o r " r i g h t " means " h a v i n g t h e p r o p e r t y P," f o r example, " b e i n g d e s i r e d " o r " b e i n g c o n d u c i v e t o t h e g r e a t e s t g e n e r a l h a p p i n e s s . " Then,."the argument i s t h a t we may a g r e e t h a t something has P, and y e t ask s i g n i f i c a n t l y , "But i s i t good?" o r " I s i t r i g h t ? " T h a t i s , we can s e n s i b l y s a y , " T h i s has P, b u t i s i t good ( o r r i g h t ) ? " But i f t h e p r o p o s e d d e f i n i -t i o n were c o r r e c t , then we c o u l d not say t h i s s e n s i b l y f o r i t would be e q u i v a l e n t to s a y i n g , " T h i s has P, but has i t P?" . . . T h e r e f o r e the d e f i n i t i o n cannot be c o r r e c t . " S i n c e we can ask the q u e s t i o n " I s a d e s i r e d l e a r n e r outcome a w o r t h -w h i l e outcome?" then we c a n n o t assume t h a t " d e s i r e d " e q u a l s "worth-w h i l e " . Need i n t h i s s e n s e o f b e i n g a d i s c r e p a n c y between a d e s i r e d and a c t u a l s t a t e i s e s s e n t i a l l y an i n d i c a t i o n o f d e g r e e o f g o a l a c h i e v e m e n t w i t h o u t n e c e s s a r i l y s e e i n g i f t h e g o a l s a r e w o r t h a c h i e v i n g . S c r i v e n has i d e n t i f i e d needs as b i o l o g i c a l . He wrote ( S c r i y e n , 47 1974b, p. 13) "True, needs a s s e s s m e n t s i n v o l v e e s t a b l i s h i n g t h a t t h e p r o d u c t a c t u a l l y f a c i l i t a t e s s u r v i v a l , h e a l t h , or some o t h e r d e f e n s i b l e need t h a t i s not a d e q u a t e l y s e r v i c e d . . ." However, i n e d u c a t i o n t h e c r i t e r i a o f s u r v i v a l e t c . , f o r e s t a b -l i s h i n g t h e worth o f t h e program's g o a l s , i s o f t e n t o o r e s t r i c t i v e . As Komisar wrote (Komi'sar, ,1961, ,p. 3 7 ) : " S u r v i v a l ... i s too m i s e r l y a c o n c e p t i o n , s i n c e t h e r e a r e d e f i c i e n c i e s which a r o u s e us w i t h o u t t h e t h r e a t o f d e a t h ( e . g . a f f e c t i o n ) . " To d e t e r m i n e i f t h e g o a l s o f t h e program, o r t h e d e s i r e d l e a r n e r outcomes o f t h e program a r e w o r t h w h i l e r e q u i r e s v a l i d a t i o n o f t h e g o a l s o r outcomes. T h i s i s . a c c o m p l i s h e d by showing: ( T a y l o r , 1961, p. 84-85, D a n i e l s , 1971, p. 1 2 ) . 1) T h a t f u l f i l l i n g t h e g o a l s e n a b l e s one t o f u l f i l l h i g h e r g o a l s . 2) T h a t f u l f i l l i n g t h e g o a l s does not c o n f l i c t , w i t h a n o t h e r p r i n c i p l e o f o u r way o f l i f e which i s o f importance t o us. 3) T h a t f u l f i l l i n g t h e g o a l s l e a d s t o b e n e f i c i a l c o n s e q u e n c e s , and 4) T h a t t h e r e a r e no grounds f o r making an e x c e p t i o n t o t h e g o a l s i n t h i s s i t u a t i o n . S c r i v e n (1966a) o f f e r e d t h e f o l l o w i n g s i m i l a r method f o r d e t e r m i n i n g i f something i s worth doing.: 1) I f something w i l l b r i n g about a s t a t e o f a f f a i r s t h a t p e o p l e 48 v a l u e , t h a t i s a good prima f a c i e r e a s o n f o r d o i n g i t . 2) I f t h e r e a r e prima f a c i e r e a s o n s f o r d o i n g something and none a g a i n s t , we s h o u l d do i t . 3) I f t h e r e i s a c o n f l i c t o f s u p p o r t a b l e prima f a c i e r e a s o n s , appeal must be made t o a g e n e r a l moral p r i n c i p l e . 3.22 S t a n d a r d 2: Goal Achievement Goal a c h i e v e m e n t i s a measure o f t h e e f f e c t i v e n e s s o f t r a n s m i t t i n g something i n a m o r a l l y a c c e p t a b l e manner t o a c h i e v e t h e w o r t h w h i l e g o a l s . In o t h e r words i t i s a check.on t h e p e r f o r m a n c e o f a p a r t i c u l a r evaluatum i n a c h i e v i n g t h e w o r t h w h i l e g o a l s . Moral c o n s i d e r a t i o n s , i n c o n s i d e r i n g t h e s t a n d a r d o f g o a l a c h i e v e m e n t , must t a k e p r e c e d e n c e o v e r any r a t i n g based on p e r f o r m a n c e . I f t h e program i s immoral t h e n i t s h o u l d be s t o p p e d r e g a r d l e s s o f how w e l l i t a p p e a r s to be a c h i e v i n g t h e g o a l s . I t i s t h i s check on m o r a l i t y which i s o f t e n n e g l e c t e d i n t h e e v a l u a t i o n o f e d u c a t i o n a l p r o d u c t s and p r o p o s a l s . I t i s a u s e f u l e x e r c i s e t o e v a l u a t e J o n a t h a n S w i f t ' s (1667-1745) "A Modest P r o p o s a l f o r P r e v e n t i n g t h e C h i l d r e n o f Poor P e o p l e f r o m Being a B u r t h e n t o t h e i r P a r e n t s o r C o u n t r y , and f o r Making them B e n e f i c i a l t o t h e P u b l i c , " u s i n g t h e major e v a l u a t i o n models, to i l l u s t r a t e t h e c o n -sequences o f o m i t t i n g a check on m o r a l i t y . Most o f t h e models would f i n d t h e p r o p o s a l q u i t e a c c e p t a b l e . T h i s i s not t o s u g g e s t t h a t c a n n a b i l i s m would be o v e r l o o k e d i n t h e e v a l u a t i o n o f an e d u c a t i o n a l 49 p r o d u c t ; b u t , o t h e r t r a n s g r e s s i o n s such as i n d o c t r i n a t i o n m i g h t be c o n v e n i e n t l y i g n o r e d o r not l o o k e d f o r , u n l e s s t h e r e i s an e x p l i c i t s t a t e m e n t r e q u i r i n g a m o r a l i t y c h e c k . 3.30 O p e r a t i o n a l C l a r i f i c a t i o n o f S t a n d a r d s In t h i s s e c t i o n each s t a n d a r d w i l l be e x p l i c a t e d i n t h e terms used to e v a l u a t e t h e t e a c h e r t r a i n i n g program, Program B. 3.31 S t a n d a r d 1: The Worth o f Program B's G o a l s The g o a l s o f t h e program can be d e r i v e d from t h e t y p e o f program b e i n g e v a l u a t e d . The e v a l u a t o r does not l o o k a t the program's p l a n s o r p e r f o r m a n c e t o d e t e r m i n e i f i t s g o a l s a r e w o r t h w h i l e ; i n s t e a d he l o o k s a t t h e c h a r a c t e r i s t i c s o f t h e o v e r a l l program g o a l s and v a l i d a t e s them as b e i n g o r not b e i n g w o r t h w h i l e . Program B i s a p a r t i c u l a r method o f t r a i n i n g t e a c h e r s , but r e g a r d l e s s o f t h e t e c h -n i q u e s used, t h e u l t i m a t e g o a l o f t h e Program i s t o t r a i n p e o p l e t o be good t e a c h e r s . I t i s t h a t goal which i s v a l i d a t e d . As T a y l o r (1961, p. 301-302) s u g g e s t e d , i f you a r e e v a l u a t i n g the a c t i v i t i e s and p o l i c i e s o f a group i n terms o f t h a t g r o u p ' s i n t e r e s t then you would e v a l u a t e t h e worth o f t h e g o a l s and the d e g r e e o f g o a l achievement as t h e y p e r t a i n e d t o o n l y t h a t g r o u p . The p o i n t o f view t o be adopted would be t h e p o i n t o f view o f t h e i n s t i t u t i o n o r , i n t h i s c a s e , t h e f a c u l t y o f e d u c a t i o n under whose a u s p i c e s Program B i s f u n c t i o n i n g . On t h e o t h e r hand, you can e v a l u a t e 50 Program B o u t s i d e o f t h e group's framework. I t i s now l e g i t i m a t e t o ask how t h e Program's a c t i v i t i e s and p o l i c i e s conform t o t h e r u l e s and s t a n d a r d s o f an e n l i g h t e n e d p u b l i c . The p o i n t o f yiew t o be adopted would be t h e e d u c a t i o n a l p o i n t o f view. . Program B, as d e s c r i b e d i n t h e S e c t i o n 3.11 (The Evaluatum) d i d not have a p a r t i c u l a r l y d e t a i l e d l i s t o f g o a l s ; t h e r e f o r e a s e t o f w o r t h w h i l e g o a l s was c o n s t r u c t e d u s i n g a w e l l t h o u g h t - o u t model o f t e a c h e r e d u c a t i o n . T h i s model was s u p p l i e d by Broudy (1965). In h i s a r t i c l e he i d e n t i f i e d t h e t e a c h e r as b e i n g a member o f t h e e d u c a t i o n a l p r o f e s s i o n and as a s p e c i a l i s t . As a member o f the e d u c a t i o n a l p r o f e s s i o n a p o t e n t i a l t e a c h e r r e q u i r e s : (1) B a s i c c o u r s e s i n h i s t o r y , p h i l o s o p h y , p s y c h o l o g y and t h e s o c i a l s c i e n c e s , as p a r t o f h i s g e n e r a l e d u c a t i o n . (2) Study o f t h e h i s t o r i c a l , p h i l o s o p h i c a l , p s y c h o l o g i c a l and s o c i o l o g i c a l f o u n d a t i o n s o f e d u c a t i o n a l p o l i c y , c u r r i c u l u m , o r g a n i z a -t i o n and s u p p o r t and s t r a t e g i e s o f t e a c h i n g and l e a r n i n g . The t e a c h e r as a s p e c i a l i s t r e q u i r e s : (3) The h i s t o r i c a l , p h i l o s o p h i c a l , p s y c h o l o g i c a l and s o c i o l o g i c a l f o u n d a t i o n s o f t h e s p e c i a l t y . (4) Study o f s u b j e c t m a t t e r o v e r and above t h a t t a k e n i n g e n e r a l e d u c a t i o n as needed f o r t h e s p e c i a l t y . (5) L a b o r a t o r y e x e r c i s e s : a b s t r a c t e d and o f t e n s c h e m a t i z e d samples o f a whole c l a s s o f r e a l t a s k s such as a made-up g r o u p i n g 51 t a s k , a m a k e - b e l i e v e t e s t , a d e m o n s t r a t i o n , e t c . Broudy r e f e r s t o t h i s as l a b o r a t o r y e x e r c i s e s . (6) C l i n i c a l e x p e r i e n c e ; e s p e c i a l l y c o n s t r u c t e d s i t u a t i o n s to de m o n s t r a t e a c t u a l examples o f s t u d i e d t h e o r y . F o r example, t h e c l a s s i c c a s e o f t h e slow l e a r n e r o r d i s c i p l i n e problem. (7) I n t e r n s h i p : t he s t u d e n t - t e a c h e r spends time w o r k i n g on a r e a l t a s k under a minimum o f s u p e r v i s i o n , i . e . t h e p r a c t i c e - t e a c h i n g segments o f t e a c h e r e d u c a t i o n . (8) R e s e a r c h : The t e a c h e r ought t o be a consumer o f r e s e a r c h and may be a p r o d u c e r o f r e s e a r c h . T h e r e f o r e he s h o u l d be i n t r o d u c e d to to h i s methods, canons, s t a t u s and p r o s p e r i t y . The f i r s t f o u r r e q u i r e m e n t s r e f e r t o t h e f o u n d a t i o n s o f t h e e d u c a t i o n a l f i e l d and a r e a o f s p e c i a l i z a t i o n i n t h i s f i e l d . As su c h t h e y c o n s t i t u t e e d u c a t i o n a l g o a l s o f a f a c u l t y o f e d u c a t i o n . R e q u i r e -ments (5) - (7) r e f e r t o t h e t e c h n o l o g y ( i . e . t h e way and means) o f a p p l y i n g t h e o r y to p r a c t i c e . Thus, t h e y can be seen as i n s t i t u t i o n a l  g o a l s , because t h e y p r i m a r i l y s e r v e t h e i n t e r e s t s o f t h e i n s t i t u t i o n ( e . g . i m p r o v i n g c l a s s r o o m p r a c t i c e ) . The t e c h n o l o g y i s n o t u n i q u e t o ' e d u c a t i o n ' , but i t i s o f i n t e r e s t t o t h e f i e l d o f e d u c a t i o n . R e s e a r c h , r e q u i r e m e n t 8, i s an i n s t i t u t i o n a l g o a l i n t h a t i t s e r v e s t h e i n t e r e s t s o f t h e f i e l d . Broudy's p a p e r , as b r i e f l y d e s c r i b e d aboye, forms the b a s i s f o r d e t e r m i n i n g t h e de g r e e t o which Program B has w o r t h w h i l e g o a l s f r o m t h e e d u c a t i o n a l and i n s t i t u t i o n a l p o i n t s o f view. 52 I t i s i m p o r t a n t t o p o i n t o u t t h a t Program B i s o n l y one y e a r o f a f o u r y e a r t e a c h e r t r a i n i n g program. T h e r e f o r e p a r t s o f Broudy's c r i t e r i a w i l l be f u l f i l l e d e i t h e r b e f o r e o r a f t e r Program B. T h i s would be t a k e n i n t o c o n s i d e r a t i o n i n t h e f i n a l , o v e r a l l e v a l u a t i o n o f t h e worth o f t h e program. 3.32 S t a n d a r d 2: The d e g r e e o f Goal Achievement Assuming t h a t t h e g o a l s o f a program have been i d e n t i f i e d and a r e w o r t h w h i l e , t h e n t he e v a l u a t o r would a t t e m p t t o d e t e r m i n e i f t h e program i s a c h i e v i n g t h e g o a l s . O f t e n , t h e program d e y e l o p e r s w i l l have w r i t t e n o u t , i n g r e a t d e t a i l , how t h e y hope t o a c h i e v e t h e ( w o r t h w h i l e ) g o a l s . I t i s u s e f u l f o r t h e e v a l u a t o r t o examine t h e s e w r i t t e n i n t e n t s t o check t h e i r v a l i d i t y . Some r e a s o n s f o r t h i s , as S c r i v e n (1974a, p. 154) p o i n t s o u t c o u l d be: (1) new programs on the market c o u l d make a s w i t c h t o them w o r t h w h i l e . (2) The p e r f o r m a n c e e v i d e n c e o f t h e e x i s t i n g program, o r o t h e r s -£0 which i t was compared, may have changed. (3) U n a n t i c i p a t e d d i f f i c u l t i e s ( p o l i t i c a l , m o r a l , economic) may a ^ i s e which have not o c c u r r e d i n o t h e r a l t e r n a t e programs o r methods (4) t h e o r i g i n a l d e c i s i o n t o r u n t h e program may have been wrong due t o poor d a t a o r poor l o g i c . I f t h i s i s done, t h e n t h e n e x t s t e p i s t o i d e n t i f y a r e a s o f i n v e s t i g a t i o n which w i l l p r o v i d e t h e e v a l u a t o r w i t h t h e d a t a n e c e s s a r y t o d e t e r m i n e whether t h e g o a l s a r e b e i n g a c h i e v e d . S c r i v e n (1974b) has c o n t r i b u t e d a g r e a t d e a l to t h i s end by d e v e l o p i n g 53 a c h e c k l i s t which can be used t o e v a l u a t e p r o d u c t s , p r o d u c e r s and p r o p o s a l s . He s u p p o r t s t h e v a l i d i t y . a n d u t i l i t y o f t h e c h e c k l i s t by making t h e f o l l o w i n g f o u r p o i n t s ( S c r i v e n , 1974b, p. 8-9): (1) E v e r y c h e c k p o i n t has a c l e a r a p r i o r i r a t i o n a l e . F a i l u r e t o meet any one o f t h e c h e c k p o i n t s i m m e d i a t e l y c a u s e s doubt about t h e q u a l i t y o f t h e e v a l u a t i o n . (2) M e d i c a l and i n d u s t r i a l r o u t i n e l y p a s s , and o f t e n a r e r e -q u i r e d t o p a s s , e v e r y c h e c k p o i n t . (3) The c h e c k l i s t has been d e v e l o p e d o u t o f t h e P r o d u c t Review P a n e l s o f 1971-72 and 1972-73 done f o r t h e N a t i o n a l C e n t e r f o r E d u c a t i o n a l Communication, on s u b c o n t r a c t t o t h e E d u c a t i o n a l T e s t i n g S e r v i c e . (4) I t has been used by s e y e r a l hundred s c h o o l a d m i n i s t r a t o r s i n e v a l u a t i n g g r a d u a t e programs, and by s t u d e n t s i n t h e e v a l u a t i o n t r a i n i n g s e m i n a r a t t h e U n i v e r s i t y o f C a l i f o r n i a , B e r k e l e y . They f r e q u e n t l y s t a t e t h a t i t i s more v a l u e t o them d o i n g e v a l u a t i o n s t h a n a n y t h i n g e l s e a v a i l a b l e i n t h e l i t e r a t u r e . Each c h e c k p o i n t w i l l be d e s c r i b e d i n g e n e r a l terms, and i n terms o f Program B. T h i s d e s c r i p t i o n w i l l be f o l l o w e d by summary v e r s i o n o f t h e c h e c k p o i n t . The summary v e r s i o n c o n s i s t s o f a m e t h o d o l o g i c a l check and a s u b s t a n t i v e check. The m e t h o d o l o g i c a l ' p a r t i s a check on whether d a t a i s a v a i l a b l e ( o r whether i t can be o b t a i n e d ) , i t s s o u r c e , and t h e q u a l i t y o f t h a t 54 d a t a . The s u b s t a n t i v e p a r t i s a de g r e e o f f u l f i l l m e n t s c a l e . I t i n d i c a t e s how w e l l t h e c h e c k p o i n t has been met. In o r d e r f o r an evaluatum t o r e c e i v e a h i g h r a t i n g on a p a r t i c u l a r c h e c k p o i n t t h e r e must be good e v i d e n c e o f good p e r f o r m a n c e ; i . e . t h e d a t a i s c o m p l e t e , and i t i n d i c a t e s good p e r f o r m a n c e by the evaluatu m . A low s c o r e on a c h e c k p o i n t i m p l i e s t h a t e i t h e r good evidencef o r good p e r f o r m a n c e o r both i s l a c k i n g . " I t does not r e q u i r e , f o r example, t h a t t h e r e i s good e v i d e n c e o f bad pe r f o r m a n c e , f o r o t h e r w i s e p r o d u c t s which t u r n e d i n no d a t a would do b e t t e r t h a n t h o s e t h a t were known t o f a r e b a d l y . " ( S c r i v e n , 1974b, p. 22) 3.33 The C h e c k l i s t f o r D e t e r m i n i n g Good Achievement 3.331 C h e c k p o i n t 1: MARKET ( D i s s e r n i n a b i l i t y ) T h i s i s not an economic c h e c k p o i n t . I t r e q u i r e s t h a t , g i v e n w o r t h w h i l e g o a l s , t h e r e a r e d i s s e m i n a t i o n p l a n s t h a t e n s u r e consumers f o r t h e e d u c a t i o n a l p r o d u c t . I t would c o n s i d e r t h e c l a r i t y , f e a s i b i l i t y and i n g e n u i t y ( f o r u t i l i z i n g a v a i l a b l e r e s o u r c e s ) o f t h e i m p l e m e n t a t i o n p l a n s . To f u l f i l l t h i s c h e c k p o i n t to a h i g h d e g r e e Program B must show t h a t i t has s p e c i f i c , f e a s i b l e p l a n s to p r o v i d e t h e s t u d e n t s w i t h ( f o l l o w i n g Broudy's (1965) c r i t e r i a ) : a) The h i s t o r i c a l , p h i l o s o p h i c a l , p s y c h o l o g i c a l and s o c i e t a l f o u n d a t i o n s o f t h e s p e c i a l t i e s . 55 b) The p r o f e s s i o n a l c o n t e n t o f t h e s p e c i a l t i e s o ver and above t h a t t a k e n i n p r e v i o u s c o u r s e s . c) L a b o r a t o r y e x e r c i s e s , c l i n i c a l e x e r c i s e s and i n t e r n s h i p s . d) Methods o f p r o c e d u r e f o l l o w e d i n p r a c t i s e , p r o f e s s i o n a l t a s k s ( g r a d i n g , m a r k i n g , making t e s t s ) , and methods o f d e a l i n g w i t h t h e a p p a r a t u s and m a t e r i a l s o f t e a c h i n g . In a d d i t i o n , t h e r e must be s t u d e n t s who want t o e n r o l l i n Program B and p r o c e d u r e s t o e n r o l l them. T h e r e must be i n s t r u c t o r s a b l e t o t e a c h t h e c o u r s e s . A s c h o o l must be a y a i l a b l e w i t h a s t a f f w i l l i n g t o t a k e on Program B; p l u s a m y r i a d o f minor d e t a i l s such as books and m a t e r i a l s b e i n g p r e s e n t , f i l m s , a c c e s s to l i b r a r i e s , a r e a s f o r s t u d y , e t c . , w h i c h , c u m u l a t i v e l y , can haye c o n s i d e r a b l e impact on t h e s u c c e s s o f a program. Summary y e r s i o n o f Market C h e c k p o i n t . M e t h o d o l o g i c a l S u b s t a n t i v e r a t -E v i d e n c e F u l f i l l m e n t c h a r a c t e r i s t i c s i n g D i s s e m i n a t i o n p l a n : V e r y l a r g e and/or v e r y i m p o r t a n t market c l a r i t y w i l l be r e a c h e d 4 f e a s i b i l i t y L a r g e and/or i m p o r t a n t market w i l l be i n g e n u i t y r e a c h e d 3 economy S i g n i f i c a n t market w i l l p r o b a b l y be S i z e r e a c h e d 2 Importance P o s s i b l e , but not p r o b a b l e , t h a t a O t h e r s i g n i f i c a n t m a r k e t w i l l be r e a c h e d 1 Inadequate e v i d e n c e t o s u g g e s t t h a t a s i g n i f i c a n t market w i l l be r e a c h e d 0 3.332 C h e c k p o i n t 2: TRUE FIELD TRIALS. How t y p i c a l a r e t h e p e o p l e , t h e s e t t i n g , t h e time 56 a l l o t t e d to t h e f i n a l v e r s i o n o f t h e program? " I t i s y e r y t e m p t i n g t o t h i n k one can e x t r a p o l a t e f r o m f i e l d t r i a l s w i t h v o l u n t e e r s c h o o l s . . . but t h i s has f r e q u e n t l y p r o v e d unsound . . . i n a c t u a l p r a c t i s e , d e a d l i n e s , overcommitment, and u n d e r f i n a n c i n g combine t o r e n d e r a l m o s t a l l p r o d u c t s d e f i c i e n t on t h i s c h e c k p o i n t . " ( S c r i v e n , 1974b, p. 11) The p a r a l l e l i n s t a t i s t i c s i s t h e c o n c e p t o f g e n e r a l i z a b i 1 i t y . I f Program B i s t o o b t a i n a h i g h s c o r e on t h i s c h e c k p o i n t , t h e n i t must be-shown t h a t t h e s c h o o l , i n w h i c h t h e program was h e l d , was not p a r t i c u l a r l y a t y p i c a l . T h i s i s t r u e r e g a r d l e s s o f whether t h e s c h o o l was e x c e p t i o n a l l y good o r e x c e p t i o n a l l y bad. I f i t was e x c e p t i o n a l l y good i t i s o b v i o u s t h a t t h e program p r o b a b l y r e c e i v e d more o f i t s s h a r e o f h e l p t o be a good program. I f i t was e x c e p t i o n -a l l y bad, t h e n t h e t e m p t a t i o n i s to say: " I f i t can work under t h o s e t e r r i b l e c o n d i t i o n s , i t c a n work anywhere!" But t h a t i s n o t n e c e s s a r i l y the c a s e , u n l e s s i t has been shown t h a t i t can work anywhere. The s a f e s t b e t i s t o r u n t h e program i n an "a v e r a g e " s c h o o l and make a d j u s t m e n t s t o t h e program's d e v e l o p e d framework i f i t s u b s e q u e n t l y ends up i n a v e r y bad o r v e r y good s i t u a t i o n . Not o n l y s h o u l d t h e s c h o o l s e t t i n g be " a v e r a g e " , but so s h o u l d t h e u n i v e r s i t y s t a f f and t h e s t u d e n t t e a c h e r s . I f p o r t a b i l i t y o f Program B i s one o f i t s aims, t h e n t h e t r i a l v e r s i o n s s h o u l d not be composed o f e x c e p t i o n a l l y good o r bad s t a f f o r e x c e p t i o n a l l y good o r bad s t u d e n t t e a c h e r s . The a r g u m e n t s . a g a i n s t e x c e p t i o n a l l y good o r bad s t a f f and s t u d e n t s p a r a l l e l t h o s e g i v e n i n t h e p r e c e e d i n g p a r a g r a p h 57 f o r t h e s c h o o l . Summary v e r s i o n o f TRUE FIELD TRIALS C h e c k p o i n t M e t h o d o l o g i c a l S u b s t a n t i v e R a t i n g E v i d e n c e F u l f i l l m e n t C h a r a c t e r i s t i c s F i n a l v e r s i o n ? P e r f e c t l y t y p i c a l 4 T y p i c a l u s e r ? Minor d i f f e r e n c e s 3 T y p i c a l a i d ? R e a s o n a b l e b e t f o r g e n e r a l i z a t i o n 2 T y p i c a l s e t t i n g ? S e r i o u s weakness 1 T y p i c a l t i m e - f r a m e ? R e l e v a n c e u n c l e a r 0 Comment: Checks s h o u l d be made on a l l t h e e v i d e n c e s o u r c e s . T h i s w i l l p r o v i d e a good base f o r t h e r a t i n g g i v e n . 3.333 C h e c k p o i n t 3: TRUE. CONSUMER T h i s c h e c k p o i n t o u t l i n e s g a i n s e x p e c t e d o f y a r i o u s groups (consumers) i n v o l v e d i n Program B. The t y p e s o f g a i n s w i l l u s u a l l y depend on t h e p o i n t o f view t a k e n i n the e v a l u a t i o n . D e c i d i n g on t h e da t a r e q u i r e s a c l e a r u n d e r s t a n d i n g o f t h e e v a l u a t i v e f u n c t i o n and " . . . which a u d i e n c e s i t i s a d d r e s s e d t o , commissioned by a n d - - r e g a r d l e s s o f t h e s e two c o n s i d e r a t i o n s — r e s p o n s i b l e t o . " ( S c r i v e n , 1974b; p. 1 2 ) . The d i f f e r e n t groups o f consumers may be i n t e r e s t e d i n d i f f e r e n t a s p e c t s o f t h e p r o j e c t , and, i n o r d e r t o p r o v i d e d a t a f o r a l l t h e r e l e v a n t g r o u p s , each a s p e c t s h o u l d be examined. T h i s i s e v a l u a t i n g t h e program from t h e p r u d e n t i a l p o i n t o f view. T h a t i s , each consumer o f Program B would i n e f f e c t be a s k i n g "What b e n e f i c i a l e f f e c t s does t h i s program have on me?" To f u l f i l l t h e r e q u i r e m e n t s o f t h i s c h e c k p o i n t , t h e e v a l u a t o r 58 would d e t e r m i n e the g o a l s , w i t h r e s p e c t t o t h e evaluatum, o f each consumer. He would t h e n g a t h e r d a t a on t h e s e g o a l s and f e e d them back t o t h e consumer f o r h i s a p p r a i s a l o f t h e b e n e f i c i a l a nd/or harmful e f f e c t s ( w i t h r e s p e c t t o t h e consumer's g o a l s ) . The e v a l u a t o r , however, must have o v e r r i d i n g judgement s i n c e he i s t h e one who t a k e s r e s p o n s i b i l i t y f o r t h e c o n c l u s i o n s . I t i s i m p o r t a n t t h a t t h e d a t a be o b j e c t i v e l y g a t h e r e d so t h a t t h e consumer c an make a v e r i f i a b l e judgement. Though t h e e v a l u a t o r t r i e s t o p r o v i d e t h e n e c e s s a r y d a t a , i t i s o b v i o u s l y n o t p o s s i b l e t o p r o v i d e " B e s t Buy" i n f o r m a t i o n i f t h e e v a l -u a t i o n i s " O v e r a l l " ( s e e T a b l e I ) . L i m i t a t i o n s o f t h i s s o r t must be p o i n t e d o u t e a r l y i n e v a l u a t i o n , and r e v i s e d up o r down as the e v a l -u a t i o n p r o c e e d s . The d i f f e r e n t consumers i n t h i s s t u d y a r e : a) The Dean o f E d u c a t i o n and h i s r e p r e s e n t a t i v e s . b) The U.B.C. s t a f f i n v o l v e d i n Program B. c) The S t u d e n t t e a c h e r s i n Program B. d) The P r i n c i p a l o f t h e s c h o o l i n w h i c h Program B o p e r a t e d . e) The T e a c h e r s i n t h e above s c h o o l . f ) The p u p i l s i n t h e above s c h o o l . g) The Sch o o l Board. h) F u t u r e consumers o f t h e program. The most i m p o r t a n t consumers f o r t h i s s t a g e o f the e v a l u a t i o n a r e t h e s t u d e n t t e a c h e r s i n Program B, and U.B.C. S t a f f , and u l t i m a t e l y 59 t h e p u p i l s i n t h e s c h o o l b e i n g exposed t o t h e s t u d e n t t e a c h e r s . Summary v e r s i o n o f TRUE CONSUMER c h e c k p o i n t . M e t h o d o l o g i c a l S u b s t a n t i v e E v i d e n c e F u l f i l l m e n t c h a r a c t e r i s t i c s R a t i n g D i s t r i c t ? F u l l d a t a on a l l r e l e v a n t "consumers" 4 P r i n c i p a l ? F a i r d a t a on a l l r e l e v a n t "consumers" 3 T e a c h e r ? Good d a t a on t h e most i m p o r t a n t "consumers" 2 S t u d e n t ? Weak d a t a on t h e most i m p o r t a n t "consumers" 1 Ta x p a y e r ? O n l y s p e c u l a t i o n about most i m p o r t a n t 0 O t h e r ? "consumers" 3.334 C h e c k p o i n t 4:. LONG TERM T h i s c h e c k p o i n t o u t l i n e s f o l l o w - u p p l a n s . These p l a n s a r e p a r t i c u l a r l y i m p o r t a n t when t h e p r o j e c t b e i n g e v a l u a t e d c a n be e x p e c t e d t o have l o n g range e f f e c t s — a s i s t h e d e s i r a b l e c a s e f o r most e d u c a t i o n a l p r o j e c t s . The f o l l o w - u p can a l s o i d e n t i f y any good o r bad s i d e e f f e c t s w h i c h may t a k e c o n s i d e r a b l e time t o s u r f a c e . Summary, v e r s i o n o f LQNGTERM, c h e c k p o i n t M e t h o d o l o g i c a l S u b s t a n t i v e E v i d e n c e F u l f i l l m e n t c h a r a c t e r i s t i c s Week t o month Good d i r e c t e v i d e n c e about t h e e f f e c t s a t l a t e r t i m e s needed. Month t o y e a r Some d i r e c t e v i d e n c e about t h e e f f e c t s a t l a t e r t i m e s needed. Year t o few F o l l o w - u p g i v e s r e a s o n a b l e s u p p o r t t o a y e a r s l a t e r c o n c l u s i o n about e f f e c t s when rieeded On-job o r l i f e - F o l l o w - u p o r o t h e r d a t a s u g g e s t s a c o n c l u s p a c e sample s i o n about e f f e c t s when needed. U s e l e s s o r no f o l l o w - u p ; no o t h e r grounds f o r i n f e r r i n g l o n g - t e r m e f f e c t s . 3.335 C h e c k p o i n t 5: SIDE EFFECTS T h i s broad c h e c k p o i n t p r o v i d e s a g e n e r a l p a t t e r n f o r R a t i n g 4 3 2 1 0 60 a f r e e r a n g i n g s e a r c h f o r any good o r bad program e f f e c t s on t h e c o n -sumers. I t i s p a r t i c u l a r l y i m p o r t a n t t h a t the e v a l u a t o r makes h i s s e a r c h under many d i f f e r e n t p o i n t s o f view. S c r i v e n (1967, pp. 77-80) has g i v e n examples o f t h e forms t h e s e s i d e - e f f e c t s may t a k e i n a new c u r r i c u l u m p r o j e c t . A p p l y i n g t h i s to t h e consumers o f Program B, t h e n examples o f s i d e e f f e c t s f o r each group a r e : 1) The U.B.C. S t a f f a) I n c r e a s e o r d e c r e a s e i n s t a t u s as a r e s u l t o f b e i n g c o n n e c t e d w i t h Program B. ( P r o m o t i o n , D e m o t i o n ) . b) Problems o r b e n e f i t s o f t r a v e l l i n g between t h e u n i v e r s i t y and t h e s c h o o l h o s t i n g Program B. c) The b e n e f i t s o r problems o f s p e n d i n g a l o t o f time on t h e Program; such a s : f a t i g u e , b e i n g a b l e t o d e v e l o p h i s / h e r own i d e a s , l o s s o f time f o r r e s e a r c h , e t c . 2) The S t u d e n t - T e a c h e r s . a) I n c r e a s e o r d e c r e a s e i n s t a t u s as a r e s u l t o f b e i n g c o n n e c t e d w i t h Program B. b) Problems o r b e n e f i t s o f b e i n g removed from t h e U n i v e r s i t y s e t t i n g ( e . g . the l i b r a r y , o t h e r p r o f e s s o r s , s t u d e n t s i n o t h e r c o u r s e s , e t c . ) c) F a t i g u e o r e n t h u s i a s m due t o h a v i n g t o t e a c h p u p i l s i n t h e h o s t s c h o o l , as w e l l as h a v i n g t o do academic work a t t h e same t i m e . 61 d) P rob l ems and b e n e f i t s o f s t u d y i n g , and w o r k i n g t o g e t h e r ( e x c h a n g i n g i d e a s , b e i n g c r owded , no q u i e t a r e a s , e n c o u r -a g i n g t h e d e s p o n d a n t , e t c . ) . e) P rob l ems and b e n e f i t s o f b e i n g i n c l o s e c o n t a c t w i t h t h e ho s t s c h o o l ' s t e a c h e r s ( s ee 2d). The Ho s t S c h o o l ' s T e a c h e r s ; a ) I n c r e a s e d o r d e c r e a s e d work l o a d ( h a v i n g a s t u d e n t t e a c h e r t o h e l p o r t o h i n d e r ) . b) I n c r e a s e d o r d e c r e a s e d s t a t u s . c ) O p p o r t u n i t y f o r p r o f e s s i o n a l d e v e l o pmen t . d) E m o t i o n a l p l e a s u r e s and p rob l ems h a v i n g s t u d e n t t e a c h e r s a lway s i n t h e s c h o o l , s t a f f - r o o m , and t h e i r c l a s s r o o m . The P r i n c i p a l a ) I n c r e a s e i n a d m i n i s t r a t i v e d u t i e s ( a l l o c a t i n g c l a s s r o o m s , c o o p e r a t i n g w i t h t h e U n i v e r s i t y s t a f f and s t u d e n t s , e t c . ) b) I n c r e a s e o r d e c r e a s e i n s t a t u s . c ) Fewer o r g r e a t e r numbers o f d i s c i p l i n e p r o b l e m s . The P u p i l s o f t h e H o s t - S c h o o l a ) D i d t h e y f i n d t h e f r e q u e n t c h a n g i n g f r o m s t u d e n t - t e a c h e r t o r e g u l a r t e a c h e r d i s t u r b i n g i n any way, o r b e n e f i c i a l i n any way. b) C o u l d t h e p u p i l s o b t a i n more i n d i v i d u a l a t t e n t i o n and h e l p as a r e s u l t o f t h e r e b e i n g t e a c h e r s , s t u d e n t - t e a c h e r and U . B . C . s t a f f i n t h e s c h o o l . 62 6) The Dean and h i s R e p r e s e n t a t i v e s a) Does t h e program p r e s e n t problems o r adyantages i n t i m e -t a b l i n g , a l l o c a t i n g s t a f f and rooms, and f i n a n c i n g . b) Has t h e Dean's o f f i c e r e c e i v e d a c c o l a d e s and/or abuse as a r e s u l t o f t h e Program f u n c t i o n i n g . 7) The S c h o o l Board a) See 6 ( a ) . b) Has t h e s c h o o l board o f f i c e r e c e i v e d a c c o l a d e s and/or abuse as a r e s u l t o f t h e program f u n c t i o n i n g under t h e Board's j u r i s d i c t i o n . The above a r e examples o f p o s s i b l e s i d e e f f e c t s . I t i s not meant t o be an e x h a u s t i v e l i s t . S i d e e f f e c t s d i f f e r f r o m p r o c e s s e f f e c t s i n t h a t t h e y a r e u n a n t i c i p a t e d and not p a r t o f t h e t r e a t m e n t . I f t h e s i d e e f f e c t s a r e p a r t o f t h e t r e a t m e n t r a t h e r t h a n a r e s u l t o f i t t h e n t h e y a r e p r o c e s s e f f e c t s . Summary v e r s i o n o f SIDE EFFECTS C h e c k p o i n t . M e t h o d o l o g i c a l S u b s t a n t i v e E v i d e n c e F u l f i l l m e n t c h a r a c t e r i s t i c s R a t i n g Comprehensive Meets a l l r e q u i r e m e n t s w e l l ; s i d e e f f e c t s s e a r c h ? o v e r a l l p o s i t i v e . 4 S k i l l e d ? G e n e r a l l y good; s i d e e f f e c t s o v e r a l l p o s i t i v e . 3 Independent? B a r e l y a c c e p t a b l e ; s i d e e f f e c t s o v e r a l l D u r i n g / E n d / p o s i t i v e o r n e u t r a l . 2 L a t e r Some s t u d y made, but i n c o m p l e t e . 1 No w o r t h w h i l e s t u d y . 0 Comments: The e v a l u a t o r s h o u l d use a l l t h e E v i d e n c e S o u r c e s . A h i g h r a t i n g on t h i s c h e c k p o i n t i n d i c a t e s t h a t the s i d e e f f e c t s a r e good and t h a t t h e q u a l i t y o f i n f o r m a t i o n on the s i d e e f f e c t s i s good. 63 3.336 C h e c k p o i n t 6: PROCESS The p r o c e s s c h e c k p o i n t i s m u l t i f a c e t e d . I t i s used t o s u b s t a n t i a t e o r i n v a l i d a t e d e s c r i p t i o n s o f t h e p r o d u c t , c a u s a l c l a i m s i n v o l v e d i n t h e e v a l u a t i o n ( t h a t t h e t r e a t m e n t caused t h e g a i n s ) and/or i t may be a r on moral q u e s t i o n s ( e . g . t h e i n t r i n s i c a l l y o r i n h e r e n t l y n e g a t i v e d i m e n s i o n s o f c r u e l t y , i n j u s t i c e and u n h a p p i n e s s , as w e l l as t h e i r o p p o s i t e s ) . The p a y - o f f emphasis i s o f p r i m a r y i m p o r t a n c e ( S c r i v e n , 1974b, p. 16) To f u l f i l l t h i s c h e c k p o i n t t o a h i g h d e g r e e r e q u i r e s t h r e e t h i n g s , t h e y a r e : a) Congruency: The program as d e s c r i b e d ( s e e S e c t i o n 3.11 "The evaluatum") and the i m p l e m e n t a t i o n p l a n s f o r t h e program ( s e e S e c t i o n 3.21, S t a n d a r d 2: MARKET) must be c o n g r u e n t w i t h what a c t u a l l y happened. b) C a u s a t i o n : I f t h e Program p r o p o n e n t s say t h a t Program B caused some e f f e c t Y, t h e n t h e r e must be a s e a r c h t o see i f Y i s p r e s e n t . T h i s i s s i m p l y a check f o r t h e p r e s e n c e o f Y, i t i s not an e x p l a n a t i o n f o r t h e cause o f Y- I f Y i s not p r e s e n t , then t h e Program's p r o p o n e n t s c l a i m t h a t Program B caused Y i s not c o r r e c t . I f Y i s p r e s e n t then t h e i r c l a i m may be c o r r e c t . The next c h e c k p o i n t , CAUSATION, d e t e r m i n e s i f o r i f not i t was Program B which caused Y. c) J u s t i c e : The moral d i m e n s i o n s o f t h e program must be o b s e r v e d . F or example t h e s e must be a s e a r c h f o r any i n j u s t i c e , u n h a p p i n e s s , c r u e l t y (and t h e i r o p p o s i t e s ) which i s p a r t o f the 64 p r o g r a m — a s opposed t o b e i n g a s i d e - e f f e c t o f t h e program ( S c r i v e n , 1974b, p. 16). Summary, V e r s i o n o f PROCESS C h e c k p o i n t . M e t h o d o l o g i c a l S u b s t a n t i v e E v i d e n c e F u l f i l l m e n t c h a r a c t e r i s t i c s R a t i n g D e s c r i p t i v e c o ngruence check? P a s s e s w i t h f l y i n g c o l o r s . 4 Ca u s a l c l u e s check? Appears s a t i s f a c t o r y . 3 Ins t r u m e n t v a l i d i t y ? R e a s o n a b l e r i s k . 2 J u d g e / o b s e r v e r r e l i a b i l i t y ? S i g n i f i c a n t o m i s s i o n ( s ) . 1 Inadequate 0 Comment: A l l o f t h e E v i d e n c e S o u r c e s s h o u l d be used by t h e e v a l u a t o r , r a t e d s e p a r a t e l y and t h e n combined i n t o an o v e r a l l p e r f o r m a n c e r a t i n g f o r PROCESS. 3.337 C h e c k p o i n t 7: CAUSATION. Whereas t h e PROCESS c h e c k p o i n t was a p r e s e n c e c h e c k f o r a p a r t i c u l a r c l a i m e d e f f e c t (E) o f Program B; CAUSATION a t t e m p t s to d e t e r m i n e i f i t was r e a l l y Program B which caused E. The pr o b l e m o f c a u s a t i o n i s e x t r e m e l y complex. As K e r l i n g e r (1973, p. 393) wrote: "... t h e s t u d y o f cause and c a u s a t i o n i s an e n d l e s s maze . . .we a g r e e t h a t c a u s a l l a w s . c a n n o t be d e m o n s t r a t e d e m p i r i c a l l y , but we a r e e q u i v o c a l about t h i n k i n g c a u s a l l y . T h e r e i s l i t t l e doubt t h a t s c i e n t i s t s do t h i n k c a u s a l l y and t h a t when t h e y t a l k o f a r e l a t i o n between "p" and "q" t h e y hope and b e l i e v e t h a t "p" causes "q",: S c r i v e n (1966b, p. 454) wr o t e : "... Can we n o t p r o c e e d f u r t h e r and d e f i n e " p o s s i b l e c a u s e s " i n terms o f some c o m b i n a t i o n o f n e c e s s a r y and s u f f i c i e n t c o n d i t i o n s , t h e s e b e i n g i n t e r p r e t e d as s i m p l e r e g u l a r i t y n o t i o n s ? The answer appea r s t o be t h a t we c a n n o t . " 65 With t h e l i m i t a t i o n t h a t when t h e word "cause" i s used, i t means " p o s s i b l e c a u s e " , S c r i v e n p r o v i d e s t h e f o l l o w i n g u s e f u l "developmental sequence" w i t h t he comment (1966b, p. 455) " [ i t ] does not e s t a b l i s h t h e common i d e a t h a t l a t e r members a r e s i m p l y complex c o m b i n a t i o n s o f t h e e a r l i e r ones." 1) B a s i c E x p e r i m e n t a l , Case: I f whenever and however C i s pr o d u c e d , E o c c u r s ; and E never o c c u r s u n l e s s C i s pro d u c e d t h e n C caused E. C i s produced a t random i n t e r v a l s . 2) B a s i c O b s e r v a t i o n Case: C i s o b s e r v e d t o o c c u r on v a r i o u s o c c a s i o n s and i s accompanied o r f o l l o w e d by E, and E never o c c u r s on o t h e r o c c a s i o n s . C i s t h e cause o f E i f we c o u l d r e d u c e t h e s i t u a t i o n t o a b a s i c e x p e r i m e n t a l c a s e , i . e . i f we c o u l d randomly c o n t r o l t h e ap p e a r a n c e o f C. I f t h i s i s n o t p o s s i b l e then t h e problem i s t o e l i m i n a t e t h e p o s s i b i l i t y t h a t some o t h e r f a c t o r X i s c a u s i n g both C and E. 3) Compound C a u s e s : Not o n l y C but a l s o D i s needed t o b r i n g about E. S c r i v e n r e f e r s t o C and D as c a u s a l f a c t o r s o r c o - c a u s e s o f E, 4) M u l t i p l e Causes: " I f C and D a r e each s u f f i c i e n t t o b r i n g about E, and n o t h i n g e l s e i s , t h e n w h i c h e v e r o c c u r s i s t h e c a u s e . I f both o c c u r , one o f them may not have had any e f f e c t on t h i s o c c a s i o n , a p o s s i b i l i t y which we check by exa m i n i n g t h e s i t u a t i o n f o r t h e p r e s -ence o f known i n t e r m e d i a t e l i n k s w hich c h a r a c t e r i z e t h e modus o p e r a n d i 6 f C and D, i . e . any s e t s o f c o n d i t i o n s "C| o r C,, o r . . ." ( o r "D^ o r D ? o r . . .") which a r e n e c e s s a r y f o r C ( o r D) t o a c t as the cause o f 66 E. T h i s t e s t does not a p p l y where no such l i n k s a r e known, and s i n c e t h a t i t i s not l o g i c a l l y n e c e s s a r y t h a t t h e r e be any (C and E may be a d j a c e n t l i n k s i n t h e c h a i n , o r d i f f e r o n l y f r o m a c e r t a i n d e s c r i p -t i v e s t a n d p o i n t , o r r e p r e s e n t " a c t i o n a t a d i s t a n c e " ) , t h e t e s t i s not p a r t o f t h e meaning, ..." ( S c r i v e n , 1966b, p. 455-56). I t i s " M u l t i p l e C a u s es" which i s . d f , p a r t i c u l a r . i n t e r e s t t o ' t h i s t h e s i s . In S-GRADING, t h e ab s e n c e o f c o n t r o l groups and t h e i n a b i l i t y o f a f o r m a t i v e e v a l u a t o r t o randomly a s s i g n p a r t i c u l a r t r e a t m e n t s " c " e l i m i n a t e p r o v i n g t he " B a s i c E x p e r i m e n t a l Case", t h e " B a s i c O b s e r v a t i o n Case" and "Compound Causes." However, t h e modus o p e r a n d i (MO) methods b r i e f l y d e s c r i b e d by S c r i v e n i n " M u l t i p l e Causes" and e l a b o r a t e d by him i n E v a l u a t i o n P e r s p e c t i v e s and P r o c e d u r e s ( S c r i v e n , 1974b, p. 68-84), p r o v i d e s a u s e f u l method o f d e t e r m i n i n g a c a u s a l i n f e r e n c e p a t t e r n . As he p o i n t o u t , t h e MO method i s r o u t i n e l y used by a n t h r o p o l o g i s t s and h i s t o r i a n s . ( S c r i v e n , 1974b, p. 6 8 ) . B r i e f l y , t h e problem i s t o d e t e r m i n e what caused E. The method o f d e t e r m i n i n g t h i s i s : St e p 1: A, B, C, D . . . can sometimes cause E. Step 2: N o t h i n g e l s e i s known t o c a u s e "E". St e p 3: B but not A, C o r D was p r e s e n t . S t e p 4: The MO o f B ( i . e . b-j, b 2 , b 3 . . ., b .), w hich i s h i g h l y d i s t i n c t i v e , was p r e s e n t , t h e n Step 5: B p r o b a b l y caused "E". ( S c r i v e n , 1974b, p. 7 3 ) . For example, i f a s t u d e n t t e a c h e r uses a p a r t i c u l a r method o f d i s c i p l i n e t h e n p o s s i b l e s o u r c e s f o r t h a t method a r e ; h i s u p b r i n g i n g 67 ( p a r e n t s , home, s c h o o l , e t c . ] , h ts s p o n s o r t e a c h e r i n . t h e s c h o o l , h i s own r e a d i n g i n h i s e d u c a t i o n Cor o t h e r ] c l a s s e s , h t s c o n t a c t w t t t i o t h e r s t u d e n t t e a c h e r s , h t s U n i v e r s i t y a d v t s o r , a n d ' r u l e s o f t h e s c h o o l he i s i n . I f he was o b s e r v e d t o c o n s i s t e n t l y use t h e r u l e s o f the s c h o o l ( t h e e f f e c t ] then a s e a r c h c o u l d be made f o r why he d i d use t h e s e r u l e s o f d i s c i p l i n e . I f i t were found t h a t t he p r i n c i p a l had c a r e f u l l y d i s c u s s e d the r u l e s of d i s c i p l i n e used i n h i s s c h o o l , t h a t he had c i r c u l a t e d a w r i t t e n l i s t of t h e s e r u l e s , t h a t t h e s p o n s o r t e a c h e r had made a p o i n t of t e l l i n g t h e s t u d e n t t e a c h e r about t he r u l e s and t h e s t u d e n t t e a c h e r s a i d t h a t he used t h o s e r u l e s because t h e y were t h e s c h o o l r u l e s , then p r o b a b l y i t was the r u l e s of t h e s c h o o l w h i c h d e t e r m i n e d ( i . e . was the cause of) t h e d i s c i p l i n e t e c h n i q u e s used by t h e s t u d e n t t e a c h e r . T h i s s e a r c h f o r the complete MO o b v i o u s l y would become more com-p l i c a t e d i f i t was found t h a t t h e d i s c i p l i n e r u l e s of t h e s c h o o l were t h e same as t h o s e espoused by the s t u d e n t t e a c h e r ' s u n i v e r s i t y a d v i s o r , Summary V e r s i o n of CAUSATION C h e c k p o i n t . M e t h o d o l o g i c a l S u b s t a n t i v e E v i d e n c e F u l f i l l m e n t C h a r a c t e r i s t i c s R a t i n g Randomized e x p e r i m e n t a l d e s i g n ? Impeccable 4 Q u a s i v e x p e r i m e n t a l d e s i g n ? Good b e t 3 Ex p o s t f a c t o ? Modus Operandi method? P l a u s i b l e b e t 2 A p r i o r i i n t e r p r e t a t i o n o f Weak b e t 1 c o r r e l a t i o n a l d a t a ? H o p e l e s s b e t 0 Comment; A randomized e x p e r i m e n t a l d e s i g n . i s t h e b e s t s o u r c e o f e v i d e n c e , The o t h e r methods a r e ranked i n d e s c e n d i n g o r d e r o f p r e f e r e n c e . 68 3.338 C h e c k p o i n t 8: STATISTICAL SIGNIFICANCE. T h i s i s t h e s t a n d a r d t h a t d e t e r m i n e s i f t h e measure-ments t a k e n on t h e p r o j e c t were a p p r o p r i a t e , and i f t h e y were a n a l y s e d a t an a c c e p t a b l e l e v e l o f s t a t i s t i c a l s i g n i f i c a n c e . S c r i v e n (1974b), p. 15) says " T h i s r e q u i r e s no g r e a t s o p h i s t i c a t i o n , and i t i s f r e q u e n t l y t h e o n l y work o f s o p h i s t i c a t i o n i n an e v a l u a t i o n d e s i g n . " In t h e next p a r a g r a p h he comments, ". . . i t i s a l l t o o e a s i l y o b t a i n e d w i t h o u t t h e r e s u l t s h a v i n g any e d u c a t i o n a l s i g n i f i c a n c e . . ." In o t h e r words, s i m p l y r e p o r t i n g s o p h i s t i c a t e d s t a t i s t i c a l t e s t r e s u l t s , does not mean an e v a l u a t i o n has been c a r r i e d o u t . Summary V e r s i o n o f STATISTICAL SIGNIFICANCE C h e c k p o i n t . M e t h o d o l o g i c a l S u b s t a n t i v e E v i d e n c e F u l f i l l m e n t c h a r a c t e r i s t i c s R a t i n g A p p r o p r i a t e a n a l y s i s ? F l a w l e s s a n a l y s i s , a s t r o n o m i c a l A p p r o p r i a t e s i g n i f i c a n c e s i g n i f i c a n c e 4 l e v e l ? High s i g n i f i c a n c e , w e l l - t e s t e d 3 R e a s o n a b l y s i g n i f i c a n t 2 M a r g i n a l s i g n i f i c a n c e 1 Not shown to be s i g n i f i c a n t 0 To d e t e r m i n e an o v e r a l l r a t i n g f o r t h e s t a n d a r d f o r g o a l a c h i e v e -ment i t i s proposed t h a t t h e i n d i v i d u a l r a t i n g s f o r each c h e c k p o i n t be added t o g e t h e r and a v e r a g e d . T h i s a v e r a g e i s then used as t h e r a t i n g f o r g o a l a c h i e v e m e n t . I t i s i m p o s s i b l e f o r a p r o d u c t t o r e c e i v e a h i g h r a t i n g i n t h i s way, and not be f u l f i l l i n g t h e w o r t h w h i l e g o a l s . Each c h e c k p o i n t has been r a t e d i n terms o f i t s own r e l e v a n c e t o t h e g o a l s and on t h e b a s i s o f good d a t a , ( i . e . good e v i d e n c e o f good p e r f o r m a n c e ) , t h e r e f o r e t h e sum o f t h e c h e c k p o i n t r a t i n g s w i l l r e f l e c t t h i s . 69 The a c t u a l r a t i n g p r o c e d u r e f o r each c h e c k p o i n t and f o r t h e combined r a t i n g o f o v e r a l l w orth w i l l be d i s c u s s e d l a t e r i n t h i s c h a p t e r . F i g u r e 3 has been p r e p a r e d t o summarize t h e p r e c e d i n g d i s c u s s i o n . T h i s F i g u r e i n c l u d e s t h e s t a n d a r d o f COSTS. F i g u r e 3. The S t a n d a r d s and T h e i r R a t i n g s STANDARD CONSIDERATIONS RATING FOR THE STANDARD 1) The worth o f t h e Program G o a l s . To be w o r t h w h i l e , t h e \ -g o a l s ^ s h o u l d : - e n a b l e one t o f u l f i l l h i g h e r g o a l s , - n o t c o n f l i c t w i t h \ a n o t h e r p r i n c i p l e o f our way o f l i f e , - l e a d t o b e n e f i c i a l c o n sequences, - n o t w a r r a n t h a v i n g an e x c e p t i o n made t o t h e g o a l s i n t h i s s i t u a t i o n . ' Maximum p r i o r i t y , a d e s p a r a t e need, e x t r e m e l y w o r t h w h i l e G r e a t i m p o r t a n c e , v e r y w o r t h -P r o b a b l y w o r t h w h i l e g o a l s . . 2 P o s s i b l y w o r t h w h i l e g o a l s . . 1 No good e v i d e n c e o f the g o a l s 2} The de-g r e e o f goal a c h i e v e -ment. -MARKET -TRUE FIELD TRIALS -TRUE CONSUMERS -LONG TERM -SIDE EFFECTS -PROCESS -CAUSATION -STATISTICAL SIGNIFICANCE C o m p l e t e l y f u l f i l l s t h e g o a l s 4 F u l f i l l s a l m o s t a l l o f t h e F u l f i l l s t h e most i m p o r t a n t a s p e c t s o f t h e g o a l s . . . 2 F u l f i l l s some but not a l l o f t h e i m p o r t a n t a s p e c t s Does n o t appear t o be d i -r e c t e d t o f u l f i l l i n g t h e 3) COSTS (not i n -c l u d e d i n t h i s t h e s i s ) -Comprehensive c o s t a n a l y s i s - E x p e r t judgement o f c o s t s -Independent judgement o f c o s t s - C o s t s f o r al1 com-p e t i t o r s B r e a k t h r o u g h f o r comparable S i g n i f i c a n t l y lower t h a n comparable p r o d u c t s . . . . 3 R e a s o n a b l e f o r comparable P r o b a b l y h i g h f o r compar-a b l e p r o d u c t s , o r some-what i n c o m p l e t e d a t a ... 1 A p p a r e n t l y e x c e s s i v e f o r com-p a r a b l e p r o d u c t s o f d a t a 70 I f t h e e v a l u a t o r was g o i n g t o examine c o s t s , o r aim a t an e v a l u a -t i v e c o n c l u s i o n o f "commendatory" o r h i g h e r ( s e e T a b l e 1 ) , th e n c o s t s t a n d a r d s would have t o be i n c l u d e d . The term " c o s t " i s e x t r e m e l y b r o a d , and i n c l u d e s not o n l y d o l l a r c o s t s , but p s y c h o l o g i c a l c o s t , l o s t o p p o r -t u n i t y c o s t , s o c i a l c o s t s , e t c . ( S c r i v e n , 1974b, pp. 19-21; H a l l e r , E . J . , 1974). In t h i s e v a l u a t i o n c o s t was not a major c o n s i d e r a t i o n as a s e p a r -a t e s t a n d a r d and w i l l not be i n c l u d e d i n subsequent d i s c u s s i o n s . The f i n a l comment i n t h i s s e c t i o n has to do w i t h t he t y p e o f s t a n d a r d s . In S e c t i o n 2.40 o f t h i s t h e s i s , s t a n d a r d s were i d e n t i f i e d as b e i n g o f i n t r i n s i c o r e x t r i n s i c v a l u e . . A c c o r d i n g t o t h e d e f i n i t i o n s o f t h e s e s t a n d a r d s ( s e e Sec. 2.40) a p r o c e s s o r p r o d u c t c a n n o t have i n t r i n s i c v a l u e ; i t c a n o n l y p r o d u c e f e e l i n g s and e x p e r i e n c e s which may have i n t r i n s i c v a l u e . Both t h e worth o f t h e g o a l s f o r Program B and t h e d e g r e e o f g o a l a c h i e v e m e n t a r e s t a n d a r d s o f e x t r i n s i c v a l u e . In t h i s i n v e s t i g a t i o n , goal worth i s a s t a n d a r d o f i n s t r u m e n t a l v a l u e because i t l e a d s t o t h e accomplishment o f h i g h e r l e v e l g o a l s o r needs ( i . e . b e i n g a good t e a c h e r ) ; and go a l a c h i e v e m e n t i s a s t a n d a r d o f i n s t r u m e n t a l v a l u e because i t i s a rank o r gr a d e o f t h e program's e f f e c t i v e n e s s i n f u l f i l l i n g t h e g o a l s . 3.40 S p e c i f i c a t i o n o f t h e C l a s s o f Comparison i n g r a d i n g . As s t a t e d i n c h a p t e r I I , s e c . 2.34 and 2.60 the c l a s s o f comparison s h o u l d be some i d e a l f o r m o f t h e program ( o r t h e b e s t o f a l l r e a l o r imagined s i m i l a r p r o g r a m s ) . The " i d e a l i z e d program" r e f e r r e d t o i n C h a p t e r II i s not o n l y what.the c l i e n t hopes t o s e e , but a l s o 71 what he s h o u l d hope to s e e . In o t h e r words, t h e e v a l u a t o r c a n n o t s i m p l y a c c e p t t h e c l i e n t ' s e x p e c t a t i o n s o f t h e program as t h e s t a n d a r d s by which the c h a r a c t e r i s t i c s o f the program a r e j u d g e d . T h i s i s p a r t i c u l a r l y t r u e i n e d u c a t i o n , where t h e program d e v e l o p e r ' s g o a l s s h o u l d be c o n g r u e n t w i t h t h o s e o f an e n l i g h t e n e d p u b l i c . T h e r e f o r e i t i s t h e d e c i s i o n o f t h e e v a l u a t o r as to the c r i t e r i a d e f i n i n g t h e s t a n d a r d s . He must be a b l e to j u s t i f y h i s r a t i n g s o f c r i t e r i a and s t a n d a r d s t o h i s c l i e n t . The e v a l u a t o r can r e a l i s t i c a l l y d e f i n e t h e a c c e p t a b l e l e v e l s t h r o u g h e x p e r t o p i n i o n , l o g i c a l a n a l y s i s and/or t h r o u g h e m p i r i c a l s t u d i e s by him o r o t h e r s . In o t h e r words t h e program i s compared t o p r e d e t e r m i n e d s t a n d a r d s o f e x c e l l e n c e ( S t a k e , 1967, p. 18) based on t h e c h a r a c t e r i s t i c s o f t h e b e s t o f a l l r e a l o r imagined s i m i l a r programs. 3.50 D e t e r m i n i n g t h e Good and Bad C h a r a c t e r i s t i c s o f t h e Evaluatum i n  S - G r a d i n g 3.51 G a t h e r i n g Data on t h e Goal A c h i e v e m e n t C h a r a c t e r i s t i c s T a b l e 2 r e p r e s e n t s a c a l e n d a r o f e v e n t s i n t h e e v a l u a t i o n o f Program B. I t shows t h e d a t e o f t h e e v e n t , t h e i n s t r u m e n t o r t e c h n i q u e used ( i n c l u d i n g v e r b a l and w r i t t e n p r e s e n t a t i o n s by t h e e v a l u a t o r ) , t h e group b e i n g q u e s t i o n e d o r b e i n g a d d r e s s e d and t h e c h e c k p o i n t s b e i n g i n v e s t i g a t e d . The i n s t r u m e n t s u s e d , and t h e q u e s t i o n e d asked i n t h e i n t e r v i e w s , e t c . , a r e appended to t h e t h e s i s . T a b l e 2. Calendar/'df Data C o l 1 e c t i n g E v e n t s 72 • Date I n s t r u m e n t o r T e c h n i q u e (1975) o r P r e s e n t a t i o n Group b e i n g Q u e s t i o n e d o r R e c e i v i n g C h e c k p o i n t s B e i n g I n v e s t i g a t e d Jan 15 Jan 31 Feb 3 Feb 27 Feb 28 Feb 28 March 4 March 6 March 7 Q u e s t i o n n a i r e (App A)' I n t e r v i e w (App.; B). , / U.B.C. S t a f f P r i n c i p a l and V i c e - P r i n c i p a l T r u e Consumer T r u e Consumer S i d e E f f e c t s P r o c e s s T r u e F i e l d T r i a l s Market L i k e r t A t t i t u d e S c a l e (App.C) ) Feb 4 R e a c t i o n form!:. (App D')/.;. Feb 7 S t a f f ; t e a c h i n g p e r f o r m -ance (form A) (App<£) q u e s t i o n n a i r e Feb 12 W r i t t e n s u b m i s s i o n from the t e a c h e r s S t u d e n t T e a c h e r s P r o c e s s Comparisons T r u e Consumer S t u d e n t T e a c h e r s F r e e Ranging Search-an open ended q u e s t i o n n a i r e S t u d e n t T e a c h e r s P r o c e s s the e v a l u a t o r S i d e E f f e c t s and U.B.C. s t a f f P r o c e s s T r u e Consumer Feb 19 V e r b a l P r e s e n t a t i o n U.B.C. S t a f f T r u e Consumer P r o c e s s Q u e s t i o n n a i r e on A l t e r n a t e Placement Appendix^F;„A V e r b a l P r e s e n t a t i o n L i k e r t A t t i t u d e S c a l e (App £)•-)•, E!v a luMoir-e e h an ^ eu^me e t a ^n9..:BG.t&e'&nfU5&clci'ei:£h-. . 'S t a f f ;ia n de t e a c he r ss a n d a= v e r b a l • p r e s e n t a t i on. R e a c t i o n form (App D)') S t a f f t e a c h i n g p e r f o r m -ance [Form B) [Appendix G)']) Q u e s t i o n n a i r e . S t u d e n t T e a c h e r s P r o c e s s S i d e E f f e c t s U.B.C. S t a f f T r u e Consumer S t u d e n t T e a c h e r s See Feb. 3. U.B.C, S t a f f T e a c h e r s V i c e - P r i n c i p a l P r i n c i p a l P r o c e s s S i d e E f f e c t s •Student t e a c h e r s See Feb. 4. S t u d e n t t e a c h e r s See Feb. 7. T a b l e 2 (Cont.) 73 Date I n s t r u m e n t o r T e c h n i q u e Group b e i n g (1975) o r P r e s e n t a t i o n Q u e s t i o n e d o r R e c e i v i n g March 20 I n t e r v i e w (App H) S t u d e n t T e a c h e r s March 25 I n t e r v i e w (App H) March 27 V e r b a l p r e s e n t a t i o n March 27 Q u e s t i o n n a i r e (App I) A p r i l 2 I n t e r v i e w (App I) A p r i l 8 O b s e r v a t i o n o f Student-t e a c h i n g (App J ) A p r i l 9 O b s e r v a t i o n o f S t u d e n t -t e a c h i n g A p r i l 10 O b s e r v a t i o n o f S t u d e n t -t e a c h i n g A p r i l 11 O b s e r v a t i o n o f S t u d e n t T e a c h i n g A p r i l 13 I n t e r v i e w (App L) A p r i l 15 Q u e s t i o n n a i r e (App M) A p r i l 29 I n t e r v i e w (App N) S t u d e n t T e a c h e r s S t u d e n t T e a c h e r s S t u d e n t T e a c h e r s S t u d e n t T e a c h e r s 6 S t u d e n t T e a c h e r s 4 S t u d e n t T e a c h e r s 2 S t u d e n t T e a c h e r s A p r i l 10 Q u e s t i o n n a i r e (App K) T e a c h e r s 2 S t u d e n t T e a c h e r s Program D i r e c t o r S c h o o l ' s p u p i l s : g r a d e s 5 & 6 U n i v e r s i t y F i e l d D i r e c t o r C h e c k p o i n t s B e i n g I n v e s t i g a t e d T r u e F i e l d t r i a l s T r u e Consumer Comparisons S i d e E f f e c t s P r o c e s s - C o n g r u e n c y , J u s t i c e See March 20 T r u e Consumer P r o c e s s - c o n g r u e n c y See March 20 P r o c e s s Causation-Modus O p e r a n d i method See A p r i l 8 See A p r i l 8 S i d e E f f e c t s See A p r i l 8 Need M a r k e t P r o c e s s T r u e F i e l d T r i a l s T r u e Consumer T r u e Consumer P r o c e s s S i d e E f f e c t s Need S i d e E f f e c t s P r o c e s s T r u e Consumer Tru e F i e l d t r i a l s 74 T a b l e 2 (Co n t . ) Date I n s t r u m e n t o r T e c h n i q u e Group b e i n g C h e c k p o i n t s B e i n g (1975) o r P r e s e n t a t i o n Q u e s t i o n e d o r I n v e s t i g a t e d R e c e i v i n g A p r i l 29 I n t e r v i e w (App 0) A p r i l 30 I n t e r v i e w (App 0) D i r e c t o r o f El e m e n t a r y E d u c a t i o n .Side E f f e c t s P r o c e s s D i r e c t o r o f S i d e E f f e c t s S t u d e n t T e a c h i ng P r o c e s s -May 2 I n t e r v i e w (App 0) A s s o c i a t e Dean S i d e E f f e c t s P r o c e s s End o f Academic y e a r : e v a l u a t i o n -t e a c h e r s , d i f f e r e n t s c h o o l . - c o n t i n u e s w i t h d i f f e r e n t s t u d e n t S e p t . 19 Q u e s t i o n n a i r e (App P) Se p t . 25 Q u e s t i o n n a i r e (App Q) O c t . 3 Q u e s t i o n n a i r e (App S) Oct . 31 I n t e r v i e w (App T) S t u d e n t T e a c h e r s S i d e E f f e c t s U.B.C. S t a f f P r o c e s s ( d e s c r i p -t i o n o f c o u r s e s ) S t u d e n t T e a c h e r s P r o c e s s - e v a l u a -t i o n o f U.B.C. Oct . 24 Q u e s t i o n n a i r e (App R) T e a c h e r s School Board S t a f f t e a c h i n g p e r f o r m a n c e . S i d e E f f e c t s P r o c e s s T r u e Consumer S i d e E f f e c t s T r u e Consumer P r o c e s s In a d d i t i o n t o t h e i n f o r m a t i o n c o l l e c t e d as i n d i c a t e d i n T a b l e . 3, L the a u t h o r a l s o a t t e n d e d Program B's d i r e c t o r s ' s t a f f m e e tings on th e f o l l o w i n g d a t e s : J a n u a r y 15, 22 F e b r u a r y 12, 19, 26 75 March 5, 12, 20 A p r i l 10 A t t e n d i n g t h e s e meetings p r o v i d e d i n f o r m a t i o n o n ^ h e c h e c k p o i n t s o f s i d e - e f f e c t s , p r o c e s s , market, and t r u e consumers. O t h e r d a t a c o l l e c t e d were censu s d a t a (1971) f o r t h e a r e a s e r v e d by t h e s c h o o l , and t h e o r i g i n a l a p p l i c a t i o n forms o f t h e s t u d e n t t e a c h e r s . These d a t a h e l p e d i n d e t e r m i n i n g a r a t i n g f o r t h e c h e c k p o i n t o f time f i e l d t r i a l s . 3.52 D e t e r m i n i n g i f t h e C h a r a c t e r i s t i c s o f t h e Program a r e Good  o r Bad, 3.521 Rating, t h e Worth o f t h e G o a l s The c h a r a c t e r i s t i c s o f t h e g o a l s a r e examined i n terms o f t h e i r w o r t h . T h e r e f o r e i n t h i s s e c t i o n arguments w i l l be p r e s e n t e d o u t l i n i n g t h e p r o c e d u r e s used i n d e t e r m i n i n g a r a t i n g f o r t h e worth o f t h e g o a l f o r t r a i n i n g p e o p l e t o be good t e a c h e r s . A complete v a l i d a t i o n o f t h e r a t i n g g i v e n a t t h e end o f t h e arguments f o r t h e need f o r t e a c h e r t r a i n i n g has not been documented h e r e . The c o n c l u s i o n s r e a c h e d can be j u d g e d by t h e i r r e a s o n a b l e n e s s and not by t h e i r c o m p l e t e n e s s . T h i s i s the t y p e o f a p p r o a c h one would use f o r h i s c l i e n t i n e v a l u a t i o n . Program B has s t a t e d t h a t i t i s a t t e m p t i n g t o c o n t r i b u t e t o t h e t r a i n i n g o f good t e a c h e r s . Perhaps i t seems t r i t e t o ask a t t h i s p o i n t "do we need t o t r a i n and e d u c a t e p e o p l e t o be good t e a c h e r s ? " 76 but t h e q u e s t i o n w i l l be posed t o m a i n t a i n c o n t i n u i t y . The o t h e r s i d e o f t h e q u e s t i o n i s " o r can any e n l i g h t e n e d but u n t r a i n e d p e r s o n s t e p i n t o a c l a s s r o o m and t e a c h ? " I f t h e q u e s t i o n o f n e e d i n g t e a c h e r s a t a l l s h o u l d a r i s e , t h e n i t would be n e c e s s a r y to p r o c e e d t h r o u g h a com-p l e t e v a l i d a t i o n u n t i l e v e r y o n e was g e n e r a l l y a g r e e d t h a t t e a c h e r s were needed. T h e r e i s no p o i n t i n a s k i n g "Do we need t o t r a i n p e o p l e to be good t e a c h e r s ? " i f t h e r e i s w i d e s p r e a d d i s a g r e e m e n t about n e e d i n g t e a c h e r s a t a l l . Assuming t h a t i t i s g e n e r a l l y a g r e e d t h a t we need good t e a c h e r s t h e n we must a d d r e s s t h e problem o f somehow e s t a b l i s h i n g i f t r a i n i n g p e o p l e t o be good t e a c h e r s i s a w o r t h w h i l e g o a l . The b e s t way t o do t h i s would be e m p i r i c a l l y i . e . randomly a s s i g n i n g a group o f p r o s p e c -t i v e t e a c h e r s t o s c h o o l s where t h e y i m m e d i a t e l y b e g i n t e a c h i n g . A f t e r a few y e a r s compare them t o a n o t h e r group who began t e a c h i n g a t t h e same time but who had been t h r o u g h a program o f t e a c h e r t r a i n i n g . L a c k i n g t h i s , however, one c o u l d l o o k ( c a u t i o u s l y ) a t p r e s e n t p r a c t i c e and e n l i g h t e n e d p r e s e n t t h o u g h t on t h e s u b j e c t . P r e s e n t p r a c t i c e i n d i c a t e s t h a t p e o p l e must be t r a i n e d t o be good t e a c h e r s . P r e s e n t t h o u g h t s u g g e s t s t h e same t h i n g . As Broudy wrote: " T h a t p u b l i c s c h o o l i n s t r u c t i o n c an be manned by b r i g h t and d e v o t e d amateurs i s t h e i l l u s i o n which m i s g u i d e s many o f t h e c r i t i c s o f t e a c h e r p r e p a r a t i o n . The s a l v a t i o n o f our s o c i e t y w i l l depend on t h e a b i l i t y o f e d u c a t i o n to e x p l o i t t h e b l e s s i n g s o f t e c h n o l o g y i n b e h a l f o f what makes l i f e w o r t h w h i l e , v i z . , t h e p o s s i b i l i t y o f h i g h grade i n d i v i d u a l e x p e r i e n c e i n something c a l l e d 'the good l i f e ' . T h a t s o l v i n g t h e s e problems w i l l 77 r e q u i r e l e s s t h a n a g e n e r a t i o n o f p r o f e s s i o n a l e d u c a t o r s t r a i n e d and ed u c a t e d i n f u l l y d e v e l o p e d p r o f e s s i o n a l s c h o o l s , I f i n d i m p o s s i b l e to b e l i e v e . " (Broudy, 1965, p. 415) Though t h e r e a r e no s p e c i f i c f o r m u l a s t h e r e a r e g e n e r a l r u l e s , o u t l i n e d i n s e c t i o n 3.21 which w i l l v a l i d a t e t h e worth o f our g o a l s w i t h as much a c c u r a c y and o b j e c t i v i t y as most measurements i n e d u c a t i o n . In making o u r judgement i t i s c o n v e n i e n t t o r a t e t h e worth o f th e g o a l s on t h e f o l l o w i n g f i v e p o i n t s c a l e . Maximum p r i o r i t y , a d e s p e r a t e need, e x t r e m e l y w o r t h w h i l e g o a l s . 4 G r e a t i m p o r t a n c e , v e r y w o r t h w h i l e g o a l s 3 P r o b a b l y w o r t h w h i l e g o a l s 2 P o s s i b l y w o r t h w h i l e g o a l s 1 No good e v i d e n c e o f t h e g o a l s h a v i n g any worth 0 The e v i d e n c e s u g g e s t s t h a t t h e r e i s more th a n j u s t " P r o b a b l y w o r t h w h i l e g o a l s " i n a program t o t r a i n p e o p l e t o be good t e a c h e r s . However, as many s c h o o l s have been o p e r a t i n g w i t h some u n t r a i n e d b ut f u n c t i o n a l t e a c h e r s on t h e i r s t a f f s i t would be d i f f i c u l t t o g i v e a r a t i n g o f "Maximum p r i o r i t y , . . .". T h e r e f o r e t h e s t a n d a r d o f g o a l worth f o r a program t o t r a i n p e o p l e t o be good t e a c h e r s i s g i v e n a r a t i n g o f " G r e a t Importance" and a r a t i n g v a l u e o f 3. N o t i c e t h a t t h e worth o f t h e g o a l s , and i t s r a t i n g , a r e i n d e p e n d e n t o f t h e program b e i n g e v a l u a t e d , i n t h e se n s e t h a t t h e g o a l s haye v a r y i n g d e g r e e s o f worth r e g a r d l e s s o f whether t h e r e i s a program d e v i s e d t o a c h i e v e t h o s e g o a l s . 78 3.522 Goal ac h i e v e m e n t and i t s r a t i n g F o r m a t i v e e v a l u a t i o n has t h e r e s p o n s i b i l i t y both o f p r o v i d i n g v a l i d i n f o r m a t i o n f o r program improvement and o f r a i s i n g q u e s t i o n s about t h e m e r i t o f t h e program a t p a r t i c u l a r p o i n t s i n time ( S c r i v e n , 1967, p. 4 1 ) . In t h e e v a l u a t i o n o f Program B i n f o r m a t i o n was c o l l e c t e d , v a l i d a t e d t h r o u g h o b s e r v a t i o n and f u r t h e r q u e s t i o n i n g , summarized, and t h e n f e d back t o t h e program p a r t i c i p a n t s so t h a t changes c o u l d be made. The changes would e i t h e r a t t e m p t t o e l i m i n a t e problems o r attempt to i n c o r p o r a t e a dvantages o f t h e program. However, even though t h e program was s t i l l i n t h e f o r m a t i v e s t a g e , an o y e r a l l e s t i m a t i o n o f worth s h o u l d be made. T h i s c o u l d be done a t n a t u r a l b r e a k s i n t h e program ( e . g . at C h r i s t m a s o r d u r i n g t h e summer h o l i d a y s o f e d u c a t i o n a l p r o g r a m s ) . The o v e r a l l e v a l u a t i o n would r e f l e c t t h e s t a t e o f the program a t t h a t p o i n t i n t i m e , and, t h e r e f o r e , so would the r a t i n g on each o f t h e goal a c h i e v e m e n t c h e c k p o i n t s . In o t h e r words i n f o r m a t i v e e v a l u a t i o n t h e e v a l u a t o r would c y c l e t h r o u g h t h e c h e c k p o i n t s a number o f t i m e s . Each t i m e t h r o u g h he would d e l e t e p r e v i o u s problems t h a t had been e l i m i n a t e d t h r o u g h changes i n the program, checkmark p o s s i b l e c o n t r i b u t i o n s t o t h e a c h i e v e m e n t o f t h e g o a l s , i d e n t i f y new problems and c o n t r i b u t i o n s , c h e c k to see i f p r e v i o u s l y i d e n t i f i e d c o n t r i b u t i v e f e a t u r e s a r e s t i l l p r e s e n t ( i f s t i l l o f use) and so on. Then, when he made h i s judgement o f t h e o v e r a l l w orth, he would c o n s i d e r t h e s t a t e o f each c h e c k p o i n t , i n terms o f t h e q u a l i t y o f d a t a ( m e t h o d o l o g i c a l ) and t h e q u a l i t y o f p e r f o r m a n c e on t h e d a t a ( s u b s t a n t i v e ) t o d e t e r m i n e a r a t i n g . 79 To d e t e r m i n e t h e o v e r a l l r a t i n g f o r go a l a c h i e v e m e n t a s i m p l e a v e r a g e i s proposed w i t h t h e c o n d i t i o n s t h a t t h e r a t i n g s h o u l d not be c a l c u l a t e d i f t h e program i s immoral o r u n j u s t . The above s t e p s a r e summarized i n F i g u r e 4. F i g u r e 4: D e t e r m i n i n g a R a t i n g f o r Goal A c hievement . SteD 1 C l a s s i f y each b i t o f i n f o r m a t i o n a c c o r d i n g t o ^ t h e c h e c k p o i n t i t f a l l s under. SteD 2 Check t o see i f t h e i n f o r m a t i o n i s t r u e , p t h r o u g h f u r t h e r o b s e r v a t i o n and q u e s t i o n i n g . Data C o l l e c t i o n . Feed t h e i n f o r m a t i o n back t o t h e a p p r o p r i a t e D a t Step 3 consumer so t h a t he c o u l d change o r i n c o r p o r a t e I p* „ .. t h e c h a r a c t e r i s t i c s i d e n t i f i e d . J i m p o r t i n g Step 4 Repeat s t e p s 1-3 as many t i m e s as n e c e s s a r y a) Summarize t h e p r e s e n t s t a t e o f t h e qual S t e p 5 °Wfak \nJ q H a ! 1 t y ° f t h e P e r f o r m a n c ' e \ E v a l u a t i o n H i n d i c a t e d by t h e d a t a . • b) Rate each c h e c k p o i n t (0-4) i l i t y "] fiance v a) Check t o see i f t h e program i s immoral o r "\ u n j u s t . I f i t i s , t h e n g i v e t h e program a SteD 6 g o a^ achieve m e n t r a t i n g o f z e r o . p b) I f t h e program i s not immoral or..unjust., t h e n a v e r a g e t h e r a t i n g s on t h e c h e c k p o i n t s . T h i s i s t h e o v e r a l l r a t i n g f o r go a l a c h i e v e m e n t . R a t i n g Goal A chievement The d a t a c o l l e c t e d i n t h e f o r m a t i v e e v a l u a t i o n o f Program B a r e c o n s i d e r e d c o n f i d e n t i a l and c a n n o t be r e p o r t e d here t o i l l u s t r a t e t h e f i n a l r a t i n g g i v e n t o t h e c h e c k p o i n t s . S c r i v e n (1974b, p. 31) has s u g g e s t e d t h e use o f a " p r o d u c t 80 e v a l u a t i o n p r o f i l e " . H i s s u g g e s t i o n i s used i n F i g u r e 5 t o i l l u s t r a t e t h e r a t i n g s o f each g o a l a c h i e v e m e n t c h e c k p o i n t and an o v e r a l l r a t i n g o f g o a l a c h i e v e m e n t f o r Program B. (see a l s o F i g u r e 2 ) . R A T 1 N G F i g u r e 5 P r o d u c t E v a l u a t i o n P r o f i l e F o r Program B ( t o end o f May '75) CO CO cu r — -(-> r — o fO u ro sz • r — to CD O rO s- %~ E o •r— O 1— OJ 4 - CO +J • r — +-> E cu LLJ CO +-> CO <4-cu T3 3 +-> <u fO • r — • r — i — co CU o CO +-> S- CD c E -a O ro CD ro •i— o o • r — s- ns H-> " r — s: u_ <_> oo Q. O 00 00 1 2 3 4 5 6 7 8 a c c e p t a b l e l e y e l L U —I •—< o o CJ3 cC Comments on t h e r a t i n g s 1) Market i s i m p o r t a n t . P l a n s f o r r e a c h i n g t h e market a r e u n i q u e , f e a s i b l e and a r e e c o n o m i c a l l y v i a b l e . D e t a i l e d program d e s c r i p t i o n s t i l l nee %ded f ,for p o r t a b i l i t y . 2) U.B.C. s t a f f v e r y e n t h u s i a s t i c and hard w o r k i n g , t h e y a r e not an av e r a g e group. T e a c h e r s seem r e a s o n a b l y a v e r a g e . S t u d e n t - t e a c h e r s were s e l e c t e d on b a s i s o f h i g h marks. Many immigrant c h i l d r e n w i t h language problems a t t h e s c h o o l — m a d e some s t u d e n t t e a c h i n g 81 o v e r l y d i f f i c u l t . 3) M i s s i n g d a t a on t h e g a i n s by t h e s c h o o l ' s p u p i l s . 4) I n f o r m a l f o l l o w - u p p l a n s , not f i n a l i z e d . 5) Good d a t a on s i d e e f f e c t s . Most bad s i d e - e f f e c t s were e l i m i n a t e d by s i m p l e a d j u s t m e n t s t o t h e program. A number o f u n a n t i c i p a t e d g a i n s by T e a c h e r s and S t u d e n t - t e a c h e r s . 6) S t u d e n t - t e a c h e r s u s i n g t e c h n i q u e s l e a r n e d i n t h e i r methods c o u r s e s . Most a r e t e a c h i n g s a t i s f a c t o r i l y . Some a r e c o n s i d e r e d by a l l consumers t o be e x c e l l e n t . Program v e r y c o n s c i o u s o f j u s t i c e and m o r a l i t y . Some d i f f i c u l t i e s i n f u l f i l l i n g a l l t h a t was e x p e c t e d due t o s k e t c h y n a t u r e o f program d e s c r i p t i o n . 7) Modus o p e r a n d i c h e c k on S t u d e n t - t e a c h e r s ' t e a c h i n g but more d a t a needed. 8) No s t a t i s t i c a l d a t a c o l l e c t e d . 3.60 O v e r a l l D e t e r m i n a t i o n o f Worth Each s t a n d a r d , g o a l worth and g o a l a c h i e v e m e n t , has now been r a t e d . The p r o b l e m now becomes t o make an o v e r a l l d e t e r m i n a t i o n o f t h e worth o f t h e e d u c a t i o n a l endeavor (Program B ) . T h i s depends on two t h i n g s ( C h a p t e r I I , s e c . 2.80): a) The d e g r e e t o which t he evaluatum f u l f i l l s t h e s t a n d a r d s a p p l i e d t o i t . b) The r e l a t i v e p r e c e d e n c e o f t h e s t a n d a r d s . C o n s i d e r i n g "E>" f i r s t : The s t a n d a r d o f Goal Worth t a k e s p r e c e d e n c e o v e r t h e s t a n d a r d o f Goal A c h i e v e m e n t . I f t h e g o a l s o f t h e program a r e 82 n o t w o r t h w h i l e , t h e n , r e g a r d l e s s o f how w e l l t h o s e g o a l s a r e b e i n g a c h i e v e d , t h e program c a n n o t r e c e i v e an o v e r a l l "good" r a t i n g . F i g u r e 6 summarizes t h i s i n t e r p r e t a t i o n o f goal worth t a k i n g p r e c e d e n c e o v e r goal achievement. F i g u r e 6 Comparing Goal Worth and Goal Achievement GOAL WORTH RATING (See F i g . 2) G o a l s n o t b e i n g a c h i e v e d G A R O C A A H T „ L. I_ j j A c h i e v e s £ N some g o a l s V G E M E Not Worthwhi1e 0 P r o b a b l y W o r t h w h i l e E x t r e m e l y W o r t h w h i l e (See . F i g . 2) A c h i e v e s a l l g o a l s The program g o a l s have not been shown t o be worth-w h i l e . The program i s not f u l f i l l i n g a u s e f u l f u n c t i o n . The g o a l s a r e worthwhile;! but the Program has not been shown t o be achiev^ i n g the g o a l s ; o r the program has been shown to) be m o r a l l y u n a c c e p t a b l e The g o a l s a r e worthwhile;! and the program has been shown to be a c h i e v i n g t h e g o a l s i n a m o r a l l y a c c e p t a b l e manner. I f t h e r a t i n g o f goal worth i s g r e a t e r t h a n o r equal t o "2" and t h e r a t i n g o f g o a l a c h i e v e m e n t i s g r e a t e r t h a n o r equal t o "2; then the Program i s a t v a r y i n g d e g r e e s o f r e a d i n e s s f o r g e n e r a l d i s -t r i b u t i o n and c o u l d now undergo a summative e v a l u a t i o n . I f the r a t i n g o f goal worth i s g r e a t e r than o r equal t o "2" and 83 t h e r a t i n g o f g o a l achievement i s l e s s than "2" t h e n t h e program needs more development b e f o r e i t i s g e n e r a l l y d i s t r i b u t e d . The r i g h t hand s i d e o f F i g u r e 6 ( i . e . Goal Worth has a r a t i n g g r e a t e r t h a n o r equal t o "2") i s expanded i n F i g u r e 7 t o i l l u s t r a t e t h e d i f f e r e n t d e s c r i p t i o n s which r e s u l t from t h e e v a l u a t i o n . I f a program has a g o a l a c h i e v e m e n t r a t i n g o f "2" o r l e s s on F i g u r e 7 then i t may be u n w o r t h w h i l e ; o r t h a t i s , t h e program may have undergone an e v a l u a t i o n and f o u n d t o not be a c h i e v i n g t h e g o a l s ; o r t h e program may be u n w o r t h w h i l e because i t i s f o u n d t o be u s i n g m o r a l l y u n a c c e p t a b l e methods; o r i t may not have undergone an e v a l u a t i o n ( o r have undergone a v e r y poor e v a l u a t i o n ) i n which c a s e no judgement can be made o f i t s o v e r a l l w o r t h . I f t h e program has a g o a l a c h i e v e m e n t r a t i n g o f "2" o r more on F i g u r e 7 t h e n i t i s w o r t h w h i l e and i n c r e a s e s i n worth, as i t i s shown to a c h i e v e more o f t h e g o a l s and as t h e g o a l s i n c r e a s e i n i m p o r t a n c e . Program B has a goal a c h i e v e m e n t r a t i n g o f about "2" and a g o a l worth r a t i n g o f "3". T h e r e f o r e i t c o u l d be d e s c r i b e d as a w o r t h w h i l e program, f u l f i l l i n g i m p o r t a n t g o a l s t o an a c c e p t a b l e l e v e l . However, t h i s s t a t e m e n t has t h e f o l l o w i n g r e s t r i c t i o n . The e v a l u a t i v e c o n c l u -s i o n b e i n g aimed f o r was an " o v e r a l l e v a l u a t i v e c o n c l u s i o n " ( s e e T a b l e 1, C h a p t e r I I ) . C o s t d a t a were not t a k e n i n t o c o n s i d e r a t i o n , nor was the program e x t e n s i v e l y compared t o o t h e r t e a c h e r t r a i n i n g programs. T h e r e f o r e , r e w o r d i n g t h e o v e r a l l d e t e r m i n a t i o n o f worth i n terms o f the " o v e r a l l e v a l u a t i v e c o n c l u s i o n " , t h e d e s c r i p t i o n becomes; 84 Figure 7:' Expansion of Figure 6 GOAL WORTH RATING J 4 Probably worthwhile goals but: Very worthwhile goals but: Extremely worthwhile goals but: 1) No data on the program or 2) The data indicates the program is not achieving any part of the goals or 3) The program is morally unacceptable. Probably worthwhile goals but: Very worthwhile goals but: Extremely worthwhile goals but: 1) Poor data on the program or 2) Data indicates the program is doing a poor job of achieving the goals or 3) The program has serious moral shortcomings, Probably worthwhile goals and: Very worthwhile goals and: Extremely worthwhile goals and:  1) Acceptable data and 2) Data indicates the goals are being acceptably achieved. Probably worthwhile goals and: Very worthwhile goals and: Extremely" worthwhile goals and: 1) Good data and 2) Data indicates the goals are almost completely being achieved by the program. Probably worthwhile goals and: Very worthwhile goals and: Extremely worthwhile goals and: 1) Excellent data and 2) Data indicates the program is completely achieving the goals. 85 "Program B a c c e p t a b l y f u l f i l l e d t h e g o a l f o r p r o d u c i n g s t u d e n t t e a c h e r s , who were competent i n t e a c h i n g ( f o r t h i s l e v e l o f t h e i r t e a c h e r t r a i n i n g ) . The s t u d e n t t e a c h e r s b e n e f i t t e d f r o m Program B; and t h e r e were no harmful e f f e c t s on t h e program p a r t i c i p a n t s . " Note t h a t t h e above d e s c r i p t i o n does not c l a i m t h a t Program B i s t h e b e s t program f o r t r a i n i n g t e a c h e r s a t t h e t h i r d y e a r l e v e l . To make t h a t s t a t e m e n t i t would be n e c e s s a r y t o f u l l y e v a l u a t e t h e c o m p a r a t i v e programs, i n c l u d i n g t h e s t a n d a r d o f c o s t , and t h e n r a n k o r d e r them and program B t o d e t e r m i n e which was b e s t . T h i s would be t h e j o b o f a summative e v a l u a t o r . The f o r m a t i v e e v a l u a t i o n i l l u s t r a t e d by t h i s t h e s i s has s i m p l y p r o v i d e d Program B's d e v e l o p e r s and s u p p o r t e r s ( f i n a n c i a l and o t h e r w i s e ) w i t h a "green l i g h t " on subsequent development. 3.70 E x t e n s i o n o f t h e Model 3.71 I n t r o d u c t i o n C h a p t e r s Two and T h r e e have e x p l i c a t e d a l o g i c a l e v a l u a t i o n model based on R-GRADING. T h i s s e c t i o n "The E x t e n s i o n o f t h e Model", has been i n c l u d e d t o show t h a t t h e model d e v e l o p e d f o r R-GRADING can be l o g i c a l l y and e a s i l y e x t e n d e d t o i n c l u d e t h e most comprehensive e v a l u a t i o n s , t h a t i s , RANKING t h e e v a l u a t a i n a c l a s s o f comparison t o d e t e r m i n e i f i t i s t h e b e s t eva l u a t u m . Ranking i n v o l v e s g r a d i n g an evaluatum a c c o r d i n g t o c e r t a i n s t a n d a r d s and then comparing t h a t o v e r a l l r a t i n g w i t h t h e r a t i n g s o b t a i n e d by o t h e r s i m i l a r programs graded a c c o r d i n g t o t h e same 86 s t a n d a r d s . The e v a l u a t a a r e t h e n r a n k e d a c c o r d i n g t o how w e l l t h e y met t h e s t a n d a r d s . T h i s t e l l s t h e c l i e n t f o r t h e e v a l u a t i o n which evaluatum i s t h e b e s t , assuming each e v a l u a t u m was e v a l u a t e d t o the same d e g r e e . S i n c e each e v a l u a t u m must have gone t h r o u g h S - g r a d i n g b e f o r e i t was ranked i n t h e c l a s s o f o t h e r r e l e v a n t e v a l u a t u m , the e v a l u a t i o n can a l s o i n f o r m t h e c l i e n t o f t h e o v e r a l l worth o f each evaluatum. T h e r e a r e t h r e e q u e s t i o n s which must be d i s c u s s e d f o r t h e e x t e n s i o n o f S - g r a d i n g t o R a n k i n g . F i r s t l y , "what p r e c e d e n c e do t h e s e s t a n d a r d s t a k e i n d e t e r m i n i n g t h e o v e r a l l worth o r m e r i t o f t h e e v a l u a t a ; " and, f i n a l l y , "how can we r a n k t h e e v a l u a t a ? " 3.72 The S t a n d a r d s t o be Used i n Ranking In C h a p t e r 3 t h e s t a n d a r d s o f Goal Worth and Goal Achievement were used as s t a n d a r d s f o r S - g r a d i n g . The l o g i c a l q u e s t i o n t h e n i s " c o u l d we use j u s t t h o s e two s t a n d a r d s f o r R a n k i n g ? " The answer i s " y e s " ; but t h i s assumes t h a t t h e c o s t o f t h e p r o j e c t i s n o t o f c o n c e r n . U n f o r t u n a t e l y c o s t i s u s u a l l y o f c o n c e r n . O b v i o u s l y i f programs X and Y f u l f i l l e d t h e s t a n d a r d o f Goal Worth t o t h e same d e g r e e , and t h e s t a n d a r d o f Goal Achievement to t h e same d e g r e e , y e t Program X c o s t more t h a n Program Y, t h e Program Y would be a b e t t e r program. T h e r e f o r e , though i t i s p o s s i b l e t o r a n k c o m p e t i t i v e programs o n l y on t h e b a s i s o f Goal Worth and Goal Achievement, i t i s more r e a s o n a b l e t o r a n k them by a l s o c o n s i d e r i n g t h e s t a n d a r d o f C o s t . 87 The s t a n d a r d o f Extended S u p p o r t may a l s o be t a k e n i n t o c o n -s i d e r a t i o n . However t h i s s t a n d a r d i s n o t as n e c e s s a r y as t h e t h r e e p r e v i o u s l y mentioned (Goal Worth, Goal A chievement and C o s t ) . Extended s u p p o r t and C o s t were i n c o r p o r a t e d i n t o S c r i v e n ' s (1974b) " P r o d u c t C h e c k l i s t " . They w i l l now be d e s c r i b e d i n more d e t a i l . 3.721 C o s t s S c r i v e n (1974b, p. 21) p r o v i d e d t h r e e c r i t e r i a f o r c o s t d a t a . The two w h i c h a r e a p p l i c a b l e t o t h i s d i s c u s s i o n a r e p r e s e n t e d below. a) C o s t d a t a must be Comprehensive. T h i s means t h a t t h e e v a l u a t o r s s h o u l d i n v e s t i g a t e such c o s t s a s : c o v e r i n g m a i n t e n a n c e c o s t s * c a p i t a l c o s t s , p s y c h i c c o s t s , d o l l a r c o s t s , c o s t s o f i n - s e r v i c e u p d a t i n g o f needed h e l p e r s , and so on. S c r i v e n ( i b i d ) a l s o s u g g e s t s t h a t : "A q u a l i t a t i v e c o s t - e f f e c t i v e n e s s a n a l y s i s , s h o u l d be a t t e m p t e d where p o s s i b l e , and, i f i t i s i m p o s s i b l e , t h e n c o s t - b e n e f i t a n a l y s i s s h o u l d be done as s y s t e m a t i c a l l y as p o s s i b l e . " b) C o s t d a t a must be v e r i f i e d . A l l t h e c o s t e s t i m a t e s and r e a l c o s t s s h o u l d be i n d e p e n d e n t l y v e r i f i e d , p r e f e r a b l y by an a c c o u n t a n t s k i l l e d i n e s t i m a t i n g c o s t s o f e d u c a t i o n a l p r o d u c t s . A summary v e r s i o n o f t h i s s t a n d a r d c o u l d be s i m i l a r t o t h e f o l l o w i n g : ( S c r i v e n , 1974b, p. 28) 88 C o n s i d e r a t i o n s R a t i n g Comprehensive c o s t a n a l y s i s ? - B r e a k t h r o u g h f o r comparable 4 E x p e r t judgement o f c o s t s ? p r o d u c t s i n d e p e n d e n t judgement o f c o s t s ? " 3 C o s t s f o r a l l c o m p e t i t o r s ? . - R e a s o n a b l e f o r comparable 2 p r o d u c t s - High f o r comparable p r o d u c t s 1 o r somewhat i n c o m p l e t e d a t a - E x c e s s i v e f o r comparable 0 p r o d u c t s o r d a t a i n c o m p l e t e A more e x t e n s i v e d i s c u s s i o n o f " C o s t s " has been p r o v i d e d by E . J . H a l l e r (1974, pp. 406-450). 3.722 Extended S u p p o r t I t i s h i g h l y d e s i r a b l e t h a t t h e r e be s y s t e m a t i c p r o c e d u r e s f o r u p d a t i n g o r u p g r a d i n g a p r o d u c t once i t i s on t h e market i n t h e l i g h t o f new i n f o r m a t i o n which c o u l d improve t h e p r o d u c t . As S c r i v e n (1974b, p. 21) p o i n t s o u t : ". . . t h i s i m p l i e s t h e n e c e s s i t y f o r a s y s t e m a t i c c o n t i n u i n g p r o c e d u r e f o r c o l l e c t i n g f i e l d d a t a . " T h i s s t a n d a r d i s d e s i r a b l e , but not n e c e s s a r y . I t s p r e s e n c e s h o u l d be a p l u s f o r t h e program, but i t s a b s e n c e s h o u l d n o t c o u n t a g a i n s t t h e program. I t would be o f g r e a t i m p o r t a n c e i f t h e r e was a s t r o n g need f o r the program and t h e r e were no a v a i l a b l e a l t e r n a t i v e s . In t h a t s i t u a t i o n t h e r e s h o u l d be a c o n t i n u o u s c o l l e c t i o n o f f i e l d t r i a l d a t a . 89 A summary v e r s i o n o f t h i s s t a n d a r d c o u l d be s i m i l a r t o t h e f o l l o w i n g : ( S c r i v e n , 1974b, p. 2 8 ) . C o n s i d e r a t i o n s P o s t m a r k e t i n g d a t a c o l l e c t i o n ? P o s t m a r k e t i n g system f o r improvement? I n - s e r v i c e t r a i n i n g ? U p - d a t i n g o f a i d s ? New u s e r s and u s e r d a t a ? 3 .73 The P r e c e d e n c e o f t h e The S t a n d a r d s i n Ranking As a r g u e d i n C h a p t e r 3 , t h e s t a n d a r d o f Goal Worth t a k e s p r e c e d e n c e o v e r f a l l t h e o t h e r s t a n d a r d s . I f t h e G o a l s o f t h e evaluatum a r e not w o r t h w h i l e then t h e evaluatum s h o u l d not e x i s t . When comparing d i f f e r e n t e v a l u a t a , i t i s assumed t h a t t h e g o a l s a r e t h e same f o r a l l o f them. The c r i t e r i u m o f s i m i l a r g o a l s i s n e c e s s a r y i n o r d e r t o compare t h e e v a l u a t a . T h e r e i s no p o i n t i n comparing Program X t o Program Y i f t h e y a r e f u l f i l l i n g d i f f e r e n t g o a l s . F o r example, one c o u l d a r g u e t h a t we need methods o f t r a n s p o r t i n g p e o p l e from P o i n t A t o P o i n t B and t h e n g i v e a r a t i n g t o t h i s g o a l ; and then c o n s i d e r t h e pr o p o s e d (or o p e r a t i n g ) methods o f f u l f i l l i n g t h a t g o a l . P o s s i b l e methods c o u l d be: by p r i v a t e c a r , by bus, by t r a i n o r whatever. The g o a l , r e g a r d l e s s o f t h e method, remains t h e same. What w i l l d e t e r m i n e t h e i n d i v i d u a l method's o v e r a l l worth, (assuming t h e g o a l s a r e w o r t h w h i l e ) , and hence t h e i r r a n k o r d e r , w i l l be t h e d e g r e e t o which t h e y each a c h i e v e t h e g o a l s , and t h e c o s t o f e a c h . T h e r e f o r e , assuming t h a t t h e e v a l u a t a -R a t i n g - E x c e l l e n t and co m p r e h e n s i v e 4 - Good and f a i r l y c o m p r e h e n s i v e 3 - M i n i m a l l y a c c e p t a b l e 2 - Weak — l e s s than adequate 1 - N e g l i g i b l e -- a p p a r e n t l y none 0 90 b e i n g compared a r e a l l t r y i n g t o f u l f i l l t h e same g o a l s , and assuming t h a t t h e g o a l s a r e w o r t h w h i l e ; t h e n t h e s t a n d a r d s o f go a l a c h i e v e m e n t and C o s t w i l l d e t e r m i n e each program's r e l a t i v e worth ( i n t h e c l a s s o f comparison) f o r t h e purpose o f r a n k i n g . (The r o l e o f Extended S u p p o r t w i l l be d i s c u s s e d l a t e r ) . I t does not appear t h a t a s i m p l e a v e r a g i n g o f t h e r a t i n g f o r Goal A chievement and o f the r a t i n g f o r C o s t can be used t o d e t e r m i n e which evaluatum i s b e s t . F i g u r e 8 w i l l h e l p c l a r i f y t h i s s t a t e m e n t . F i g u r e W i l l u s t r a t e s t h e same p r o c e d u r e t h a t was used i n d e t e r m i n i n g t h e o v e r a l l worth i n S - g r a d i n g u s i n g Goal Worth and Goal A c hievement. The m a t r i x has been d i v i d e d i n t o f o u r q u a d r a n t s . I t i s assumed i n t h e f o l l o w i n g d i s c u s s i o n t h a t t h e e v a l u a t i o n was c o m p l e t e , so t h a t a low r a t i n g i s n o t due t o l a c k o f i n f o r m a t i o n on t h a t s t a n d a r d . Each q u a d r a n t w i l l now be d e s c r i b e d : Quadrant 1: ( h i g h c o s t , low goal a c h i e v e m e n t ) I f t h e c o s t i s h i g h and t h e evaluatum i s not a c h i e v i n g t h e g o a l s , t h e n i t seems r e a s o n a b l e t o r e j e c t i t from f u r t h e r c o n s i d e r a t i o n . The c o s t o f a program i s a r e l a t i v e s t a n d a r d , i . e . h i g h c o s t means t h a t an evaluatum c o s t s much more th a n some o t h e r evaluatum w i t h i n t h e a v a i l a b l e r e s o u r c e s . I f a l l t h e e v a l u a t a b e i n g compared f a l l i n t o t h i s q u a d r a n t , then t h e y s h o u l d a l l be r e j e c t e d , and a new method f o r a c h i e v i n g t h e g o a l s s h o u l d be s e a r c h e d f o r . Quadrant 2: ( h i g h c o s t , g o a l s b e i n g a c h i e v e d ) I n . t h i s q u a d r a n t , t h e c o s t i s h i g h , but t h e evaluatum i s a c h i e v i n g t h e g o a l s . The q u e s t i o n i s : "Why i s t h e evaluatum c o s t i n g 91 Figure 8. Combining the Goal Achievement and Cost Ratings Rating for Goal Achievement 0 1 2 3 . 4 R 0 a Quadrant 1 Quadrant 2 t I ' g Reject evaluatum. Why i s the evaluatum costing so much? r Quadrant 3 Quadrant 4 o 3 Why i s the evaluatum The evaluatum i s s t not achieving the goals? acceptable. 4 so much?" It may be achieving the goals only because of a high investment of resources and materials. An evaluatum"which has an equal or greater goal achievement rating ( i . e . a rating of "2" or more) and lower cost i s obviously superior. If a l l the evaluata f a l l i n t h i s quadrant, then they should be closely examined as to methods of cutting the costs. Quadrant 3: (low cost, low goal achievement) Here the cost i s acceptable but the evaluata which f a l l in this quadrant are not achieving the goals. The reasons an evaluatum 92 i n q u a d r a n t 3 may not be a c h i e v i n g t h e g o a l s may be e i t h e r because i t i s not a "good" program ( i . e . p o o r l y d e s i g n e d , e t c . ) ; o r i t may be because i t has n o t u t i l i z e d s u f f i c i e n t r e s o u r c e s . I f . a l l t h e e v a l u a t a b e i n g compared f a l l i n t h i s q u a d r a n t t h e n t h e y s h o u l d be c l o s e l y examined to d e t e r m i n e i f i t i s due t o b e i n g p o o r l y d e s i g n e d programs, o r due t o i n s u f f i c i e n t use o f r e s o u r c e s . 3.74 Ranking t h e E v a l u a t a Quadrant 4: ( R e a s o n a b l e t o low c o s t , r e a s o n a b l e t o h i g h g o a l a c h i e v e m e n t ) T h i s q u a d r a n t d e f i n e s t h e a r e a o f a c c e p t a b i l i t y . I t i s o n l y here t h a t a s i m p l e a v e r a g e o f t h e r a t i n g s f o r c o s t and g o a l a c h i e v e m e n t s h o u l d be c a l c u l a t e d t o d e t e r m i n e t h e rank o r d e r o f t h e e v a l u a t a . The e v a l u a t u m w i t h t h e h i g h e s t a v e r a g e r a t i n g i s the " b e s t buy" ( s e e T a b l e 1, C h a p t e r 2 ) . I f two o r more e v a l u a t a have t h e same h i g h e s t a v e r a g e r a t i n g i t i s s u g g e s t e d t h a t t h a t one w i t h t h e h i g h e s t g o a l a c h i e v e m e n t r a t i n g be chosen as t h e b e s t . I f t h e r e i s s t i l l a t i e f o r t h e " b e s t buy" ( i . e . two o r more have t h e same go a l a c h i e v e m e n t / r a t i n g and t h e same c o s t r a t i n g ) t h e n t h a t one w i t h p l a n s f o r "extended s u p p o r t " s h o u l d be g i v e n the h i g h e s t r a t i n g . Thus, t h e s t a n d a r d o f extended s u p p o r t i s seen as a f i n a l t i e - b r e a k e r . I f none o f t h e e v a l u a t a b e i n g r a n k e d f a l l i n Quadrant 4 o f F i g u r e ,'8, t h e y can s t i l l be r a n k e d e i t h e r w i t h r e s p e c t t o c o s t s or w i t h r e s p e c t t o g o a l a c h i e v e m e n t . However, t h i s r a n k o r d e r w i l l e s t a b l i s h t h e b e s t o f a bad l o t and does not mean t h e " b e s t " evaluatum i s a "good" evaluatum. CHAPTER IV Data Cp11ectjon_Technique_s. i n t h e A p p l i c a t i o n  °f, t h e E v a l u a t i o n Model. 4.00 I n t r o d u c t i o n T h i s c h a p t e r c a t a l o g u e s t h e d a t a g a t h e r i n g . t e c h n i q u e s used i n . the a p p l i c a t i o n o f t h e e v a l u a t i o n model d e s c r i b e d i n C h a p t e r s Two and T h r e e . A l l o f t h e q u e s t i o n n a i r e s and o t h e r t e c h n i q u e s , used t o g a t h e r d a t a i n the e v a l u a t i o n o f Program B, have been appended to t h e t h e s i s . The f i r s t p a r t o f C h a p t e r Four i l l u s t r a t e s two s t a g e s i n t h e d a t a g a t h e r i n g p r o c e d u r e s . The f i r s t s t a g e was an o v e r a l l p o r t r a y a l o f Program B and t h e second s t a g e was d a t a c o l l e c t i o n on s p e c i f i c a s p e c t s o f t h e Program. These s t a g e s a r e r e l a t e d t o t h e s t a n d a r d o f Goal Achievement by T a b l e ,3. The f i n a l p a r t o f t h e C h a p t e r i l l u s t r a t e s how t h e c o l l e c t e d d a t a was r e c o r d e d and v a l i d a t e d . 4.10 I n i t i a l Stage of. Data C o l l e c t i o n The e v a l u a t i o n - o f Program B began a f t e r t h e program had been i n o p e r a t i o n f o r t h r e e months. T h e r e f o r e t h e i n i t i a l s t a g e o f d a t a c o l l e c t i o n had two p u r p o s e s . F i r s t l y t o d e t e r m i n e which check-p o i n t s i n S c r i v e n ' s E v a l u a t i o n C h e c k l i s t s h o u l d be c o n s i d e r e d f i r s t ; and s e c o n d l y , t o p r o v i d e a g e n e r a l i d e a o f t h e b e n e f i t s - o f t h e Program to t h e s t u d e n t t e a c h e r s a t t h i s s t a g e i n t h e Program's development. 93 94 The f i r s t p urpose was a c c o m p l i s h e d by a s k i n g each o f t h e mem-be r s o f t h e development team which o f t h e C h e c k p o i n t s on S c r i v e n ' s E v a l u a t i o n C h e c k l i s t s h o u l d be c o n s i d e r e d f i r s t , ( see Appendix A) The r e s p o n s e s t o t h e C h e c k l i s t i d e n t i f i e d major a r e a s o f i n t e r e s t and c o n c e r n , from t h e p o i n t o f view o f t h e d e v e l o p e r s , and p r o v i d e d d i r e c t i o n f o r s e t t i n g i n i t i a l p r i o r i t i e s i n d a t a - c o l l e c t i o n . The second p u r p o s e , t h a t i s t o d e t e r m i n e t h e i n i t i a l b e n e f i t s o f t h e program, was a c c o m p l i s h e d by q u e s t i o n i n g t h r e e groups o f p e o p l e -t h e s t u d e n t t e a c h e r s , t h e s c h o o l a d m i n i s t r a t o r s and t h e s c h o o l ' s p u p i l s . The s t u d e n t t e a c h e r s were a s k e d t o r e s p o n d , on a f i v e p o i n t s c a l e , t o a s e t o f a t t i t u d e s t a t e m e n t s on t h e t e a c h e r t r a i n i n g p r o -c e s s . The a t t i t u d e s c a l e ( s e e Appendix C) was d e v e l o p e d u s i n g t h e u s u a l t e c h n i q u e s ( L i k e r t , 1932). I t was a d m i n i s t e r e d t w i c e , o v e r a p e r i o d o f a o n e month, to t h e s t u d e n t t e a c h e r s i n Program B, t o t h e s t u d e n t t e a c h e r s i n t h e e q u i v a l e n t y e a r o f t h e r e g u l a r t e a c h e r t r a i n -i n g program, and t o t h e s t u d e n t t e a c h e r s i n t h e e q u i v a l e n t y e a r o f t h e t r a n s f e r t e a c h e r - t r a i n i n g program (see U.B.C. 1974-75 c a l e n d a r f o r more d e t a i l s o f t h e p r o g r a m s ) . T h i s p r o v i d e d a means o f compar-i n g t h e a t t i t u d e s o f t h e program B s t u d e n t • t e a c h e r s w i t h t h e a t t i t u d e s i n t he two comparison programs ( t h e r e g u l a r and the t r a n s f e r program). A d m i n i s t e r i n g t h e a t t i t u d e s c a l e t w i c e gave an e s t i m a t e o f s t a b i l i t y o f group r e s p o n s e on the q u e s t i o n n a i r e . The s t u d e n t t e a c h e r s were a l s o a s k e d t o f i l l i n a " r e a c t i o n form" ( s e e Appendix D). T h i s p r o v i d e d t h e e v a l u a t o r w i t h t h e i r o v e r a l l r a t i n g o f t h e m e r i t o f t h e program, and w i t h a g r e a t d e a l o f i n f o r m a -t i o n on g e n e r a l c o m p l a i n t s and p r a i s e s . 95 The second group t o be q u e s t i o n e d was the s c h o o l a d m i n i s t r a t i o n , t h a t i s , t h e p r i n c i p a l and v i c e - p r i n c i p a l of t h e s c h o o l . They were i n t e r v i e w e d a c c o r d i n g t o t h e l i s t of q u e s t i o n s g i v e n i n Appendix B. The t h i r d group to be q u e s t i o n e d was t h e p u p i l s of t h e s c h o o l . T h i s was a c c o m p l i s h e d by t a l k i n g t o i n d i v i d u a l s o r s m a l l groups d u r i n g the r e c e s s o f l u n c h b r e a k s . The c a n d i d r e s p o n s e s of t h i s group p r o v i d e d a g r e a t d e a l of g e n e r a l i n f o r m a t i o n , n o t o n l y on t h e o p e r a -t i o n of Program B, as seen t h r o u g h t h e p u p i l s e y e s , but a l s o on t h e g e n e r a l s c h o o l c l i m a t e . T h e r e a r e two o b v i o u s groups o m i t t e d from t h i s i n i t i a l s u r v e y , the t e a c h e r s i n t h e s c h o o l and t h e program d e v e l o p e r s . The t e a c h e r s had been exposed t o numerous s u r v e y s a l r e a d y . The S c h o o l Board and t h e P r o v i n c i a l Government had both been u n u s u a l l y a c t i v e i n s e n d i n g o u t q u e s t i o n n a i r e s i n t h e 1974-75 y e a r . F u r t h e r , an e x t e r n a l group o f e v a l u a t o r s , commissioned by the F a c u l t y o f E d u c a t i o n a t U.B.C, was e v a l u a t i n g a l l t h e a l t e r n a t e programs o f f e r e d i n t e a c h e r educa-t i o n i n t h e F a c u l t y . The l a t t e r group o f e v a l u a t o r s had asked t h e t e a c h e r s t o f i l l i n an e x t e n s i v e q u e s t i o n n a i r e on Program B j u s t p r i o r t o t h e i n t e r n a l e v a l u a t i o n b e g i n n i n g . T h e r e f o r e , i t was c o n s i d e r e d p r u d e n t t o d e l a y g i v i n g any q u e s t i o n n a i r e s t o t h e t e a c h e r s a t t h e b e g i n n i n g and i n s t e a d t o r e l y on i n f o r m a l " c h a t s " w i t h i n d i v i d u a l t e a c h e r s . The program d e v e l o p e r s were a l s o o m i t t e d from t h i s i n i t i a l d a t a g a t h e r i n g because o f t h e i r f r a n k n e s s i n t h e i r w eekly m e e t i n g s . These m e e t i n g s were a t t e n d e d by the e v a l u a t o r . 96 T h i s o v e r v i e w o f Program B was v a l u a b l e f o r a number o f r e a s o n s . I t p r o v i d e d g e n e r a l d a t a on t h e f e e l i n g s o f a l l t h e consumers toward t h e program; i t s u g g e s t e d major a r e a s o f c o n c e r n o r n e g l e c t f o r more d e t a i l e d i n v e s t i g a t i o n ; i t a l l o w e d t h e e v a l u a t o r t o i n t r o d u c e him-s e l f and o u t l i n e t h e g e n e r a l scheme o f e v a l u a t i o n t o most o f t h e consumer g r o u p s ; and i t p r o v i d e d g e n e r a l f e e d b a c k t o a l l t h e groups i n v o l v e d , p a r t i c u l a r l y t o t h e program d e v e l o p e r s . T h i s i n i t i a l s t a g e o f d a t a c o l l e c t i o n c o u l d a l s o be c a l l e d a ' p o r t r a y a l ' s t a g e o f d a t a c o l l e c t i o n . I t i s p o r t r a y a l t h a t S t a k e (1972a) a d v o c a t e d when he w r o t e : "In t h e more o r d i n a r y and modest s i t u a t i o n ... we s h o u l d l i m i t o u r e v a l u a t i o n aims t o what we can do and t o what t h e c l i e n t needs most. What many c l i e n t s need i s a c r e d i b l e , t h o r o u g h r e p r e s e n t a t i o n o f what t h e program i s , i n c l u d i n g i n f o r m a t i o n about who l i k e s what about i t . " P o r t r a y a l i s u s e f u l (see p r e v i o u s p a r a g r a p h ) but i s o f t e n o n l y a s t e p toward an e v a l u a t i o n . ( F o r an example o f t h i s a p p r o a c h see Gleadow ( 1 9 7 5 ) ) . T h e r e must be a second s t a g e o f d a t a c o l l e c t i o n i n o r d e r t o make c o n c l u s i o n s o f the worth o f t h e program. An e s t i m a t e o f worth may be p o s s i b l e u s i n g p o r t r a y a l o r i n i t i a l d a t a c o l l e c t i o n , but i t would be a guess and e x t r e m e l y d i f f i c u l t i f not i m p o s s i b l e t o j u s t i f y . 4.20 Second Stage o f Data C o l l e c t i o n A f t e r t h i s i n i t i a l , ' p o r t r a y a l ' s t a g e o f d a t a c o l l e c t i o n t h e d a t a g a t h e r i n g was d i r e c t e d t o s p e c i f i c a r e a s o f the Program. Each 97 o f t h e t e c h n i q u e s used w i l l be now b r i e f l y d i s c u s s e d . The t e c h n i q u e s have been c a t e g o r i z e d i n T a b l e 3 i n t h i s C h a p t e r . 4.21 E v a l u a t i o n o f U.B.C. Instructors,'-.Teaching. P e r f o r m a n c e S c r i v e n (1973) has condensed many o f t h e c o r r e l a t e s o f good t e a c h i n g p e r f o r m a n c e i n t o a s h o r t l i s t o f p r i m a r y i n d i c a t o r s . I t was t h e s e p r i m a r y i n d i c a t o r s t h a t p r o v i d e d t h e s o u r c e o f many o f t h e q u e s t i o n s asked on Form A (Appendix E) and l a t e r i n the program, Form B (Appendix G). These two forms were f i l l e d o u t by the s t u d e n t t e a c h e r s d i r e c t l y o n t o IBM mark c a r d s f o r ease o f a n a l y s i s . Seven s t u d e n t - t e a c h e r s were randomly a s s i g n e d t o t h e e v a l u a t i o n o f two i n s t r u c t o r s which were randomly p a i r e d f o r each s t u d e n t - t e a c h e r , l i o t h e r words each i n s t r u c t o r was e v a l u a t e d by seven s t u d e n t - t e a c h e r s . S u b s e q u e n t l y , i t was f e l t t h a t t h i s method was too ti m e consuming. T h e r e f o r e , t h e q u e s t i o n n a i r e was r e v i s e d t o t h e form i l l u s t r a t e d i n Appendix Q. In t h i s f i n a l form, t h e s t u d e n t - t e a c h e r gaye an o v e r a l l r a t i n g t o t h e t e a c h i n g p e r f o r m a n c e o f t h e i n s t r u c t o r s and i n d i c a t e d i f t h e r e were any i n s t r u c t o r s who were e x c e p t i o n s ( e i t h e r b e t t e r t h a n o r worse th a n t h e r e s t o f t h e group) t o t h e o v e r a l l r a t i n g . T h i s s i m p l i f i e d t h e f i l l i n g o u t o f the q u e s t i o n n a i r e , and t h e c o l l e c t i o n and i n t e r p r e t a t i o n o f d a t a , as w e l l as i d e n t i f i e d e x c e p t i o n a l i n s t r u c t o r s . The q u e s t i o n n a i r e s a l s o i n c l u d e an o v e r a l l e v a l u a t i o n o f t h e t e a c h i n g p e r f o r m a n c e o f each i n s t r u c t o r . S c r i v e n (1973, p. 13) wrote about t h i s o v e r a l l judgment o f m e r i t : 98 "Even though we do not know e x a c t l y what i t s r e l i a b i l i t y i s , a t l e a s t (a) t h e r e s u l t s c a n n o t so e a s i l y be w r o n g l y i n t e r p r e t e d and (b) t h e chances t h a t i t ' s n e g a t i v e l y c o r r e l a t e d w i t h good t e a c h i n g a r e p r o b a b l y lower t h a n w i t h any o t h e r s i n g l e q u e s t i o n ( o r c o m b i n a t i o n o f a few q u e s t i o n s ) . " T h i s r e a s o n i n g can a l s o be a p p l i e d t o o t h e r a r e a s o f c o n c e r n ; as a r e s u l t an o v e r a l l judgement was asked f o r . o n a number o f o t h e r a s p e c t s o f t h e program. The e v a l u a t i o n o f t h e U.B.C. i n s t r u c t o r s ' t e a c h i n g p e r f o r m a n c e was a p r o c e s s check. 4.22 Q u e s t i o n n a i r e t o S t u d e n t t e a c h e r s on t h e i r A l t e r n a t e  Placement (Appendix F) D u r i n g t h e c o u r s e o f Program B, t h e s t u d e n t t e a c h e r s were p l a c e d i n o t h e r s c h o o l s f o r a two week p r a c t i c u m . The t y p e o f s c h o o l was chosen by t h e s t u d e n t t e a c h e r . T h i s q u e s t i o n n a i r e p r o -v i d e d a check on t h e p e r c e i v e d s u c c e s s o f t h a t e x p e r i e n c e . 4.23 I n t e r v i e w o f S t u d e n t T T e a c h e r s (Appendix H) Th r e e randomly chosen s t u d e n t t e a c h e r s were i n t e r v i e w e d o v e r t h e space o f a few weeks. Each i n t e r v i e w t o o k a p p r o x i m a t e l y o n e - h a l f hour. The i n t e r v i e w s were r e c o r d e d on t a p e w i t h t h e p e r -m i s s i o n o f t h e i n t e r v i e w e e . 4.24 F i n a l q u e s t i o n n a i r e f o r t h e S t u d e n t T e a c h e r s (Appendix I) The s o u r c e o f a l l t h e q u e s t i o n s on t h i s q u e s t i o n n a i r e was t h e o r i g i n a l d e s c r i p t i o n s o f t h e program ( s e e C h a p t e r 3-Sec. 3.11 99 "The E v a l u a t u m " ) . I t s purpose was to o b t a i n a f i n a l s u b j e c t i y e c o n g r u e n c y c h e c k between what was s a i d was g o i n g t o o c c u r , and what a c t u a l l y o c c u r r e d . 4.25 O b s e r v a t i o n o f S t u d e n t T e a c h i n g F i f t e e n o f the s t u d e n t t e a c h e r s were o b s e r v e d for a t l e a s t one l e s s o n e a c h , by t h e e v a l u a t o r , d u r i n g t h e i r l a s t c l a s s r o o m t e a c h i n g e x p e r i e n c e . T h e r e were two a r e a s o f c o n c e r n , F i r s t l y a p r o c e s s check on c l a s s r o o m j u s t i c e was needed and s e c o n d l y an i. • • a t t e m p t t o c o n n e c t o b s e r v e d s t u d e n t t e a c h e r b e h a v i o r s i n t h e c l a s s -room and t h e i r Program B e x p e r i e n c e , t h a t i s , a c a u s a l check u s i n g t h e "modus o p e r a n d i " method (see Ch a p t e r 3, s e c t i o n 3.337) was a t t e m p t e d . C l a s s r o o m j u s t i c e d a t a was c o l l e c t e d by d i r e c t o b s e r v a t i o n , w i t h p a r t i c u l a r a t t e n t i o n b e i n g p a i d t o t h e s t u d e n t t e a c h e r s ' methods o f d i s c i p l i n e . I t was f e l t t h a t d i s c i p l i n e t e c h n i q u e s would be good i n d i c a t o r s o f t h e f a i r n e s s w i t h which t h e s t u d e n t t e a c h e r s ' : d e a l t w i t h t h e p u p i l s . T h e r e f o r e , each time a p u p i l o r p u p i l s were d i s c i p l i n e d , a note was made on t h e r e a s o n f o r t h e d i s c i p l i n i n g , t h e t y p e o f d i s c i p l i n e , t e c h n i q u e used, and whether t h e d i s c i p l i n e t e c h -n i q u e was e f f e c t i v e . The d i s c i p l i n e t e c h n i q u e s were c a t e g o r i z e d b e f o r e h a n d ( s e e Appendix J ) . A f t e r t h e l e s s o n , t h e s t u d e n t t e a c h e r was a s k e d by t h e e v a l u a t o r why c e r t a i n t e c h n i q u e s were used i n p a r t i c u l a r s i t u a t i o n s . The p u p i l s were a s k e d i n a q u e s t i o n n a i r e i f t h e y f e l t t h e y were f a i r l y t r e a t e d by t h e s t u d e n t t e a c h e r s (Appendix 100 M q u e s t i o n s 3 and 7) t o p r o v i d e a n o t h e r check o f c l a s s r o o m j u s t i c e . The modus o p e r a n d i check was p e r f o r m e d by r e c o r d i n g a l l t h e t e c h n i q u e s used by t h e s t u d e n t t e a c h e r and t h e n i n t e r v i e w i n g t he s t u d e n t t e a c h e r a f t e r t h e l e s s o n t o d e t e r m i n e where he/she g o t t h e i d e a f o r a p a r t i c u l a r technique.(Was i t h i s / h e r own i d e a ? Was i t a v a r i a t i o n o f an i d e a from a p r o f e s s o r ? a t e a c h e r ? a t e x t b o o k ? Was .is t h e s p o n s o r t e a c h e r ' s i d e a ? Was t h e t e c h n i q u e f r e e l y used o r was t h e s t u d e n t t e a c h e r t o l d t o use t h a t t e c h n i q u e ? ) . G e n e r a l l y , t h e number o f d i f f e r e n t t e c h n i q u e s used i n a l e s s o n was q u i t e l i m i t e d , so t h a t r e c o r d i n g them and i n t e r v i e w i n g t h e s t u d e n t t e a c h e r a f t e r t h e l e s s o n was not a t i m e consuming o r onerous t a s k . The r e s u l t o f t h i s was an i n d i c a t i o n o f t h e c o n t r i b u t i o n Program B made i n t r a n s -f e r r i n g c l a s s r o o m t e c h n i q u e s t o t h e s t u d e n t t e a c h e r s . E v a l u a t i n g t h e e f f e c t i v e n e s s o f t h e s t u d e n t t e a c h e r s was an a r e a o f p a r t i c u l a r d i f f i c u l t y . T h e r e were no i n s t r u m e n t s d e v e l o p e d d u r i n g t he p e r i o d o f e v a l u a t i o n d e s c r i b e d by t h i s t h e s i s , though i n i t i a l work on t h e i n s t r u m e n t s was begun i n t h i s p e r i o d . However methods have now been d e v e l o p e d and a r e b e i n g f i e l d t e s t e d (as t h i s t h e s i s i s b e i n g w r i t t e n ) i n t h e second y e a r o f Program B. F o r t h e sake o f c o m p l e t e n e s s , t h e p r o p o s a l and i n s t r u m e n t s b e i n g used a r e appended t o t h i s t h e s i s as Appendix U). 4.26 S i d e E f f e c t s o f Program B ( T e a c h e r ' s Q u e s t i o n n a i r e ) (Appendix K) As n o t e d i n t h e c o v e r i n g l e t t e r t o t h i s q u e s t i o n n a i r e , t h i s 101 was t h e o n l y q u e s t i o n n a i r e g i v e n t o t h e t e a c h e r s . I t s purpose was to d e t e r m i n e t h e p o s i t i v e and n e g a t i v e s i d e e f f e c t s as p e r c e i v e d by t h e t e a c h e r s . 4.27 I n t e r v i e w with. Program B's D i r e c t o r (Appendix L) T h i s i n t e r v i e w was t a p e r e c o r d e d w i t h t h e p e r m i s s i o n o f t h e i n t e r v i e w e e . I t p r o v i d e d i n f o r m a t i o n on t h e i n i t i a l p l a n n i n g o f t h e program, and t h e r e a s o n s f o r i t s i n i t i a t i o n . It;.was a l s o used t o d e t e r m i n e t h e v a l i d i t y o f t h e f i e l d t r i a l d a t a a l r e a d y c o l l e c t e d . 4.28 Q u e s t i o n n a i r e f o r . S c h o o l P u p i l s (Appendix M) T h i s q u e s t i o n n a i r e was g i v e n t o two grade f i v e and two grade s i x c l a s s e s . Each q u e s t i o n was r e a d t o t h e p u p i l s . I f t h e y were not c l e a r on t h e meaning o f a q u e s t i o n t h e y were asked t o ask f o r c l a r i f i c a t i o n . I t p r o v i d e d i n f o r m a t i o n on t h e t e a c h i n g e f f e c t -i v e n e s s o f t h e s t u d e n t t e a c h e r s , as p e r c e i v e d by t h e p u p i l s . 4.29 Interviewjwitjx U..B.C.. A d m i n i s t r a t o r s , (Appendix N) The F i e l d D i r e c t o r o f S t u d e n t T e a c h i n g , t h e D i r e c t o r o f El e m e n t a r y Ed, t h e D i r e c t o r o f S t u d e n t T e a c h i n g ( E l e m e n t a r y ) , and the A s s o c i a t e Dean were a l l i n t e r v i e w e d ( s e p a r a t e l y ) and each was asked t h e f o u r q u e s t i o n s w r i t t e n i n Appendix N. In t h e i n t e r v i e w s and q u e s t i o n n a i r e s , u s u a l l y more th a n one 102 o f t h e c h e c k p o i n t s o f goal achievement was a d d r e s s e d . To i l l u s t r a t e t h i s , T a b l e 3 has been p r e p a r e d . The s t a n d a r d o f goal w o r t h , and a c a t e g o r y o f " e v a l u a t i v e c o n c l u s i o n " have a l s o been added t o t h e t a b l e . The " e v a l u a t i v e c o n c l u s i o n " c a t e g o r y r e f e r s to a q u e s t i o n w h ich a s k s t h e r e s p o n d a n t t o make an o v e r a l l judgement o f some a s p e c t o f t h e program. The numbers under each c a t e g o r y o f t h e C h e c k l i s t i n T a b l e 3, r e f e r t o the q u e s t i o n s o f t h e i n t e r v i e w o r on t h e q u e s t i o n n a i r e which were o b t a i n i n g i n f o r m a t i o n a p p l i c a b l e t o t h a t c a t e g o r y . 4.210 O t h e r I n f o r m a t i o n S o u r c e s T a b l e 3 s u g g e s t s t h a t t h e r e a r e a r e a s w h i c h were n e g l e c t e d by t h e i n t e r v i e w s and the q u e s t i o n n a i r e s . Some o f t h e s e , such as l o n g t e r m , s t a t i s t i c a l s i g n i f i c a n c e and c r i t i c a l c o m p a r i s o n s were n e g l e c t e d because o f t h e c h a r a c t e r o f t h i s e v a l u a t i o n . As s t a t e d e a r l i e r i n t h i s t h e s i s , t h i s s t u d y was a f o r m a t i v e e v a l u a t i o n i n S-GRADING. T h e r e f o r e c o m p a r i s o n groups were not o f c o n c e r n e x c e p t f o r e s t a b l i s h i n g a r e f e r a n t p o i n t f o r t h e a t t i t u d e q u e s t i o n n a i r e (Appendix C ) . T h e r e were no l o n g term p l a n s b e i n g d e v e l o p e d a t t h e time o f t h e e v a l u a t i o n so t h e r e was no d a t a t o c o l l e c t on t h i s c h e c k p o i n t ( n o t e t h a t i t a l s o r e c e i v e d a v e r y low r a t i n g on t h e p r o f i l e (Chapt. I l l S e c t i o n 3 . 5 2 2 ) ) . No measures which c o u l d have s t a t i s t i c a l p r o c e d u r e s m e a n i n g f u l l y a p p l i e d t o them were t a k e n , e x c e p t f o r t h e a t t i t u d e q u e s t i o n n a i r e a d m i n i s t e r e d t o t h e s t u d e n t t e a c h e r s ; t h e r e f o r e t h e c h e c k p o i n t o f s t a t i s t i c a l s i g n i f i c a n c e was TABLE 3 Data Sources for the Goal Achievement Checkpoints* ApDendix Questionnaire Interview ' No. , A X B X C X D X E X F X G X I J K L M Observation Respondant Developers School Admin. Student teach. Student teach. Student teach. Student teach. Student teach. Student teach. Student teach. Student teach. School teach. Program Direc. School pupils U.B.C. Admin-istrators Goal Worth 13 2,12 1 Market Field True Long- Side Effects Trials Consumer term 1 20 1,3,4, 5,6,13 3 6,7 3,5,6, 7 - n -•, 8,9,16, 7 * ° ' n 18,19 y c 9,10 5,6,7, 10,11 8,10 5,9 17 5 4,5 1,2,15,20 13 1-4 5,6,14, 16,17 Process Causation Statistical Evaluative Significance Conclusions 6 2,3 All 2 1-14 1,5-9 1-12 4,9,10,11 12,15,20 1-8,11,12 All 5 1-4,6,7, 8,10,11, 12,13,15, 18,19,20 2,3 * The numbers under each checkpoint refer to the question numbers on the questionnaires noted for each appendix. 7 All Yes 1 page 2 10,11 page 2 14 21 i—1 o 104 g i v e n a z e r o r a t i n g . The o t h e r c h e c k p o i n t s w i t h a p p a r e n t l y s m a l l amounts o f d a t a a r e market and t r u e f i e l d t r i a l s . But t h a t i s because t he d a t a f o r t h e s e c h e c k p o i n t s were c o l l e c t e d by means o t h e r t h a n q u e s t i o n n a i r e o r f o r m a l i n t e r v i e w s . .Most o f t h e i n f o r m a t i o n f o r market ( t h a t i s , t h e p l a n s f o r i m p l e m e n t a t i o n o f the program) was c o l l e c t e d from t h e o r i g i n a l r e c r u i t m e n t m a t e r i a l s s e n t t o t h e s t u d e n t t e a c h e r s , and t h r o u g h t h e weekly meet4ngs~wfl'thtthe> p r o g r a m ^ d e v e l . o p e r s ^ d u r i n g which t h e program p l a n n i n g was r e v i s e d and/or p l a n n e d . Much o f t h e t r u e f i e l d t r i a l i n f o r m a t i o n was o b t a i n e d t h r o u g h (a) examining census d a t a f o r t h e a r e a s e r v e d by t h e s c h o o l and comparing t h i s w i t h t h e r e s t n o f t h e c i t y ; (b) exam i n i n g t h e t r a n s c r i p t s o f marks and o r i g i n a l a p p l i c a t i o n forms o f t h e s t u d e n t t e a c h e r s who a p p l i e d and were a c c e p t e d , as w e l l as t h o s e who a p p l i e d and were r e j e c t e d ; and ( c ) a s k i n g t h e a d m i n i s -t r a t o r s i n t h e f a c u l t y o f e d u c a t i o n t o compare t h e a b i l i t y o f Program B's p r o f e s s o r s as a group, t o t h e r e s t o f t h e f a c u l t y . The c h e c k p o i n t o f t r u e consumers was u n s a t i s f a c t o r i l y d e v e l o p e d . Gains o r l o s s e s o f the s t u d e n t t e a c h e r s were measured t h r o u g h t h e i r marks on t h e i r t e a c h i n g a b i l i t y and c o u r s e s , and t h r o u g h t h e i r a t t i -t udes and o v e r a l l e v a l u a t i o n s o f the program. However, s c h o o l p u p i l g a i n s o r l o s s e s , t e a c h e r g a i n s o r l o s s e s , U.B.C. p r o f e s s o r s ' g a i n s o r l o s s e s , s c h o o l a d m i n i s t r a t i o n g a i n s o r l o s s e s , U n i v e r s i t y g a i n s o r l o s s e s , and School Board g a i n s o r l o s s e s were not e s t a b l i s h e d i n s u f f i c i e n t d e t a i l . T h e r e was some e v i d e n c e on t h i s c h e c k p o i n t f o r a l l o f t h e s e groups however i t was s c a n t y . Only e s t i m a t e s o f s p e c i f i c 105 g a i n s o r l o s s e s c o u l d be made. 4.30 P r a c t i c a l P r o c e d u r e s f o r t h e O r g a n i z a t i o n o f Data A d i f f i c u l t and i m p o r t a n t t a s k o f t h e e v a l u a t o r i s t o o r g a n i z e and condense t h e d a t a he c o l l e c t s . The f o l l o w i n g p r o c e d u r e i s p r e s e n t e d as a f e a s i b l e and w o r k a b l e method d e v e l o p e d f o r t h e t y p e o f e v a l u a t i o n model d i s c u s s e d i n t h i s t h e s i s . F o r each c h e c k p o i n t (market, t r u e consumer, e t c . ) a c h a r t i s p r e p a r e d as i l l u s t r a t e d i n F i g u r e 9"below: F i g u r e <9. Format f o r O r g a n i z a t i o n o f Data C h e c k p o i n t : Date Data S o u r c e I n f o r m a t i o n V a l i d a t i o n Check Comment Each o f t h e he a d i n g s i n F i g u r e S w i l l now be d i s c u s s e d : Date: The d a t e t h e i n f o r m a t i o n was c o l l e c t e d . 106 Data S o u r c e : The d a t a s o u r c e would i n d i c a t e t h e p e r s o n ( s ) who s u p p l i e d t h e d a t a ; t h e t e c h n i q u e used t o c o l l e c t t h e d a t a ( e . g . ; q u e s t i o n n a i r e , i n t e r v i e w , i n f o r m a l d i s c u s s i o n , e t c . ) ; and t h e q u e s t i o n asked o r r e s p o n d e d t o ( i d e n t i f i e d by number i f i t was on a q u e s t i o n n a i r e o r i n a s t r u c t u r e d i n t e r v i e w ) . I n f o r m a t i o n : An a c c u r a t e summary s t a t e m e n t o f t h e i n f o r m a t i o n c o l l e c t e d , o r , i f t h e i n f o r m a t i o n c a n n o t be summarized, a r e f e r e n c e t o t h e raw d a t a s h o u l d be p r e s e n t e d . T h e r e s h o u l d a l s o be a judgement on t h e p a r t o f t h e e v a l u a t o r as t o whether t h e i n f o r m a t i o n i s p o s i t i v e , n e g a t i v e o r n e u t r a l ( e q u a l amounts, o f p o s i t i v e n e s s and n e g a t i v e n e s s ) w i t h r e s p e c t t o a p a r t i c u l a r c h e c k p o i n t i n Need F u l f i l l m e n t . V a l i d a t i o n Check: I t i s i m p o r t a n t t h a t t h e r e be a c h e c k to d e t e r m i n e i f t h e i n f o r m a t i o n o b t a i n e d was v a l i d . T h i s u s u a l l y means f u r t h e r q u e s t i o n i n g o f program p a r t i c i p a n t s o r o b s e r v e r s , and/or o b s e r v a t i o n by t h e e v a l u a t o r . Comment: T h i s c o u l d be a comment by t h e e v a l u a t o r about what s h o u l d be done w i t h the i n f o r m a t i o n and whether o r not what s h o u l d be done has been done. For example, i f t h e i n f o r m a t i o n was n e g a t i v e t h e n t h e e v a l u a t o r m i g h t make a n o t e t h a t t h a t a s p e c t o f t h e program s h o u l d be r e p o r t e d t o t h e program d e v e l o p e r s and t h e p r o b l e m s h o u l d be e l i m i n a t e d . L a t e r , he would check to see i f i t was e l i m i n a t e d . I f t h e program was changed so as to e l i m i n a t e t h e problem t h e n t h e e v a l u a t o r i n h i s f i n a l r e p o r t would use t h e i n f o r m a t i o n as a c a v e a t t o f u t u r e program u s e r s , 107 but he would not g i v e a lower r a t i n g t o t h e c h e c k p o i n t because o f i t . An example (Figure!©)) i s now o f f e r e d t o c l a r i f y t h e above d e s c r i p t i o n . A l l t h e d a t a i s f i c t i t i o u s . F i g u r e 1®. Example o f Format, f o r O r g a n i z a t i o n o f Data C h e c k p o i n t : S i d e E f f e c t s Date Data S o u r c e I n f o r m a t i o n V a l i d a t i o n Check Comment Feb. 15 S t u d e n t t e a c h -e r s , Q u e s t i o n -n a i r e 1, Ques-t i o n 3. "Were t h e r e any un-e x p e c t e d p o s i -t i v e o r nega-t i v e e x p e d -i e n c e s i n t h i s program." 12 o f 20 s t u -dent t e a c h e r s i n d i c a t e d t h a t t h e t e a c h e r s asked them t o p a r t i c i p a t e i n P a r e n t - t e a c h e r n i g h t . The s t u d e n t - t e a c h -e r s t h o u g h t t h e e x p e r i e n c e v e r y worth-w h i l e ( j u d g e d p o s i t i v e ) ( s e e summary o f a l l t h e comments on t h i s , on page 125 o f d a t a notebooks) a) The t e a c h e r s were asked f o r t h e i r . o p i n i o n s o f t h i s p a r t i c i -p a t i o n by t h e s t u d e n t - t e a c h e r s . They i n d i c a t e d t h e y t h o u g h t t h a t i t was v a l u a b l e f o r t h e S.T.'s t o p a r t i c i p a t e - no n e g a t i v e f e e l i n g . b) Some o f t h e p a r e n t s who came t o t h e p a r e n t -t e a c h e r i n t e r -views were ques-t i o n e d . They i n -d i c a t e d t h e y d i d not f i n d t h e p r e s e n c e o f t h e s t u d e n t - t e a c h e r s d i s t r a c t i n g . R e p o r t t h i s t o t h e p r o -gram d i r e c -t o r s , (done Feb. 20) D e c i s i o n was made to implement t h i s i n f u t u r e programs. 108 The advantage o f t h i s a p p r o a c h t o r e c o r d i n g t h e d a t a i s t w o - f o l d . F i r s t l y , i t t r e m e n d o u s l y s i m p l i f i e s t h e c o n d e n s i n g o f i n f o r m a t i o n when i t comes t i m e t o make a f i n a l r e p o r t ; and s e c o n d l y , i t c l e a r l y o u t l i n e s a l l t h e d a t a which went i n t o g i v i n g a r a t i n g t o a p a r t i c u l a r c h e c k p o i n t . T h i s i s e x t r e m e l y i m p o r t a n t , f o r t h e e v a l u a t o r must be a b l e t o v e r i f y each c o n c l u s i o n , recommendation and r a t i n g he makes. T h i s i s r e l a t i v e l y easy t o do u s i n g t h e above f o r m a t . 4.40 C o n f i d e n t i a l i t y o f Data I f t h e d a t a s o u r c e s a r e p e o p l e o r t h e i r r e c o r d s t h e n s t r i c t c o n f i d e n t i a l i t y must be m a i n t a i n e d . A l l o f t h e p e o p l e i n v o l v e d i n an e v a l u a t i o n must be a s s u r e d , and b e l i e v e t h a t a s s u r a n c e , t h a t t h e i n f o r m a t i o n c o l l e c t e d w i l l r emain anonymous as t o t h e i n d i v i d u a l . In o t h e r words t h e e v a l u a t o r c o u l d i d e n t i f y t he s o u r c e o f i n f o r m a t i o n as t h e t e a c h e r s , but not i n d i c a t e what i n d i v i d u a l t e a c h e r s had t o s a y . T h i s can d e c r e a s e t h e power o f s t a t i s t i c a l t e c h n i q u e s which r e l y on p r e - p o s t t e s t i n g o r c o r r e l a t i o n measures; but t h e l o s s i n s t a t i s t i c a l power due t o m a i n t a i n i n g anonymity was t h o u g h t t o be j u s t i f i e d by t h e g a i n i n c a n d i d i n f o r m a t i o n . CHAPTER V Summary and C o n c l u s i o n s 5.00 Summary T h i s s t u d y has e x p l o r e d the p o s s i b i l i t y o f d e v i s i n g a method o f e v a l u a t i o n , based on t h e l o g i c a l method o f Paul W. T a y l o r (1961) i n c o m b i n a t i o n w i t h the more p r a c t i c a l a p p r o a c h o f S c r i v e n (1974b) which c o u l d be used i n the e v a l u a t i o n o f a t e a c h e r t r a i n i n g program. The goal o f t h e s t u d y was t o b r i d g e the gap between the t h e o r e t i c a l c o n c e p t s o f T a y l o r (1961) and S c r i v e n , and the p r a c t i c a l problems o f e v a l u a t i n g an e d u c a t i o n a l program. A fundamental assumption o f t h i s s t u d y was t h a t e v a l u a t i o n i s a l o g i c a l p r o c e d u r e r e g a r d l e s s o f what i s b e i n g e v a l u a t e d . ". . . i t i s t h e l o g i c a l method which a r a t i o n a l p e r s o n would f o l l o w i f he were t r y i n g t o corne t o a c a r e f u l r e f l e c t i v e d e c i s i o n about the v a l u e o f s o m e t h i n g . " ( T a y l o r , 1961, p. 4 ) . The method o f e v a l u a t i o n i n c l u d e s a d o p t i n g a s e t o f s t a n d a r d s w i t h i n a p a r t i c u l a r p o i n t o f view, the o p e r a t i o n a l c l a r i f i c a t i o n o f t h o s e s t a n d a r d s , s p e c i f y i n g a c l a s s o f c o m p a r i s o n , g a t h e r i n g d a t a on t h e c h a r a c t e r i s t i c s o f t h e e v a l uatum, and d e d u c i n g from the d a t a g a t h e r e d and the s t a n d a r d s chosen t h e degree to which the e v a l u a t u m f u l f i l l s t h e s t a n d a r d s . The p r o d u c t o f an e v a l u a t i o n i s a c o n c l u s i o n t h a t t h e e v a l u a t u m has a c e r t a i n v a l u e , worth o f m e r i t . T h a t c o n c l u s i o n depends on the r e l a t i v e p r e c e d e n c e o f the s t a n d a r d s , and t h e degree to which the e v a l u a t u m 109 no f u l f i l l s t h e s t a n d a r d s . The s t a n d a r d s used i n t h i s s t u d y were: (1) The s t a n d a r d o f g o a l worth and (2) the s t a n d a r d o f goal a c h i e v e m e n t . These s t a n d a r d s were used f o r b r o a d l y c a t e g o r i z i n g t h e two main components o f any good e d u c a t i o n a l program, ( i . e . something w o r t h w h i l e i s b e i n g t r a n s m i t t e d  i n a m o r a l l y a c c e p t a b l e manner). The s t a n d a r d s were used i n t h e e v a l u a t i o n o f a t e a c h e r t r a i n i n g program (Program B) from t he educa-t i o n a l , i n s t i t u t i o n a l , moral and l e g a l p o i n t s o f view. The methodology d e v e l o p e d i n t h e s t u d y , was i l l u s t r a t e d by a p p l y i n g i t t o t h e f o r m a t i v e e v a l u a t i o n o f a s c h o o l - b a s e d , t e a c h e r t r a i n i n g program. The e v a l u a t i o n was an example o f S-GRAUING. T h a t i s , t h e t e a c h e r t r a i n i n g program was n o t d i r e c t l y compared t o o t h e r , e x i s t i n g , 0 - s i m i l a r programs; r a t h e r , i t was graded a c c o r d i n g t o t h e degree t o which i t f u l f i l l e d a s e t o f a p p r o p r i a t e s t a n d a r d s . When GRADING, t h e e v a l u a t o r d e t e r m i n e s a s e t o f c r i t e r i a o r f u l f i l l m e n t c h a r a c t e r i s t i c s which o p e r a t i o n a l l y c l a r i f y t h e s t a n d a r d s used i n t h e e v a l u a t i o n . The c r i t e r i a a r e d e r i v e d from an i d e a l f o r m o f t h e program b e i n g e v a l u a t e d , and w i l l t h e r e f o r e d i f f e r f r o m one typ e o f e d u c a t i o n a l p r o d u c t t o the n e x t . I n t h i s s t u d y , t h e c r i t e r i a used were t h o s e s u g g e s t e d by Broudy i n h i s paper on the p r o f e s s i o n a l p r e p a r a t i o n o f t e a c h e r s . (Broudy, 1965). The t e a c h e r t r a i n i n g program (Program B) was e v a l u a t e d by comparing both t h e g o a l s o f Program B and the method and degree o f a c h i e v i n g t h o s e g o a l s t o Broudy's c r i t e r i a ; and, on the b a s i s o f t h a t c o m p a r i s o n , j u d g i n g whether t h e < ••~\ 111 program's g o a l s were w o r t h w h i l e and b e i n g a c h i e v e d i n a m o r a l l y a c c e p t a b l e manner. I f the c l i e n t f o r the e v a l u a t i o n q u e s t i o n s t h e f i n a l judgement o f w o r t h , then the e v a l u a t o r would f i r s t show t h a t the program met the c r i t e r i a t o the d e g r e e i n d i c a t e d by t h a t f i n a l judgement o f w o r t h . I f t he c l i e n t q u e s t i o n n e d the c r i t e r i a used i n d e f i n i n g the s t a n d a r d s , then t h e e v a l u a t o r must show t h a t t h o s e s t a n d a r d s were a p p r o p r i a t e f o r t h e program b e i n g e v a l u a t e d , and t h a t meeting t h e c r i t e r i a used to o p e r a t i o n a l i z e t h e s t a n d a r d s i n d i c a t e d m e r i t o r i o u s p e r f o r m a n c e . F i g u r e ] i T c l a r i f i e s t he e v a l u a t i o n p r o c e d u r e s d e v e l o p e d i n t h i s s t u d y . More i n f o r m a t i o n can be o b t a i n e d by r e f e r r i n g t o t h e s e c t i o n s o f t h e t h e s i s n o t e d i n each s t e p o f t h e F i g u r e . F i g u r e T/b s h o u l d be c a u t i o u s l y i n t e r p r e t e d as a temporal sequence. In p r a c t i s e , the e v a l u a t o r may e n t e r t h e program a f t e r the g o a l s have been e s t a b l i s h e d and a f t e r t h e program has begun o p e r a t i o n . In t h a t c a s e , he would.have t o g a t h e r d a t a on t h e g o a l s and the goal a c h i e v e m e n t c r i t e r i a a t the same t i m e . S e c t i o n 3.70 showed how the e v a l u a t i o n p r o c e d u r e s d e v e l o p e d f o r S-GRADING c o u l d be l o g i c a l l y e x t e n d e d f o r RANKING. Appendix H h a s been i n c l u d e d t o summarize t h a t e x t e n s i o n . 112 F i g u r e 11 S t e p s i n S-GRADING i n FORMATIVE EVALUATION Determine the e v a l u a t u m and the p o i n t s o f view t o be t a k e n i n the E v a l u a t i o n . ( S ec. 3.11, Sec. 3.12) E s t a b l i s h t he g o a l s o f t h e e v a l - * uatum ( S e c . 3.31) 1 < Determine, the wor uatum's g o a l s . (S t h o f the e v a l -e c . 3.521)-A r e the g o a l s w o r t h w h i l e ? (Sec. 3.521) YES E s t a b l i s h the goal a c h i e v e m e n t c r i t e r i a . ( S e c . 3.'32%)  Determine t h e " d e g r e e t o which the e v a l u a t u m i s a c h i e v i n g the g o a l s . ( S ec. 3.522) . Is t h e e v a l u a t u m a c h i e v i n g t h e g o a l s ? ( S e c . 3.522) YES Determine the,,overal 1 worth o f t h e e v a l u a t u m ( S e c . 3.60) NO YES A r e o t h e r . g o a l s b e i n g pursued? NO F u r t h e r e v a l u a t i o n would be p o i n t l e s s . "  R e v i s e the evaluatum. F •NO-1 Is t h e - e v a l u a t u m a c h i e v i n g • ( o t h e r g o a l s ? lis i t p o s s i b l e t o r e v i s e (the evaluatum? YES E v a l u a t u m has w o r t h w h i l e g o a l s p u t i s n o t a c h i e v i n g t h o s e g o a l s ,  S T O p C o n c l u s i o n : X was t h e t r e a t m e n t , X r e s u l t e d i n the achievement o f t h e w o r t h w h i l e g o a l s Y, the s u b j e c t s ( S ' s ) d e s i r e d , e n j o y e d o r were b e n e f i t t e d by Y. STOP 113 5.10 C o n c l u s i o n s A g e n e r a l : ' c o n c l u s i o n o f t h i s s t u d y i s t h a t T a y l o r ' s (1961) a n a l y s i s o f t h e method and p r o d u c t o f e v a l u a t i o n can p r o v i d e a framework f o r t h e e v a l u a t i o n o f an e d u c a t i o n a l program. More s p e c i f i c a l l y when T a y l o r ' s a n a l y s i s i s combined w i t h S c r i v e n ' s e x t e n s i v e work and c h e c k l i s t , a f e a s i b l e model o f e v a l u a t i o n r e s u l t s , w hich r e a d i l y p r o d u c e s a d e f e n -s i b l e , o v e r a l l e s t i m a t i o n o f worth f o r an e d u c a t i o n a l p r o d u c t . T a b l e 4 has been p r e p a r e d t o p o i n t o u t how t h e model o f e v a l u a t i o n d e v e l o p e d i n t h i s t h e s i s d i f f e r s from the more p o p u l a r models o f educa-t i o n a l e v a l u a t i o n . On t h e l e f t o f t h e t a b l e a r e t h e l i m i t a t i o n s o f some o f the p r e s e n t e d u c a t i o n a l e v a l u a t i o n models (see S e c t i o n 2.10). On the r i g h t o f the t a b l e a r e the c o n t r i b u t i o n s o f t h i s t h e s i s toward overcoming t h o s e l i m i t a t i o n s . T a b l e 4. Comparison o f E v a l u a t i o n P r o c e d u r e s . L i m i t a t i o n s o f some p r e s e n t models o f e v a l u a t i o n (see S e c . 2.10). C o n t r i b u t i o n s o f t h i s t h e s i s toward overcoming the l i m i t a t i o n s . 1) T y l e r ' s " M o d e l a) Focuses on the p r o d u c t and n o t on the p r o c e s s o f e v a l u a t i o n . a) Both p r o d u c t and p r o c e s s a r e o f c o n c e r n i n the e v a l u a t i o n ( s e e ' p a r t i c u l a r l y S e c . 3.336, p. 6 3 ) . b) Focuses d i r e c t l y on measure-a b l e g o a l s and o b j e c t i v e s b u t makes no d i r e c t j u d g e -ments on the worth o f t h o s e g o a l s b) D e t e r m i n i n g t h e worth o f t h e g o a l s i s one o f the most i m p o r t a n t s t e p s i n the e v a l u a t i o n . I f a program has w o r t h l e s s g o a l s then i t has no m e r i t ( s e e Sec. 3.21, p. 45, 3.60, p. 81) i 114 T a b l e 4 ( C o n t . ) 2) A c c r e d i t a t i o n . a) Does n o t c o n n e c t the o b s e r v e d c h a r a c t e r i s t i c s o f the p r o -gram to t h e i r consequences on the program p a r t i c i p a n t s . a) A l l d a t a g a t h e r e d i s j u d g e d as t o whether i t i s c o n t r i b -u t i n g t o the a c h i e v e m e n t o f the w o r t h w h i l e g o a l s . ( S e c . 3.521, p. 75; Sec. 3.522, p. 7 8 ) . 3) S t u f f l e b e a m ' s CIPP Model. a) P r o v i d e s d a t a t o d e c i s i o n makers but makes no d e c i s i o n on the worth o f t h a t d a t a o r on the worth o f t h e p r o d u c t b e i n g e v a l u a t e d . a) Both the q u a l i t y o f d a t a and the o v e r a l l worth o f t h e p r o -gram a r e j u d g e d . (Sec. 3.32, p. 52, 3.60, p. 81) 4) S t a k e ' s Model. a ) ^ I t i s n o t c l e a r how t h e worth o f the program g o a l s i s d e t e r m i n e d . a) See l b above and S e c t i o n 3.521, p. 75. 5) A l l Models. a) T h e r e i s no c l e a r l y d e f i n e d method o f combining and w e i g h t i n g the d a t a i n o r d e r t o a r r i v e a t an o v e r a l l de-t e r m i n a t i o n o f worth f o r t h e program. a) A c l e a r , l o g i c a l system o f w e i g h t i n g t h e d a t a o f the c r i t e r i a and c o m b i n i n g t h e s e t o d e t e r m i n e t h e o v e r a l l worth has been p r e s e n t e d ( S e c t i o n s 3.522, p. 78, S e c . 3.60, p. 81.) b) The models do not have a s p e c -i f i c check on t h e moral a c c e p t -a b i l i t y o f the program. b) A check o f t h e moral a c c e p t a b i l i t y o f t h e program i s an i n t e g r a l p a r t o f the e v a l u a t i o n . I f a program i s immoral, then t h a t a l o n e i s s u f f i c i e n t r e a s o n t o make the program u n a c c e p t a b l e . ( S e c . 3.22, S e c . 3.60) 115 S c r i v e n ' s r e c e n t c h e c k l i s t f o r e v a l u a t i n g p r o d u c t s and p r o p o s a l s ( S c r i v e n 1974b) p r o v i d e d many o f the c h e c k p o i n t s used i n o p e r a t i o n a l -i z i n g the s t a n d a r d o f Goal Achievement (Sec. 3.33). However, two changes were made i n S c r i v e n ' s C h e c k l i s t t o make i t more c o n g r u e n t w i t h T a y l o r ' s (1961) a n a l y s i s o f the p r o c e s s o f e v a l u a t i o n . F i r s t l y , t h e C h e c k p o i n t "Need" was r e c o n c e p t u a l i z e d and g i v e n s e p a r a t e s t a t u s as a s t a n d a r d i n t h e e v a l u a t i o n . As a s t a n d a r d i t was reworded t o "goal w orth". I t was reworded because "need" has a m u l t i p l i c i t y o f meanings ( e . g . , as an o b j e c t i v e ; a n e c e s s i t y , a d e f i c i e n c y , an o b l i g a t i o n ( K o m i s a r , 1 9 6 1 ) ) , and u s i n g any s i n g l e meaning may be t o o r e s t r i c t i v e . A g o a l , on the o t h e r hand, can be any i n t e n d e d outcome, t h e worth o f which c a n be d e t e r m i n e d by a p p l y i n g t h e s t r a t e g y o u t l i n e d i n S e c t i o n 3.21. Goal w o rth was g i v e n s e p a r a t e s t a t u s as a s t a n d a r d i n t h e e v a l u a t i o n f o r two r e a s o n s : (1) To u n d e r l i n e the im p o r t a n c e o f d e t e r m i n i n g whether t h e g o a l s o f a program a r e w o r t h w h i l e , (2) To c l a r i f y t h e c o n t r i b u -t i o n o f "goal worth" i n the o v e r a l l m e r i t r a t i n g o f t h e program. S e c o n d l y , i n t h i s t h e s i s t h e C h e c k p o i n t o f " C r i t i c a l C omparisons" was d e l e t e d f r o m S c r i v e n ' s C h e c k l i s t . S c r i v e n e x p l a i n s , " C r i t i c a l Comparisons" i n two ways. F i r s t l y , i t i s used t o d e t e r m i n e t h e a p p r o p r i a t e n e s s o f the h y p o t h e t i c a l o r i d e a l form o f t h e program used i n S - G r a d i n g ; and s e c o n d l y i t i s an a c t u a l comparison o f how t h e program b e i n g e v a l u a t e d compares w i t h o t h e r , s i m i l a r programs. T h a t i s , how i t would be RANKED. The second e x p l a n a t i o n o f " C r i t i c a l Comparisons" i s a p r o c e d u r a l t e c h n i q u e , and, though i t r e s u l t s i n a judgement o f t h e r e l a t i v e 116 v a l u e o f a program, i t does not c o n t r i b u t e d i r e c t l y t o the worth o f the program. T h e r e f o r e i t s h o u l d not be i n c l u d e d i n the C h e c k l i s t . F o r example, a poor program, when compared t o o t h e r s i m i l a r programs, may be t h e b e s t o f t h e l o t ; b u t t h a t judgement does n o t make i t b e t t e r than i t was b e f o r e i t was compared t o t h e o t h e r programs. The f i r s t e x p l a n a t i o n o f " C r i t i c a l Comparison", t h a t i t i s an i n d i c a t i o n o f t h e a p p r o p r i a t e n e s s o f the i d e a l f o r m o f the; program used i n S - G r a d i n g , * - i ^ a n r e v a l u a t i o n o f the e v a l u a t i o n . The i d e a l form o f the program chosen f o r S-Grading w i l l d e t e r m i n e t h e c r i t e r i a used i n e s t a b l i s h i n g t h e s t a n d a r d s . I f the c r i t e r i a were so easy t o a t t a i n t h a t even a poor program c o u l d g e t h i g h marks, then t h a t f a u l t r e f l e c t s on the e v a l u a t i o n n o t on the program b e i n g e v a l u a t e d . The r e a l i s m o f the c r i t e r i a i s r e f l e c t e d i n t h e d e f e n s i b i 1 i t y o f t h e i d e a l t o b o t h t h e c l i e n t and the e d u c a t i o n a l community. The p a r a l l e l i n s t a t i s t i c s , t o t h e p r e c e e d i n g p a r a g r a p h , i s a d i s c u s s i o n o f TYPE I and TYPE I I e r r o r s . Suppose we worded t h e o u t -comes o f an e v a l u a t i o n i n t h e f o l l o w i n g s t a t i s t i c a l manner: H G: The program i s a ppor program. H-|: The program i s a good program. I f our c r i t e r i a a r e so easy t o a t t a i n t h a t we r e j e c t H 0 i n f a v o u r o f H-| when we s h o u l d have a c c e p t e d H 0, t h e n a k i n d o f TYPE I e r r o r has been committed. I f , on the o t h e r hand, we a r e so s e v e r e d i n our c h o i c e o f c r i t e r i a f o r the e v a l u a t i o n t h a t we a c c e p t H 0 when we s h o u l d have r e j e c t e d i t , then a k i n d o f TYPE II e r r o r has been committed. 117 T h e r e f o r e , as i n s t a t i s t i c s , the c h o i c e o f c r i t e r i a f o r an e v a l u a t i o n i s an a t t e m p t t o m i n i m i z e b o t h TYPE I and TYPE II e r r o r s . As s u c h , the c h e c k p o i n t o f C r i t i c a l Comparisons can be used as a method o f s c r u t i n i z i n g t he c h o i c e o f c r i t e r i a , but does n o t d i r e c t l y c o n t r i b u t e t o the worth o f the program b e i n g e v a l u a t e d . F o r t h i s r e a s o n and t h e r e a s o n p r e v i o u s l y g i v e n , " C r i t i c a l C omparisons" was o m i t t e d from the c h e c k l i s t o u t l i n e d i n Sec. 3.33. The e f f e c t o f the e v a l u a t o r e n t e r i n g a program i s an i m p o r t a n t c o n c e r n which was not f u l l y i n v e s t i g a t e d i n t h i s t h e s i s . However, i t was a p p a r e n t t h a t some pro b l e m s , which would have remained as i n t e r n a l g r u m b l i n g s and been a c c e p t e d by t h e program p a r t i c i p a n t s w i t h o u t comment, r a p i d l y g a i n e d more s t a t u r e than t h e y d e s e r v e d , because t h e r e was an e v a l u a t o r p r e s e n t who r e c o r d e d them. The appearance o f t h e e v a l u a t o r tended t o make many uncommitted p a r t i c i p a n t s o f t h e program more c r i t i c a l , and many committed p a r t i c i p a n t s more d e f e n s i v e . A l t h o u g h t h e methodology o f e v a l u a t i o n developed,; by t h i s t h e s i s was o n l y a p p l i e d t o a t e a c h e r t r a i n i n g program, i t would a p p e a r t h a t i t c o u l d e a s i l y be extended t o the e v a l u a t i o n o f o t h e r e d u c a t i o n a l programs. The two s t a n d a r d s u s e d , g o a l w o r t h and g o a l a c h i e v e m e n t , a r e s t a n d a r d s which a p p l y t o a l l e d u c a t i o n a l programs. The g e n e r a l e x p l i c a t i o n o f e v a l u a t i o n i n C h a p t e r Two can be a p p l i e d t o the e v a l u a -t i o n o f any p r o d u c t . The f o r m o f the e v a l u a t i o n i l l u s t r a t e d by t h i s t h e s i s was S-GRADING. However, as shown i n Sec. 3.70, t h e e x t e n s i o n t o more comprehensive 118 e v a l u a t i o n s i n RANKING i s l o g i c a l l y a p p a r e n t . F i n a l l y , t he methodology d e v e l o p e d i s seen as p r o v i d i n g a l i n k between t h e t h e o r e t i c a l views o f e v a l u a t i o n and t h e i r a p p l i c a t i o n t o a p a r t i c u l a r e v a l u a t i o n s t u d y . 118a REFERENCES A i k e n , W.M. 1942. The S t o r y o f the E i g h t Year S t u d y . Harper & B r o t h e r s , N.Y., N.Y. -ATkijn, M i , K l e i n , S., F e r n s t e r m a c h e r , G. 1968. The C e n t e r s Changing E v a l u a t i o n Model. E v a l u a t i o n Comment, O c t o b e r , 1968. Baker, Eva. 1974. " F o r m a t i v e E v a l u a t i o n o f I n s t r u c t i o n " , E v a l u a t i o n  i n E d u c a t i o n , ed. W.J. Popham.''McCutchan P u b l i s h i n g Corp. B e r k e l e y , C a l i f o r n i a . 1974. pp. 635-584. Bloom, B.S. 1969. "Some T h e o r e t i c a l I s s u e s R e l a t i n g t o E d u c a t i o n a l E v a l u a t i o n " , E d u c a t i o n a l E v a l u a t i o n : New R o l e s , New Means, ed. R. T y l e r . The 68th Annual Yearbook o f t h e NSSE. U n i v e r s i t y o f C h i c a g o P r e s s , C h i c a g o , 111. 1969. pp. 26-50. Broudy, H a r r y S. 1965. " C r i t e r i a f o r t h e P r o f e s s i o n a l P r e p a r a t i o n o f T e a c h e r s . " J o u r n a l o f T e a c h e r E d u c a t i o n . l!6)(4), 1965. pp. 408-415. D a n i e l s , LeRoi B. 1971. The J u s t i f i c a t i o n o f C u r r i c u l a . A paper p r e p a r e d f o r d i s c u s s i o n a t the AERA 1971 Annual meeting i n New York C i t y ( s e s s i o n B 1 4 ) . Evans, J.W. 1974. " E v a l u a t i n g E d u c a t i o n a l Programs - A r e We G e t t i n g Anywhere?" E d u c a t i o n a l R e s e a r c h e r , V o l . 3, No. 8, S e p t . , 1974. pp. 7-12. F r a n k e n a , W.K. 1973. E t h i c s . P r e n t i c e - H a l l , I n c . Englewood C l i f f s , New J e r s e y . 2nd E d i t i o n . G l a s s , G.V. 1969. The Growth o f E v a l u a t i o n Methodology. L a b o r a t o r y o f E d u c a t i o n a l R e s e a r c h , U n i v e r s i t y o f C o l o r a d o . G l a s s , G.V. 1975. "A Paradox about E x c e l l e n c e o f S c h o o l s and t h e P e o p l e i n Them." E d u c a t i o n a l "Researcher. March 1975, pp. 9-13. Gleadow, Norman E. 1975. "The M o b e r l y T o t s Summer S c h o o l Program 1975" R e s e a r c h R e p o r t ( i n P r i n t ) . E v a l u a t i o n and R e s e a r c h , E d u c a t i o n S e r v i c e s Group, Vancouver Sch o o l B o a rd, 1595 West 10th Ave. Vancouver, B.C. Canada Guba, Egon G. 1969. "The F a i l u r e o f E d u c a t i o n a l E v a l u a t i o n " E d u c a t i o n a l T e c h n o l o g y , May 1969. pp. 29-38. 118b H a l l , 1966. "The A p p l i e d S o c i o l o g i s t and O r g a n i z a t i o n a l S o c i o l o g y " i n S o c i o l o g y i n A c t i o n . Ed. A.B. S h o s t a k . Dorsey P r e s s . H a l l e r , E . J . 1974. " C o s t A n a l y s i s f o r E d u c a t i o n a l Program E v a l u a t i o n , " E v a l u a t i o n i n E d u c a t i o n . Ed. W.J. Popham. McCutchan P u b l i s h i n g C o r p o r a t i o n , B e r k e l e y , C a l i f o r n i a , pp. 406-449. Hayes, Samuel P. ( J r . ) 1969. E v a l u a t i n g Development P r o j e c t s . ( 5 t h E d i t i o n ) U n i t e d N a t i o n s E d u c a t i o n a l , S c i e n t i f i c and C u l t u r a l O r g a n i z a t i o n . P l a c e de F o n t e r o y , P a r i s . H e m p h i l l , John K. 1969. "The R e l a t i o n s h i p between R e s e a r c h and E v a l u a t i o n S t u d i e s " , E d u c a t i o n a l E v a l u a t i o n , New R o l e s , New  Means. Ed. R. T y l e r . The 68th Annual Yearbook o f t h e NSSE. U n i v e r s i t y o f C h i c a g o P r e s s , C h i c a g o , 111. pp. 189-220. House, E r n e s t R. 1973. S c h o o l E v a l u a t i o n : The P o l i t i c s and P r o c e s s . McCutchen P u b l i s h i n g C o r p o r a t i o n . C a l i f o r n i a , U.S.A. K e r l i n g e r , F r e d N. 1973. F o u n d a t i o n s o f B e h a v i o r a l R e s e a r c h . 2nd Ed. H o l t , R i n e h a r t and W i n s t o n , I n c . T o r o n t o , Ont. Komisar, B. Paul 1961. Needs and t h e N e e d s - C u r r i c u l u m . Language  and Concepts i n E d u c a t i o n . S mith, B.O. and E n n i s , R.H~ ( E d ' s ) pp. 24-43. Rand M c N a l l e y & Co., C h i c a g o . L i k e r t , R. 1932. "The method o f C o n s t r u c t i n g an A t t i t u d e S c a l e " A r c h i v e s o f P s y c h o l o g y , 140? pp. 44-53. M e t f e s s e l , N.S., M i c h a e l , W.B. 1967. "A Paradigm I n v o l v i n g M u l t i p l e C r i t e r i o n Measures f o r E v a l u a t i n g the E f f e c t i v e n e s s o f S c h o o l Programs" E d u c a t i o n a l and P s y c h o l o g i c a l Measurement. 27. p. 931-943. P e t e r s , R.S. 1966. E t h i c s and E d u c a t i o n . George A l l e n & Unwin L t d . London. Popham, W.J. 1969. " E d u c a t i o n a l Needs Assessment." In C u r r i c u l u m  T heory Network Monograph-Supplement C u r r i c u l u m E v a l u a t i o n :  P o t e n t i a l i t y and R e a l i t y Ed. J o e l Weiss. The O n t a r i o I n s t i t u t e f o r S t u d i e s i n E d u c a t i o n . 1972. p. 22-32. Popham, W.J. 1972. An E v a l u a t i o n Guidebook. The I n s t r u c t i o n a l O b j e c t i v e s Exchange, Lo.scAngeles, C a l i f o r n i a . Popham, W.J. 1974. E d i t o r . E v a l u a t i o n i n E d u c a t i o n : C u r r e n t A p p l i - c a t i o n s . McCutchan P u b l i s h i n g Corp. B e r k e l e y , C a l i f o r n i a . 118c P r o v u s , M. 1969. " E v a l u a t i o n o f Ongoing Programs i n t h e P u b l i c S c h o o l System". E d u c a t i o n a l . E v a l u a t i o n : New R o l e s , New Means. Ed. Paul T y l e r P a r t I I : The 68th Yearbook o f the NSSE. U n i v e r s i t y o f C h i c a g o P r e s s , C h i c a g o , 111, pp. 242-283. P r o v u s , M. 1970. " E v a l u a t i o n o r R e s e a r c h , R e s e a r c h o r E v a l u a t i o n ? " E d u c a t i o n a l T e c h n o l o g y , Aug. 1970. pp. 50-54. P r o v u s , M. 1972. " E v a l u a t i o n as P u b l i c P o l i c y " , C u r r i c u l u m T h e o r y  Network. Monograph Supplement C u r r i c u l u m E v a l u a t i o n : P o t e n t i a l i t y  and R e a l i t y . Ed. J o e l Weiss. The O n t a r i o I n s t i t u t e f o r S t u d i e s i n E d u c a t i o n . 1972. Pp. 33-44. R o s s i , P. 1966. "Boobytraps and P i t f a l l s i n the E v a l u a t i o n o f S o c i a l A c t i o n Programs", P r o c e e d i n g s : Annual M e e t i n g o f the American  S t a t i s t i c a l A s s o c i a t i o n , Washington, D.C. ^'j^ S c r i v e n , M. 1966a. V a l u e C l a i m s i n t h e S o c i a l S c i e n c e . Bounder, C o l o r a d o , S o c i a l S c i e n c e E d u c a t i o n C o n s o r t i u m P u b l i c a t i o n No. 123. S c r i v e n , M. 1966b. "Causes, C o n n e c t i o n s and C o n d i t i o n s i n H i s t o r y , " P h i l o s o p h y o f E d u c a t i o n a l R e s e a r c h . Ed. H.S. Broudy, R.H. E n n i s , L . I . K r i n e r m a n . John W i l e y & Sons, I n c . T o r o n t o 1973. pp. 439-458. S c r i v e n , M. 1967. "The Methodology o f E v a l u a t i o n " . P e r s p e c t i v e s o f  C u r r i c u l u m E v a l u a t i o n . AERA Monograph s e r i e s on C u r r i c u l u m Development No. 1, 1967. C h i c a g o 111. Radn M c N a l l y . pp. 39-83. S c r i v e n , M. 1972. " E v a l u a t i o n : Noble P r o f e s s i o n and P e d e s t r i a n P r a c t i s e . " C u r r i c u l u m E v a l u a t i o n P o t e n t i a l . A C u r r i c u l u m T h e o r y  Network Monograph. T o r o n t o : O n t a r i o I n s t i t u t e f o r S t u d i e s i n E d u c a t i o n , pp. 132-139. S c r i v e n , M. 1973. "The E v a l u a t i o n o f T e a c h i n g a t B e r k e l e y " D r a f t paper ( n o t a f i n a l v e r s i o n ) . 10/10/73. S c r i v e n , M. 1974a. "The E v a l u a t i o n o f E d u c a t i o n a l G o a l s , I n s t r u c t i o n a l P r o c e d u r e s and Outcomes, o r , t h e Human Cometh", Program Development  i n E d u c a t i o n , Ed. B l a n e y , Horsego and M c i n t o s h . ( C e n t e r f o r Con-t i n u i n g E d u c a t i o n , U n i v e r s i t y o f B.C. 1974. V a n c o u v e r , B.C. Canada), pp. 134-162. S c r i v e n , M. 1974b. " E v a l u a t i o n P e r s p e c t i v e s and P r o c e d u r e s " , E v a l - u a t i o n i n E d u c a t i o n : C u r r e n t A p p l i c a t i o n s . Ed. W.J. Popham. McCutchan P u b l i s h i n g C o r p o r a t i o n , B e r k e l e y , C a l i f o r n i a , pp. 1-94. 118d S j o r g e n , Douglas D. 1970. "Measurement T e c h n i q u e s i n E v a l u a t i o n " Review o f E d u c a t i o n a l R e s e a r c h . 40. No. 2. A p r i l 1970. pp. 301-320. S t a k e , R.E. 1967. "The Contenance o f E d u c a t i o n a l E v a l u a t i o n " T e a c h e r s  C o l l e g e R e c o r d . 68. pp. 523-540. S t a k e , R?71971. An . E v a l u a t i o n o f TCITY, t h e Twin C i t y I n s t i t u t e f o r  T a l e n t e d Youth. S t a k e , R. 1972a. An Approach t o t h e E v a l u a t i o n o f I n s t r u c t i o n a l  Programs. Paper d e l i v e r e d a t AERA Annual M e e t i n g A p r i l 4, 1972. i n C h i c a g o (ERIC ED064350). S t a k e , R.E. 1972b. " R e s p o n s i v e E v a l u a t i o n " . U n i v e r s i t y o f I l l i n o i s , u n p u b l i s h e d m a n u s c r i p t . S t a k e , R.E. 1972c. "An Approach t o t h e E v a l u a t i o n o f I n s t r u c t i o n a l Programs" A paper d e l i v e r e d a t the AERA Annual m e e t i n g , A p r i l 4, 1972. i n C h i c a g o . (ERIC a c c e s s no: ED 064350) S t u f f l e b e a m , D.L. 1968. "Toward a S c i e n c e o f E d u c a t i o n a l E v a l u a t i o n " E d u c a t i o n a l T e c h n o l o g y , J u l y 30, 1968. pp. 5-12. T a y l o r , P.W. 1961. Normative D i s c o u r s e . P r e n t i c e - H a l l , I n c . Englewood C l i f f s , N.J. T a y l o r , P.A. and M a g u i r e , T.0. 1966. "A T h e o r e t i c a l E v a l u a t i o n Model" Manitoba J o u r n a l o f E d u c a t i o n R e s e a r c h . 1966, ]_, pp. 12-17. T y l e r , R. 1942. B a s i c P r i n c i p l e s o f C u r r i c u l u m and I n s t r u c t i o n . U n i v e r s i t y o f C h i c a g o P r e s s , C h i c a g o . 134?r T y l e r , R.W. 1969. E d i t o r . E d u c a t i o n a l E v a l u a t i o n : New R o l e s , New  Means. P a r t I I o f the 68th Annual Yearbook o f the NSSE. U n i v e r -s i t y o f C h i c a g o P r e s s . C h i c a g o , 111. ; U.S. Dept. o f H e a l t h , E d u c a t i o n and W e l f a r e . ' 1959. A c c r e d i t a t i o n i n  H i g h e r E d u c a t i o n . U.S. Government P r i n t i n g O f f i c e , Washington, D.C. Welch, Wayne W. 1969. " C u r r i c u l u m E v a l u a t i o n " . Review o f E d u c a t i o n a l  R e s e a r c h . V o l . 39, No. 4, O c t . 1969. pp. 429-443. Welch, Wayne W. 1972. " E v a l u a t i o n o f the PSNS c o u r s e : I : Desi g n and Imp l e m e n t a t i o n " J o u r n a l o f R e s e a r c h i n S c i e n c e T e a c h i n g 9^  pp. 139-145. 118e Worthen, B. 1968. "Toward a Taxonomy o f E v a l u a t i o n D e s i g n s " . Edu- c a t i o n a l T e c h n o l o g y , Aug.,1'5, p. 3-9. Worthen and S a n d e r s . 1973. E d i t o r s . E d u c a t i o n a l E v a l u a t i o n : T h e o r y  and P r a c t i c e . C h a r l e s A. Jones P u b l i s h i n g . C o . W o r t h i n g t o n , O h i o . Y t 1 119 Appendix A Q u e s t i o n n a i r e f o r t h e Program D e v e l o p e r s S u b j e c t : E v a l u a t i o n P r i o r i t i e s 120 Program B EVALUATION PRIORITIES The persons involved in Program B should help determine the direction and emphasis of an internal evaluation. Therefore, you are asked to indicate on the scale, following each of the topics in the checklist, the Importance and emphasis the topic should receive, This is not a test or attitude survey. The results will be used to help determine initially, which topics should be investigated, the depth of the investigation and the sequence of investigation, 1) MARKET Is there an implementation plan for ensuring a market for the project? Possible considerations should be: 1. Portability of the program. 2. Clarity, feasibility, ingenuity and economy of the imple-mentation plan. 3. Size of the market for Program B. 4. Importance of the market for Program B. 5. Communications between Program B and potential consumers. Importance of evaluating Market (check one) 4 Extremely important: a comprehensive check should be made. 3 Important: check as thoroughly as you can. 2 Definitely do some checking, 1 Check only if there is time. _ 0 Don't waste time on it, Any comments? 2) TRUE FIELD TRIALS (performance) How typical is the performance of the Program B program in /s'.v'Yschool of the final, portable version. For example: 1. Are the student-teachers in Program B typical of their group? 2. Are the U.B.C. staff members a representative sample? 121 3. Is the School representative? 4. Are any major changes seen for the final version? Importance of evaluating True Field Trials: (check one) 4 Extremely important: a comprehensive check should be made. 3 Important: check as thoroughly as you can. 2 Definitely so some checking, 1 Check only if there is time, 0 Don't waste time on it. Any comments? 3. TRUE CONSUMER (performance) How do significant consumer groups gauge the performance of Program B. Examples of significant consumers would be: 1. The student-teachers, 2. The U.B.C. staff. 3. The*Lr.'\-i3st/;,5» school teachers and principal, 4. The students in the school. 5. Vancouver School Board. Importance of evaluating True Consumers: (check one) 4 Extremely important: a comprehensive check should be made. 3 Important: check as thoroughly as you can. 2 Definitely do some checking. 1 Check only if there is time. 0 Don't waste time on it. Any comments? 122 CRUCIAL COMPARISONS [ p e r f o rman c e ) HOY/ does P rogram B p e r f o r m a s compared t o o t h e r t e a c h e r t r a i n i n g p rograms a t U . B . C ? I s P rogram B a c h i e v i n g t h e same e x p e c t a t i o n s as o t h e r t e a c h e r - t r a i n i n g p rograms a t U . B . C ? A c r i t i c a l c o m p e t i t o r m i g h t be t h e r e g u l a r 3 r d y e a r t e a c h e r s ' t r a i n i n g p rog ram. Impo r t an ce o f e v a l u a t i n g c r u c i a l c o m p a r i s o n s : ( c he c k one) 4 E x t r e m e l y i m p o r t a n t : a c omp rehen s i v e c he c k s h o u l d be made. — 3 ^ I m p o r t a n t : check as t h o r o u g h l y as you c an ; 2 D e f i n i t e l y do some c h e c k i n g . 1 Check o n l y i f t h e r e i s t i m e . 0 D o n ' t w a s t e t i m e on i t . Any comments? SIDE EFFECTS ( p e r f o rman c e ) A r e t h e r e any u n a n t i c i p a t e d s i d e e f f e c t s t o t h e program? T h i s c h e c k p o i n t wou ld l o o k f o r ( f o r e x a m p l e ) : 1. Any u n a n t i c i p a t e d b e n e f i t s o r p rob l ems t o t h e s t u d e n t - t e a c h e r s . 2. Any u n a n t i c i p a t e d b e n e f i t s o r p r ob l ems t o t h e s t u d e n t s . 3. Any u n a n t i c i p a t e d b e n e f i t s o r p rob l ems t o t h e s c h o o l and i t s s t a f f . 4. Any u n a n t i c i p a t e d b e n e f i t s o r p rob l ems t o t h e U n i v e r s i t y and t h e U . B . C s t a f f . Impo r t ance o f e v a l u a t i n g S i d e E f f e c t s : ( check one) 4 E x t r e m e l y i m p o r t a n t : a c omp rehen s i v e check s h o u l d be made. 3 I m p o r t a n t : i c h e e f e - a s e t h o r o u g h l y as you c a n . 2 D e f i n i t e l y do some c h e c k i n g . 1 Check o n l y i f t h e r e I s t i m e . 0 D o n ' t wa s t e t i m e on i t . S h o u l d t h i s be done : d u r i n g ; a t t h e end o f ; a f t e r P rogram B? Any comments? 123 PROCESS (p e r f o r m a n c e ) Is t h e Program B program a c h i e v i n g what t t s e t o u t t o do? Is t h e r e any i n j u s t i c e ; any u n h a p p i n e s s ; any c r u e l t y o r t h e i r c o n v e r s e s which a r e p a r t o f t h e program? Examples o f t h t s c h e c k p o i n t a r e : 1. A r e t h e s t u d e n t s a t the s c h o o l b e i n g t a u g h t e f f e c t i v e l y by i . . s t h e e s t u d e n t ^ t e a c h e r s ? 2. Are t h e s t u d e n t - t e a c h e r s b e i n g t a u g h t e f f e c t i v e l y by t h e U.B.C. s t a f f ? 3. Is t h e r e any c r u e l t y , u n h a p p i n e s s o r I n j u s t i c e i n t h e program? 4. Is t h e r e the c o n v e r s e o f "3" i n the program? 5. Are t h e r e any moral i m p l i c a t i o n s t o c o n s i d e r ? 6. Are o b s e r v a t i o n and d e s c r i p t i o n c o n g r u e n t ? Importance o f e v a l u a t i n g P r o c e s s : (check one) 4 E x t r e m e l y i m p o r t a n t : a comprehensive check s h o u l d be made. 3 Important: check as t h o r o u g h l y as you can. 2 D e f i n i t e l y do some c h e c k i n g . 1 Check o n l y i f t h e r e i s t i m e . 0 Don't waste time on i t . Any comments? CAUSATION (p e r f o r m a n c e ) A r e the o b s e r v e d and r e p o r t e d outcomes a t t r i b u t a b l e t o Program B? F o r example: 1, Have the c l a s s r o o m t e c h n i q u e s used by t h e s t u d e n t - t e a c h e r s been l e a r n e d t h r o u g h Program B? 2, Are the b e h a v i o r s i d e n t i f i e d under " P r o c e s s " t h e r e s u l t o f the 124 Program B program? Importance o f e v a l u a t i n g C a u s a t i o n : [check, one) 4 E x t r e m e l y i m p o r t a n t : a comprehensive check s h o u l d be made. 3 Important: check as t h o r o u g h l y as you can. 2 D e f i n i t e l y so some c h e c k i n g . 1 Check o n l y i f t h e r e t s t i m e . 0 Don't waste t i m e on i t . Any comments? 8, (a) The t o p i c you a r e p e r s o n a l l y most I n t e r e s t e d i n i s number (b) The t o p i c you c o n s i d e r most n e c e s s a r y t o the e v a l u a t i o n o f Program B i s number . Any comments? Appendix B I n t e r v i e w o f School A d m i n i s t r a t i o n Q u e s t i o n s asked; 126 1) What needs do you see Program B f u l f i l l i n g ? 2) Is Program B d o i n g t h e t h i n g s you e x p e c t e d i t t o do? I f Yes? What a r e t h o s e t h i n g s ? I f NO? What i s b e i n g o m i t t e d ? 3) Are you c l e a r about y o u r r o l e i n Program B? 4) A r e t h e r e any u n a n t i c i p a t e d p o s i t i v e e f f e c t s o f t h e program so f a r ? I f y e s , what a r e t h e y ? 5) A r e t h e r e any u n a n t i c i p a t e d n e g a t i v e e f f e c t s o f t h e program so f a r ? I f y e s , what a r e t h e y ? 6) What a r e y o u r g e n e r a l i m p r e s s i o n s about t h e U.B.C. s t a f f and t h e i r f u n c t i o n i n g i n the program? 7) What a r e y o u r g e n e r a l i m p r e s s i o n s about t h e s t u d e n t t e a c h e r s ? 8) Any o t h e r comments? 127 Appendix C L i k e r t A t t i t u d e S c a l e : t o measure g e n e r a l a t t i t u d e s toward t h e t e a c h e r t r a i n i n g p r o c e s s o f t h e s t u d e n t . t e a c h e r s . 128 This is not a test. Consider each item carefully before answering, Your answers will remain anonymous. Record all your answers on the computer answer card (depicted below) i 1 i ! i ! l 1 1 0 0 '0 rt J 0 *-* w* O •0 19 ;o 3 6 3 ii 6 0 B n Li o rt r» i i c c § § § § 3 8 r3 § g g C3 r» 6 8 o n C t? r: 0 rt 5 i» § § P rt L5 5 A IS r» 6 i j r. o ^ L» § u o rt rt 7 u r> p p '7 <7 |7 L. o U rt I7 B g g g r: i/ o S fS <3 i« I? 1? 1? § Ps o C8 § ? <-5 § 6 !? p? p? rt !*3 i a 2 5; & i 7 ;« •? >e> it 12 is i * ts l i i? iz n to n z> zr. u 2~ ;1 - :1 •2 7 '2 !3 ii 3 I R Q i i3! *3; p. 1 G i •-' i O J O 1 O fi i a i ii! (i! (i: fi i 'i ... ! o 1 , G G | G 1 ij rt I rt i rt ? 1 '? BIBS is vANSWER FIELD.? 2«,27. :<>," f c 1 ' I (~> I rt I rt ( rt J rt I C I I a I ij G , i; e8;3i iii; q I p : G I G' ; r ^ i c •i!U P.! P. ^ n I • S i aiM ; 3 3 | 3-1 ! 35 ! 35 | 3?| 33 | 33 i 40 : 41 i « I 4] j « | 4b j «6 j 47 : SS ! <3 Li i i G | o :5 I [5 B|B B L? i n • | u j u i (5 ! C L3113 i l3 § r> , p I rt '•' : I-' I 1 L. i G o BIGlBiBiB i2 -2 :2 ; o - rt • rt 3 I iJ • 13 : ,3 I* i u | 0 . 0 •I :5 I n rt i i I , •6 ! '6 j 15 rt ; rt n f; ; ,C .5 B B B s u o rt a B B B Q 3 n i f i : i ' § § § g g ?2 o g g g >2 u rt • rt ' rt 2 . 2 ; 2 ^ ^ , ^ C3 C3 g g P3 g g g g rt G4 S rj 8 8 8 8 8 rt d 8 rt U i o |3 r5 rt '5 £ 8 B s rt i rt j 5 ; ,5 , --, UNIVEHS.Tr C " 6RIT;SH COLUMBIA Do not put any marks in the section called "identification number", or in the section called "answer card number". Use the answer field to record your answers. There are 50 columns for answers, but you will only use the first 18 columns. For each item in the folic ./ing form there is a corresponding 5-choice column. Record your response to each item by blackening the appropriate bubble in the column corresponding to the item. See the following example: EXAMPLE: 1. EDUCATION IS FUN. Answer 1 - Strongly disagree 2 - Disagree 3 - Undvicided -»-4 - Agree 5 - Strongly agree BLACKEN bubble 4 in column 1 as shown below: Item 1 USE PENCIL ONLY! Answer (4) I 0) 129 RESPONSE CODE 1 SD S t r o n g l y d i s a g r e e w i t h i t e m . 2 D D i s a g r e e w i t h i t e m , 3 U U n d e c i d e d . 4 A Agree w i t h i t e m . 5 SA S t r o n g l y a g r e e w i t h i t e m . PLEASE USE PENCIL ONLY: Note: TEACHER TRAINING r e f e r s t o y o u r whole program o f c o u r s e s , c l a s s e s , t e a c h i n g p r a c t i c e , e t c . i . e . i t i n c l u d e s a l l a s p e c t s o f y o u r program. STUDENT TEACHING r e f e r s t o y o u r c l a s s r o o m t e a c h i n g p r a c t i c e o n l y . THE STATEMENTS: PLEASE RESPOND TO THESE ON YOUR COMPUTER ANSWER  CARD, 1, I HOPE TO KEEP IN TOUCH WITH MY PRESENT CLASSMATES IN THE FUTURE. 2, U.B.C, HAS MADE A SIGNIFICANT CONTRIBUTION TO TEACHING IN B.C. 3, STUDENT TEACHING ISN'T WORTH THE EFFORT. 4, UNIVERSITY IS AN ENJOYABLE EXPERIENCE. 5, I LOOK FORWARD TO MY STUDENT TEACHING EXPERIENCE, 6, GENERALLY SPEAKING, MY EDUCATION COURSES ARE USELESS. 7, MY U.B.C. INSTRUCTORS ARE FINE PEOPLE. 8, TEACHING IS AN ENJOYABLE EXPERIENCE. 9, MY CLASSMATES ARE A VALUABLE SOURCE OF IDEAS FOR TEACHING. 10. MY SPONSOR TEACHERS HAVE HELPED ME DEVELOP MANY USEFUL TEACHING SKILLS. 11. TEACHING SCHOOL IS A VERY IMPORTANT JOB. 12. TEACHER TRAINING IS BORING. 13. AS A WHOLE, MY EDUCATION COURSES ARE INTERESTING. 14. MY SPONSOR TEACHERS ARE GENERALLY ENTHUSIASTIC ABOUT TEACHING. 15. TEACHER TRAINING IS IMPORTANT. 16. MY EDUCATION COURSES ARE NOT HELPING ME TO BECOME A GOOD TEACHER. 17. TEACHER TRAINING IS AN ENJOYABLE EXPERIENCE. 18. MY SPONSOR TEACHERS HAVE A STRONG INFLUENCE ON MY TEACHING. Appendix D R e a c t i o n Form 131 REACTION FORM Date C i r c l e t h e number below which b e s t r e p r e s e n t s y o u r f e e l i n g s about Program B a t t h i s t i m e . 1 2 3 4 5 6 7 v e r y n e u t r a l v e r y n e g a t i v e p o s i t i v e B r i e f l y d e s c r i b e any i n c i d e n t s t h a t you saw as p o s i t i v e , and any t h a t you saw as n e g a t i v e r e l a t i n g t o y o u r c u r r e n t r e a c t i o n t o Program B. A n y t h i n g e l s e you'd l i k e t o mention? 132 Appendix E E v a l u a t i o n o f UBC i n s t r u c t o r s ' t e a c h i n g p e r f o r m a n c e . Form A. f o r m A RESPONSE CODE 1 SD S t r o n g l y d i s a g r e e w i t h i t e m . 2. D D i s a g r e e w i t h i t e m . 3. U U n d e c i ded, 4, A Agree w i t h i t e m . 5. SA S t r o n g l y a g r e e w i t h i t e m . COURSE INSTRUCTOR_ IDENTIFICATION No; COURSE INSTRUCTOR__ IDENTIFICATION No.. 1) T h i s i n s t r u c t o r g i v e s u n s a t i s f a c t o r y answers t o s t u d e n t s ' q u e s t i o n s . 2) T h i s i n s t r u c t o r o f t e n m i s s e s t h e p o i n t o f a q u e s t i o n asked by a s t u d e n t . 3) T h i s i n s t r u c t o r seldom v a r i e s h i s method o f i n s t r u c t i o n . 4) T h i s i n s t r u c t o r ' s p r e s e n t a t i o n s i n c l a s s a r e p o o r l y o r g a n i z e d . 5) T h i s i n s t r u c t o r i s v e r y c a r e f u l n o t t o o f f e n d h i s s t u d e n t s w i t h h i s remarks, 6) T h i s i n s t r u c t o r i s i m p a t i e n t w i t h h i s s t u d e n t s , 7) The l e v e l o f d i f f i c u l t y o f t h i s c o u r s e i s about r i g h t . 8) The i n s t r u c t o r seldom checks t o see i f s t u d e n t s u n d e r s t a n d what he i s t e a c h i n g . 9) T h i s i n s t r u c t o r c l e a r l y e x p l a i n s d i f f i c u l t i d e a s . 10) T h i s i n s t r u c t o r uses humour e f f e c t i v e l y i n h i s t e a c h i n g . 11) T h i s i n s t r u c t o r seldom engages h i s s t u d e n t s i n c l a s s d i s c u s s i o n s . 12) The r e a d i n g m a t e r i a l s i n t h i s c o u r s e ( t e x t , o t h e r s o u r c e s ) a r e o f l i t t l e use. 13) I d e v e l o p e d my a b i l i t y t o marshal o r i d e n t i f y main p o i n t s o r c e n t r a l i s s u e s i n t e a c h i n g from t h i s c o u r s e . 14') T h i s c o u r s e i s o f d e f i n i t e v a l u e f o r a p e r s o n who w i s h e s t o become a t e a c h e r . I f you have any comments then p l e a s e w r i t e them on t h e back o f t h i s page w i t h t h e name o f t h e i n s t r u c t o r y o u r comments p e r t a i n t o . OVERALL EVALUATION OF INSTRUCTORS 134 1, p r o f e s s o r A 2. 5. p r o f e s s o r ^ E \ 6 . 1 p r o f e s s o r B 3. p r o f e s s o r C 4. p r o f e s s o r D professor F. g [3 p s> f*> ,0 -0 |0 U L» U r» O f» II I 1 u O l> o p 5 ^8 g " ^ § g '". i"; p. p. 0 5 5 1? r?R? P P p b U O t» 0 Cs § L? § C8 o p p in e 8 g § § O p p p !8 !3 15 Po °~ 9 B 9 1-3 ;9 !•} !*(»• 0 « ~ < F p , 1 1 1 0 p. *— PC T > < V-F ir 13 )«•; 15- ts »r J£ n ?s J I 2;: ?* 3r o i O ' j „ I ,J i2: ? ! i w I p 1 n ' C3|&|i ;•> 1 4 ; i R ! 1 ! !i IJ j 6 b' ! C } i § i ^ A N S W E R 1 p.: a ! Q ; ;.4 1 i 11 1 !5 ! £ ! 5 ' ;5 2;.; £ 7 . I! P I E L D J ! 3 j •! P i 5 j Q i p. i .1 i a : !i O I u 1 w p'i$!S p | P p '5! p I P i ;s I i5! £ j '5 i -5 p 1 a ; 2 . I 33 i 34 ! 35 j 36 ; 3 ; i 33 j 33 ( 4U : 41 i 4? ; 43 < 44 | 4 i I 45 ; 47 • 52 , 12 i p - .'2 (] I h 0 | 0 P P ;2 2 ••j! ;5 g|ej§ P ! P i J i J i d B (5 i S ;4 i !4 0 I L. ! C U IU ] li p p I p :2 1 2 I !2 o 0 j a 5!r3ir3 p I p i p i3!i5il5 0 p r; n i p I! p n 1 i p fj f; b ii u li U 0 Li ij g g g g g P2 g p :2 o g 2^ 2 p 2 p b3 g r C3 g" P3 g g IT 8 p ii p. ii S p. D g g Pi g g P b4 3 «_» 4 o r; c • S p Ps Ps p 5 |5 P^  g •P5 !5 c -^ 1 :5 .5 UNIV£H5.Tr C" ERITiSM COLUMBIA PLEASE RATE EACH INSTRUCTOR IN THE APPROPRIATE COLUMN (ILLUSTRATED ABOVE). How good a j o b i s the i n s t r u c t o r d o i n g i n g e t t i n g a c r o s s t o you the knowledge, u n d e r s t a n d i n g , s k i l l s and a t t i t u d e s t h a t appear t o be th e g o a l s o f th e c o u r s e ? (In a n s w e r i n g , t r y not t o be un d u l y i n f l u e n c e d by y o u r l i k i n g o r d i s l i k e f o r t h e i n s t r u c t o r as a p e r s o n ; o r by y o u r f e e l i n g s about t h e c o u r s e c o n t e n t , i f t h a t wasn't something which t h e i n s t r u c t o r c o u l d choose f r e e l y . ) P l e a s e use t h e f o l l o w i n g s c a l e f o r y o u r r e s p o n s e s on th e computer c a r d : 1) O u t s t a n d i n g l y bad 2) Below ave r a g e 3) Average 4) Above a v e r a g e 5) O u t s t a n d i n g l y good 1 3 5 Appendix F Q u e s t i o n n a i r e t o s t u d e n t - t e a c h e r s on t h e i r a l t e r n a t e p l acement. PROGRAM B EVALUATION 136 This questionnaire deals with your most recent experience in alternate placements. 1, During your alternate placement were you given the placement you requested? YES NO 2, Which of the following was your placement? OPEN AREA ; FREE SCHOOL FAMILY GROUPING OTHER [PLEASE SPECIFY1) 3, Was the placement more demanding or less demanding of your time andtenergy than you had anticipated it would be? MORE DEMANDING LESS DEMANDING WHAT I EXPECTED UNDECIDED COMMENTS: 4. Did you find your experience beneficial? YES No UNDECIDED Please comment: 5. Were your sponsor teachers expecting you? YES NO UNDECIDED Comments: 137 D i d you have a c l e a r i d e a o f y o u r t e a c h i n g d u t i e s - b e f o r e y o u r a l t e r n a t e e x p e r i e n c e began? YES NO Comments: D i d y o u r s p o n s o r t e a c h e r have a d u t i e s when you a r r i v e d : YES Comments: c l e a r i d e a o f y o u r t e a c h i n g NO G e n e r a l l y s p e a k i n g , d i d you a c t u a l l y do what you had e x p e c t e d t o do i n y o u r a l t e r n a t e e x p e r i e n c e ? YES NO Clf "NO" p l e a s e comment) Comments: D i d you f e e l you were s u p e r v i s e d o f t e n enought by: a, YOUR SPONSOR TEACHER YES NO UNDECIDED b. YOUR U.B.C. ADVISOR YES NO UNDECIDED Comments: Do you t h i n k t h i s a l t e r n a t e e x p e r i e n c e s h o u l d be c o n t i n u e d i n f u t u r e Program B's? YES, WITH CHANGES YES, WITHOUT CHANGES NO, IT SHOULD BE DISCONTINUED 138 10. ( C o n t i n u e d ) , I f you checked "YES, WITH CHANGES", what changes would you s u g g e s t ? I f you checked "NO, IT SHOULD BE DISCONTINUED", would you p l e a s e I n d i c a t e why you f e e l t h a t way? 11. C o n s i d e r i n g e v e r y t h i n g , what i s y o u r o v e r a l l e v a l u a t i o n o f t h e a l t e r n a t e placement? EXTREMELY VALUABLE VALUABLE HAD SOME VALUE _^  HAD LITTLE VALUE A COMPLETE. WASTE OF TIME 12, Any o t h e r comments? Appendix Q E v a l u a t i o n o f UBC I n s t r u c t o r s -t e a c h i n g p e r f o r m a n c e Form B 140 COURSE: COURSE: 1 2 3 4 5 SD D U A SA FORM B S t r o n g l y d i s a g r e e w i t h i t e m . D i s a g r e e w i t h i t e m . U n d e c i d e d . Agree w i t h i t e m . S t r o n g l y a g r e e w i t h i t e m , INSTRUCTOR: INSTRUCTOR: IDENTIFICATION NO. IDENTIFICATION NO. 1) The i n s t r u c t o r a p p e a r s i n t e r e s t e d i n t e a c h i n g t h i s c o u r s e , 2) O v e r a l l , t h i s i s a good c o u r s e , 3) The p l a n f o r t h i s c o u r s e t s c l e a r l y e v i d e n t . 4) The i n s t r u c t o r a l l o w s a few s t u d e n t s t o m o n o p o l i z e c l a s s d i s c u s s i o n s . 5) The i n s t r u c t o r i s seldom a v a i l a b l e f o r a s s i s t a n c e o u t s i d e o f c l a s s t i m e . 6) The i n s t r u c t o r o f t e n s t i m u l a t e s s t u d e n t s t o u n d e r t a k e a d d i t i o n a l work i n t h e c o u r s e on t h e i r own, 7) The i n s t r u c t o r seldom asks q u e s t i o n s which s t i m u l a t e d i s c u s s i o n . 8) The i n s t r u c t o r seems competent i n t h e s u b j e c t m a t t e r o f t h i s c o u r s e . 9) The i n s t r u c t o r f r e q u e n t l y p o i n t s o u t how d i f f e r e n t p a r t s o f t h e c o u r s e a r e r e l a t e d . 10) I can r e c o g n i z e good and bad t e a c h i n g t e c h n i q u e s because o f t h i s c o u r s e . 0 11) T h i s c o u r s e i n c r e a s e d my awareness o f i s s u e s r e l a t e d t o t h e s u b j e c t m a t t e r . 12) T h i s c o u r s e i s o f d e f i n i t e v a l u e f o r a p e r s o n who w i s h e s t o become a t e a c h e r . I f you have any comments then p l e a s e w r i t e them on t h e back o f t h i s page w i t h t h e name o f t h e i n s t r u c t o r y o u r comments p e r t a i n t o . 141 O v e r a l l E v a l u a t i o n o f I n s t r u c t o r s ( s e e Appendix £ ) 1 P r o f e s s o r 2 P r o f e s s o r 3 P r o f e s s o r 4 P r o f e s s o r 5 P r o f e s s o r e t c . A B O D E PLEASE RATE EACH INSTRUCTOR IN THE APPROPRIATE COLUMN (ILLUSTRATED ABOVE) How good a j o b i s t h e i n s t r u c t o r d o i n g i n g e t t i n g a c r o s s t o you t h e knowledge, u n d e r s t a n d i n g , s k i l l s and a t t i t u d e s t h a t appear t o be the g o a l s o f t h e c o u r s e ? ( In a n s w e r i n g , t r y n o t t o be u n d u l y i n f l u -enced by y o u r l i k i n g o r d i s l i k e f o r t h e I n s t r u c t o r as a p e r s o n ; o r by y o u r f e e l i n g s about t h e c o u r s e c o n t e n t , i f t h a t wasn't something which t h e i n s t r u c t o r c o u l d choose f r e e l y . ) P l e a s e use t h e f o l l o w i n g s c a l e f o r y o u r r e s p o n s e s on t h e computer" c a r d : 1. O u t s t a n d i n g l y bad. 2. Below a v e r a g e . 3. A v e r a g e . 4. Above a v e r a g e . 5. O u t s t a n d i n g l y good,. Appendix H I n t e r v i e w o f s t u d e n t ; ; t e a c h e r s 143 I n t e r v i e w Q u e s t i o n s 1) D i d you have any s e r i o u s p r o b l e m s : a) In f i n d i n g a p l a c e t o l i v e ? b) With t r a n s p o r t a t i o n t o t h e school? c) With t h e p u p i l s you t a u g h t ? d) With t h e UBC p r o f e s s o r s ? e) With t h e t e a c h e r s f ) With y o u r f e l l o w s t u d e n t - t e a c h e r s ? g) A n y t h i n g e l s e ? 2) Was Program B more c o s t l y t o you than a 3 r d y e a r r e g u l a r program a t t h e U n i v e r s i t y would have been? 3) Would i t m a t t e r t o you where t h e s c h o o l was l o c a t e d i n terms o f c o s t ? T r a n s p o r t a t i o n ? Type o f s t u d e n t s ? 4) How d i d you come t o know about Program B? 5) Why d i d you a p p l y t o Program B? 6) Why do you want t o be a t e a c h e r ? 7) D i d Program B meet y o u r e x p e c t a t i o n s ? 8) Were t h e f o l l o w i n g p h y s i c a l f a c i l i t i e s a t t h e s c h o o l a d e q u a t e : a) Washrooms? b) Lunchrooms, s t a f f r o o m s ? c) A r e a s t o r e l a x , smoke, t a l k ? d) Work and p r e p a r a t i o n a r e a s ? 9) D i d you have s u f f i c i e n t a c c e s s t o books and s u p p l i e s f o r : a) Your u n i v e r s i t y c o u r s e work? b) Your t e a c h i n g ? 10) D i d you f e e l t h a t t h e work l o a d ( c o u r s e s , c l a s s r o o m t e a c h i n g , s u p e r v i s i o n , e t c . ) was e x c e s s i v e ? 11) Do you t h i n k t h a t h a v i n g methods c o u r s e s a t t h e s c h o o l i s an advantage o v e r h a v i n g them a t the u n i v e r s i t y ? Why? 144 12) O v e r a l l , were t h e method c o u r s e s u s e f u l ? -- How were t h e y u s e f u l o r why were t h e y n o t u s e f u l ? 13) What do t h e g o a l s o f Program B seem t o be? — Do you t h i n k i t has succ e e d e d i n r e a c h i n g t h e s e g o a l s ? — On what e v i d e n c e a r e you b a s i n g y o u r o p i n i o n ? 14) Would you recommend Program B as i t i s t o o t h e r p o t e n t i a l s t u d e n t - t e a c h e r s ? Why o r why n o t ? 15) Would you t a k e Program B a g a i n i f you were back i n Semptember o f 1974? Why o r why not? 16) Is Program B e s p e c i a l l y s u i t e d f o r c e r t a i n t y p e s o f s t u d e n t t e a c h e r s ? What t y p e ? v.. • 17) Do you t h i n k t h e y e a r f o l l o w i n g t h i s w i l l be: ~ v a l u a b l e ? -- i n t e r e s t i n g ? 18) D i d any i n d i v i d u a l s have a l a r g e n e g a t i v e o r p o s i t i v e e f f e c t on the d i r e c t i o n the program t o o k ? 19) Was t h i s s c h o o l i d e a l f o r Program B? Why o r why no t ? 20) C o u l d t h e r e have been more s t u d e n t - t e a c h e r i n p u t i n t o t h e program? I f s o , when o r where? I f n o t , a r e you s a t i s f i e d w i t h t h e amount o f i n p u t you had? Appendix I F i n a ' l Q u e s t i o n n a i r e f o r S t u d e n t Teache 146 Program B E v a l u a t t o n D i d Program B l i n k t h e methods c o u r s e s t o a c t u a l t e a c h i n g ? f r e q u e n t l y n e v e r D i d t h e program r e l a t e t h e s t u d y o f d e v e l o p m e n t a l p s y c h o l o g y t o immediate e x p e r i e n c e s ? f r e q u e n t l y n e v e r D i d t h e f o u n d a t i o n c o u r s e f o c u s on s c h o o l based i s s u e s and problems? f r e q u e n t l y n e v e r D i d t h e p s y c h o l o g y component i n c r e a s e y o u r s e n s i t i v i t y t o t h e needs and r e q u i r e m e n t s o f y o u r s t u d e n t s ? Yes No _ Un d e c i d e d . D i d t h e f o u n d a t i o n s component e n a b l e you t o make c l e a r , w e l l -founded d e c i s i o n s as t o what g o a l s a t e a c h e r ought t o pursue ? Yes No _^ U n d e c i d e d D i d t h e f o u n d a t i o n s component e n a b l e you t o promote t h e moral development o f y o u r s t u d e n t s ? Yes No U n d e c i d e d D i d t h e f o u n d a t i o n s component g i v e you a c l e a r u n d e r s t a n d i n g o f the n a t u r e o f t h e t a s k o f d e v e l o p i n g t h e a b i l i t y t o t h i n k ? Yes No U n d e c i d e d D i d you have s u f f i c i e n t o p p o r t u n i t y t o t r y o u t y o u r own i d e a s i n c l a s s ? Yes No U n d e c i d e d Do you f e e l t h e U.B.C. p r o f e s s o r s a r e e n t h u s i a s t i c about Program B's approach t o t e a c h e r - t r a i n i n g ? Yes No Un d e c i d e d Do you f e e l t h e t e a c h e r s a t ( s c h o o l name)are e n t h u s i a s t i c about Program B's approach t o t e a c h e r - t r a i n i n g ? n: .-:Yes'' . '•; - N o ' r . ^  , Undecided"-.: : ;: ; 147 11. Do you t h i n k program B has been f l e x i b l e ? Yes No _^ U n decided _^  12. Do you t h i n k Program B has been v a r i e d ? Yes No ^ U n d e c i d e d 13. Were you a b l e t o make as much use o f t h e U.B.C. l i b r a r y as you wanted t o ? Yes No U n d e c i d e d 14. C o n s i d e r i n g e v e r y t h i n g , what i s y o u r o v e r a l l e v a l u a t i o n o f Program B? EXCELLENT GOOD O.K. POOR A WASTE OF TIME 148 Appendix J Discipl ine techniques 149 DISCIPLINE TECHNIQUES 1. Simple c o n t r o l a. A l o o k o r a frown. b. Shake o f head o r o t h e r m i l d p h y s i c a l g e s t u r e . c . M i l d v e r b a l r e p r o o f . d. Movement t o t r o u b l e a r e a . e. S h o u t i n g a t s t u d e n t ( s ) . 2. I n d i v i d u a l c o n f e r e n c e w i t h p u p i l . 3. P a y i n g f o r , r e b u i l d i n g o r r e p l a c i n g damaged i t e m , 4. Lo s s o f some p r i v e l e g e . 5. D e t e n t i o n a f t e r s c h o o l . 6. D i s m i s s a l from c l a s s . a. Sent t o o f f i c e . b. Sent t o h a l l w a y . c. O t h e r a r e a . 7. I s o l a t i o n i n t h e c l a s s r o o m , i . e . s e n t t o a c o r n e r , e t c . 8. P u n i s h i n g a whole group o r c l a s s . 9. E x t r a work o r t a s k s . 10. E n f o r c e d a p o l o g i e s . 11. P h y s i c a l c o n t a c t ~ s h a k i n g , s t r i k i n g , f o r c e f u l l y d i r e c t i n g . 12. T h r e a t o f any o f the above. 13. O t h e r . Appendix K S i d e E f f e c t s o f Program B ( T e a c h e r s ' Q u e s t i o n n a i r e ) 151 Program B's Evaluation Dear Teacher, During the past few months of Program B an internal evaluation, as you know, has been carried out. Knowing how loaded down with paper-work and forms you are, I have not bothered you with any questionnaires or opionnaires — until now. This is the f i r s t and the last questionnaire that I wi l l give to you. Its purpose is to attempt to establish i f there have been any unanticipated benefits, and/or problems which have resulted from the operation of Program B in your school. These unanticipated aspects of the program (or side-effects) are extremely important. Will you please put your responses to this questionnaire in the envelope provided for them in the of f ice . Please do not put your name on your response. Thank you, Norman Gleadow 152 SIDE EFFECTS OF PROGRAM B T h i s form i s t o d e t e r m i n e i f t h e r e a r e any u n a n t i c i p a t e d e f f e c t s o f Program B. These e f f e c t s a r e t h e b e n e f i t s and/or the problems w h i c h were n o t f o r s e e n when the program was i n i t i a t e d . P l e a s e do n o t put y o u r name on t h i s form so as t o m a i n t a i n y o u r anonymity. 1. A r e y o u aware o f any UNANTICIPATED BENEFITS o r any UNANTICIPATED PROBLEMS o f Program B t o the STUDENT-TEACHERS? UNANTICIPATED BENEFITS UNANTICIPATED PROBLEMS 2. A r e you aware o f any UNANTICIPATED BENEFITS o r UNANTICIPATED PROBLEMS o f Program B t o t h e SCHOOL'S STUDENTS? UNANTICIPATED BENEFITS UNANTICIPATED PROBLEMS I f you have any a d d i t i o n a l comments p l e a s e p u t them on the back. 153 3. Are y o u aware o f any UNANTICIPATED BENEFITS o r any UNANTICIPATED PROBLEMS o f Program B t o the SCHOOL'S TEACHERS? UNANTICIPATED BENEFITS UNANTICIPATED PROBLEMS 4. A r e you aware o f any UNANTICIPATED BENEFITS o r any UNANTICIPATED PROBLEMS o f Program B t o t h e U.B.C. PROFESSORS? UNANTICIPATED BENEFITS UNANTICIPATED PROBLEMS 5. A r e you aware o f a n y t h i n g you would c o n s i d e r i n j u s t , i m moral, o r c a u s i n g s e r i o u s u n h a p p i n e s s as a r e s u l t o f Program B? Appendix L I n t e r v i e w w i t h Program B's D i r e c t o r 155 Q u e s t i o n s Asked. 1) How d i d the i d e a o f Program B o r i g i n a t e ? When? What was t h e impetus f o r Program B. 2) What d i d Program B hope t o do t h a t wasn't b e i n g done by the t h i r d y e a r r e g u l a r c o u r s e a t U . B . C ? 3) Why were the t h i r d y e a r t r a n s f e r s t u d e n t s chosen? 4) How were p e r s p e c t i v e s t u d e n t - t e a c h e r s n o t i f i e d o f Program B? 5) How were the s t u d e n t - t e a c h e r s f i n a l l y s e l e c t e d f o r Program B? How many were t u r n e d down? Why? 6) How d i d t h e UBC s t a f f f o r Program B g e t chosen? 7) How was the h o s t s c h o o l chosen? 8) What c o n n e c t i o n has the s c h o o l b o a r d w i t h the program? 9) Has t h e r e been any u n s o l i c i t e d , w r i t t e n f e e d - b a c k on the s t u d e n t -teachers;': p e r f o r m a n c e ? 10) Do you t h i n k t h e average t h i r d y e a r s t u d e n t c o u l d o p e r a t e w e l l i n Program B? 11) What do you t h i n k c o n s t i t u t e s t h e i d e a l s t u d e n t - t e a c h e r f o r Program B? 12) Do you know o f any r e s e a r c h r e s u l t s on programs s i m i l a r t o Program B? 13) Do you t h i n k i t would be f e a s i b l e t o have a l l t e a c h e r - t r a i n i n g done v i a Program B? Why o r why not ? 14) What i s t h e most e x p e n s i v e p a r t o f Program B as compared t o t h e r e g u l a r t h i r d y e a r Program? 15) What a r e some o t h e r expenses which a r e a s s o c i a t e d w i t h Program B? 16) What do you t h i n k a r e t h e g r e a t e s t s t r e n g t h s and weaknesses o f Program B? 156 Appendix M Q u e s t i o n n a i r e f o r School P u p i l s . 157 Your Grade QUESTIONNAIRE FOR STUDENTS We would l i k e you t o answer some q u e s t i o n s about the s t u d e n t - t e a c h e r s who have t a u g h t you o v e r t h e p a s t s c h o o l y e a r . P l e a s e do NOT put y o u r name on t h i s p a p e r . I f you w i s h t o make any w r i t t e n comments, p l e a s e do so i n the spaces under each q u e s t i o n . 1. D i d the s t u d e n t - t e a c h e r s t e a c h too f a s t ? YES NO 2. D i d the s t u d e n t - t e a c h e r s t e a c h t o o slow? YES NO 3. D i d t h e y g i v e you f a i r t e s t s and exams? YES NO 4. D i d you u n d e r s t a n d what they were t r y i n g t o t e a c h y o u i n c l a s s ? YES NO 5. D i d any o f them e v e r h e l p you w i t h s c h o o l work o u t s i d e o f c l a s s ? YES NO 6. C o u l d t h e s t u d e n t - t e a c h e r s c o n t r o l t h e k i d s who d i d n ' t behave? YES NO 7. Were t h e y e v e r u n f a i r t o you? YES NO I f y o u answered y e s t o t h i s q u e s t i o n , would you p l e a s e t e l l us how the s t u d e n t - t e a c h e r s were u n f a i r ? 8. D i d you f i n d them i n t e r e s t i n g t o l i s t e n t o ? YES NO 9. D i d you f i n d i t easy t o l e a r n t h i n g s f r o m them? YES NO 10. Were the s t u d e n t - t e a c h e r s ' l e s s o n s i n t e r e s t i n g ? YES NO 11. D i d t h e y g i v e you good answers t o y o u r q u e s t i o n s ? YES NO 158 12. D i d you f i n d i t easy t o g e t t h e s t u d e n t - t e a c h e r s ' a t t e n t i o n i n c l a s s ? YES NO 13. D i d t h e y l e a r n y o u r name? YES NO 14. D i d you f i n d them e a s y t o t a l k t o ? YES NO 15. D i d you e v e r t a l k t o any o f t h e s t u d e n t - t e a c h e r s i n the c l a s s r o o m ? YES NO 16. D i d you e v e r t a l k t o any o f the s t u d e n t - t e a c h e r s o u t s i d e o f the c l a s s r o o m ? YES NO 17. D i d you f i n d s w i t c h i n g b a c k - a n d - f o r t h f r o m y o u r r e g u l a r t e a c h e r to t h e s t u d e n t - t e a c h e r s c o n f u s i n g ? YES NO 18. P l a c e a check-mark b e s i d e t h o s e s u b j e c t s which a s t u d e n t - t e a c h e r t a u g h t you ART ENGLISH OR LANGUAGE ARTS ( r e a d i n g , s p e l l i n g , MATH o r ARITHMETIC w r i t i n g ) SCIENCE P.E. SOCIAL STUDIES MUSIC OTHER (WHAT IS?IT? ) 19. Which o f t h e above s u b j e c t s do you t h i n k t h e s t u d e n t - t e a c h e r s t a u g h t t h e b e s t ? ' . 159 Which o f the above s u b j e c t s do y o u t h i n k t he s t u d e n t - t e a c h e r s t a u g h t t h e w o r s t ? ' . D i d t h e s t u d e n t - t e a c h e r s do a good j o b o f t e a c h i n g you? YES NO NOT SURE Appendix N I n t e r v i e w w i t h UBC A d m i n i s t r a t o r s . 161 Q u e s t i o n s Asked. 1) What rieed do you see Program B f u l f i l l i n g ? 2) Have you had any s p e c i f i c f e e d b a c k on Program B's p a r t i c i p a n t s ? I f s o , what? 3) What do you see as t h e s t r o n g and weak p o i n t s o f Program B? 4) What i s y o u r own judgment as t o t h e o v e r a l l p e r f o r m a n c e o f Program B? 5) Would you d e s c r i b e t h e U.B.C. p r o f e s s o r s i n v o l v e d i n Program B as an/average group, a poor group o r an above a v e r a g e group? Appendix 0 S t u d e n t T e a c h e r s ' Q u e s t i o n n a i r e ( S i d e E f f e c t s ) 163 SIDE EFFECTS - STUDENTS 1. Please i n d i c a t e your o v e r a l l evaluation of the r e g i s t r a t i o n procedures. E x c e l l e n t , couldn't be smoother / / Good, hardly any d i f f i c u l t i e s / / Acceptable, there were a few problems / / Not good, there were many problems / / T e r r i b l e , complete confusion / / Are there any good and/or bad points about the r e g i s t r a t i o n procedure you would l i k e to comment on? If so, please use the space below. 2. (a) Please i n d i c a t e your method of tr a n s p o r t a t i o n to School from your place of .residence. Walking (or running) J_ B i c y c l e l_ Motorcycle l_ Your own car ]_ In a car pool l_ i 164 Bus / / Other (Please specify) (b) About how long does i t take you to get to from your place of residence? (c) School Less than 15 minutes Between 15 -and 30 minutes Between 30 and 45 minutes Between 45 and 60 minutes More than an hour / / Are you experiencing any problems with your present trans p o r t a t i o n arrangements from your place of residence to -. - N. School? Yes / / No / / If "yes" please i n d i c a t e what those problems are. Were you given an adequate o r i e n t a t i o n to the school? No / / Yes / / Not Sure / / (a) (b) If "No" please i n d i c a t e how i t was inadequate. If "Not Sure" please i n d i c a t e , i n general terms, why you f e e l uneasy about the o r i e n t a t i o n . Do you have adequate work space i n Yes / 7 No / 7 Comments School? Are the washroom f a c i l i t i e s adequate? Yes / / No / / Comments Are arrangements f o r eating lunch, drinking coffee, smoking, etc. acceptable to you? Yes / 7 No / 7 Comments 166 7. Are there any problems or i r r i t a t i o n s which are " i n the bud" and should be d e a l t with now before they got worst? Yes / / No / / I f "Yes" please o u t l i n e those problems. 8. Is there anything e l s e you would l i k e to comment on? Please do i n the space provided below. / Appendix P U.B.C. S t a f f Q u e s t i o n n a i r e on C o u r s e s 168 I n s t r u c t o r s ' Questionnaire A Course: Instr u c t o r Course w i l l be given i n F a l l Term / / Spring Term / / Hours of i n s t r u c t i o n per week hrs. • 1. Recommended textbook (s) i f any 2. Is (are) the textbook(s) a v a i l a b l e at the u n i v e r s i t y Bookstore? Yes / / No / / Comments: 3. Which part(s) of the textbook(s) w i l l you be emphasizing? A l l / 7; or ? 169 How w i l l you be measuring the student's progress i n your course (please check a l l that apply)? Comments: F i n a l exam (written) / / F i n a l exam (oral) l_ / Mid term exam / / Other exams (or t e s t s ) Observation / / (Please give approximate frequency ) (Please i n d i c a t e the type of observational s e t t i n g you w i l l be arranging). Short w r i t t e n assignments (one page or less ) l_ / Longer written assignments (short essays) j_ / Major pr o j e c t s / / Other (please s p e c i f y ) Any other comments? Appendix Q S t u d e n t - t e a c h e r s ' Q u e s t i o n n a i r e on E v a l u a t i o n o f U.B.C. I n s t r u c t o r s ' T e a c h i n g Performance. 171 EVALUATION OF UBC STAFF TEACHING PERFORMANCE In t h i s p a r t i c u l a r questionnaire, you are being asked to rate the , UBC s t a f f , as a group, on a number of teaching c r i t e r i a . If there are any i n d i v i d u a l s t a f f members who you f e e l do not f i t into a r a t i n g f o r the re s t of the group then please indi c a t e , under the heading of exceptions, who those i n d i v i d u a l s are and why they do not f i t into your o v e r a l l r a t i n g . 1. How frequently do UBC s t a f f members come l a t e to class without an adequate explanation? / / / / / / / / / / very frequently sometimes never exceptions? No / / o r Yes _/ / : please i n d i c a t e who the exceptions are and why they are exceptions. 172 2. How w e l l i s the s t a f f s t i c k i n g to what you were led to expect i n .their courses? l—l l_l l_l LJ LJ very well sometimes very deviate poorly exceptions? No / / o r Yes ]_ / : please ind i c a t e who the exceptions are and why they are exceptions. 3. How much opportunity do you get to ask questions i n class? 1—1 l_l l_l LJ LJ ample occasional i n s u f f i c i e n t opportunity opportunity opportunity exceptions? No / / or Yes / / : please i n d i c a t e who the exceptions are and why they are exceptions. 1 7 3 4. How well organized are the UBC s t a f f member's presentations i n c l a s s . / / / / / / / / / / very well organized O.K. very poorly organized exceptions? No / / or Yes / I : please indicate who the exceptions are, and why they are exceptions. 5. Do the UBC s t a f f members know t h e i r subject material? Yes / / No / / Don't Know / / exceptions? No / / or Yes / /: please indicate who the exceptions are, and why they are exceptions. 174 6. How useful are the assignments you have had so far? L—l L_l l_l LJ LJ very useful useless exceptions? No / / or Yes ]_ / : please indicate what the exceptions are, and why they are exceptions. 7. How worthwhile have you found the textbooks you bought and are presently using? (Please take into consideration the cost of the books, the assignments (readings, problems, etc.) you have had using the textbook, the reading l e v e l , the content, etc.) l—l LJ LJ LJ LJ very O.K. worthless worthwhile exceptions? No / / or Yes ]_ / : please indicate what the exceptions are, and why they are exceptions. 175 8. Are the UBC s t a f f member r e a d i l y a v a i l a b l e when you need them? Yes. / / No / / Don't Know / / exceptions? No / / or Yes / / : please i n d i c a t e who the exceptions are, and why they are exceptions. What i s your o v e r a l l evaluation of the job the UBC s t a f f members are doing i n getting across to you the knowledge, understanding, s k i l l s and at t i t u d e s that appear to be the goals of the courses? (In answering, t r y not to be unduly influenced by your l i k i n g or d i s l i k i n g f o r the in s t r u c t o r s as people; or by your f e e l i n g s about the courses' content i f that i s n ' t something the i n s t r u c t o r s could choose f r e e l y ) Outstandingly good / / Above average / / Average / / Below Average / / Outstandingly bad / / exceptions? No / / or Yes / / : please indicate who the exceptions are, and why they are exceptions. 176 10. .If you have any other comments, please do so: Appendix R T e a c h e r s ' Q u e s t i o n n a i r e 2 178 Dear T e a c h e r , T h i s q u e s t i o n n a i r e a s k s a number of q u e s t i o n s about how has been o p e r a t i n g ^ y o u r s c h o o l . The q u e s t i o n -n a i r e l o o k s more f o r m i d a b l e t h a n i t a c t u a l l y i s . The q u e s t i o n s r e q u i r e j u s t a s i m p l e checkmark; however, i f you w i s h to add to y o u r a n s w e r s , p l e a s e do so i n t h e "comments" box under each q u e s t i o n . When you c o m p l e t e t h i s q u e s t i o n n a i r e , p l e a s e put i t i n t h e m a i l s l o t i n y o u r o f f i c e . I t would be a p p r e -c i a t e d i f you c o u l d c o m p l e t e them w i t h i n a week. Thank you f o r y o u r t i m e . Y o u r s t r u l y , Norman Gleadow E v a l u a t o r egr 179 TEACHER'S QUESTIONNAIRE PLEASE INDICATE YOUR ANSWER BY PUTTING A CHECKMARK IN THE APPROPRIATE BOX. PLEASE DO NOT PUT YOUR NAME ON THIS FORM. IF YOU WISH TO MAKE ANY COMMENTS, PLEASE DO SO IN THE BOX PROVIDED. 1. The student-teachers have been taking a number of University courses at your school. Please place checkmarks beside those u n i v e r s i t y courses f o r which you have received course OUTLINES; Music Math Psychology ' Science S o c i a l Comments: 2. Do you f i n d the material you receive about the courses being taught the student teachers of any value? Studies Other _____ (Please specify) P.E. Language Arts Not applicable / /, Yes /__/,. No / / Haven't had time to read i t / / Comments: 4 180 3. Are you being kept aware of the student-teachers' course ASSIGNMENTS? Yes, f o r a l l courses / / Yes, f o r most courses / / Yes, f o r about h a l f the courses / / Yes, but for less than h a l f the courses / / No, have not received any information of t h i s type / / Comments: 4. In your opinion, have the U.B.C. professors adequately consulted with you with respect to: (a) Placement of student-teachers with sponsor teachers? Yes / / No / / Not applicable / / Comments: (b) Teaching time f o r the student-teacher working with you? Yes / / No l_ / Not applicable / / Comments: 5 181 (c) Appropriate teaching strategies (methods, classroom management, material to cover, etc.) f o r your student teacher? Yes / / No / / Not applicable / / Comments: (d) The evaluation of your student-teachers' teaching effectiveness. Yes / / No / / Not applicable / / Comments: 5. Are you clear about your r o l e i n the . program? (Please check one of the following:) No, I'm t o t a l l y confused about what I'm expected to do. / / I think I know what I'm expected to do but there are s t i l l some areas of confusion. / / Yes, I have a very clear idea of what I am expected to do. / / Comments: 6 182 6. Generally, how would you rate the communication between the teachers . . and the U.B.C. s t a f f . /__/ /_/ LJ LJ I I Excellent Good " F a i r Poor T e r r i b l e Comments: 7. Are the student-teachers courteous / / / / / / / / / / Always Sometimes Never Comments: 8. Are the student-teachers appropriately dressed? / / / / / / LJ LJ Always Sometimes Never Comments: 7 1 8 3 Do.you f e e l the student-teachers are seperatlng themselves from the operation of the school ( i . e . staying i n t h e i r room too much, not mixing s u f f i c i e n t l y with the teachers, etc.) Yes / / No / / Undecided / / 10. Do you approve of the student-teachers' behavior i n a l l t h e i r contacts with the pupils i n your school? approve / / disapprove / / undecided I f If you disapprove of t h e i r behavior, would you please'' indic a t e your reasons. 11. Are y o u . s a t i s f i e d with the student-teachers' preparation for teaching? Yes / / No / / undecided / / I f you are not s a t i s f i e d with t h e i r preparation, please indic a t e how they could prepare better. 8 184 12. Generally, how would you rate the o v e r a l l rapport between the student-teachers and the teachers? / / / / / / / / /__/ excellent good f a i r poor non-existent 13. Do you f e e l that your pupils are getting behind i n t h e i r work because of the program? Yes / / No / / Undecided / / If you answered "yes" would you please ind i c a t e the se v e r i t y of the problem. 14. Have you found Program B too t i r i n g ? Yes / / No / / Not Sure / / If yes, i n what way has i t been too t i r i n g ? 9 185 15. Do you have adequate time to do your own lesson planning, marking, etc. Yes / / No / / Not Sure / / Comments: 16. Did you f i n d the Communications Workshop useful? Yes / / No / / Didn't attend / / Undecided / / 17. What i s your o v e r a l l evaluation of the program as i t has operated i n your school f o r the past month? Excellent / / Good / / O.K. / / Poor / / " T e r r i b l e / / 18. Are there any minor or major problems you f e e l should be taken care of now before they get any worse? Yes / / No / / 10 186 18. continued If you answered "Yes" please i n d i c a t e what these problems are. 19. Are there any s p e c i f i c p o s i t i v e aspects of Program B you have noticed i n the past month? Yes / / No / / If you answered "yes" w i l l you please ind i c a t e what those p o s i t i v e aspects are. 20. If you have any other comments, please do so. 187 Appendix S I n t e r v i e w w i t h D i r e c t o r o f E l e m e n t a r y E d u c a t i o n a t S c h o o l Board O f f i c e s . 188 Q u e s t i o n s Asked. (I) Have you r e c e i v e d any i n f o r m a t i o n on b e n e f i c i a l e f f e c t s o f the program from: P a r e n t s ? S t u d e n t s ? T e a c h e r s ? P r i n c i p a l s ? A d m i n i s t r a t o r s ? 2) Have you r e c e i v e d any i n f o r m a t i o n on harmful e f f e c t s o f t h e program from: P a r e n t s ? S t u d e n t s ? T e a c h e r s ? P r i n c i p a l s ? A d m i n i s t r a t o r s ? 3) Does t h e program c o s t the s c h o o l D i s t r i c t money? time? manpower? Is i t worth t h e c o s t ( i f any)? 4) Has i t i n c r e a s e d y o u r w o r k l o a d u n a c c e p t a b l y ? 5) Do you s e e any need f o r t h i s a p p r oach t o t e a c h e r - t r a i n i n g ? Why ,q\ o r why n o t ? v -6) What a s p e c t o f the program a r e y o u most i n t e r e s t e d i n ? 7) Do you have any p l a n s f o r e v a l u a t i n g the program? I f s o , what? I f n o t , why n o t ? 8) What i s y o u r o v e r a l l i m p r e s s i o n o f Program B as i t has o p e r a t e d to d a t e ? 9) Any o t h e r comments? 189 Appendix I f E v a l u a t i o n o f the S t u d e n t T e a c h e r s E f f e c t i v e n e s s i n Program B (a P r o p o s a l s u b m i t t e d t o and a c c e p t e d by the t e a c h e r s h o s t i n g Program B) 190 Proposal for the Evaluation of the Student Teachers  Effectiveness i n the Program Research has not yet established usable connections between observed teacher behaviors i n the classroom and educational gains by the p u p i l s . Class-room observational instruments are l e g i t i m a t e tools f or obtaining an accurate account of what i s taking place i n the classroom but they do not neces s a r i l y i d e n t i f y v a r i a b l e s which can be used to judge e f f e c t i v e teachers. At best descriptions of a teacher's classroom technique are only secondary indications of teaching ef f e c t i v e n e s s . For example, one teacher may run a very " t i g h t " classroom with well developed systems of management and regulation, while another teacher may have a very "loose" classroom. The two teachers would have d i f f e r e n t p r o f i l e s on any observational instrument which described the teaching act, yet both might be described as being "good" teachers. If we are to evaluate teaching effectiveness , then we must go beyond descriptions of the teaching act and search for primary i n d i c a t o r s of "good" teaching. This i s not meant to suggest that there are no secondary- or d e s c r i p t -ive indicators of good teaching. Some weight i n the evaluation can be given to a small number of secondary ind i c a t o r s chosen on a common sensical,or moral basis, and which can be expected to at l e a s t weakly c o r r e l a t e with good teach-ing. -Therefore, a major feature of the evaluation plan i s extreme parsimony i n the data-gathering,, combined wittu extreme care i n i d e n t i f y i n g the use to which the data w i l l be put. Parsimony i s advocated for the sake of v a l i d i t y . Only data which can be demonstrated to be connected with teaching merit should be c o l l e c t e d and then only when there i s a legitimate use for i t . -2-191 The major uses of the evaluation of student-teaching include: 1. Evaluation for self-improvement. 2. Evaluation to a s s i s t the sponsor teacher and U.B.C. adviser with the process of improving the student teacher's teaching performance. 3. Evaluation to advise other possible consumers, such as pupils d e s i r i n g teacher assistance i n other school a c t i v i t i e s . 4. Evaluation f o r personnel decisions by school and U.B.C. s t a f f s . 5. Evaluation by external audiences interested i n , e.g., success of the Project. Description of the Evaluation Instruments (These instruments are attached to these proposal) I. Primary Indicators (1) Overall Evaluation of Teaching. This i s a simple r a t i n g on a f i v e point s c a l e , ranging from out-standingly good to outstandingly bad of the student teacher's teaching a b i l i t y . This s i n g l e r a t i n g has two main advantages. F i r s t l y , i t cannot be e a s i l y misinterpreted, and secondly, the chances that a r a t i n g i s not rel a t e d to an i n d i c a t i o n of teaching a b i l i t y are extremely remote. I d e a l l y t h i s r a t i n g s h a l l be given by the pupils being taught by the student teacher. However, i n an elementary school, this presents s p e c i a l d i f f i c u l t i e s , p a r t i c u l a r l y i n the lower grades where the pupils may not f u l l y understand what i s expected of them. In t h i s s i t u a t i o n , we may have to r e l y on ratings given the student teacher by the sponsor teach-ers and the U.B.C. advisors. Since these people are not primary consumers of the teaching of the student teacher, then the o v e r a l l evaluative r a t i n g must be based on as many classroom v i s i t s as possible - c e r t a i n l y not on the basis of a s i n g l e v i s i t . 192 (2) Pu p i l Gains A teacher's main function i s to meet the educational needs of his/her p u p i l s . In other words, as a r e s u l t of the teaching a c t , the pupils should have gained some combination of knowledge, s k i l l , dispos-i t i o n or a t t i t u d e . I t i s inconceivable to c a l l someone a good teacher who's pupils do not "learn". Therefore, i t i s proposed that the student teachers use some method of determining whether the pupils have learned anything while exposed to the student teacher's teaching. To give t h i s some experimental v a l i d i t y a simple pre-test, post-test procedure i s suggested. That i s , i n consultation with the sponsor teacher, a test w i l l be made up on a p a r t i c u l a r u n i t the student teacher w i l l be teaching. (Dr. Ed. 311 course w i l l provide the techniques needed f o r test construction.) This t e s t w i l l be given to the pupils before the unit i s taught and a f t e r the un i t i s taught. Using simple s t a t i s t i c a l techniques i t can be determined whether or not there Was pu p i l gain as measured by the t e s t . Secondary Indicators 1) Minimal teaching ob l i g a t i o n s Minimal teaching ob l i g a t i o n s are duties or minimum requirements f o r .teachers. F u l f i l l i n g these duties does not normally count f o r the teacher, but not f u l f i l l i n g them does account against the teacher. T y p i c a l l y these minimal obligations are being on time for the class he/she i s to teach (except when i l l e t c . ) , t e s t i n g and marking pupils f a i r l y , knowing the subject matter of what he/she i s teaching, etc. The best judges of whether the student teacher i s f u l f i l l i n g these minimal o b l i g a t i o n s are the primary consumers of h i s teaching, the p u p i l s . However, as f o r the o v e r a l l evaluation, the r e s p o n s i b i l i t y f o r determining i f the minimal ob l i g a t i o n s -4-193 are being f u l f i l l e d may have to f a l l on the sponsor teachers and U.B.C. advisors. 2) Out-of-Class Contributions In the program, the student teachers are part of the school. The school i s not simply a geographical l o c a t i o n , d i f f e r e n t from that of the U n i v e r s i t y , where classes are held. Therefore, the student-teachers should do more than teach and attend classes, and they should teach other than by being i n a classroom. Their public and professional s e r v i c e to the school, s t a f f and pupils through e x t r a - c u r r i c u l a r a c t i v i t i e s should be acknowledged. The school administration, the sponsor teacher and the U.B.C. advisors would probably be the best judges of the degree of out-of - c l a s s contributions. 3) Description of Teaching "Description of Teaching" i s perhaps a misnomer f o r t h i s s e c t i o n . The teaching act i s not to be described*, instead, there i s a check made f o r undesirable extremes i n the student teachers' teaching s t y l e . These extremes are common s e n s i c a l . For example, i t i s u n l i k e l y that a good teacher would constantly use o r a l or written language which was too d i f f i c u l t f o r the pupils to understand; or i t i s u n l i k e l y that a good teacher would be cons i s t e n t l y disorganized. As before, i t would be d e s i r -able to have the pupils respond to th i s c h e c k l i s t ; however, i f that i s not possible, then the sponsor teachers, and U.B.C. advisors should complete the c h e c k l i s t on the basis of many classroom v i s i t s . 4) Performance i n University Courses. The student teachers are also students of the University. Therefore, they have c e r t a i n standards to achieve i n t h e i r u n i v e r s i t y courses. Whether or not a student teacher has met the standards f o r a p a r t i c u l a r course w i l l -5-194 be determined by the professor teaching that course. These marks are not indica t o r s of the student teachers teaching e f f e c t i v e n e s s , but they are necessary f o r e s t a b l i s h i n g an o v e r a l l evaluation of the student teachers' performance i n the program. The Evaluation Procedures Each sponsor teacher and each U.B.C. advisor w i l l receive a booklet for each student teacher under his/her supervision. The booklet w i l l be organ-ized as follows: COVER - Student teachers' name, grade he/she i s teaching, Sponsor Teachers' or U.B.C. Advisor's name. Page 6 - Overall evaluation of teaching r a t i n g scale. Page 5 - Minimal teaching o b l i g a t i o n s c h e c k l i s t Page 4 - Out-of-Class Contributions Page 1,2 and 3 - Description of teaching c h e c k l i s t . The measurement of p u p i l gains w i l l be handled by the student teachers themselves. The grades obtained on the U.B.C. courses w i l l be c o l l e c t e d by the evaluator when those marks are a v a i l a b l e . The various ratings w i l l be made on the basis of as many classroom observations as possible by the Sponsor Teacher and the U.B.C. advisor. A l l of t h i s data w i l l be entered on the student teacher's teaching evaluation form. This teaching evaluation form (T-form) w i l l be an averaging and condensing of the above information from the sponsor teacher, U.B.'C. advisor and ( i f possible) the pu p i l s . -6- 195 Inputs to T-Form Method of Obtaining Inputs 1. P u p i l estimate of teaching merit i f possible. Otherwise, sponsor teacher and U.B.C. advisor over-a l l evaluation of teaching. 2. Estimates of classroom j u s t i c e , e.g., appropriate quizzes and materials, f a i r hearing f o r complaints, etc.* 3. Non-classroom con t r i b u t i o n to teaching. 4. Learning gains due to student teacher. 5. Course-work i n u n i v e r s i t y courses re l a t e d to teaching. 1. Single question c a l l i n g f o r an evaluation of the teaching s k i l l on a simple A-F scale ( s p e c i a l consideration f o r K-3). 2. Program of adequate random v i s i t s to y i e l d r e l i a b l e judgments, plus d e t a i l e d study of materials, quizzes, etc. V i s i t s by school s t a f f and U.B.C. s t a f f . 3. School s t a f f , peers, and possibly p u p i l s . 4. Controlled study using pre- and post-tests etc., done by student teacher with advice of evaluation s p e c i a l i s t . 5. U.B.C. s t a f f . * I f i t i s r e a l l y necessary to do so, school and U.B.C. s t a f f could estimate merit of teaching s t y l e of student teacher. Each of these inputs contributes something i n the d i r e c t i o n of good teaching. The optimal model for weighting these inputs i s equal weighting. This means that i n a summative evaluation of the student teacher each input has equal weight. Any evaluation of a student teacher by the pupils ( i f performed) should be made a v a i l a b l e to the student teacher. Anonymity of the pupils should be assured. Hopefully, i n the years to come, a better way to evaluate teaching w i l l come about. But, meanwhile, l e t ' s not use something worse. 196 EVALUATION OF STUDENT TEACHERS STUDENT TEACHER NAME DATE (plense put your numa i n the appropriate space) SPONSOR TBACKER U.B.C. ADVISOR (please i n d i c a t e whon the student teacher wao ob-eerved, and what subject and grade he/she was ten chine) &ate of ob-servation Subject being taught grac e 197 Plaa'-e r».te each of th« following ctancrii't j tii<n' of the stu'1.-;it t u i o l.trii teaching on the sc a l e beuide each question. SA o atrongly Bji r e a I e x c e p t i o n a l l y good perform-ance ) A = ftj=;re3 ( scceptable performance) U » undecided D = disagree (performance not at Rn acceptable l e v e l ) SD = s t r o n g l y disagree ( e x c e p t i o n a l l y poor per-formance ) 1. The student taacher presentd the content of the le s s o n at un appropriate pace, comments: SA A U D SD 2. The student teacher's lessons are pres-ented i n a well organised manner, comments: — — SA S3 3. The student teacher's o r a l and written E n g l i s h are appropriate f o r the grpde l e v -e l he/she i s teaching. SA A U SD 198 4. The student teecher has an e f f e c t i v e and I fair system of classroom c o n t r o l ( d i s c i p l i n e ) . SA A U D SD comments: 5. The student teacher teaches the content of the lesson at an approx^riate l e v e l of d i f f i c u l t y f o r the grade l e v e l . SA A U D SD comments: 6. The student teacher uses a u d i o - v i s -u a l m a t e r i a l when appropriate. SA A U D SD comments: 7. The student teachor accepts and acts on c o n s t r u c t i v e c r i t i c i s m . SA A U D SD comments: 8. The p u p i l s react p o s i t i v e l y to the- | student teacher. SA A U D SD comments: 199 A ?. The p u p i l s appear to be l e a r n i n g the course uvtterial being taught then by the otudar,t teecher, SA A TJ D SD W i l l you pleaso i;ive the evidence you used to determine the r a t i n g given i i n no. 9. 10. Please comment on any other t r a i t s which you think are c o n t r i b u t i n g to, o r decreaaine the student teacher's teaching e f f e c t i v e n e s s . 200 OUT-GF-CLASr: CCI'TRI3UTIC?I 1. Kns the student teacher contributed to the organ-i z a t i o n and/or operation of school sports? never seldonu , sometimes frequently very frequently 2. Han the student teacher contributed to or p a r t i c -ipated on school committees? | [ | | [ [ — [ j never seldom ^  sometimes frequently very frequently 3. Han the student teacher spent o u t - o f - c l a s s time ( f o r example, lunch time or a f t e r school) h e l p i n g p u p i l s i n t h o i r school work? tzz • EZI • t z i never seldom>-: sometimes frequently very frequently 4. Kan the student teacher contributed any time or ef-f o r t to organised s o c i a l events i n the school, such as p l a y s , s k i t s , assemblies, pageants, etc.? [ZH • tZ3 • LZI ' C J never seldom*-. sometimes frequently very don't frequently know 5. Has the student teacher volunteered f o r or p a r t i c i p a t e d i n any e x t r a - c u r r i c u l s r a c t i v i t i e s not covered by the above four questions? I f so ple-.se b r i e f l y note the nature of the a c t i v i t y , and the amount of p a r t i c i p a t i o n . don't know don' t know don't know 201 MINIMAL STUDKHT TEACHEH OBLIGATIONS X. Th.i student teacher spends an adequate amount of time c o r i u l t i n g rfith the sponsor teacher. • • . • yes r.o undecided 2. Thj student teacher f a i l e d to appear^on occasions without udequ-.'.te explanations; or was freauehtly very l a t e . • ' • • y e 3 no undecided 3. The student teacher gives the pupile enough op p c r t u n i t i e o to a3k qujHtiona. • • O yea no undecided A. Ihe student teacher gives s a t i s f a c t o r y responses to the p u p i l s ' questions. • • • yes no undecided 5. The student teacher seems s e r i o u s l y l a c k i n g i n knowledge of the subject n a t t e r he/she i s teaching. • • • y e B no undecided 6. Ihe student teacher i s a v a i l a b l e to t h e p u p i l s , when needed, for out-of-class questions or extra help. yes no undecided 7. The t e s t s and marking of the student toacber eeem f a i r . • • • yes no undecided 6. Furthur comments on the above r a t i n g s , or furthur i n f o r -mation? . 202 OVERALL EVALUATION" CF TEACHING W i l l y o u please g i v e y o u r o v e r a l l r a t i n g of the job the B t u d e n t t e a c h e r i e doing i n c o m m u n i c a t i n g the knowledge, u n d e r s t a n d i n g , s k i l l s , and a t t i t u d e s t h a t seem to be a p p r o p r i a t e f o r these, g r a d e l e v e l s . I n a n s -w e r i n g t r y not to be i n f l u e n c e d by y o u r l i k i n g or d i s -l i k i n g f o r the B t i i d e n t t e a c h a r as a p e r s o n . Outstandingly Above Average Below Outstandingly Good Average Average Bad CD Comments to c l a r i f y the above rating?. Appendix U Summary o f E v a l u a t i o n S t e p s i n RANKING. 204 S t e p s i n RANKING. (See Sec. 3.70 o f . t h i s t h e s i s ) F u r t h e r e v a l u a t i o n would be p o i n t l e s s A l l t h e programs s h o u l d be r e v i s e d . NO Are t he e v a l u a t a a t t e m p t i n g t o a c h i e v e o t h e r g o a l s ? yes NO A r e some o f the e v a l u a t a a c h i e v -i n g t h e g o a l s t o an a c c e p t a b l e d e g r e e ? NO YES Determine t he e v a l u a t a and the p o i n t s o f view t o be ta k e n i n the e v a l u a t i o n . z E s t a b l i s h t h e g o a l s o f t h e e v a l u a t a . A r e t h e eva ua t a a t t e m p t i n g t o a c h i e v e the same g o a l s ; A r e t h e goa NO YES s worthwhi YES NO Determine t he degree t o which each e v a l u a t u m i s a c h i e v i n g t h e g o a l s . A r e a l l t h e e v a l u a t a a c h i e v i n g ] the g o a l s t o an a c c e p t a b l e d e g r e e ? E l i m i n a t e t h o s e p r o -grams n o t a c h i e v i n g the g o a l s f r o m f u r t h e r c o n s i d e r a t i o n . YES Determine t he r e l a t i v e c o s t s o f t h e e v a l u a t a a c h i e v i n g t h e g o a l s . E l i m i n a t e t h o s e e v a l u a t a which a r e a t t e m p t i n g t o a c h i e v e d i f -f e r e n t g o a l s . F u r t h e r e v a l u a -t i o n would be p o i n t l e s s . -*-(go t o next page) 205 A r e some o f t h e e v a l u a t a b e i n g o p e r a t e d a t a c c e p t a b l e l e v e l s ? NO NO YES. (from p r e v i o u s page) 1 A r e t he e v a l u a t a b e i n g o p e r a t e d a t a c c e p t a b l e c o s t l e v e l s . Rank o r d e r t h e e v a l -u a t a on t h e b a s i s o f goal a c h i e v e m e n t and p o i n t o u t the s h o r t -comings o f the e v a l -u a t a i n terms o f t h e i r e x c e s s i v e costs.' YES Average the r a t i n g s f o r go a l a c h i e v e m e n t and c o s t f o r each e v a l u a t u m o p e r a t i n g a t a c c e p t -a b l e c o s t . A r e t h e r e any t i e s f o r t h e h i g h e s t a v e r a g e r a t i n g ? YES .1 NO • Evaluatum w i t h h i g h e s t a v e r a g e r a t i n g i s the " B e s t Buy".! P i c k the e v a l uatum w i t h t h e h i g h e s t g o a l a c h i e v e m e n t r a t i n g as t h e "B e s t Buv" 1 Is t h e r e s t i l l a t i e ? YES The e v a l u a t u m w i t h t he b e s t p l a n s f o r ex-ten d e d s u p p o r t i s t h e b e s t buy, I f t h e r e i s s t i l l a t i e the n t h e t i e d e v a l u a t a a r e o f equal w o r t h . Note: 1 " B e s t Buy" means t h a t t h e ev a l u a t u m , X, o f f e r s t h e b e s t p e r f o r m a n c e f o r l e s s c o s t t h a n .the e v a l u a t a t o which i t i s b e i n g compared. 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            data-media="{[{embed.selectedMedia}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
https://iiif.library.ubc.ca/presentation/dsp.831.1-0093721/manifest

Comment

Related Items