UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

The development and application of a methodology for program evaluation Gleadow, Norman E. 1976

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-UBC_1976_A8 G59_4.pdf [ 7.89MB ]
Metadata
JSON: 831-1.0093721.json
JSON-LD: 831-1.0093721-ld.json
RDF/XML (Pretty): 831-1.0093721-rdf.xml
RDF/JSON: 831-1.0093721-rdf.json
Turtle: 831-1.0093721-turtle.txt
N-Triples: 831-1.0093721-rdf-ntriples.txt
Original Record: 831-1.0093721-source.json
Full Text
831-1.0093721-fulltext.txt
Citation
831-1.0093721.ris

Full Text

THE DEVELOPMENT AND APPLICATION OF A METHODOLOGY FOR PROGRAM EVALUATION by  Norman E. Gleadow, B.Sc,  University of B.C. (1967)  A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS i n the Department of Science Education  We accept t h i s thesis as conforming to the required standard  THE UNIVERSITY OF BRITISH COLUMBIA May, 1976.  In p r e s e n t i n g t h i s  thesis  an advanced degree at the L i b r a r y s h a l l I  f u r t h e r agree  in p a r t i a l  fulfilment of  the requirements f o r  the U n i v e r s i t y of B r i t i s h Columbia,  make it  freely available  that permission  for  I agree  r e f e r e n c e and  f o r e x t e n s i v e copying o f  this  that  study. thesis  f o r s c h o l a r l y purposes may be granted by the Head of my Department or by h i s of  this  written  representatives. thesis  It  is understood that  f o r f i n a n c i a l gain s h a l l  permission.  Department of The U n i v e r s i t y o f B r i t i s h  Columbia  2075 W e s b r o o k P l a c e V a n c o u v e r , Canada V6T 1W5  Date  {If^l  / f / ^  copying or p u b l i c a t i o n  not be allowed without my  Abstract The purpose o f t h i s s t u d y was t o d e v e l o p a p r a c t i c a l m e t h o d o l o g y o f program e v a l u a t i o n from c o n s i d e r a t i o n s o f what the e v a l u a t i v e a c t r e q u i r e d on l o g i c a l g r o u n d s . P.W.  The m e t h o d o l o g y d e v e l o p e d was based on  T a y l o r ' s (1961) a n a l y t i c a l t r e a t m e n t o f e v a l u a t i o n and M.  Scriven's  e x t e n s i v e w r i t i n g s on the s u b j e c t . The s t u d y i n v o l v e d two broad p h a s e s .  The f i r s t phase was a  d i s c u s s i o n o f T a y l o r ' s and S c r i v e n ' s thoughts on e v a l u a t i o n showing where they a r e c o m p a t i b l e and o v e r l a p . ideas developed  The second phase showed how  i n the f i r s t phase were o p e r a t i o n a l i z e d and adapted  the to  the e v a l u a t i o n o f a t e a c h e r t r a i n i n g program a t the U n i v e r s i t y o f B r i t i s h Columbia. A g e n e r a l c o n c l u s i o n o f t h i s s t u d y was t h a t T a y l o r ' s a n a l y s i s o f the method and p r o d u c t o f e v a l u a t i o n p r o v i d e d a s u i t a b l e framework f o r the e v a l u a t i o n o f an e d u c a t i o n a l program.  More s p e c i f i c a l l y when  T a y l o r ' s a n a l y s i s was combined w i t h S c r i v e n ' s e x t e n s i v e work, a f e a s i b l e model o f e v a l u a t i o n r e s u l t e d w h i c h r e a d i l y p r o d u c e d a d e f e n s i b l e , o v e r a l l e s t i m a t i o n o f w o r t h f o r an e d u c a t i o n a l  product.  Although the methodology of e v a l u a t i o n developed  i n t h i s study  was  o n l y a p p l i e d t o a t e a c h e r t r a i n i n g program, i t c o u l d e a s i l y be e x t e n d e d to the e v a l u a t i o n o f o t h e r e d u c a t i o n a l programs o r p r o d u c t s .  TABLE OF CONTENTS Page LIST OF TABLES  .  .  .  .  .  .  .  .  v i  v i i  LIST OF FIGURES ACKNOWLEDGEMENTS  .  .  v i i i  • Chapter I  THE INTRODUCTION . . . 1.00 Purpose o f the' S t u d y 1.10 S t a t e m e n t o f t h e G e n e r a l Problem. ...... I. 1.20 D e f i n i t i o n o f Terms •• « ' 1.21 E d u c a t i o n a l E v a l u a t i o n 1.22 Summative E v a l u a t i o n . 1.23 F o r m a t i v e E v a l u a t i o n 1.30 S t a t e m e n t o f t h e S p e c i f i c P r o b l e m 1.40 Overview o f t h e S t u d y . 1.50 D e l i m i t a t i o n o f t h e S t u d y II  :  . • ." 3 3 , 3 3 4 5  THE THEORETICAL FRAMEWORK FOR AN EVALUATION MODEL . . 2.00 I n t r o d u c t i o n 2.10 P r e s e n t D i f f i c u l t i e s i n E v a l u a t i o n 2.11 • T y l e r ' s Approach t o E v a l u a t i o n 2.12 The A c c r e d i t a t i o n Model 2.13 S t u f f l e b e a m ' s C o n t e x t , I n p u t , P r o c e s s and P r o d u c t (CIPP) Model 2.14 S t a k e ' s Model . 2.15 Summary 2.20 E v a l u a t i o n : The T h e o r e t i c a l Framework . • • • • 2.30 The P r e c o n d i t i o n s . .' . . . . . . •.' . . . . 2.31 The Evaluatum . 2.32 P o i n t o f View . . ' .... 2.33 G r a d i n g o r Ranking 2.40 L e v e l 1 o f S - G r a d i n g : A d o p t i o n o f S t a n d a r d s . . 2.41 I n t r i n s i c V a l u e 2.42 E x t r i n s i c V a l u e 2.50 L e v e l I I o f S - G r a d i n g : O p e r a t i o n a l C l a r i f i c a t i o n o f the Standards in Grading 2.60 L e v e l I I I : S p e c i f i c a t i o n o f t h e C l a s s o f Comparison i n G r a d i n g  i i..  1 1 1 3  7 7 7 10 11 12 15 16 17 19 19 19 21 28 28 28 31 32  Table of Contents (Cont.) Page 2.70  L e v e l IV: D e t e r m i n a t i o n o f the Good and Bad C h a r a c t e r i s t i c s o f t h e Evaluation i n S - G r a d i n g 2.71 G a t h e r i n g Data on the C h a r a c t e r i s t i c s . . . . 2.72 D e t e r m i n i n g i f the C h a r a c t e r i s t i c s a r e Good o r Bad 2.80 L e v e l V: The O v e r a l l Judgement o f Worth . . . . I I I APPLICATION OF THE MODEL TO THE DESIGN OF AN ACTUAL EVALUATION STUDY:. . 3.00 I n t r o d u c t i o n 3.10 E s t a b l i s h i n g t h e P r e c o n d i t i o n s 3.11 The Evaluatum 3.111 L e t t e r t o P o t e n t i a l S t u d e n t s f o r Program B 3.112 Program D e s c r i p t i o n 3.113 D e s c r i p t i o n o f t h e O f f e r e d C o u r s e s i n Program B 3.12 The P o i n t o f View 3.13 G r a d i n g 3.20 A d o p t i o n o f S t a n d a r d s 3.21 S t a n d a r d 1: The Worth o f t h e Program G o a l s . 3.22 S t a n d a r d 2: Goal Achievement 3.30 O p e r a t i o n a l C l a r i f i c a t i o n o f S t a n d a r d s . . . . 3.31 S t a n d a r d 1: The Worth o f Program B's G o a l s . 3.32 S t a n d a r d 2: The Degree o f Goal A c h i e v e m e n t . 3.33 The C h e c k l i s t f o r D e t e r m i n i n g Good •;' '• Achievement 3.331 C h e c k p o i n t 1: Market 3.332 C h e c k p o i n t 2: T r u e F i e l d T r i a l s 3.333 C h e c k p o i n t 3: T r u e Consumer 3.334 C h e c k p o i n t 4: Long Term 3.335 C h e c k p o i n t 5: S i d e E f f e c t s . . . . . . . . . 3.336 C h e c k p o i n t 6: P r o c e s s 3.337 C h e c k p o i n t 7: C a u s a t i o n 3.338 C h e c k p o i n t 8: S t a t i s t i c a l S i g n i f i c a n c e . . 3.40 S p e c i f i c a t i o n o f the C l a s s o f Comparison in Grading 3.50 D e t e r m i n i n g t h e Good and Bad C h a r a c t e r i s t i c s of the Evaluatum i n S-Grading 3.51 G a t h e r i n g Data on the Goal Achievement Characteristics 3.52 D e t e r m i n i n g i f the C h a r a c t e r i s t i c s o f t h e Program a r e Good o r Bad 3.521 R a t i n g the Worth o f the G o a l s  iii.  33 33 34 36 38, 38 38 38 39 39 40 41 43 44 45 48 49 49 52 54 54 55 57 59 59 63 64 68 70 71 71 75 75  Table of Contents  (Cont.) Page  3.522 Goal Achievement and i t s R a t i n g 3.60 O v e r a l l D e t e r m i n a t i o n o f Worth 3.70 E x t e n s i o n o f the Model 3.71 I n t r o d u c t i o n 3.72 The S t a n d a r d s t o be Used i n Ranking 3.721 C o s t s . . . 3.722 Extended S u p p o r t 3.73 The Precedence o f t h e S t a n d a r d s i n Ranking 3.74 Ranking the E v a l u a t a IV  V  .  DATA COLLECTION TECHNIQUES IN THE APPLICATION OF THE EVALUATION MODEL 4.00 I n t r o d u c t i o n . 4.10 I n i t i a l S t a g e o f Data C o l l e c t i o n 4.20 Second S t a g e o f Data C o l l e c t i o n . 4.21 E v a l u a t i o n o f U.B.C I n s t r u c t o r s ' , T e a c h i n g Performance . . . . . 4.22 Q u e s t i o n n a i r e t o S t u d e n t T e a c h e r s o r t h e i r A l t e r n a t e Placement 4.23 I n t e r v i e w o f S t u d e n t T e a c h e r s . . . . . . . . 4.24 F i n a l Q u e s t i o n n a i r e f o r the S t u d e n t T e a c h e r s 4.25 O b s e r v a t i o n o f S t u d e n t T e a c h i n g . 4.26 S i d e E f f e c t s of-Program B . . . 4.27 I n t e r v i e w w i t h Program B's D i r e c t o r 4.28 Q u e s t i o n n a i r e f o r School P u p i l s 4.29 I n t e r v i e w w i t h U.B.C. A d m i n i s t r a t o r s . . . . 4.210 O t h e r I n f o r m a t i o n S o u r c e s 4.30 P r a c t i c a l P r o c e d u r e s f o r the O r g a n i z a t i o n o f Data 4.40 C o n f i d e n t i a l i t y o f Data SUMMARY AND CONCLUSIONS 5.00 Summary 5.10 C o n c l u s i o n s  . . .  REFERENCES , APPENDICES A. B.  78 81 85 85 86 87 88 89 92  .  93 93 93 96 97 98 98 98 99 100 101 101 101: 102 105 108 109 109 113 llfto.-*  ,  .  Q u e s t i o n n a i r e f o r t h e Program D e v e l o p e r s , Subject: Evaluation P r i o r i t i e s Interview o f School A d m i n i s t r a t i o n  iv.  n 9  119  12  5  Table of Contents (Cont.) Page C.  Likert Attitude Scale (to measure general attitudes toward the teacher training process of the student teachers)  127  D. Reaction Form E.  130  Evaluation of U.B.C. Instructors' Teaching Performance: Form A  132  F. Questionnaire to Student Teachers on Their Alternate Placement . . . . . . . !35 G. Evaluation of U.B.C. Instructors' Teaching Performance: Form B  139  H. Interview of Student Teachers  ^2  I. Final Questionnaire for Student Teachers J. Discipline Techniques  . . . . . . ^45  K. 'Side Effects of Program B (Teacher's Questionnaire) . *50 L.  Interview with Program B's Director  "^54  M. Questionnaire for School Pupils N. Interview with U.B.C. Administrators  ^-->6 ........  0. Student Teacher's Questionnaire (Side Effects) •P. 'U.B.C. Staff Questionnaire on Courses  ...  l  o  2  -.167/  Q. Student Teachers' Questionnaire on the Evaluation _ of U.B.C. Instructors Teaching Performance . . *-<° R. Teachers' Questionnaire S. Interview with Director of Elementary Education at School Board Offices T. -Proposal for the Evaluation of Student Teachers Teaching Effectiveness U.  Summary of Evaluation Steps in Ranking  v.  1 8 7  1 8  °  2 0  ^  LIST OF TABLES  Table  Page  1  Taxonomy o f E v a l u a t i o n Types  25  2  C a l e n d a r o f Data C o l l e c t i n g E v e n t s  3  Data S o u r c e s f o r t h e Goal Achievement C h e c k p o i n t s  4  Comparison o f E v a l u a t i o n P r o c e d u r e s  vi.  . . .  72-74 103 113-114  LIST OF FIGURES Figure  Page  1  Developing E v a l u a t i o n Designs  14^  2  An O u t l i n e o f t h e L o g i c a l P r o c e s s  27  3  The S t a n d a r d s and t h e i r R a t i n g s  69  4  D e t e r m i n i n g a R a t i n g f o r Goal Achievement  79  5  P r o d u c t E v a l u a t i o n P r o f i l e f o r Program B  80  6  Comparing Goal Worth and Goal Achievement R a t i n g . . . .  82  7  Expansion of Figure 6  84  8  Combining the Goal Achievement and C o s t R a t i n g s  9  Format f o r O r g a n i z a t i o n o f Data  105  10  Example o f Format f o r O r g a n i z a t i o n o f Data  107  11  Steps i n S-Grading i n Formative E v a l u a t i o n .  112  vii.  ....  91  ACKNOWLEDGEMENTS I w i s h t o e x p r e s s my s i n c e r e t h a n k s t o P r o f e s s o r W a l t e r B. B o l d t , who i n t r o d u c e d me t o t h e f i e l d o f e d u c a t i o n a l e v a l u a t i o n , and whose s c h o l a r s h i p and i n t e l l e c t u a l i n t e g r i t y were a c o n s t a n t i n s p i r a t i o n .  viii.  DEDICATION t h e memory o f my f a t h e r Frank R. Gleadow  CHAPTER I The  Introduction  1.00 Purpose o f t h e Study Many p r e s e n t e v a l u a t i o n models a r e u n s a t i s f a c t o r y on t h e o r e t i c a l grounds because they l a c k c l a r i t y as t o t h e n a t u r e o f t h e evaluative act.  The p u r p o s e o f t h e p r e s e n t s t u d y i s t o shed some  l i g h t on t h i s problem by d e v e l o p i n g a p r a c t i c a l model o f e v a l u a t i o n based on an a n a l y s i s o f t h e o r e t i c a l i d e a s a b o u t e v a l u a t i o n p r o cedures and t o d e m o n s t r a t e how t h e r e s u l t i n g model c o u l d be used i n a p a r t i c u l a r case.  The problem i s i m p o r t a n t b e c a u s e much o f what  is currently called evaluation f a i l s to f u l f i l l the evaluation f u n c t i o n s t o be s e r v e d , on t h e o r e t i c a l g r o u n d s .  1.10 S t a t e m e n t o f the_.Gener.a1 P r o b l em y.- -,: ' ;  "One c a n be a g a i n s t e v a l u a t i o n o n l y i f one c a n show t h a t i t is.improper, t o seek an answer t o q u e s t i o n s a b o u t t h e m e r i t o f e d u c a t i o n a l i n s t r u m e n t s , w h i c h would i n v o l v e showing t h a t t h e r e a r e no l e g i t i m a t e a c t i v i t i e s ( r o l e s ) i n w h i c h t h e s e q u e s t i o n s c a n be r a i s e d . " ( S c r i v e n , 1967, p. 41) The g e n e r a l p r o b l e m t o be i n v e s t i g a t e d i n t h e s t u d y i s t o determine the nature o f t h e procedures  o f e v a l u a t i o n on t h e o r e t i c a l  grounds and t o a t t e m p t t o s p e c i f y t h e e v a l u a t i v e f u n c t i o n s t o be s e r v e d i n p r a c t i c a l terms t h r o u g h i l l u s t r a t i o n and a p p l i c a t i o n t o the e v a l u a t i o n o f a t e a c h e r - t r a i n i n g program. E v a l u a t i o n as p r a c t i s e d f o r t h e p a s t few y e a r s i s n o t  1  ,2 succeeding i n i t s perceived f u n c t i o n .  Evans (1974, pp. 7, 11, 12)  i n d i c a t e s t h a t though e v a l u a t i o n i s ". . . now a l l t h e r a g e i t f a c e s many new  problems.  The problems r e v o l v e around  ..." difficul-  t i e s o f c o l l e c t i n g d a t a , u n r e a l i s t i c e x p e c t a t i o n s on t h e p a r t o f p o l i c y m a k e r s , u n w i l l i n g n e s s , o f p e o p l e t o be e v a l u a t e d and so o n . Guba (1969) i n d i c t s e v a l u a t i o n f o r i t s f a i l u r e t o f i n d any d i f f e r ences among programs, even when, t h e r e a r e o b v i o u s d i f f e r e n c e s , and s u g g e s t s t h a t t h e whole t e c h n i q u e o f e v a l u a t i o n s h o u l d be q u e s t i o n e d and examined.  Bloom (1969, p. 46) who equates e v a l u a t i o n w i t h  t e s t i n g and m e a s u r i n g , w r i t e s o f t h e p o t e n t i a l l y d e s t r u c t i v e a s p e c t s o f e v a l u a t i o n when t e a c h e r s and s t u d e n t s b e l i e v e t h e p r o c e d u r e s t o be u n f a i r ( p . 4 5 ) . In a r e c e n t paper, G l a s s (1975) c r i t i c i s e s e v a l u a t i o n as t a k i n g w e a k l y d e f i n e d s t a n c e s when t h r e a t e n e d and t h e n r e a c t i n g t o t h i s i n h e r e n t weakness w i t h e i t h e r a h u m a n i s t i c approach o f g e n t l e s e l f - e v a l u a t i o n ; o r by r e g a r d i n g " r e s i s t a n c e s " t o e v a l u a t i o n as mere t e c h n i c a l problems overcome by t e c h n i c a l means; o r by r e t r e a t i n g i n t o a v a s t g r a y a r e a between. These a r e some o f t h e c r i t i c i s m s which have been l e v e l e d a t evaluation.  They r a i s e two fundamental  q u e s t i o n s : what i s e v a l u a -  t i o n ; and how c a n e v a l u a t i o n be c a r r i e d o u t ?  Many e v a l u a t i o n  models d e a l w i t h t h e f i r s t q u e s t i o n by p o s i n g a n o t h e r w h i c h i s g e n e r a l l y o f t h e form; "what i s i t t h a t we a r e d o i n g t o d a y t h a t we c a l l e v a l u a t i o n ? "  From t h e answer t o t h i s q u e s t i o n i s d i s t i l l e d  a model w h i c h i s then used f o r e v a l u a t i o n .  In o t h e r words t h e  model i s a c l a r i f i c a t i o n o f p r e s e n t p r a c t i s e , and n o t n e c e s s a r i l y evaluation.  The second p a r t o f t h e q u e s t i o n i s u s u a l l y answered  3  i n a s i m i l a r way and e m b r o i l s r e s e a r c h e r s i n the e v a l u a t i o n v s . r e s e a r c h i m b r o g l i a ( P r o v u s , 1970: Hemphill 1969, p. 190; Welch 1969, p. 440).  T h e r e f o r e to work from p r e s e n t p r a c t i s e i s not a  s a t i s f a c t o r y method o f d e f i n i n g e v a l u a t i o n and w i l l not be used i n t h i s t h e s i s e x c e p t where p r e s e n t p r a c t i s e i s t h e same as t h e t h e o r e t i c a l p o s i t i o n d e v e l o p e d i n Chapter  2.  1.20 D e f i n i t i o n o f Terms  1.21  Educational E v a l u a t i o n : Educational E v a l u a t i o n i s the  p r o c e d u r e o f j u s t i f y i n g a v a l u e c l a i m about t h e m e r i t o r worth o f an e d u c a t i o n a l p r o d u c t ( c o u r s e , c u r r i c u l u m , e t c . )  The r e s u l t  o f t h i s procedure i s a c o n c l u s i o n o f the worth o f the product.  1.22 Summative E v a l u a t i o n : T h i s i s an o v e r a l l e v a l u a t i o n o f a completed  e d u c a t i o n a l p r o d u c t and s e r v e s t h e needs o f the c l i e n t ,  and more i m p o r t a n t l y , the u l t i m a t e c o n s u m e r ( s ) o f the p r o d u c t .  1.23 F o r m a t i v e E v a l u a t i o n : T h i s t y p e o f e v a l u a t i o n t a k e s p l a c e a t i n t e r m e d i a t e p o i n t s i n the d e v e l o p m e n t o f a p r o d u c t  and  s e r v e s t h e needs o f t h e d e v e l o p e r s f o r m o d i f y i n g and i m p r o v i n g product.  1.30 S t a t e m e n t o f t h e S p e c i f i c Problem The s p e c i f i c problem to be d e a l t w i t h i n t h i s t h e s i s i s t o  the  4  d e v e l o p a p r a c t i c a l m e t h o d o l o g y o f e v a l u a t i o n based on the l o g i c a l c o n s i d e r a t i o n s p r e s e n t e d by T a y l o r (1961) and S c r i v e n (1967) and the e v a l u a t i v e f u n c t i o n s t o be s e r v e d ^ i n t h e form o f t h e c h e c k l i s t f o r the e v a l u a t i o n o f P r o d u c t s , p r o d u c e r s and p r o p o s a l s proposed  by  S c r i v e n (1974b); and t o a p p l y t h e r e s u l t s t o t h e e v a l u a t i o n o f a t e a c h e r - t r a i n i n g program a t the U n i v e r s i t y o f  B.C.  The s p e c i f i c p r o b l e m can be f u r t h e r broken down t o a number o f sub-problems f o r t h e p u r p o s e s o f the  study:  a) An a n a l y s i s o f the e v a l u a t i o n p r o c e s s based on T a y l o r ' s p h i l o s o p h i c a l p o s i t i o n (Chapter I I ) . b) E s t a b l i s h i n g the e v a l u a t i o n f u n c t i o n s t o be s e r v e d and the v a l i d i t y of Scriven's Checklist. c) A d a p t i o n o f S c r i v e n ' s C h e c k l i s t t o t h e e v a l u a t i o n o f a t e a c h e r - t r a i n i n g program ( C h a p t e r I I I ) . d) Development o f i n s t r u m e n t s and t e c h n i q u e s f o r c o l l e c t i n g t h e n e c e s s a r y e v a l u a t i o n data» ( C h a p t e r s I I I and I V ) . e) Making a f i n a l e v a l u a t i v e c l a i m a b o u t t h e t e a c h e r - t r a i n i n g s program (Chapter I I I ) . f ) C r i t i q u e of the e f f e c t i v e n e s s of the c h e c k l i s t f o r t h i s p u r p o s e (Chapter V ) .  1.40 O v e r v i e w o f t h e Study The t h e s i s comprises  the d e v e l o p m e n t and i l l u s t r a t i o n o f a  methodology o f e v a l u a t i o n based on P.W.  T a y l o r ' s (1961) a n a l y t i c a l  t r e a t m e n t o f e v a l u a t i o n and M. S c r i v e n ' s (1967, 1974b) e x t e n s i v e  5  o p e r a t i o n a l i z a t i o n o f a c o m p a t i b l e model o f e v a l u a t i o n . t h r e e broad phases i n t h i s p r o c e s s . d i s c u s s i o n o f P.W.  The f i r s t phase i n v o l v e s a  T a y l o r ' s and S c r i v e n ' s thoughts on e v a l u a t i o n  showing where t h e y a r e c o m p a t i b l e and o v e r l a p . Chapter  There are  T h i s i s done i n  II.  The second phase shows the method by which t h e i d e a s i n Chapter  developed  II were o p e r a t i o n a l i z e d and adapted t o a p a r t i c u l a r  e v a l u a t i o n p r o j e c t ; t h a t i s the e v a l u a t i o n o f a program i n t e a c h e r training.  T h i s was done i n Chapter I I I . Chapter  III also i l l u s t r a t e s  how t h e d e v e l o p e d model can be e x t e n d e d . The t h i r d phase i s a d i s c u s s i o n o f the d a t a c o l l e c t i o n t e c h niques used i n a p p l y i n g the e v a l u a t i o n model d e v e l o p e d .  Chapter  IV  i l l u s t r a t e s t h o s e t e c h n i q u e s . : C h a p t e r V summarizes the p r e v i o u s f o u r c h a p t e r s and makes f i n a l recommendations on t h e o v e r a l l usef u l n e s s o f t h e scheme.  1.50 D e l i m i t a t i o n o f the Study The E v a l u a t i o n t e c h n i q u e i s d e v e l o p e d around t h e e v a l u a t i o n o f an e d u c a t i o n a l p r o d u c t .  T h e r e f o r e the e x t e n s i o n o f t h e scheme  t o o t h e r a r e a s , such as t e a c h i n g e f f e c t i v e n e s s , has not been u n d e r taken i n the p r e s e n t  study.  The e v a l u a t i o n a p p l i e d was a f o r m a t i v e e v a l u a t i o n .  The o v e r a l l  statement o f worth a r r i v e d a t i n Chapter I I I , f o r t h e p a r t i c u l a r program b e i n g e v a l u a t e d , r e p r e s e n t s an i n t e r i m c o n c l u s i o n and not a f i n a l c o n c l u s i o n on the w o r t h o f the program.  Data were c o l l e c t e d  6  representing good performance and bad performance.  To present a  balanced picture both types of data should be revealed; however, one of the functions of a formative evaluation i s to provide information, in a summarized and judged (as to v a l i d i t y and worth) format so that program changes can be made to better f u l f i l l program goals.  the  To reveal only the good performance data and not the  bad would be misleading; but to also present the bad would not contribute to the evaluation model being developed and could v i o l a t e c o n f i d e n t i a l i t y . Therefore the actual data have not been included in this study. F i n a l l y , the instruments developed for the evaluation of the program being evaluated are not generalizable. discussed in more d e t a i l  Since this  is  in Chapter IV, i t w i l l only be b r i e f l y  noted here that the questionnaires are s p e c i f i c for the particular program.  CHAPTER II The T h e o r e t i c a l Framework f o r an E v a l u a t i o n Model 2.00  Introduction C h a p t e r Two e x p l i c a t e s t h e t h e o r e t i c a l framework f o r e v a l u a t i o n .  The f i r s t p a r t o f t h e c h a p t e r b r i e f l y examines a number o f commonly used a p p r o a c h e s t o e d u c a t i o n a l e v a l u a t i o n , and p o i n t s out the s h o r t comings o f t h o s e a p p r o a c h e s .  The second p a r t o f the Chapter  T a y l o r ' s (1961) l o g i c a l e x a m i n a t i o n  of e v a l u a t i v e procedures  Scriven's very p r a c t i c a l d e s c r i p t i o n s of e v a l u a t i o n .  with  The r e s u l t i n g  amalgam provides, t h e s t r u c t u r e o f e v a l u a t i o n u n d e r l y i n g t h i s  2.10 P r e s e n t P i f f i c u l t i e s J n  combines  study.  Evaluation  Many o f t h e . p r e s e n t l y used models o f e v a l u a t i o n have been d e v e l o p e d from an e x p e r i e n c e base.  T a y l o r and M a g u i r e (1966, p.  12)  wrote: "In many ways, t h e e f f i c i e n c y o f e v a l u a t i o n has been hampered by a l a c k o f c l e a r g u i d i n g p r i n c i p l e s i n d i c a t i n g what e v a l u a t i o n i s p o t e n t i a l l y a b l e t o a c c o m p l i s h . • What p r i n c i p l e s t h e r e a r e , e x i s t m a i n l y as e x p e r i e n t i a l l y induced p r o c e d u r e s and t e c h n i q u e s h e l d by t h e few who work i n t h e a r e a . " The danger i n d e v e l o p i n g e v a l u a t i o n models from i s t h a t the meaning o f " e v a l u a t i o n " i s never examined.  experience The r e s u l t  i s a broad r a n g e o f models w h i c h p u r p o r t t o be d o i n g t h e same t h i n g , i . e . e v a l u a t i n g , but when examined a r e r e a l l y d o i n g something v e r y d i f f e r e n t such as d e s c r i b i n g , o r d a t a g a t h e r i n g , o r m e a s u r i n g o r  7  8 simply rubber stamping.  T h e s e may a l l be p r o c e d u r e s  in evaluation,  but n o t one o f them c a n s i n g l y be l a b e l l e d as e v a l u a t i o n . The weakness o f e x p e r i e n t i a l l y based e v a l u a t i o n m o d e l s , and the lack o f understanding  o f t h e meaning o f e v a l u a t i o n has r e s u l t e d  i n a g r e a t d e a l o f c r i t i c i s m o f " e v a l u a t i o n " as p r e s e n t l y a p p l i e d . On t h i s p o i n t Guba (1969, p. 29) noted t h a t : "The t r a d i t i o n a l methods o f e v a l u a t i o n have f a i l e d e d u c a t o r s i n t h e i r attempts t o a s s e s s t h e impact o f i n n o v a t i o n s i n o p e r a t i n g s y s t e m s . Indeed, t h e e v i d e n c e p r o d u c e d . . . has c o n t r a d i c t e d t h e e x p e r i e n t i a l e v i d e n c e o f t h e p r a c t i t i o n e r . I n n o v a t i o n s have p e r s i s t e d i n e d u c a t i o n n o t because o f t h e s u p p o r t i n g e v i d e n c e o f e v a l u a t i o n but d e s p i t e i t . " T h i s i n d i c t m e n t i s due t o t h e narrowness w i t h which some practitioners define evaluation.  Guba i s s u g g e s t i n g t h a t many  e v a l u a t i o n s a r e n o t complete o r c o m p r e h e n s i v e , and they t e n d t o be l i m i t e d t o v a r i a b l e s w h i c h c a n be " s c i e n t i f i c a l l y " measured and s t a t i s t i c a l l y treated. A n o t h e r c r i t i c i s m i s ". . . one o f t h e p r i m a r y l a t e n t f u n c t i o n s o f many e v a l u a t i o n s t u d i e s [ i s ] a l e g i t i m i z a t i o n p r o c e s s f o r p r e d e t e r mined p o l i c i e s . " (House, 1973, p. 8 0 ) . House's c r i t i c i s m p o i n t s o u t one o f t h e most s e r i o u s  omissions  i n many " e v a l u a t i o n s " ; t h a t i s , f a i l i n g t o d e t e r m i n e whether t h e g o a l s o f t h e program a r e w o r t h w h i l e .  The a c c e p t a n c e o f program  g o a l s a s being prima f a c i e w o r t h w h i l e  g o a l s i s a major weakness i n  many e v a l u a t i o n s .  Instead o f l e g i t i m i z i n g predetermined p o l i c i e s ,  t h e e v a l u a t o r s h o u l d be d e t e r m i n i n g are legitimate.  i f t h e predetermined  policies  9  The u n c l e a r n a t u r e o f e v a l u a t i o n p r o c e d u r e s , and t h e l i n k i n g o f e v a l u a t i o n w i t h c r i t i c i s m has r e s u l t e d i n d e f e n s i v e p o s t u r i n g by t h o s e b e i n g e v a l u a t e d and by the e v a l u a t o r s .  Stake (1967, p.  524)  wrote t h a t t h e e d u c a t o r ' s d i s d a i n o f e v a l u a t i o n was due t o h i s s e n s i t i v i t y to c r i t i c i s m .  The e d u c a t o r uses smoke s c r e e n s such as  " n a t i o n a l norm c o m p a r i s o n s " and "academic freedom" t o a v o i d e x p o s u r e through e v a l u a t i o n .  P r o v u s (1970, p. 54) s u g g e s t e d  that the  e d u c a t i o n a l a d m i n i s t r a t i o n has c o n t r i b u t e d t o the f a i l u r e o f e d u c a t i o n a l e v a l u a t i o n by, among o t h e r t h i n g s , not e x p l i c a t i n g h i s v a l u e a s s u m p t i o n s , or by not s e t t i n g p e r f o r m a n c e s t a n d a r d s ;  but  goes on to say t h a t p a r t o f the r e a s o n i s the u n c l e a r n a t u r e o f a n a l y s i s , r e s e a r c h and e v a l u a t i o n i n t h e problem s o l v i n g p r o c e s s . G l a s s (1975, p. 10) s t r o n g l y c r i t i c i z e s t h e e v a l u a t o r f o r r e t r e a t i n g t o t h r e e "straw men"  t e n e t s when under a t t a c k .  The  t e n e t s a r e i d e n t i f i e d as b e i n g t h e s a n c t i t y o f s c i e n c e , the p u b l i c ' s r i g h t t o know and a l l feedback  is beneficial.  In s p i t e o f h i s  c r i t i c i s m , G l a s s says ( i b i d ) ". . . e v a l u a t i o n can f i n d a s t r o n g e r f a i t h , even i f the s c o p e o f i t s a p p l i c a t i o n i s somewhat The c o n c e p t o f e v a l u a t i o n has been used to d i f f e r e n t procedures.  reduced."  descrifee-many , ;  Most o f t h e p r e s e n t models o f e v a l u a t i o n a r e  d e r i v e d from c o n c e p t s o f e d u c a t i o n a l measurement.  They a v o i d  p u t t i n g v a l u e judgements on d a t a o r p r o d u c t s by u s i n g r i g o r o u s research design which supposedly  i s " v a l u e f r e e " , o r by  the e v i d e n c e and g i v i n g i t t o someone e l s e t o j u d g e .  bundling-up  Brief descrip-  t i o n s o f t h e most common models o f e v a l u a t i o n f o l l o w i n s u b s e q u e n t  10 sections of this  chapter.  2.11 T y l e r ' s A p p r o a c h t o E v a l u a t i o n T y l e r f i r s t proposed  hisrrmodel f o r t h e e v a l u a t i o n o f  l e a r n i n g e x p e r i e n c e s d u r i n g t h e 1930's.  I t e v o l v e d from h i s p a r t i c i -  p a t i o n i n t h e , " E i g h t - y e a r Study" ( A i k e n , 1942) and was p u b l i s h e d i n 1949.  originally  H i s model i n v o l v e s t h e f o l l o w i n g s i x s t e p s ( T y l e r ,  1942): 1)  E s t a b l i s h t h e broad g o a l s or o b j e c t i v e s o f t h e program.  2)  C l a s s i f y the o b j e c t i v e s .  3)  D e f i n e the o b j e c t i v e s i n b e h a v i o r a l t e r m s .  4)  Suggest s i t u a t i o n s i n which achievement of the o b j e c t i v e s  can be shown; 5)  D e v e l o p or s e l e c t measurement t e c h n i q u e s ,  and,  6)  G a t h e r s t u d e n t p e r f o r m a n c e d a t a and compare i t w i t h t h e  behaviorally-stated objectives. A p p r o a c h e s s i m i l a r to T y l e r ' s a r e found i n M e t f e s s e l ' s Michael's  and  (1967), "A P a r a d i g m I n v o l v i n g M u l t i p l e C r i t e r i o n Measures  f o r E v a l u a t i n g t h e E f f e c t i v e n e s s o f School Programs", and W. James Popham's (1972) An E v a l u a t i o n G u i d e b o o k . A l l o f t h e s e methods would have the e v a l u a t o r l o o k i n g e x c l u s i v e l y a t t h e o b j e c t i v e s o f the program.  almost  The bases o f the  o b j e c t i v e s a r e not t h e f o c a l p o i n t i n the e v a l u a t i o n .  Popham  (1972),  i n h i s g u i d e b o o k , does m e n t i o n a needs a s s e s s m e n t , but i t r e a l l y i s m a i n l y a r e s o u r c e s i n v e n t o r y and c l a r i f i c a t i o n o f p r e d e t e r m i n e d  needs,  11 goals or objectives. Welch used t h e T y l e r i a n a p p r o a c h when he e v a l u a t e d a c o u r s e callede-'Thysical Science f o r the non-science Student."  In h i s  e v a l u a t i o n he was a t t e m p t i n g t o d e t e r m i n e i f the o b j e c t i v e s o f t h e c o u r s e had been a c h i e v e d . (Welch, p. 140, 1972).  He c a l l s t h i s a summative e v a l u a t i o n  But he omits d e t e r m i n i n g whether t h e c o u r s e  o b j e c t i v e s a r e w o r t h w h i l e g o a l s , and hence, whether t h e program has m e r i t .  2.12 The A c c r e d i t a t i o n Model The f o l l o w i n g quote i s a t y p i c a l  definition  of accreditation:  " A c c r e d i t a t i o n , as a p p l i e d i n e d u c a t i o n , i s t h e r e c o g n i t i o n a c c o r d e d t o an i n s t i t u t i o n t h a t meets t h e s t a n d a r d s o r c r i t e r i a e s t a b l i s h e d by a competent agency o r a s s o c i a t i o n . I t s g e n e r a l purpose i s t o promote and i n s u r e h i g h q u a l i t y i n e d u c a t i o n a l programs." (U.S. Dept. o f H e a l t h , E d u c a t i o n and W e l f a r e , 1959, p. 3) The a r t i c l e (1959, p. 4) goes on to o u t l i n e f o u r s t e p s i n f u l l y developing a c c r e d i t i n g procedures.  They a r e :  1)  Establishment o f standards or c r i t e r i a .  2)  Inspection of institutions  by competent a u t h o r i t i e s t o  d e t e r m i n e whether t h e y meet t h e e s t a b l i s h e d s t a n d a r d s o r c r i t e r i a . 3)  P u b l i c a t i o n of a l i s t of institutions  t h a t meet t h e s t a n d a r d s  or c r i t e r i a . 4)  P e r i o d i c r e v i e w s t o a s c e r t a i n whether a c c r e d i t e d i n s t i t u -  t i o n s c o n t i n u e t o meet t h e s t a n d a r d s o r c r i t e r i a . . A c c r e d i t a t i o n appears t o be v e r y s i m i l a r to e v a l u a t i o n ,  12 p a r t i c u l a r l y t h e f i r s t two s t e p s above.  Howeverj t h e i n v a l i d  a s s u m p t i o n made i n a c c r e d i t a t i o n i s t h a t the f u l f i l l m e n t o f t h e s t a n d a r d s and c r i t e r i a i n S t e p 1 n e c e s s a r i l y b e n e f i t the l e a r n e r i n t h e i n s t i t u t i o n . As G l a s s r i g h t l y p o i n t s out; "The a c c r e d i t a t i o n c r i t e r i a r e f l e c t t h e i n t e r e s t s o f a d m i n i s t r a t o r s ; a t t e n t i o n i s g i v e n t o the p r o c e s s e s o r means o f e d u c a t i o n and i s opposed t o i t s c o n s e q u e n c e s on l e a r n e r s . " ( G l a s s , 1969, p. 20) 2.13 S t u f f l e b e a m ' s  Context,  Input, Process and P r o d u c t  (CIPP)  Model, In t h i s model, e v a l u a t i o n i s d e f i n e d a s , ". . . the p r o v i s i o n o f i n f o r m a t i o n t h r o u g h formal means, such as c r i t e r i a , measurement, and s t a t i s t i c s , t o s e r v e as r a t i o n a l bases f o r making judgements i n d e c i s i o n s i t u a t i o n s . " ( S t u f f l e b e a m , 1958, p. 6) In o t h e r words, the e v a l u a t o r i s e s s e n t i a l l y an appendage t o the decision-maker.  T h i s d e f i n i t i o n i s o p e r a t i o n a l i z e d t h r o u g h t h e CIPP  model, a b o u t w h i c h worthen (1968, p. 3) w r i t e s : " D e c e p t i v e l y s i m p l e when viewed i n o u t l i n e f o r m , t h e s t r u c t u r e c o n s i s t s o f a l i s t o f no l e s s than twenty-two d e c i s i o n s i t u a t i o n s w h i c h a r e common to most e v a l u a t i o n d e s i g n s . Upon even c u r s o r y i n s p e c t i o n , however, i t i s a p p a r e n t t h a t t h e s t r u c t u r e i m p l i e s much more than a s i m p l e l i s t o f twenty-two i t e m s w h i c h e v a l u a t o r s must keep i n mind, f o r a t each o f the d e c i s i o n p o i n t s i d e n t i f i e d i n the s t r u c t u r e , t h e e v a l u a t o r needs t o make a c h o i c e among t h e available alternatives." Stufflebeam  i d e n t i f i e s four classes of decisions:  programming, i m p l e m e n t i n g , and r e - c y c l i n g ; and s u g g e s t s  planning, four  13 evaluation s t r a t e g i e s f o r those d e c i s i o n s . a)  The s t r a t e g i e s a r e :  C o n t e x t e v a l u a t i o n ^ : w h i c h i d e n t i f i e s t h e g o a l s and needs  to be f u l f i l l e d by t h e program, and t h e e n v i r o n m e n t where t h e program w i l l b) procedures c)  operate.  Input e v a l u a t i o n : w h i c h i d e n t i f i e s and a s s e s s e s  alternative  and d e s i g n s f o r a t t a i n i n g program o b j e c t i v e s . P r o c e s s e v a l u a t i o n s which m o n i t o r s t h e on-going  program f o r  problems o r d e v i a t i o n s from program d e s i g n . d) determines  Product, e v a l u a t i o n ^ w h i c h i s summative e v a l u a t i o n , and t h e e x t e n t t o w h i c h t h e g o a l s and o b j e c t i v e s have been met.  Each s t r a t e g y i s then s u b - d i v i d e d i n t o twenty-two c a t e g o r i e s - the same f o r each s t r a t e g y — t o g i v e t h e e v a l u a t i o n d e s i g n .  This i s  i l l u s t r a t e d i n F i g u r e 1 (Worthen 1968, p. 4 ) . T h e r e a r e two o b j e c t i o n s t o t h i s model. practical consideration.  The f i r s t i s a  T h e r e c o u l d be a s many as 88 d e c i s i o n s  s i t u a t i o n s ( f o u r c l a s s e s o f d e c i s i o n s w i t h twenty-two d e c i s i o n s i t u a t i o n s i n each c l a s s ) w h i c h s e e m i n g l y w o u l d t a k e a n army o f data gatherers to f u l f i l l . The second o b j e c t i o n i s t h a t u s e o f t h e CIPP model would f a i l t o p r o v i d e an o v e r a l l e v a l u a t i o n .  The e v a l u a t o r u s i n g t h e  CIPP model would p r o v i d e i n f o r m a t i o n t o d e c i s i o n makers, b u t would make no d e c i s i o n on t h e w o r t h o f t h a t i n f o r m a t i o n o r t h e w o r t h o f t h e p r o d u c t being e v a l u a t e d .  As G l a s s (1969, p. 36) w r o t e :  "Being o f a s s i s t a n c e t o t h e program p e r s o n n e l — s o t h e y may b e t t e r conduct t h e i r b u s i n e s s — i s a proximate a i m o f e v a l u a t i o n ; t h e u l t i m a t e a i m o f an e v a l u a t i o n i s t o d e c i d e q u e s t i o n s o f w o r t h . "  14  Figure 1 Developing Evaluation  Designs  The l o g i c a l s t r u c t u r e o f e v a l u a t i o n d e s i g n i s t h e same f o r a l l types o f e v a l u a t i o n , whether c o n t e x t , i n p u t , p r o c e s s o r p r o d u c t e v a l u a t i o n . The p a r t s , b r i e f l y , a r e as f o l l o w s : A. Focusing the Evaluation 1. I d e n t i f y t h e m a j o r l e v e l ( s ) o f d e c i s i o n - m a k i n g t o be s e r v e d , e.g., l o c a l , s t a t e , o r n a t i o n a l . 2. F o r each l e v e l o f d e c i s i o n - m a k i n g , p r o j e c t t h e d e c i s i o n ; s i t u a t i o n s t o be s e r v e d and d e s c r i b e each one i n terms o f i t s l o c u s , ? f o c u s , t i m i n g , and c o m p o s i t i o n o f a l t e r n a t i v e s . 3. D e f i n e c r i t e r i a f o r e a c h d e c i s i o n s i t u a t i o n by s p e c i f y i n g v a r i a b l e s f o r measurement and s t a n d a r d s f o r u s e i n t h e judgment o f a l t e r n a t i v e s . 4. D e f i n e p o l i c i e s w i t h i n w h i c h t h e e v a l u a t i o n must o p e r a t e . B. C o l l e c t i o n o f I n f o r m a t i o n 1. S p e c i f y t h e s o u r c e o f t h e i n f o r m a t i o n t o be c o l l e c t e d . 2. S p e c i f y t h e i n s t r u m e n t s and methods f o r c o l l e c t i n g t h e needed i n f o r m a t i o n . 3. S p e c i f y t h e sampling p r o c e d u r e t o be employed. 4. S p e c i f y t h e c o n d i t i o n s and s c h e d u l e f o r i n f o r m a t i o n c o l l e c t i o n . C. O r g a n i z a t i o n o f I n f o r m a t i o n 1. S p e c i f y a format f o r t h e i n f o r m a t i o n w h i c h i s t o be c o l l e c t e d . 2. S p e c i f y a means f o r c o d i n g , o r g a n i z i n g , s t o r i n g , and r e t r i e v i n g information. D. A n a l y s i s o f I n f o r m a t i o n 1. S p e c i f y t h e a n a l y t i c a l p r o c e d u r e s t o be employed. 2. S p e c i f y a means f o r p e r f o r m i n g t h e a n a l y s i s . E. Reporting o f Information 1. D e f i n e t h e a u d i e n c e s f o r t h e e v a l u a t i o n r e p o r t s . 2. S p e c i f y means f o r p r o v i d i n g i n f o r m a t i o n ^ t o i t h e . a u d i e n c e s . 3. S p e c i f y * t h e f o r m a t " f o r e v a l u a t i o n r e p o r t s a n d / o r r e p o r t i n g sessions. 4. S c h e d u l e t h e r e p o r t i n g o f i n f o r m a t i o n . F. Administration o f the evaluation 1. Summarize t h e e v a l u a t i o n s c h e d u l e . 2. D e f i n e s t a f f and r e s o u r c e r e q u i r e m e n t s and p l a n s f o r m e e t i n g these requirements. 3. S p e c i f y means f o r meeting p o l i c y r e q u i r e m e n t s f o r conduct o f t h e evaluation. 4. E v a l u a t e t h e p o t e n t i a l o f t h e i n f o r m a t i o n d e s i g n f o r p r o v i d i n g i n f o r m a t i o n w h i c h i s v a l i d , r e l i a b l e , c r e d i b l e , t i m e l y , and pervasive. .5. S p e c i f y and s c h e d u l e means f o r p e r i o d i c u p d a t i n g o f t h e e v a l uation design. 6. P r o v i d e a budget f o r t h e t o t a l e v a l u a t i o n program. ;  15  It i s p r e c i s e l y the question o f worth that evaluators using the CIPP model a v o i d . Provus (1969) and A l k i n ' s (UCLA, 1968) models both c l o s e l y resemble Stufflebeam's  CIPP model.  S c r i v e n (1972) s t r o n g l y y c r i t i c i z e d t h e CIPP model.  He s t a t e d  t h a t what i s c a l l e d e v a l u a t i o n i n t h e CIPP model i s n o t e v a l u a t i o n at a l l .  Context e v a l u a t i o n i s e s s e n t i a l l y a market-survey, input  e v a l u a t i o n i s a survey o f r e s o u r c e o p t i o n s , p r o c e s s e v a l u a t i o n i s u s u a l l y s o c i a l m o n i t o r i n g and b o o k k e e p i n g , and p r o d u c t  evaluation  appears t o be a d i l u t e m i x t u r e o f summative and f o r m a t i v e e v a l u a t i o n - - c l a i m i n g t o do b o t h , but a c t u a l l y , n o t d o i n g e i t h e r ( I b i d , p. 134-135). S c r i v e n ' s most g e n e r a l c r i t i c i s m o f t h e CIPP model i s ( I b i d , p. 3 6 ) ; " [ t h e CIPP model i s ] a b o u t t h e most c o m p l i c a t e d and c o n f u s i n g way o f a n a l y s i n g t h e p r a c t i c a l p r o c e d u r e s o f e v a l u a t i o n t h a t I can imagine, and i t ' s c e r t a i n l y t h e most c o m p l i c a t e d one t h a t I've e v e r seen." 2.14 S t a k e ' s Model In 1967 Stake proposed  h i s e v a l u a t i o n model ( S t a k e , 1967).  The model f o c u s s e d on t h e d e s c r i p t i o n and judgement o f e d u c a t i o n a l programs.  The d e s c r i p t i v e p a r t i s d i v i d e d i n t o " i n t e n t s " ( t h e g o a l s  a n d / o r o b j e c t i v e s ) and o b s e r v a t i o n s  (what t h e e v a l u a t o r l e a r n s  through e m p i r i c a l m e a s u r e s ) . The judgement m a t r i x i s a l s o d i v i d e d i n t o two p a r t s — s t a n d a r d s  and j u d g e m e n t s . The s t a n d a r d s can e i t h e r  be a b s o l u t e ( i . e . c r i t e r i o n l e v e l s ) o r r e l a t i v e ( i . e . compared t o o t h e r , s i m i l a r p r o g r a m s ) , b u t must p e r t a i n t o t h e t h i n g being  16 evaluated.  The judgements a r e the a s s i g n i n g o f w e i g h t s to the  s t a n d a r d s and j u d g i n g t h e m e r i t o f the p r o d u c t under c o n s i d e r a t i o n . Stake a l s o says t h a t t h e program s h o u l d have a r a t i o n a l e ; but i t i s not c l e a r from h i s paper (The Countenance o f E d u c a t i o n a l E v a l u a t i o n , S t a k e , 1967), whether t h e e v a l u a t o r makes any judgements o f the r a t i o n a l e . He says "The e v a l u a t o r asks h i m s e l f o r o t h e r j u d g e s w h e t h e r t h e p l a n d e v e l o p e d by t h e e d u c a t o r c o n s t i t u t e s a l o g i c a l step i n the implementation 1967, p. 1 3 ) .  o f the b a s i c p u r p o s e s . "  (Stake,  However, t h i s i s not a judgement o f t h e w o r t h o f t h e  program g o a l s as t h e y have been o u t l i n e d i n the program's r a t i o n a l e - r a t h e r , i t i s a judgement o f t h e l o g i c o f the i m p l e m e n t a t i o n  plans.  In an a p p l i c a t i o n o f h i s model Stake (1971, p. 4) w r i t e s : " E v a l u a t o r s have an o b l i g a t i o n t o r a i s e t h e q u e s t i o n s , 'Were the r i g h t g o a l s pursued?' D i f f e r e n t p e o p l e have d i f f e r e n t i d e a s , o f c o u r s e , as t o what t h e r i g h t g o a l s a r e . S t i l l , g o a l s and p r i o r i t i e s s h o u l d be e v a l u a t e d . " He goes on t o s a y , about the p a r t i c u l a r p r o j e c t he i s e v a l u a t i n g "In t h e eyes o f t h e e v a l u a t o r s (my e m p h a s i s ) ,  the T c i t y g o a l s a r e  worthy g o a l s , s u i t a b l y d i s c u s s e d and r e a s o n a b l y o p e r a t i o n a l i z e d . " ( S t a k e , 1971, p. 4)  However, the l o g i c a l c o n n e c t i o n between t h e  g o a l s and w o r t h o f the g o a l s has s t i l l not been e x p l i c a t e d , f o r one can s t i l l ask, "The g o a l s a r e w o r t h w h i l e really  i n h i s eyes, but a r e t h e y  worthwhile?"  2.15 Summary The problem o f d e c i d i n g what one s h o u l d do when one  \  17  e v a l u a t e s s p r i n g s from an i m p r e c i s e i d e a o f what the word " e v a l u a t i o n " means.  T h i s has caused many o f the s h o r t c o m i n g s  d e s c r i b e d above.  o f the models  U n f o r t u n a t e l y , i n none o f the r e c e n t  compilations  o f e v a l u a t i o n t h e o r y (Popham; 1974, wbrthen & S a n d e r s , 1973;  Tyler,  1969) i s t h e r e a s e r i o u s e f f o r t to deal w i t h t h i s p r o b l e m .  Therefore  the r e s t o f t h i s Chapter w i l l be an a t t e m p t t o p r o v i d e a l o g i c a l base f o r the meaning and use o f t h e word " e v a l u a t i o n " . 2.20 E v a l u a t i o n : The T h e o r e t i c a l Framework "The p r o c e s s o f e v a l u a t i o n c o n s i s t s i n t r y i n g t o d e t e r m i n e t h e v a l u e o f s o m e t h i n g . As a p r o d u c t o r outcome o f t h a t p r o c e s s , an e v a l u a t i o n i s a s e t t l e d o p i n i o n t h a t something has a c e r t a i n v a l u e . " ( T a y l o r , 1961, p. 3) T a y l o r ' s views a r e p a r a l l e l e d by S c r i v e n (1967), whose c o n t r i b u t i o n i s to provide a methodological treatment.  framework f o r t h e a n a l y t i c a l  S c r i v e n c o n s i d e r s t h a t the goal o f e v a l u a t i o n i s to  d e t e r m i n e the m e r i t o r worth o f some e n t e r p r i s e , and s u g g e s t s  we  can a t t a i n t h a t goal by g a t h e r i n g and c o m b i n i n g needs r e l a t e d p e r f o r m a n c e d a t a , weighing  and c o m b i n i n g t h a t - d a t a and j u s t i f y i n g  our c o n c l u s i o n o f worth by showing t h a t our  data-gathering  i n s t r u m e n t s and s e l c t i o n o f c r i t e r i a a r e v a l i d ( S c r i v e n ,  1967,  p. 40-41). T a y l o r (1961, p. 9, 10) has i d e n t i f i e d f i v e s t e p s i n evaluation.  To t h e s e f i v e s t e p s one c o u l d add a p r e l i m i n a r y s e t o f  p r e c o n d i t i o n s to c l a r i f y the f i v e s t e p s w h i c h f o l l o w .  Therefore  an  18 e v a l u a t i o n on l o g i c a l grounds r e q u i r e s : 1)  The P r e c o n d i t i o n s : a)  evaluated).  I d e n t i f y t h e evaluatum : (what i t i s t h a t i s b e i n g _ "\ • . )  b)  D e c i d e on whether you a r e g o i n g t o : (i)  Grade a c c o r d i n g t o s t a n d a r d s ( S - G r a d i n g ) ( t h a t i s e v a l u a t i n g i n terms o f some a b s o l u t e standards);  ( i i ) Rank ( t h a t i s S-Grade and t h e n compare and rank o r d e r t h e e v a l u a t u m i n a c o m p a r i s o n group o f s i m i l a r e v a l u a t a which have a l s o been S-Graded); ( i i i | G r a d e a c c o r d i n g to r u l e s (R-Grading) (that i s g r a d e t h e e v a l u a t u m as o b e y i n g o r not o b e y i n g a set of applicable rules) c) 2)  D e t e r m i n e t h e P o i n t o f View o r P o i n t s o f View.  Adoption of a standard or r u l e , or set of standards or  r u l e s , f o r e v a l u a t i n g the evaluatum. 3)  Operational c l a r i f i c a t i o n of the standards or r u l e s .  4)  S p e c i f i c a t i o n o f the c l a s s o f comparison.  5)  D e t e r m i n i n g t h e good o r bad c h a r a c t e r i s t i c s o f t h e  evaluatum. 6)  D e d u c i n g , from (3) and ( 5 ) , t h e d e g r e e t o which t h e  e v a l u a t u m on t h e whole f u l f i l l s o r f a i l s t o f u l f i l l t h e s t a n d a r d s ; o r , in the case of r u l e s , the r e l a t i v e importance of the r u l e s the evaluatup  p a s s e s o r f a i l s (A r u l e can e i t h e r be obeyed o r v i o l a t e d .  19 T h e r e f o r e t h e r e a r e no d e g r e e s o f ' c o m p l i a n c e o r v i o l a t i o n . ) T h e s e s i x s t e p s w i l l not;be d i s c u s s e d i n more d e t a i l .  2.30 The P r e c o n d i t i o n s t o an E v a l u a t i o n I f an e v a l u a t o r i s t o p e r f o r m an e v a l u a t i o n , t h e n t h r e e p r e c o n d i t i o n s must be s a t i s f i e d .  The f i r s t o f t h e s e i s t h a t t h e r e must  be something t o e v a l u a t e ( i . e . , an e v a l u a t u m ) ; t h e second i s t h a t t h e e v a l u a t o r must adopt a p o i n t o f view, and t h e t h i r d i s t h a t t h e e v a l u a t o r must d e c i d e i f he i s g o i n g t o grade o r rank a c c o r d i n g t o rules or standards.  2.31 The  Evaluatum  T h a t which i s t o be e v a l u a t e d has been l a b e l l e d by T a y l o r as t h e evaluatum  ( T a y l o r , p. 4, p. 23) and i s a n y t h i n g which can be  ranked o r g r a d e d ( d i s c u s s e d l a t e r i n t h i s c h a p t e r ) , o r which  has  bad and/or good-making c h a r a c t e r i s t i c s , to be d e t e r m i n e d by t h e evaluator.  For example, t h e evaluatum f o r t h i s t h e s i s i s a t e a c h e r -  t r a i n i n g program a t U.B.C.  2.32 P o i n t o f View T a y l o r says t h a t : " T a k i n g a c e r t a i n p o i n t o f view i s n o t h i n g but a d o p t i n g c e r t a i n canons o f r e a s o n i n g as t h e framework w i t h i n which v a l u e judgements a r e t o be j u s t i f i e d ; t h e canons o f r e a s o n i n g d e f i n e t h e p o i n t o f view." ( T a y l o r , 1961, p. 109) and  20 " T h i s means p r e c i s e l y t h a t t h e j u d g e o r e v a l u a t o r has a d o p t e d a s e t o f r u l e s o f r e l e v a n c e and v a l i d i n f e r e n c e t h a t r e c o g n i z e o n l y c e r t a i n r e a s o n s as r e l e v a n t and good." ( T a y l o r , 1961, p. 109) The above e x p l a n a t i o n s were e x e m p l i f i e d by T a y l o r as f o l l o w s : "We cannot e v a l u a t e a c a r as good o r bad, o r as b e t t e r o r worse than a n o t h e r c a r , u n l e s s we know what p o i n t o f v i e w we.-are t o t a k e ... Is i t to be e v a l u a t e d as a p i e c e o f m e c h a n i c a l engineering? As an o b j e c t o f a e s t h e t i c c o n t e m p l a t i o n ? . . . Each p o i n t o f v i e w we t a k e makes a d i f f e r e n c e , s i n c e d i f f e r e n t s t a n d a r d s (norms) a r e a p p r o p r i a t e t o d i f f e r e n t p o i n t s o f view." ( T a y l o r , 1961, p. 5, 6) I t would be unusual t o be a b l e to d e c i d e t h e o v e r a l l w o r t h o f any program by e v a l u a t i n g i t from o n l y one p o i n t o f v i e w . T a y l o r ' s example above, a c a r e f u l s h o p p e r would not buy an o n l y on t h e b a s i s o f mechanical aesthetic contemplation.  Even i n automobile  e n g i n e e r i n g , o r o n l y on t h e b a s i s o f  Nor i s i t r e a s o n a b l e t o compare t e a c h e r -  t r a i n i n g programs o n l y on t h e b a s i s o f t h e i r o r g a n i z a t i o n , o r o n l y on the b a s i s o f t h e i r r e l a t i v e c o s t s .  A l l r e a s o n a b l e p o i n t s o f view  must be t a k e n i n t o c o n s i d e r a t i o n . T a y l o r has i d e n t i f i e d e i g h t b a s i c p o i n t s o f view i n a c i v i l i z e d c u l t u r e ; they a r e : t h e m o r a l , the a e s t h e t i c , the s c i e n t i f i c , the r e l i g i o u s , t h e e c o n o m i c , t h e p o l i t i c a l , the l e g a l , and t h e p o i n t o f view o f e t i q u e t t e o r custom.  He d e r i v e s t h e s e p o i n t s o f view from  e i g h t c o r r e s p o n d i n g m a j o r i n s t i t u t i o n s and a c t i v i t i e s p l u s the e d u c a t i o n a l i n s t i t u t i o n s ; but as he p o i n t s o u t : "There i s no s i n g l e p o i n t o f view c o r r e s p o n d i n g t o t h e educational i n s t i t u t i o n s of a s o c i e t y , since education i s a process which may t a k e p l a c e w i t h i n any p o i n t o f v i e w . Thus t h e r e i s moral education, aesthetic education, i n t e l l e c t u a l education, r e l i g i o u s e d u c a t i o n , and so on." (1961, p. 300)  the  21 T a y l o r ' s r e a s o n i n g does not sound t h e d e a t h k n e l l f o r t h e evaluation of educational i n s t i t u t i o n s .  I t simply a l e r t s the  e v a l u a t o r t o t h e m u l t i p l i c i t y o f p o i n t s o f view which must be a t l e a s t c o n s i d e r e d i n t h e e v a l u a t i o n . F o r example, i s t h e t e a c h e r t r a i n i n g program immoral  (moral p o i n t o f v i e w ) , c r a s s ( a e s t h e t i c  point of view), i l l e g a l " ( l e g a l point of view), expensive  (economic  p o i n t o f v i e w ) , and so on. 2.33 G r a d i n g o r Ranking The t h i r d p r e - c o n d i t i o n i s to d e c i d e what form t h e evaluation w i l l take.  There are three p o s s i b i l i t i e s : grading  a c c o r d i n g t o s t a n d a r d s , g r a d i n g a c c o r d i n g t o r u l e s , and r a n k i n g „. according to standards.  D a n i e l s (1971, p. 6) g i v e s a p a r t i c u l a r l y  l u c i d a c c o u n t o f t h e s e t h r e e f o r m s , so h i s d e s c r i p t i o n i s r e p r o d u c e d i n i t s e n t i r e t y below: "Form 1 G r a d i n g a c c o r d i n g t o s t a n d a r d s . ( L e t us c a l l t h i s S-grading.) Form 2 G r a d i n g a c c o r d i n g to r u l e s . ( L e t us c a l l t h i s R - g r a d i n g . ) Form 3 R a n k i n g . B e f o r e we see how t h e s e t h r e e f o r m s , S - g r a d i n g , R - g r a d i n g and Ranking, c o u l d be used t o e v a l u a t e c u r r i c u l a , l e t us l o o k a t each i n a somewhat simpler context. S - g r a d i n g : Suppose we a r e j u d g i n g c a r s f o r t h e i r c o m f o r t . I t makes sense t o . s a y . t h a t Car 1 i s u n c o m f o r t a b l e , Car 2 i s f a i r l y c o m f o r t a b l e , Car 3 i s c o m f o r t a b l e , Car 4 i s v e r y c o m f o r t a b l e and Car 5 i s e x t r e m e l y c o m f o r t a b l e . This-shows us t h e two main f e a t u r e s o f S - g r a d i n g : a) The c r i t e r i o n we use as t h e b a s i s f o r our e v a l u a t i o n i s a s t a n d a r d , b) T h i n g s can f u l f i l l s t a n d a r d s i n d i f f e r i n g d e g r e e s . Thus, we can e v a l u a t e t h i n g s on a s i m p l e t w o - p a r t s c a l e ( e . g . , c o m f o r t a b l e o r u n c o m f o r t a b l e ) o r on a m u l t i p l e - l e v e l s c a l e (as i n our example o f t h e f i v e c a r s ) . R - g r a d i n g : Suppose, however, t h a t we a r e c o n c e r n e d t o d i s c o v e r whether o r not t h e h e a d l i g h t s on o u r c a r s a r e c o r r e c t l y a d j u s t e d as s e t down by a law. Now we a r e e v a l u a t i n g a c c o r d i n g to r u l e s and t h e n o t i o n o f d e g r e e s i s not a p p l i c a l b e ; e i t h e r t h e h e a d l i g h t s conform w i t h t h e r u l e s o r t h e y do n o t ; t h e y a r e r i g h t o r wrong, c o r r e c t o r incorrect.  22  R a n k i n g : In o r d e r t o rank t h i n g s we must go t h r o u g h two s t e p s : a) we f i r s t o f a l l S-grade t h e t h i n g s t o be r a n k e d , as we d i d f o r i n s t a n c e w i t h t h e f i v e c a r s , b) We t h e n compare the d e g r e e t o w h i c h the t h i n g s f u l f i l l t h e s t a n d a r d ( R - g r a d i n g c a n n o t be used f o r R a n k i n g ) , and rank them. Thus, i n r a n k i n g o u r c a r s from t h e p o i n t o f view o f c o m f o r t , we might say s o m e t h i n g l i k e t h e f o l l o w i n g ; Car 1 - w o r s t , Car 3 - a v e r a g e , Car 5 - b e s t . " I t i s n o t u n r e a s o n a b l e t o s u g g e s t t h a t any e d u c a t i o n a l p r o d u c t can  undergo a l l t h r e e f o r m s , R - g r a d i n g , S - g r a d i n g and R a n k i n g , i n  i t s e v o l u t i o n from an i n i t i a l i n s p i r a t i o n t o t h e f i n i s h e d p r o d u c t . For example, suppose a group o f p r o f e s s o r s w i s h e d t o i n i t i a t e a t e a c h e r - t r a i n i n g program based on a p a r t i c u l a r model o f t e a c h e r training.  F i r s t l y , t h e y would have t o d e t e r m i n e , among o t h e r t h i n g s ,  i f t h e i r p r o p o s e d program ( t h e e v a l u a t u m ) was f e a s i b l e i n terms o f the u n i v e r s i t y .  In o t h e r words, would t h e s t u d e n t s be t a u g h t t h e  r e q u i r e d number o f c o u r s e s ( s p e c i f i e d number o f u n i t s ) ; do t h e p r o s p e c t i v e s t u d e n t s have t h e r e q u i r e d minimum e n t r a n c e q u a l i f i c a t i o n s (for t h i s p a r t i c u l a r year o f u n i v e r s i t y study); w i l l the p r o f e s s o r s ' t e a c h i n g r e s p o n s i b i l i t i e s be a t l e a s t t h e minimum r e q u i r e d ? T h e s e a r e r u l e s which a r e e i t h e r f u l f i l l e d o r n o t f u l f i l l e d .  In t h e example  g i v e n , t h i s f i r s t s t e p , though u s u a l l y i n f o r m a l l y done, c o u l d be i n t e r p r e t e d as R-GRADING. Once t h e program has been t h r o u g h i n f o r m a l R - g r a d i n g , i t i s o p e r a t i o n a l i z e d and i s no l o n g e r a p r o p o s a l .  The program i s i n a  d e v e l o p m e n t a l p e r i o d o r f o r m a t i v e s t a g e . The main c o n c e r n o f t h e program p r o p o n e n t s i s whether o r n o t t h e s t a n d a r d s t h e y s e t f o r t h e program a r e b e i n g met.  They a s k :  " I s t h e o p e r a t i o n a l program t h e  same as t h a t program we o r i g i n a l l y c o n c e p t u a l i z e d ? " be used i n t h e e v a l u a t i o n come f r o m t h i s v e r s i o n .  The s t a n d a r d s t o Various character-  i s t i e s o f t h e program a r e t h e n g r a d e d a c c o r d i n g t o t h o s e s t a n d a r d s . S t a k e (1967) has p r o p o s e d t h i s method when he d i s c u s s e s j u d g i n g t h e c h a r a c t e r i s t i c s o f a program w i t h r e s p e c t t o a b s o l u t e s t a n d a r d s .  He  s a y s t h a t each s e t o f a b s o l u t e s t a n d a r d s , when f o r m a l i z e d , would i n d i c a t e a c c e p t a b l e and m e r i t o r i o u s l e v e l s o f p e r f o r m a n c e f o r each characteristic.  This f o r m a l i z a t i o n i s i l l u s t r a t e d i n Chapter I I I  of t h i s t h e s i s . The p r o c e s s o f d e t e r m i n i n g whether a program i s good o r bad w i t h r e s p e c t t o a b s o l u t e s t a n d a r d s i s S-GRADING. The f i n a l s t a g e i n t h e program's e v a l u a t i o n , i s t o d e t e r m i n e how i t compares t o t h e o t h e r , s i m i l a r t e a c h e r - t r a i n i n g programs ( i . e . t o t h e members o f t h e c l a s s o f c o m p a r i s o n ) .  available  T h e r e a r e two  c r i t e r i a which must be t a k e n i n t o a c c o u n t t o d e c i d e whether t h e e v a l u a t u m i s b e t t e r o r worse than s o m e t h i n g e l s e , o r i s t h e b e s t o r worst i n the class o f comparisons.  They a r e :  "... whether t h e e v a l u a t u m i s good on t h e whole, o r bad on t h e whole, when graded a c c o r d i n g t o t h e g i v e n s t a n d a r d s , and whether i t s o v e r a l l goodness o r badness outweighs t h e o v e r a l l goodness or badness o f each member o f t h e c l a s s o f c o m p a r i s o n . " ( T a y l o r , 1961, p. 15, 16) Of c o u r s e , each member o f t h e c l a s s o f c o m p a r i s o n has a l s o been g r a d e d a c c o r d i n g t o t h e same s e t o f s t a n d a r d s . T h e r e f o r e , an e v a l u a t u m can be graded as "bad" on t h e whole, y e t be b e t t e r than any o f t h e members o f t h e c l a s s o f c o m p a r i s o n s , o r t h e b e s t o f a bad l o t . O r , s i m i l a r l y , i t c o u l d be g r a d e d as good, and ranked as a v e r a g e , e t c . The p r o c e s s o f d e t e r m i n i n g t h e c o m p a r a t i v e v a l u e o f a program i s c a l l e d RANKING.  24 S c r i v e n has argued t h a t we s h o u l d d e t e r m i n e t h e s u p e r i o r i t y o r i n f e r i o r i t y o f the e v a l u a t u m t o the c o m p e t i t i o n b e f o r e we have comp l e t e l y e v a l u a t e d the e v a l u a t u m .  ( S c r i v e n , 1967)  must have gone through a p r o c e s s o f RANKING.  In o t h e r words we  In a l a t e r a r t i c l e he  p r e s e n t e d a taxonomy o f d i f f e r e n t e v a l u a t i v e c o n c l u s i o n s , ( S c r i v e n , 1974a, p. 146) p r e s e n t e d here as T a b l e 1. In p r a c t i s e , a p a r t i c u l a r e v a l u a t i v e c o n c l u s i o n i s aimed f o r ( o r a r r i v e d a t ) somewhere between a " P r e e v a l u a t i v e , " and a "Best-Buy" evaluative conclusion.  A n y t h i n g l e s s than a "Best-Buy e v a l u a t i v e  c o n c l u s i o n " , c o n s t i t u t e s l e s s than a c o m p l e t e e v a l u a t i o n ; t h e r e f o r e i t i s i m p o r t a n t t h a t the c l i e n t f o r the e v a l u a t i o n knows e x a c t l y what s o r t o f e v a l u a t i o n he i s g e t t i n g , and how much i t f a l l s s h o r t o f a complete e v a l u a t i o n ( i f , indeed, i t does). An e v a l u a t i v e c o n c l u s i o n o f "Commendatory" i n S c r i v e n ' s t a x onomy, i s t h e h i g h e s t l e v e l p o s s i b l e u s i n g S - g r a d i n g .  The c o n t r o l  group i m p l i e d i n t h e , " u s u a l l y a d e q u a t e p r e m i s e s " o f T a b l e 1, i s the i d e a l form o f the e v a l u a t u m .  I t i s a g a i n s t the s t a n d a r d s o f t h a t  i d e a l t h a t the e v a l u a t u m i s graded. The l a s t t h r e e e v a l u a t i v e c o n c l u s i o n s : L a u d a t o r y ,  I d e a l , and  B e s t - B u y , i n v o l v e r a n k i n g t h e e v a l u a t a , o r making o v e r a l l statements.  comparative  Therefore, these three conclusions i l l u s t r a t e increas-  i n g l y c o m p r e h e n s i v e RANKING. T h i s p a r t i t i o n i n g o f S c r i v e n ' s taxonomy does not e x c l u d e f i r s t f o u r e v a l u a t i v e c o n c l u s i o n s ( P r e e v a l u a t i v e , Minimal  the  evaluative,  O v e r a l l e v a l u a t i v e and Commendatory) from t h e p r o c e s s o f RANKING.  If  TABLE 1 Taxonomy o f E v a l u a t i o n Types E v a l u a t i o n Type Preevaluative (Goal o r criterion achievement) Minimal evaluative  Usual V e r b a l E x p r e s s i o n "The t r e a t m e n t X had t h e e f f e c t Y on t h e p o p u l a t i o n o f s t u d e n t s , S, i n c o n d i t i o n s C; and Y was the goal o r shows t h a t .the'.goal was a c h i e v e d . " "X had a good e f f e c t ( o n S i n C)."  Overall evaluative  "X had an o v e r a l l good effect."  Commendatory  "X was w o r t h d o i n g . " T h i s i s a l m o s t a subcase o f o v e r a l l e v a l uation, i f costs are taken as a harmful effect. "X was t h e b e s t c h o i c e . "  Laudatory  25  Usually Adequate Premises 1. X was t h e t r e a t m e n t . 2. X c a u s e d Y. 3. Y i m p l i e s t h a t t h e g o a l - , was a c h i e v e d . 1. X c a u s e d Y. 2. S o r non-Ss ( d e s i r e d , 1 e n j o y e d , were benef i t t e d by) Y. 1. As f o r m i n i m a l , p l u s , 2. Y had no harmful e f f e c t s dnSSs; o r ndn-Ss, or, 3. Y had much l e s s s i g n i f i c a n t harmful e f f e c t s on S s . 1. As f o r o v e r a l l , p l u s , 2. The c o s t o f X was manageable. 3. Y was worth t h e c o s t . 1  As f o r commendatory, plus, 2. No o t h e r t r e a t m e n t , on d a t a which was a v a i l able, appeared as costeffective. Ideal "X was t h e b e s t 1. As f o r commendatory, p o s s i b l e treatment. plus, 2. No o t h e r t r e a t m e n t was i n f a c t as c o s t - e f f e c tive. Best-Buy "X was a B e s t - B u y . " 1 As f o r commendatory, plus, 2. X i s a member o f a group which o f f e r s t h e best o r almost the best performance f o r s i g n i f i c a n t l y l e s s c o s t than t h e i r performance peers. T h e s e p r e m i s e s g i v e prima f a c i e s u p p o r t f o r e v a l u a t i v e c o n c l u s i o n s , not d e d u c t i v e s u p p o r t , b u t e v a l u a t i o n - - l i k e * s c i e n c e - - o n l y needs prima f a c i e i n f e r e n c e . 1  26 t h e e v a l u a t o r i s o n l y RANKING, then he would base h i s rank o r d e r o f t h e v a r i o u s e v a l u a t a on an i n c r e a s i n g l y c o m p r e h e n s i v e l i s t o f s t a n d a r d s , as he goes from a p r e e v a l u a t i v e t o a Best-Buy c o n c l u s i o n . An i n t e r n a l , f o r m a t i v e e v a l u a t o r ( i . e . t h e e a r l y  developmental  s t a g e o f t h e p r o j e c t i s b e i n g e v a l u a t e d by someone i n v o l v e d i n , and committed t o t h e p r o j e c t ( S c r i v e n , 1967, p . 45) i s u s u a l l y  concerned  w i t h t h e e a r l y s t a g e s o f S-GRADING. I n o t h e r words, he i s g r a d i n g the e v a l u a t u m on t h e b a s i s o f a s e r i e s o f s t a n d a r d s d e r i v e d from some i d e a l , o r b e s t p o s s i b l e , form o f t h e e v a l u a t u m .  On t h e o t h e r hand,  i f t h e e v a l u a t o r i s an e x t e r n a l , summa.tive e v a l u a t o r ( i . e . t h e f i n a l marketed v e r s i o n i s being e v a l u a t e d by someone e x t e r n a l t o t h e p r o j e c t ) , then he i s r a n k i n g t h e e v a l u a t u m i n t h e c l a s s o f c o m p a r i s o n . T h e r e i s a n i n t e r m e d i a t e s t e p near t h e end o f t h e f o r m a t i v e s t a g e o f the e v a l u a t i o n .  A t t h i s t i m e , an e x t e r n a l f o r m a t i v e e y a l u a t o r c o u l d  be engaged t o perform t h e l a s t s t a g e s o f S - g r a d i n g and e a r l y s t a g e s o f r a n k i n g b e f o r e t h e e v a l u a t u m went t o t h e summative e v a l u a t i o n . The p a r t i c u l a r problem examined by t h i s t h e s i s was an e v a l u a t i o n o f a t e a c h e r - t r a i n i n g program.  T h i s program was i n t h e developmental  s t a g e , and t h e a u t h o r o f t h i s t h e s i s was p r i m a r i l y an i n t e r n a l , formative evaluator.  T h e r e f o r e , t h e form o f t h e e v a l u a t i o n was  S-GRADING, and aimed a t an o v e r a l l e v a l u a t i v e c o n c l u s i o n (see T a b l e 1 ) . S i n c e t h i s was t h e c a s e , t h e n t h e r e s t o f T a y l o r ' s l o g i c a l  process  o f e v a l u a t i o n w i l l be examined a s i t a p p l i e s t o S-GRADING. F i g u r e 2 has been p r e p a r e d t o a i d i n t h e f o l l o w i n g d e s c r i p t i o n o f T a y l o r ' s p h i l o s o p h i c a l p o s i t i o n . . T h i s f i g u r e s h o u l d n o t be r i g i d l y  Fig- 2 An O u t l i n e o f the L o g i c a l P r o c e s s o f E v a l u a t i o n The Preconditions  Evaluatum  I  The P o i n t o f View G r a d i n g or Ranking? I n t r i n s i c Value  Level 1  Level 2 Level 3  Adoption of Standards  \  E x t r i n s i c Value  Operational C l a r i f i c a t i o r of Standards S p e c i f i c a t i o n of the Class of  Comparisons G a t h e r i n g Data bh the C h a r a c t e r i s t i c s  D e t e r m i n a t i o n o f t h e Good Level 4  and Bad C h a r a c t e r i s t i c s of the E v a l u a t i o n Determining i f the Characteristics are .Good o r Bad  Level 5  Overall Determination of Worth o f the Evaluatum  28  i n t e r p r e t e d as a f l o w c h a r t d e p i c t i n g a temporal sequence o f e v e n t s . The p r o c e s s o f e v a l u a t i o n does b e g i n w i t h an evaluatum* and end w i t h an o v e r a l l e s t i m a t i o n o f goodness o r badness; however, t h e i n t e r v e n i n g s t e p s may o c c u r i n any o r d e r and r e p e t i t i v e l y , b e f o r e a r r i v i n g a t t h e final evaluative conclusion.  Nevertheless, i t i s a necessary condi-  t i o n i n T a y l o r ' s vciew o f e v a l u a t i o n , t h a t a l l t h e s t e p s i n F i g u r e 1 must be completed b e f o r e an e v a l u a t i o n can be l o g i c a l l y v e r i f i e d .  2.40 L e v e l 1 of, S - G r a d i n g : A d o p t i o n  o f S t a n d a r d s (See F i g . 1)  T a y l o r i d e n t i f i e s two t y p e s o f s t a n d a r d s .  They a r e s t a n d a r d s o f  i n t r i n s i c v a l u e and s t a n d a r d s o f e x t r i n s i c v a l u e .  ( T a y l o r 1961,  pp.  20-25).  2.41  I n t r i n s i c Value S t a n d a r d s o f i n t r i n s i c v a l u e a r e used t o e v a l u a t e t h i n g s  which a r e v a l u a b l e i n and f o r t h e m s e l v e s  ( D a n i e l s , 1971, p. 9) and  t h e r e b y s a t i s f y i n g t h e f o l l o w i n g two c o n d i t i o n s . F i r s t l y t h e y must be used o n l y i n e v a l u a t i n g i m m e d i a t e l y f e l t o r p e r c e i v e d q u a l i t i e s o f o u r own e x p e r i e n c e ; and s e c o n d l y , t h e y must be s t a n d a r d s o f n o n d e r i v a t i v e v a l u e ( T a y l o r , 1961, p. 23). I f an evaluatum  produced  f e e l i n g s o f p l e a s a n t n e s s , then t h o s e f e e l i n g s would have i n t r i n s i c value.  2.42 E x t r i n s i c V a l u e Whereas s t a n d a r d s o f i n t r i n s i c v a l u e a r e o f n o n d e r i v a t i v e  29  value, standards of e x t r i n s i c value are of d e r i v a t i v e value, that i s , d e p e n d a n t on the v a l u e o f o t h e r t h i n g s .  T a y l o r has s u b - d i v i d e d  e x t r i n s i c v a l u e i n t o i n h e r e n t v a l u e , i n s t r u m e n t a l v a l u e and contributive value. I f an evaluatum  ( T a y l o r , 1961, pp. 26-32, 310-311) i s j u d g e d i n terms o f s t a n d a r d s o f i n h e r e n t v a l u e ,  then the e v a l u a t o r would l o o k f o r e v i d e n c e t h a t i t produces which a r e j u d g e d as i n t r i n s i c a l l y good.  experiences  I n h e r e n t v a l u e d i f f e r s from  i n t r i n s i c v a l u e i n t h a t the former must be o t h e r than a q u a l i t y o f our own  experience.  "Works o f a r t and t h i n g s o f n a t u r a l beauty may a l s o be s a i d to be good on the ground t h a t one who l o o k s at. them n o r m a l l y has a good o r r e w a r d i n g e x p e r i e n c e . Then, we may say t h a t they have i n h e r e n t goodness". ( F r a n k e n a , 1973, p. 81) In o t h e r words an o b j e c t ' s i n h e r e n t v a l u e depends on i t s c a p a c i t y to produce i n t r i n s i c v a l u e o f someone's e x p e r i e n c e i n r e s p o n s e to i t . ( T a y l o r , 1961, p. 28) To have i n s t r u m e n t a l v a l u e , the evaluatum must be e f f e c t i v e i n b r i n g i n g about a g i v e n end w h i c h i s i n t r i n s i c a l l y o r e x t r i n s i c a l l y valuable.  As D a n i e l s s a i d , (1971, p. 9 ) ,  " A - t h i n g has i n s t r u m e n t a l v a l u e i f i t can be used to o b t a i n some t h i n g o r c o n d i t i o n o t h e r than i t s e l f w h i c h i s , f o r some r e a s o n o r o t h e r , c o n s i d e r e d t o be o f v a l u e . " In the b r o a d e s t sense o f i n s t r u m e n t a l v a l u e an e v a l u a t o r would grade or rank a t e a c h e r - t r a i n i n g program a c c o r d i n g t o i t s e f f e c t i v e n e s s i n e n a b l i n g s t u d e n t s t o become good t e a c h e r s .  In a narrower  sense  30 he c o u l d examine methods c o u r s e s ( o r p a r t s o f t h o s e methods c o u r s e s ) f o r t h e i r s u c c e s s i n t e a c h i n g t h e s t u d e n t some p a r t i c u l a r t e c h n i q u e of value. To f u l f i l l a s t a n d a r d o f c o n t r i b u t i v e v a l u e , a " p a r t " must c o n t r i b u t e to a good w h o l e .  The more n e c e s s a r y i t i s t o t h e whole,  t h e more c o n t r i b u t i v e v a l u e i t has.  For example, p r o f e s s o r s o f  e d u c a t i o n s h o u l d c o n t r i b u t e g r e a t l y t o t h e p r o d u c t i o n o f good t e a c h e r s , t h e r e f o r e t h e y s h o u l d have h i g h c o n t r i b u t i v e v a l u e i n a t e a c h e r - t r a i n i n g program and would be j u d g e d i n terms o f s t a n d a r d s of "contribution". As T a y l o r i n d i c a t e s , the above t h r e e k i n d s o f s t a n d a r d s o f e x t r i n s i c v a l u e a r e dependent upon i n t r i n s i c v a l u e .  He says  (1961,  p. 3 2 ) : " F i r s t , a t h i n g may have i n s t r u m e n t a l v a l u e i n v i r t u e o f the f a c t t h a t i t i s a means t o an end which has c o n t r i b u t i v e v a l u e as p a r t o f a whole. T h i s whole i n t u r n e i t h e r may have i n h e r e n t v a l u e i n i t s e l f o r may have i n s t r u m e n t a l v a l u e as a means t o an end which has i n h e r e n t o r i n t r i n s i c v a l u e . " In o t h e r words, t h e f i n a l appeal f o r t h e r a i s o n d ' e t r e o f a n y t h i n g i s t h a t some one o r some group o f p e o p l e d e r i v e s i n t r i n s i c v a l u e from it.  "Thus we a r r i v e f i n a l l y a t t h i n g s which we j u d g e t o be d e s i r a b l e  in themselves. p. 32)  T h e i r v a l u e i s i n t r i n s i c to them." ( T a y l o r , 1961,  Or as Frankena (1973, p. 81-82) wrote:  "We a l s o sometimes say t h a t t h i n g s a r e good; d e s i r a b l e , o r w o r t h w h i l e i n t h e m s e l v e s , as ends, i n t r i n s i c a l l y . When someone a s k s 'What i s good f o r ; t h e answer may be g i v e n by t r y i n g to  31 e x h i b i t i t s u s e f u l n e s s , e x t r i n s i c v a l u e , o r i n h e r e n t goodness; b u t one may a l s o t r y t o show ( a n d h e r e t h e f i n a l a p p e a l must be ' T r y i t and see') t h a t i t i s e n j o y a b l e o r o t h e r w i s e good i n i t s e l f . . . In f a c t , i t i s hard t o s e e how money, c a r s , and o t h e r m a t e r i a l p o s s e s s i o n s , even p a i n t i n g s , c a n have any goodness o r v a l u e a t a l l , e x t r i n s i c o r i n h e r e n t , i f t h e e x p e r i e n c e s t h e y make p o s s i b l e a r e n o t i n some way e n j o y a b l e o r good i n t h e m s e l v e s . " The above s t a t e m e n t s about i n t r i n s i c v a l u e do n o t mean t h a t an e d u c a t i o n a l p r o d u c t i s n o t w o r t h w h i l e i f i t d o e s n ' t have i n t r i n s i c value.  However, i f t h e program does n o t have i n t r i n s i c v a l u e , t h e n  i t s h o u l d be shown t h a t a c h i e v i n g t h e g o a l s o f t h e program i s instrumental i n f u l f i l l i n g i n t r i n s i c value.  some a s p e c t o f l i v i n g which w i l l have  I f t h e r e i s no a p p a r e n t c o n n e c t i o n between t h e  program g o a l s and f u t u r e a t t a i n m e n t o f i n t r i n s i c v a l u e , t h e n t h e worthwhileness  o f t h e program g o a l s s h o u l d be s e r i o u s l y q u e s t i o n e d .  2.50 L e v e l I I o f S-GRADING (See F i g . 1 ) : O p e r a t i o n a l C l a r i f i c a t i o n o f the Standards i n Grading "This c o n s i s t s i n a s e t o f statements to t h e e f f e c t t h a t i f an o b j e c t 0 has c h a r a c t e r i s t i c s C, i t f u l f i l l s a c e r t a i n s t a n d a r d S to a c e r t a i n degree D; t h a t i f 0 has c h a r a c t e r i s t i c s C , i t f u l f i l l s S t o a g r e a t e r ( l e s s e r ) 'degree D'; e t c . " ( T a y l o r , 1961, p. 10) Once a s t a n d a r d has been g e n e r a l l y i d e n t i f i e d as a s t a n d a r d o f i n t r i n s i c o r e x t r i n s i c v a l u e , then i t must be. e l a b o r a t e d i n such terms t h a t p e r m i t t h e e v a l u a t o r t o s a y : " c h a r a c t e r i s t i c C f u l f i l l s s t a n d a r d S t o d e g r e e D."  This intermediate step i n the process o f  e v a l u a t i o n has been c a l l e d " c o n c e p t u a l i z i n g " by S c r i v e n ( S c r i v e n , 1974a, p. 1 4 2 ) . He sees  t h i s s t e p i n e v a l u a t i o n as a p r o c e s s o f  32 both " d e v i s i n g an a p p r o p r i a t e taxonomy and m e a s u r i n g t h e s p e c i f i c ' p e r f o r m a n c e i n terms o f t h e d i m e n s i o n s o f t h e taxonomy."  (Scriven,  1974a, p. 1 3 7 ) . I n Chapter I I I i t w i l l be shown how each s t a n d a r d c h o s e n i n t h e e v a l u a t i o n o f a t e a c h e r - t r a i n i n g program i s e l a b o r a t e d so a s t o a l l o w some r a t i n g o f t h e d e g r e e t o w h i c h t h e e v a l u a t u m , met that  standard.  2.60 L e v e l I l l — S p e c i f i c a t i o n o f t h e . C l a s s o f Comparison i n G r a d i n g In S-GRADING, t h e c l a s s o f c o m p a r i s o n i s some i d e a l form o f t h e evaluatum.  T a y l o r ' s example (1961, p. 9) o f t h e S-GRADING o f  p a i n t i n g s ended ". . . i s [ t h e p a i n t i n g ] good as compared w i t h what a p a i n t i n g o f t h i s type s h o u l d b e ? " In t h e i n s t a n c e o f t h e S-GRADING o f a t e a c h e r - t r a i n i n g program t h e p a r a l l e l q u e s t i o n becomes: " I s t h e program good as compared w i t h what a t e a c h e r - t r a i n i n g program a t t h i s l e v e l s h o u l d be?" S - g r a d i n g , a s argued e a r l i e r , i s o n l y an i n t e r m e d i a t e s t e p i n t h e complete e v a l u a t i o n o f a p r o d u c t .  The s t r o n g e s t e v a l u a t i v e c o n c l u s i o n  which c a n be r e a l i z e d i s "Commendatory" (See T a b l e 1 ) . The c h o i c e o f S-GRADING o r RANKING i s an o p t i o n i n t h e f i r s t f o u r e v a l u a t i v e c o n c l u s i o n s o f T a b l e 1, and i s p r i m a r i l y based on c i r c u m s t a n c e .  If  t h e r e i s a r e a d i l y i d e n t i f i e d c o m p a r i s o n g r o u p ( s ) p r e s e n t , and i f the e v a l u a t o r has a c c e s s t o t h e c o m p a r i s o n g r o u p ( s ) e a r l y enough t o p l a n a p a r a l l e l e v a l u a t i o n scheme f o r t h e e v a l u a t a , then he s h o u l d p r o b a b l y use RANKING.  S c r i v e n (1967, p. 64) a r g u e s :  33 "Comparative e v a l u a t i o n s a r e o f t e n v e r y much e a s i e r t h a n noncomparative e v a l u a t i o n s , because we can o f t e n use t e s t s w h i c h y i e l d d i f f e r e n c e s i n s t e a d o f h a v i n g t o f i n d a n a b s o l u t e s c a l e and then e v e n t u a l l y compare t h e a b s o l u t e s c o r e s . " I f , on t h e o t h e r hand, t h e comparison  group(s) i s not e a s i l y  i d e n t i f i a b l e and/or t h e e v a l u a t o r cannot g a i n a c c e s s t o i t e a r l y enough t o p e r f o r m e v a l u a t i o n s on i t then S-GRADING i s t h e b e s t alternative i n the early stages.  However, t h e e v a l u a t o r s h o u l d be  c o n s i d e r i n g methods o f u s i n g , o r c r e a t i n g ( S c r i v e n , 1967, p. 6 9 ) , comparison  groups i f a n y t h i n g l i k e a complete e v a l u a t i o n i s t o be  accomplished.  2.70 L e v e l _IV--Determi n a t i o n o f t h e Good and Bad C h a r a c t e r i s t i c s o f t h e Eva 1 uatuhrrj• n"SSGRADING? T h i s p a r t i c u l a r s t e p i n a l o g i c a l e v a l u a t i o n c a n be broken down i n t o two s u b p r o c e s s e s ; t h e y a r e : a)  G a t h e r i n g d a t a on t h e c h a r a c t e r i s t i c s .  b)  Determining  i f t h e c h a r a c t e r i s t i c s a r e good o r bad.  2.71 G a t h e r i n g Data on t h e C h a r a c t e r i s t i c s This sub-process techniques to gather data.  involves using various  psychometric  In t h i s p a r t i c u l a r s t u d y , t h e g e n e r a l  t e c h n i q u e s used were: a L i k e r t a t t i t u d e s c a l e , q u e s t i o n n a i r e s , o b s e r v a t i o n , formal and i n f o r m a l i n t e r v i e w s , e x a m i n a t i o n s  of records  o f m e e t i n g s , and e x a m i n a t i o n s o f o t h e r i n t e r n a l e v a l u a t i o n s ( o v e r a l l and o f s t u d e n t t e a c h i n g ) .  M e t f e s s e l and M i c h a e l (1967. p. 937-943)  34 provided a very comprehensive l i s t of p o s s i b l e techniques f o r the e v a l u a t i o n o f school programs.  The t e c h n i q u e s  w i l l be e l a b o r a t e d i n C h a p t e r I I I , S e c t i o n  2.72 D e t e r m i n i n g  used i n t h i s s t u d y  3.51.  i f the C h a r a c t e r i s t i c s a r e Good o r Bad  Each c h a r a c t e r i s t i c , on which data has been c o l l e c t e d , i s j u d g e d as good, bad o r n e u t r a l (equal amounts o f goodness and b a d n e s s ) . T h i s judgement depends not o n l y on t h e p o i n t o f view b e i n g t a k e n , a l s o the value system i n the p a r t i c u l a r p o i n t . o f view.  but  Thus w h i l e a  c h a r a c t e r i s t i c w h i c h i s j u d g e d as good f r o m the i n t e l l e c t u a l p o i n t o f view, may be bad f r o m t h e economic p o i n t o f v i e w , i t i s a l s o the c a s e t h a t a c o u r s e i n home economics w h i c h d e a l t w i t h the p r e p a r a t i o n  of  pork f o r the t a b l e may be r e l i g i o u s l y q u i t e a c c e p t a b l e f o r an A n g l i c a n , but r e l i g i o u s l y r e p u g n a n t t o an o r t h o d o x  Jew.  The p r o b l e m s o f e v a l u a t o r and c l i e n t w o r k i n g a t  cross-purposes  due to c o n f l i c t i n g v a l u e systems or p o i n t s o f view a r e m a g n i f i e d international education.  in  The f o l l o w i n g q u o t e i s i n c l u d e d as a s c a l e d -  up, but p a r t i c u l a r l y good, example o f t h e s e  concerns:  "There a r e f r e q u e n t l y s e v e r a l i n t e r e s t s i n a s i n g l e d e v e l o p ment p r o j e c t , t h e i n t e r e s t o f t h e r e s p o n s i b l e o r g a n i z a t i o n (government agency, b u s i n e s s e n t e r p r i s e , p r i v a t e i n s t i t u t i o n , e t c . ) b e i n g , i n t h e o r y , paramount and u s u a l l y a l s o i n p r a c t i c e . I f i n t e r g o v e r n m e n t a l o r g a n i z a t i o n s o r f o r e i g n governments o r p r i v a t e f o u n d a t i o n s a l s o p a r t i c i p a t e i n t h e p r o j e c t , t h e y may be assumed t o have t h e i r o b j e c t i v e s to a c h i e v e , and t h e s e may o r may not be i d e n t i c a l w i t h t h o s e o f t h e r e s p o n s i b l e o r g a n i z a t i o n . Moreover, i n d i v i d u a l experts, although n o m i n a l l y , and u s u a l l y g o v e r n e d by the g o a l s o f e m p l o y i n g o r g a n i z a t i o n s , sometimes have t h e i r own and d i f f e r e n t o b j e c t i v e s ( s u c h as t e s t i n g out a pet t h e o r y , o r d e m o n s t r a t i n g the s u p e r i o r i t y o f a p a r t i c u l a r  35 method o r p r o d u c t ) . The p e o p l e l i k e l y t o be a f f e c t e d by a development p r o j e c t a l s o have t h e i r own g o a l s , which may o r may not be r e f l e c t e d i n t h e p o l i c i e s o f t h e i r government. A government may change, t o o , o r a new o f f i c i a l be a p p o i n t e d d u r i n g t h e l i f e o f a p r o j e c t , and as a r e s u l t d i f f e r e n t g o a l s may be e s t a b l i s h e d . " (Hayes, 1969, pp. 29-30) In a s i m i l a r v e i n , S c r i v e n (1974a, p. 138)  wrote:  "The d e v e l o p e r o r t e a c h e r always has one c o n c e p t u a l i z a t i o n o f t h e d a t a i n mind: I f he ( o r she) f e e l s he has been s u c c e s s f u l , he o r she n a t u r a l l y sees t h e d a t a as ' d e m o n s t r a t i n g s u c c e s s i n a c h i e v i n g such and such g o a l s , ' and hence ( s i n c e t h o s e g o a l s would not have been adopted had i t not been f e l t t h e y had m e r i t ) t h e p r o j e c t i s j u d g e d m e r i t o r i o u s . But t h e r e a r e many o t h e r ways t o s e e most p r o j e c t s . I t i s j u s t as i m p o r t a n t f o r t h e e v a l u a t o r , as opposed to t h e d e v e l o p e r , t o r e t a i n an open mind a b o u t t h e l e g i t i m a c y o f r a d i c a l l y d i f f e r e n t i n t e r p r e t a t i o n s o f d a t a , as i t i s f o r t h e s c i e n t i s t r e a d i n g a r e s e a r c h paper i n which t h e a u t h o r p r o p o s e s t h a t c e r t a i n e x p e r i m e n t a l r e s u l t s support his theory." I t i s f o r t h e s e r e a s o n s t h a t t h e e v a l u a t o r must be s u r e t h a t t h e c l i e n t f o r t h e e v a l u a t i o n u n d e r s t a n d s the p o i n t o f view b e i n g t a k e n i n t h e e v a l u a t i o n ; and t h a t t h e v a l u e systems o f t h e c l i e n t and evaluator are f a i r l y congruent.  When''the e v a l u a t o r makes a judgement  t h a t a p a r t i c u l a r ^ c h a r a c t e r i s t i c , "C", i s "good" a c c o r d i n g to s t a n d a r d "S", t h e n he must have arguments which w i l l s u p p o r t h i s c o n t e n t i o n ; he must be a b l e t o p o i n t a t v a r i o u s e v i d e n c e s f o r "C" and, t h r o u g h them, v e r i f y h i s v a l u e judgement.  ( T a y l o r , 1961, p. 70)  In o t h e r words, the e v a l u a t o r must have c o m p l e t e r and v a l i d e v i d e n c e a b o u t a p a r t i c u l a r c h a r a c t e r i s t i c and he must have good r e a s o n ( g i v e n t h e e v i d e n c e ) f o r s a y i n g t h e c h a r a c t e r i s t i c i s good, bad o r n e u t r a l . I f t h e e v a l u a t o r i s c h a l l e n g e d on h i s judgement, he would f i r s t show t h a t h i s e v i d e n c e was complete and v a l i d  36 (verification).  I f he was s t i l l c h a l l e n g e d , he would then show t h a t  his r e a s o n s f o r j u d g i n g t h a t t h a t d a t a on t h a t c h a r a c t e r i s t i c meant t h e c h a r a c t e r i s t i c was good, bad o r n e u t r a l , were good r e a s o n s (validation). S c r i v e n (1974a, p. 139) has c a l l e d t h i s p r o c e s s , w h i c h l e a d s from t h e c o n c e p t u a l i z e d i n t e r m e d i a t e c o n c l u s i o n s to t h e e v e n t u a l c o n c l u s i o n s of worth or m e r i t , c r e d e n t i a l i n g .  2.80  L e v e l V-- The O v e r a l l Judgement o f Worth T h e r e a r e two f a c t o r s t h a t d e t e r m i n e whether an evaluatum  be graded as good or bad on t h e whole.  i s to  The f i r s t f a c t o r , i s t h e  degree to w h i c h t h e evaluatum f u l f i l l s a l l the s t a n d a r d s a p p l i e d t o it.  As T a y l o r (1961, p. 8 ) * w n t e s ' ;  "That depends on what s t a n d a r d s one i s a p p e a l i n g t o , . . . how c l e a r l y t h o s e s t a n d a r d s a r e d e f i n e d , t o what e x t e n t the degrees to w h i c h t h e y can be f u l f i l l e d a r e m e a s u r a b l e , and how " d i s t a n t " from r e a l i t y i s one's i d e a l . " The second f a c t o r i s t h e r e l a t i v e p r e c e d e n c e o f the When the evaluatum  i s e v a l u a t e d a c c o r d i n g to a s e t o f s t a n d a r d s ,  s t a n d a r d s may be r a n k e d i n o r d e r o f d e c r e a s i n g i m p o r t a n c e S^, S^, . . ., S  n  standards.  where  ( i . e . S-j,  i s the most i m p o r t a n t s t a n d a r d and S  the l e a s t i m p o r t a n t s t a n d a r d ) .  those  n  is  T h a t i s t o say, when we a r e making an  o v e r a l l judgement o f v a l u e , g r e a t e r weight i s g i v e n t o the f a c t t h a t the evaluatum f u l f i l l s S^ t h a it t o t h e f a c t t h a t i t f u l f i l l s S,,. the evaluatum  f u l f i l l s S-, and S  9  If  t o t h e same d e g r e e , i t s f u l f i l l m e n t  37  o f S-j adds more t o i t s o v e r a l l goodness t h a n i t s f u l f i l l m e n t o f S  2  ( T a y l o r , 1961, p. 1 3 ) .  CHAPTER I I I A p p l i c a t i o n o f the Model to The D e s i g n o f an A c t u a l E v a l u a t i o n , S t u d y 3.00  Introduction C h a p t e r T;hree a p p l i e s the model o f e v a l u a t i o n d e v e l o p e d i n  C h a p t e r Two to the d e s i g n o f an e v a l u a t i o n s t u d y o f an u n d e r g r a d u a t e t e a c h e r t r a i n i n g program. standards,  The c h a p t e r  i l l u s t r a t e s the c h o i c e o f  data c o l l e c t i o n on the s t a n d a r d s ,  c o n c l u s i o n can be r e a c h e d .  and how an  evaluative  Each s e c t i o n o f the C h a p t e r i l l u s t r a t e s  a p r a c t i c a l a p p l i c a t i o n o f the s i m i l a r l y headed, but t h e o r e t i c a l sections i n Chapter  3.10  Two..  E s t a b l i s h i n g the  3.11  Preconditions  The E v a l u a t u m The e v a l u a t u m i s an u n d e r g r a d u a t e t e a c h e r t r a i n i n g program  s u b s e q u e n t l y r e f e r r e d t o as Program B.  A d e s c r i p t i o n o f the Program  does not n e c e s s a r i l y p o r t r a y t h e Program as i t a c t u a l l y but r a t h e r as i t was i n t e n d e d o p e r a t i o n , the g e o g r a p h i c a l etc.  to o p e r a t e ,  operated  including a time scale of  l o c a t i o n , the p e r s o n n e l  requirements,  D e t e r m i n i n g the d e g r e e to which the i n t e n d e d program and  a c t u a l program were c o n g r u e n t i s a f u n c t i o n of the  evaluator.  The f o l l o w i n g i s a d e s c r i p t i o n o f the e v a l u a t u m based on m a t e r i a l s p r e p a r e d f o r the Program. 38  the  the  39 3.111  L e t t e r t o P o t e n t i a l S t u d e n t s f o r Program B.  Dear S t u d e n t : The F a c u l t y o f E d u c a t i o n has i n i t i a t e d s e v e r a l new programs f o r t h e p r o f e s s i o n a l t r a i n i n g o f t e a c h e r s . One o f t h e s e new programs, t i t l e d "Program B" i s d e s c r i b e d i n t h e accompanying l e t t e r . B r i e f l y , what we i n t e n d t o do i s have a group o f 25 s t u d e n t s spend t h e e n t i r e y e a r a t s c h o o l X, a Vancouver e l e m e n t a r y s c h o o l . A l l u n i v e r s i t y c o u r s e s and a l l s t u d e n t t e a c h i n g w i l l be u n d e r t a k e n t h e r e . The a d m i n i s t r a t i o n and s t a f f o f School X a n f f a c u l t y members from U.B.C. have c o o p e r a t i v e l y p l a n n e d t h e program and a r e v e r y e n t h u s i a s t i c about t h i s new a p p r o a c h t o t e a c h e r e d u c a t i o n . As a 3 r d y e a r t r a n s f e r s t u d e n t , y o u a r e e l i g i b l e t o be a p a r t o f t h i s new program. The program i s l i m i t e d t o 25 s t u d e n t s and we hope t o f i l l t h e s e c t i o n b e f o r e t h e end o f t h e Vancouver s c h o o l y e a r . I f y o u a r e i n t e r e s t e d p l e a s e c o n t a c t me i m m e d i a t e l y so a p r e l i m i n a r y meeting o r two may be a r r a n g e d . Sincerely,  Program B D i r e c t o r , Faculty of Education U n i v e r s i t y o f B.C. 3.112 Program D e s c r i p t i o n Program B w i l l be c o n d u c t e d e n t i r e l y i n a p a r t i c i p a t i n g e l e m e n t a r y s c h o o l , b u t s t u d e n t s w i l l be a b l e t o make u s e o f t h e U n i v e r s i t y l i b r a r i e s . I t has been agreed t h a t i t w i l l c o n t a i n : 1. 2. 3. 4. 5.  a heavy component o f a c t u a l c l a s s r o o m e x p e r i e n c e ( E d u c a t i o n 397) p r o f e s s i o n a l methods c o u r s e s a p s y c h o l o g i c a l component a f o u n d a t i o n s component a general elementary education course  The form which t h e c l a s s r o o m component w i l l t a k e w i l l depend on t h e s c h o o l s t a f f and t h e i r recommendations, b u t a f l e x i b l e , v a r i e d , t e a c h i n g program t h a t i n c l u d e s e x p e r i e n c e s a t s e v e r a l l e y e l s o f s c h o o l l i f e i s e n v i s i o n e d . The methods c o u r s e s w i l l be i n t e r e s t e d i n t h e s p e c i f i c r e q u i r e m e n t s o f each a r e a , t h e m a t e r i a l s a v a i l a b l e , and c u r r i c u l u m development. D u r i n g t h e t i m e d e v o t e d t o p r o f e s s i o n a l  40  methods the following courses w i l l be covered: Ed. 304  Language Arts  3 units  Professor A  Ed. 321  Science Education  1-1/2 units  Professor B  Ed. 323  Art Education  1 unit  Professor C  Ed. 325  Physical  1 unit  Professor D  Ed. 370  Mathematics  1-1/2 units  Professor E  Ed. 371  Mathematics.Education  1-1/2 units  Professor E  Education  The psychology component w i l l be designed to increase the s e n s i t i v i t y of the pre-service teachers to the needs and requirements of children not as consumers of science, mathematics, or any other content area, but rather as human beings. This component is also designed to help the pre-service teachers understand some of the problems associated with learning and evaluation. To this end the theoretical material associated with Education 310 and 311 w i l l be covered. The foundations component w i l l be designed to accomplish three major tasks: 1. To enable pre-service teachers to make c l e a r , well-founded decisions as to what goals they, as teachers, ought to pursue. 2. To enable them to promote, in an e t h i c a l l y responsible manner, the moral development of their students. 3. To give them a clear understanding of the nature of the task of developing the a b i l i t y to think. To this end the theoretical material associated with Ed. 400 w i l l be covered. The general elementary education component (Education 449) w i l l contain many of the common elements of the methods courses. It w i l l also deal with much of what i s currently done in student teaching seminars at the University. Further i t w i l l deal with such topics as teaching strategies, classroom organization and c u r r i c u l a r planning.  3.113 Description of the offered courses in Program B (Taken from the Calendar of the University of B.C. 1974/75). Ed. 304: Curriculum and Instruction study of (a) the curriculum organization p a r t i c u l a r l y in the intermediate grades; in these subjects and grades. Ed. 321: Curriculum and Instruction study of (a) the curriculum organization grades; (b) techniques and strategies of these grades.  in the Language A r t s . - - A in the language arts (b) techniques of instruction in Elementary Science. - - A in science for the elementary instruction in science for  41 Ed. 323: C u r r i c u l u m and I n s t r u c t i o n i n A r t I . Ed. 325: C u r r i c u l u m and I n s t r u c t i o n i n P h y s i c a l E d u c a t i o n . -- A s t u d y o f (a) the c u r r i c u l u m o r g a n i z a t i o n i n p h y s i c a l e d u c a t i o n f o r t h e e l e m e n t a r y g r a d e s ; (b) t e c h n i q u e s o f i n s t r u c t i o n i n p h y s i c a l e d u c a t i o n for these grades. Ed. 370: Mathematics f o r E l e m e n t a r y T e a c h e r s . -- A s t u d y o f t h e method and s t r u c t u r e o f e l e m e n t a r y s c h o o l m a t h e m a t i c s as a d i s c i p l i n e . Ed. 371: Methods o f t e a c h i n g E l e m e n t a r y School Mathematics. -A s t u d y o f m a t e r i a l s , s k i l l s , s t r a t e g i e s and c u r r i c u l u m o r g a n i z a t i o n . Ed. 310: Growth and Development. -- R e s e a r c h as i t a p p l i e s t o t h e e l e m e n t a r y s c h o o l c h i l d . Not open t o s t u d e n t s who have t a k e n E d u c a t i o n 331. Ed. 311: The Nature and Measurement o f L e a r n i n g . — A s t u d y o f l e a r n i n g and t h e t e c h n i q u e s o f e v a l u a t i o n as t h e y a p p l y to t h e element a r y s c h o o l c h i l d . Not open t o s t u d e n t s who have t a k e n E d u c a t i o n 331. Ed. 400: P h i l o s o p h y o f E d u c a t i o n . -- An i n t r o d u c t o r y c o u r s e i n which c o n s i d e r a t i o n i s g i v e n t o t h e p h i l o s o p h i c a l f o u n d a t i o n s o f educat i o n and t o t h e p r a c t i c a l b e a r i n g s o f t h e o r y upon c u r r i c u l u m c o n t e n t and c l a s s r o o m p r a c t i c e i n our s c h o o l s .  3.12 The P o i n t o f View In C h a p t e r I I , S e c t i o n 2.32, t h e e i g h t b a s i c p o i n t s o f view as o u t l i n e d by T a y l o r (1961; p. 300) were g i v e n .  They a r e t h e m o r a l ,  the a e s t h e t i c , the i n t e l l e c t u a l , the r e l i g i o u s , the economic, the p o l i t i c a l , t h e l e g a l and t h e p o i n t o f view o f e t i q u e t t e o r custom. c o n s i d e r s t h e s e p o i n t s o f view to be b a s i c because: ". . always s u b j e c t and i s u s u a l l y them. Second, the sense t h a t and a c t i v i t i e s ( T a y l o r , 1961,  Taylor  "  :;  . t h e conduct o f any g i v e n i n d i v i d u a l i n a c u l t u r e i s t o a v a l u e system b e l o n g i n g to a t l e a s t one o f them s u b j e c t t o v a l u e systems b e l o n g i n g t o more than one o f t h e y a r e t h e d o m i n a n t p o i n t s o f view i n a c u l t u r e , i n t h e y s e t t h e v a l u e s o f the major s o c i a l i n s t i t u t i o n s which c a r r y on t h e c i v i l i z a t i o n o f t h e c u l t u r e . " p. 300)  M o r e o v e r , i n a d d i t i o n to the e i g h t b a s i c p o i n t s o f view t h e r e a r e many n o n b a s i c p o i n t s o f view c o r r e s p o n d i n g t o p a r t i c u l a r group interests.  42 " . . . i t i s a p p r o p r i a t e to j u d g e t h e o r g a n i z a t i o n i n terms o f i t s group i n t e r e s t , t h a t i s , i n terms o f t h e purposes o f the group as a whole (as d i s t i n c t f r o m t h e s e l f - i n t e r e s t o f each member o f t h e g r o u p ) . " ( T a y l o r , 1961, pp. 301-302) ' A f a c u l t y o f e d u c a t i o n e x i s t s f o r t h e c l e a r l y d e f i n e d purpose o f p r o d u c i n g good t e a c h e r s , ( u s i n g t h e b e s t t e c h n i q u e s p o s s i b l e ) , and can l e g i t i m a t e l y be j u d g e d a c c o r d i n g to i t s a b i l i t y to p e r f o r m t h a t function.  S i m i l a r l y , so can any p a r t i c u l a r program o f t e a c h e r  educa-  t i o n i n t h e f a c u l t y . T h e r e f o r e , t h e e v a l u a t o r can e v a l u a t e a program o f t e a c h e r e d u c a t i o n from the i n s t i t u t i o n a l p o i n t o f view, t h a t i s , from t h e p o i n t o f view o f a f a c u l t y o f e d u c a t i o n .  T h i s p o i n t o f view  would encompass t h e l o g i s t i c s o f t e a c h e r e d u c a t i o n , such as o p p o r t u n i t i e s f o r p r a c t i s e t e a c h i n g , a v a i l a b l e methods c o u r s e , p r o f e s s o r s a v a i l a b l e to t e a c h t h e s t u d e n t s , f i n a n c i n g o f programs and so on. In o t h e r words, t h e e v a l u a t o r would use t h e s e c r i t e r i a f o r j u d g i n g t h e a c t i v i t i e s and p o l i c i e s o f a program o p e r a t i n g under t h e a u s p i c e s of a f a c u l t y of education. It i s a l s o necessary f o r the e y a l u a t o r to judge a teacher e d u c a t i o n program from t h e e d u c a t i o n a l p o i n t o f view.  I f we a r e educa-  t i n g p e o p l e t o be t e a c h e r s , r a t h e r t h a n s i m p l y t r a i n i n g them t o be t e a c h e r s , t h e n t h e p r a c t i s e s and p o l i c i e s o f a program o f t e a c h e r e d u c a t i o n must be j u d g e d i n terms o f t h e i r e d u c a t i o n a l worth. I t i s n e c e s s a r y t h a t a l l programs be e v a l u a t e d from t h e moral p o i n t o f view and t h e l e g a l p o i n t o f view "...  s i n c e [ t h e Program's]  purposes a r e to a c h i e v e c e r t a i n g o a l s o r t o p u r s u e c e r t a i n ends w i t h o u t v i o l a t i n g the-moral code o f t h e s o c i e t y and w i t h o u t b r e a k i n g S o c i e t y ' s laws." ( T a y l o r , 1961, p.  300)  43 T h e r e f o r e , Program B w i l l be e v a l u a t e d from t h e p o i n t o f view of the f a c u l t y of education (that i s , the i n s t i t u t i o n a l point o f v i e w ) , from t h e e d u c a t i o n a l p o i n t o f view, and f r o m t h e moral and l e g a l p o i n t s o f view. Some o t h e r b a s i c p o i n t s o f v i e w w h i c h c o u l d be c o n s i d e r e d i n t h e e v a l u a t i o n a r e t h e economic, t h e p o l i t i c a l , t h e i n t e l l e c t u a l , and the p o i n t o f view o f e t i q u e t t e o r custom.  However, Program c o s t s  were n o t a c o n s i d e r a t i o n i n t h i s e v a l u a t i o n so no e v a l u a t i o n was done f r o m t h e economic p o i n t o f view; t h e i n t e l l e c t u a l p o i n t o f view was i n c l u d e d i n t h e e d u c a t i o n a l p o i n t o f view; and t h e p o l i t i c a l and e t i q u e t t e o r custom p o i n t s o f view were beyond t h e scope o f t h i s t h e s i s .  3.13 G r a d i n g The method o f e v a l u a t i o n d e s c r i b e d by t h i s t h e s i s i s S-GRADING. ( s e e C h a p t e r I I , S e c t i o n 2.33).  T h a t i s , Program B i s n o t  d i r e c t l y compared t o o t h e r e x i s t i n g programs i n terms o f i t s g o a l s and p e r f o r m a n c e . R a t h e r , Program B i s g r a d e d a c c o r d i n g t o t h e d e g r e e to w h i c h i t f u l f i l l s a s e t o f s t a n d a r d s which a r e c o n s i d e r e d by t h e e v a l u a t o r t o be a p p r o p r i a t e t o t h e Program.  S-GRADING.is  reasonable  i n f o r m a t i v e e v a l u a t i o n , where t h e e v a l u a t o r must make judgements as t o t h e worth o f t h e program f o r t h e p u r p o s e s o f i m p r o v i n g i t and where c o m p a r i s o n w i t h o t h e r Programs i s n o t a m a j o r c o n c e r n . R e f e r r i n g t o T a b l e I i n C h a p t e r I I , i t was shown how t h e most c o m p r e h e n s i v e e v a l u a t i v e c o n c l u s i o n which c o u l d be r e a c h e d i n S-GRADING  44  (when c o s t s have not been t a k e n i n t o a c c o u n t ) , i s t h e " O v e r a l l Evaluative Conclusion."  T h e r e f o r e , t h e e v a l u a t i o n d e s c r i b e d by t h i s  t h e s i s attempted t o answer t h e f o l l o w i n g q u e s t i o n : "Did t r e a t m e n t X p r o v i d e d by Program B, have t h e e f f e c t o f p r o d u c i n g  undergraduate  s t u d e n t t e a c h e r s ( S ) , competent i n t h e c l a s s r o o m (Y) ( f o r t h i s l e v e l o f u n i v e r s i t y t r a i n i n g ) ; w i t h t h e p r e m i s e s t h a t X was the t r e a t m e n t and i t caused Y; t h e S's e n j o y e d , d e s i r e d and b e n e f i t t e d by Y, and Y had no harmful e f f e c t s on S's ( o r non S ' s ) ? "  3.20 A d o p t i o n o f S t a n d a r d s I t i s u s e f u l t o l o o k a t a g e n e r a l l y agreed upon d e s c r i p t i o n o f e d u c a t i o n i n o r d e r to e s t a b l i s h t h e broad s t a n d a r d s t o be used i n t h i s e v a l u a t i o n . P e t e r s (1966, p. 3) p r o v i d e d t h i s d e f i n i t i o n when he wrote: " [ E d u c a t i o n ] i m p l i e s t h a t something w o r t h - w h i l e i s b e i n g o r has been i n t e n t i o n a l l y t r a n s m i t t e d i n a m o r a l l y a c c e p t a b l e manner." processes.  Thus, t h e r e a r e two c r i t e r i a f o r e v a l u a t i n g e d u c a t i o n a l The f i r s t i s t h e worth o f t h e p r o c e s s ( o r p r o d u c t ) g o a l s  and t h e second i s t h e " t r a n s m i s s i o n i n a m o r a l l y a c c e p t a b l e manner." "... to e d u c a t e someone i m p l i e s not o n l y some s o r t o f achievement, but a l s o one t h a t i s w o r t h - w h i l e . I t a l s o i m p l i e s t h a t the manner o f d o i n g t h i s s h o u l d not be m o r a l l y o b j e c t i o n a b l e . " ( P e t e r s , 1966, p. 4) I t i s t h i s e s s e n t i a l d u a l i t y o f end p r o d u c t and p r o c e s s which  has  been o f t e n o v e r l o o k e d i n e d u c a t i o n a l e v a l u a t i o n . S e r i y e n (1967, p. 51) wrote,  45  "One o f t h e r e a c t i o n s to t h e t h r e a t o f e v a l u a t i o n . . . was the extreme r e l a t i v i z a t i o n o f e v a l u a t i o n r e s e a r c h . The s l o g a n became: How w e l l does t h e c o u r s e a c h i e v e i t s g o a l s ? i n s t e a d o f How good i s t h e c o u r s e ? But i t i s o b v i o u s t h a t i f t h e g o a l s a r e n ' t worth a c h i e v i n g t h e n i t i s u n i n t e r e s t i n g how w e l l t h e y a r e a c h i e v e d . . . Thus e v a l u a t i o n p r o p e r must i n c l u d e as an equal p a r t n e r w i t h t h e measuring o f p e r formance a g a i n s t g o a l s , p r o c e d u r e s f o r t h e e v a l u a t i o n o f t h e g o a l s . " T h e r e f o r e , f o r t h e e v a l u a t i o n model d e v e l o p e d i n t h i s t h e s i s , s t a n d a r d s w i l l be used.  The f i r s t s t a n d a r d i s goal worth.  two  The  degree to which P r o g r a n r B f u l f i l l s o r f a i l s t o f u l f i l l t h i s s t a n d a r d i s dependent on the worth o f t h e Program's g o a l s . i s goal achievement.  The second s t a n d a r d  The degree to which Program B f u l f i l l s o r f a i l s  to f u l f i l l t h i s s t a n d a r d i s d e p e n d e n t on whether o r not t h e o p e r a t i o n o f t h e Program r e s u l t s i n t h e g o a l s b e i n g a c h i e v e d . There i s one r e s t r i c t i o n p l a c e d on t h e s t a n d a r d o f g o a l a c h i e v e - , ment.  T h a t i s , i t s degree o f f u l f i l l m e n t s h o u l d o n l y be d e t e r m i n e d i f  the g o a l s have been shown t o be  worthwhile.  3.21 S t a n d a r d 1: The Worth o f t h e Program G o a l s One a p p r o a c h t o e s t a b l i s h i n g t h e worth o f a s e t o f g o a l s i s to d e t e r m i n e t h e e x t e n t to which t h e y a r e based on d e f e n s i b l e needs and/or wants.  F o r some, d e f e n s i b l e needs a r e seen as a r i s i n g out o f  the d i s c r e p a n c y between a d e s i r e d s t a t e o f a f f a i r s i n e d u c a t i o n and the c u r r e n t s t a t e of a f f a i r s .  Both Popham (1969, p. 23) and  Provus  (1.972, p. 37) have a d v o c a t e d t h i s d i s c r e p a n c y as a measure o f e d u c a t i o n a l need.  Popham w r o t e t h a t d e s i r e d l e a r n e r outcome minus  c u r r e n t l e a r n e r s t a t u s e q u a l s an e d u c a t i o n a l need.  Provus,  similarly,  46  w r o t e t h a t t h e d i s c r e p a n c y between i d e a l i z e d performance, o f c i t i z e n s and a c t u a l p e r f o r m a n c e e q u a l s need.  However, t h i s does not  n e c e s s a r i l y d e t e r m i n e need i n e d u c a t i o n .  Suppose the " d e s i r e d l e a r n e r  outcomes" or " i d e a l i z e d p e r f o r m a n c e o f c i t i z e n s " have not been shown ( i n the c o n t e x t o f a p a r t i c u l a r e d u c a t i o n a l e n v i r o n m e n t ) t o be w o r t h w h i l e g o a l s ; o r , suppose they a r e i n f a c t w o r t h l e s s g o a l s ; i n t h a t c a s e , the d i s c r e p a n c y between c u r r e n t l e a r n e r s t a t u s and t h o s e c o u l d not be d e s c r i b e d as an e d u c a t i o n a l be c o u n t e r e d ,  need.  goals  T h i s argument c o u l d  o f c o u r s e , by s a y i n g t h a t d e s i r e d l e a r n e r outcomes  would never be d e s i r e d u n l e s s t h e y were w o r t h w h i l e .  T h i s can be  shown t o be f a l l a c i o u s by u s i n g the " o p e n - q u e s t i o n argument." F r a n k e n a (1973, p. 99) wrote about t h i s t y p e o f argument: !  "Suppose t h a t a d e f i n i s t h o l d s t h a t "good" o r " r i g h t " means "having the p r o p e r t y P," f o r example, " b e i n g d e s i r e d " o r " b e i n g c o n d u c i v e to t h e g r e a t e s t g e n e r a l h a p p i n e s s . " Then,."the argument i s t h a t we may a g r e e t h a t something has P, and y e t ask s i g n i f i c a n t l y , "But i s i t good?" or " I s i t r i g h t ? " T h a t i s , we can s e n s i b l y say, " T h i s has P, but i s i t good ( o r r i g h t ) ? " But i f t h e p r o p o s e d d e f i n i t i o n were c o r r e c t , then we c o u l d not say t h i s s e n s i b l y f o r i t would be e q u i v a l e n t to s a y i n g , " T h i s has P, but has i t P?" . . . T h e r e f o r e the d e f i n i t i o n cannot be c o r r e c t . " S i n c e we can ask the q u e s t i o n " I s a d e s i r e d l e a r n e r outcome a w o r t h w h i l e outcome?" then we c a n n o t assume t h a t " d e s i r e d " e q u a l s while".  "worth-  Need i n t h i s s e n s e o f b e i n g a d i s c r e p a n c y between a d e s i r e d  and a c t u a l s t a t e i s e s s e n t i a l l y an i n d i c a t i o n o f d e g r e e o f goal achievement without n e c e s s a r i l y seeing i f the goals are worth achieving. S c r i v e n has i d e n t i f i e d needs as b i o l o g i c a l . He w r o t e ( S c r i y e n ,  47 1974b, p. 13) "True, needs a s s e s s m e n t s i n v o l v e e s t a b l i s h i n g t h a t t h e p r o d u c t a c t u a l l y f a c i l i t a t e s s u r v i v a l , h e a l t h , or some o t h e r d e f e n s i b l e need t h a t i s not a d e q u a t e l y s e r v i c e d . . ." However, i n e d u c a t i o n t h e c r i t e r i a o f s u r v i v a l e t c . , f o r e s t a b l i s h i n g t h e worth o f t h e program's g o a l s , i s o f t e n t o o r e s t r i c t i v e . As Komisar w r o t e (Komi'sar, ,1961, ,p. 3 7 ) : " S u r v i v a l ... i s too m i s e r l y a c o n c e p t i o n , s i n c e t h e r e a r e d e f i c i e n c i e s which a r o u s e us w i t h o u t t h e t h r e a t o f d e a t h ( e . g . affection)." To d e t e r m i n e i f t h e g o a l s o f t h e program, o r t h e d e s i r e d l e a r n e r outcomes o f t h e program a r e w o r t h w h i l e r e q u i r e s v a l i d a t i o n o f t h e g o a l s o r outcomes.  T h i s i s . a c c o m p l i s h e d by showing: ( T a y l o r , 1961,  p. 84-85, D a n i e l s , 1971, p. 1 2 ) . 1)  That f u l f i l l i n g t h e g o a l s e n a b l e s one t o f u l f i l l  higher  goals. 2)  T h a t f u l f i l l i n g t h e g o a l s does not c o n f l i c t , w i t h a n o t h e r  p r i n c i p l e o f our way o f l i f e which i s o f importance t o us.  and  3)  That f u l f i l l i n g the g o a l s l e a d s to b e n e f i c i a l consequences,  4)  T h a t t h e r e a r e no grounds f o r making an e x c e p t i o n t o t h e  goals i n this s i t u a t i o n . S c r i v e n (1966a) o f f e r e d t h e f o l l o w i n g s i m i l a r method f o r d e t e r m i n i n g i f something 1)  i s worth doing.:  I f something w i l l b r i n g about a s t a t e o f a f f a i r s t h a t p e o p l e  48  v a l u e , t h a t i s a good prima f a c i e r e a s o n f o r d o i n g i t . 2)  I f t h e r e a r e prima f a c i e r e a s o n s f o r d o i n g something  and  none a g a i n s t , we s h o u l d do i t . 3)  I f t h e r e i s a c o n f l i c t o f s u p p o r t a b l e prima f a c i e r e a s o n s ,  appeal must be made t o a g e n e r a l moral  principle.  3.22 S t a n d a r d 2: Goal A c h i e v e m e n t Goal a c h i e v e m e n t i s a measure o f t h e e f f e c t i v e n e s s o f t r a n s m i t t i n g something worthwhile g o a l s .  i n a m o r a l l y a c c e p t a b l e manner t o a c h i e v e t h e  In o t h e r words i t i s a c h e c k . o n t h e  o f a p a r t i c u l a r evaluatum  performance  i n a c h i e v i n g the w o r t h w h i l e g o a l s .  Moral  c o n s i d e r a t i o n s , i n c o n s i d e r i n g t h e s t a n d a r d o f g o a l a c h i e v e m e n t , must t a k e p r e c e d e n c e o v e r any r a t i n g based on p e r f o r m a n c e .  I f t h e program  i s immoral then i t s h o u l d be stopped r e g a r d l e s s o f how w e l l i t a p p e a r s to be a c h i e v i n g t h e g o a l s . I t i s t h i s c h e c k on m o r a l i t y w h i c h i s o f t e n n e g l e c t e d i n t h e e v a l u a t i o n o f e d u c a t i o n a l p r o d u c t s and p r o p o s a l s .  It i s a useful  e x e r c i s e to e v a l u a t e J o n a t h a n S w i f t ' s (1667-1745) "A Modest P r o p o s a l f o r P r e v e n t i n g the C h i l d r e n o f Poor P e o p l e from Being a Burthen t o t h e i r P a r e n t s o r C o u n t r y , and f o r Making them B e n e f i c i a l t o the P u b l i c , " u s i n g t h e m a j o r e v a l u a t i o n m o d e l s , to i l l u s t r a t e the consequences o f o m i t t i n g a c h e c k on m o r a l i t y . f i n d the proposal q u i t e acceptable.  Most o f t h e models would  T h i s i s not t o s u g g e s t t h a t  c a n n a b i l i s m would be o v e r l o o k e d i n t h e e v a l u a t i o n o f an e d u c a t i o n a l  49 p r o d u c t ; b u t , o t h e r t r a n s g r e s s i o n s such as i n d o c t r i n a t i o n might  be  c o n v e n i e n t l y i g n o r e d o r not l o o k e d f o r , u n l e s s t h e r e i s an e x p l i c i t s t a t e m e n t r e q u i r i n g a m o r a l i t y check.  3.30 O p e r a t i o n a l C l a r i f i c a t i o n o f S t a n d a r d s In t h i s s e c t i o n each s t a n d a r d w i l l be e x p l i c a t e d i n t h e terms used to e v a l u a t e t h e t e a c h e r t r a i n i n g program, Program B. 3.31 S t a n d a r d 1: The Worth o f Program B's G o a l s The g o a l s o f t h e program can be d e r i v e d from t h e t y p e o f program b e i n g e v a l u a t e d .  The e v a l u a t o r does not l o o k a t the program's  p l a n s o r performance to d e t e r m i n e i f i t s g o a l s a r e w o r t h w h i l e ; i n s t e a d he l o o k s a t t h e c h a r a c t e r i s t i c s o f t h e o v e r a l l program g o a l s and v a l i d a t e s them as b e i n g o r not being w o r t h w h i l e .  Program B i s a  p a r t i c u l a r method o f t r a i n i n g t e a c h e r s , but r e g a r d l e s s o f t h e t e c h n i q u e s used, t h e u l t i m a t e goal o f t h e Program i s to t r a i n p e o p l e t o be good t e a c h e r s .  I t i s t h a t goal which i s v a l i d a t e d .  As T a y l o r (1961, p. 301-302) s u g g e s t e d , i f you a r e e v a l u a t i n g the a c t i v i t i e s and p o l i c i e s o f a group i n terms o f t h a t group's i n t e r e s t then you would e v a l u a t e t h e worth o f t h e g o a l s and the d e g r e e o f g o a l achievement  as t h e y p e r t a i n e d t o o n l y t h a t group.  The p o i n t o f view to be adopted would be t h e p o i n t o f view o f t h e i n s t i t u t i o n o r , i n t h i s c a s e , t h e f a c u l t y o f e d u c a t i o n under whose a u s p i c e s Program B i s f u n c t i o n i n g . On t h e o t h e r hand, you can e v a l u a t e  50  Program B o u t s i d e o f t h e g r o u p ' s framework.  I t i s now l e g i t i m a t e t o  ask how t h e Program's a c t i v i t i e s and p o l i c i e s conform t o t h e r u l e s and s t a n d a r d s o f an e n l i g h t e n e d p u b l i c . The p o i n t o f yiew t o be adopted would be t h e e d u c a t i o n a l p o i n t o f v i e w . . Program B, a s d e s c r i b e d i n t h e S e c t i o n 3.11 (The Evaluatum) d i d not have a p a r t i c u l a r l y d e t a i l e d l i s t o f g o a l s ; t h e r e f o r e a s e t o f w o r t h w h i l e g o a l s was c o n s t r u c t e d u s i n g a w e l l t h o u g h t - o u t model o f teacher education.  T h i s model was s u p p l i e d by Broudy (1965).  In  h i s a r t i c l e he i d e n t i f i e d t h e t e a c h e r a s b e i n g a member o f t h e e d u c a t i o n a l p r o f e s s i o n and as a s p e c i a l i s t . As a member o f t h e e d u c a t i o n a l p r o f e s s i o n a p o t e n t i a l t e a c h e r requires: (1)  B a s i c c o u r s e s i n h i s t o r y , p h i l o s o p h y , p s y c h o l o g y and t h e  social sciences, as part o f h i s general (2)  education.  Study o f t h e h i s t o r i c a l , p h i l o s o p h i c a l , p s y c h o l o g i c a l and  sociological foundations o f educational p o l i c y , curriculum, organizat i o n and s u p p o r t and s t r a t e g i e s o f t e a c h i n g and l e a r n i n g . The t e a c h e r as a s p e c i a l i s t r e q u i r e s : (3)  The h i s t o r i c a l , p h i l o s o p h i c a l , p s y c h o l o g i c a l and s o c i o l o g i c a l  foundations o f the s p e c i a l t y . (4)  Study o f s u b j e c t m a t t e r over and above t h a t t a k e n i n g e n e r a l  e d u c a t i o n a s needed f o r t h e s p e c i a l t y . (5)  L a b o r a t o r y e x e r c i s e s : a b s t r a c t e d and o f t e n  schematized  samples o f a whole c l a s s o f r e a l t a s k s such a s a made-up g r o u p i n g  51 t a s k , a m a k e - b e l i e v e t e s t , a d e m o n s t r a t i o n , e t c . Broudy r e f e r s t o t h i s as l a b o r a t o r y e x e r c i s e s . (6)  C l i n i c a l experience; e s p e c i a l l y constructed s i t u a t i o n s to  d e m o n s t r a t e a c t u a l examples o f s t u d i e d t h e o r y . c a s e o f t h e slow l e a r n e r o r d i s c i p l i n e (7)  F o r example, t h e c l a s s i c  problem.  I n t e r n s h i p : t h e s t u d e n t - t e a c h e r spends time working on a r e a l  t a s k under a minimum o f s u p e r v i s i o n , i . e . t h e p r a c t i c e - t e a c h i n g segments of teacher education. (8)  R e s e a r c h : The t e a c h e r ought t o be a consumer o f r e s e a r c h and  may be a p r o d u c e r o f r e s e a r c h .  T h e r e f o r e he s h o u l d be i n t r o d u c e d t o  to h i s methods, canons, s t a t u s and p r o s p e r i t y .  The f i r s t f o u r r e q u i r e m e n t s r e f e r t o t h e f o u n d a t i o n s o f t h e e d u c a t i o n a l f i e l d and a r e a o f s p e c i a l i z a t i o n i n t h i s f i e l d . they c o n s t i t u t e educational goals o f a f a c u l t y o f education.  As such Require-  ments (5) - (7) r e f e r t o t h e t e c h n o l o g y ( i . e . t h e way and means) o f a p p l y i n g t h e o r y t o p r a c t i c e . Thus, t h e y c a n be seen as i n s t i t u t i o n a l g o a l s , because t h e y p r i m a r i l y s e r v e t h e i n t e r e s t s o f t h e i n s t i t u t i o n (e.g. improving classroom p r a c t i c e ) . The technology i s not unique t o 'education', but i t i s o f i n t e r e s t to the f i e l d o f education.  Research,  r e q u i r e m e n t 8, i s an i n s t i t u t i o n a l goal i n t h a t i t s e r v e s t h e i n t e r e s t s of the f i e l d . Broudy's paper, as b r i e f l y d e s c r i b e d aboye, forms t h e b a s i s f o r d e t e r m i n i n g t h e degree t o which Program B has w o r t h w h i l e g o a l s from t h e e d u c a t i o n a l and i n s t i t u t i o n a l p o i n t s o f view.  52 I t i s i m p o r t a n t t o p o i n t o u t t h a t Program B i s o n l y one y e a r o f a f o u r y e a r t e a c h e r t r a i n i n g program.  T h e r e f o r e p a r t s o f Broudy's  c r i t e r i a w i l l be f u l f i l l e d e i t h e r b e f o r e o r a f t e r Program B.  This  would be t a k e n i n t o c o n s i d e r a t i o n i n t h e f i n a l , o v e r a l l e v a l u a t i o n o f t h e worth o f t h e program.  3.32 S t a n d a r d 2: The d e g r e e o f Goal A c h i e v e m e n t Assuming t h a t t h e g o a l s o f a program have been i d e n t i f i e d and a r e w o r t h w h i l e , t h e n t h e e v a l u a t o r w o u l d a t t e m p t t o d e t e r m i n e i f t h e program i s a c h i e v i n g t h e g o a l s .  O f t e n , t h e program d e y e l o p e r s  w i l l have w r i t t e n o u t , i n g r e a t d e t a i l , how t h e y hope t o a c h i e v e t h e (worthwhile) goals.  I t i s u s e f u l f o r t h e e v a l u a t o r t o examine t h e s e  w r i t t e n i n t e n t s to check t h e i r v a l i d i t y .  Some r e a s o n s f o r t h i s , a s  S c r i v e n (1974a, p. 154) p o i n t s o u t c o u l d be: (1) new programs on t h e market c o u l d make a s w i t c h t o them w o r t h w h i l e . (2) The p e r f o r m a n c e e v i d e n c e o f t h e e x i s t i n g program, o r o t h e r s -£0 w h i c h i t was compared, may have changed.  (3) U n a n t i c i p a t e d d i f f i c u l t i e s ( p o l i t i c a l , m o r a l ,  economic) may a ^ i s e which have n o t o c c u r r e d i n o t h e r a l t e r n a t e programs o r methods (4) t h e o r i g i n a l d e c i s i o n t o r u n t h e program may have been wrong due t o p o o r d a t a o r p o o r l o g i c . I f t h i s i s done, t h e n t h e n e x t s t e p i s t o i d e n t i f y a r e a s o f i n v e s t i g a t i o n which w i l l p r o v i d e t h e e v a l u a t o r w i t h t h e d a t a n e c e s s a r y t o d e t e r m i n e whether t h e g o a l s a r e b e i n g a c h i e v e d . S c r i v e n (1974b) has c o n t r i b u t e d a g r e a t deal t o t h i s end by d e v e l o p i n g  53 a c h e c k l i s t which can be used to e v a l u a t e p r o d u c t s , p r o d u c e r s and proposals.  He s u p p o r t s t h e v a l i d i t y . a n d u t i l i t y o f t h e c h e c k l i s t by  making t h e f o l l o w i n g f o u r p o i n t s ( S c r i v e n , 1974b, p. 8-9): (1)  E v e r y c h e c k p o i n t has a c l e a r a p r i o r i r a t i o n a l e . F a i l u r e  t o meet any one o f t h e c h e c k p o i n t s i m m e d i a t e l y c a u s e s doubt about t h e q u a l i t y of the e v a l u a t i o n . (2)  M e d i c a l and i n d u s t r i a l r o u t i n e l y p a s s , and o f t e n a r e r e -  quired to pass, every checkpoint. (3)  The c h e c k l i s t has been d e v e l o p e d out o f t h e P r o d u c t Review  P a n e l s o f 1971-72 and 1972-73 done f o r t h e N a t i o n a l C e n t e r f o r E d u c a t i o n a l Communication, on s u b c o n t r a c t t o t h e E d u c a t i o n a l T e s t i n g Service. (4)  I t has been used by s e y e r a l hundred s c h o o l a d m i n i s t r a t o r s  i n e v a l u a t i n g g r a d u a t e programs, and by s t u d e n t s i n t h e e v a l u a t i o n t r a i n i n g seminar a t t h e U n i v e r s i t y o f C a l i f o r n i a , B e r k e l e y .  They  f r e q u e n t l y s t a t e t h a t i t i s more v a l u e t o them d o i n g e v a l u a t i o n s than anything e l s e a v a i l a b l e i n the l i t e r a t u r e .  Each c h e c k p o i n t w i l l be d e s c r i b e d i n g e n e r a l terms, and i n terms o f Program B. the checkpoint.  T h i s d e s c r i p t i o n w i l l be f o l l o w e d by summary v e r s i o n o f The summary v e r s i o n c o n s i s t s o f a m e t h o d o l o g i c a l  check and a s u b s t a n t i v e check. The m e t h o d o l o g i c a l ' p a r t i s a check on whether d a t a i s a v a i l a b l e ( o r whether i t can be o b t a i n e d ) , i t s s o u r c e , and t h e q u a l i t y o f t h a t  54  data.  The s u b s t a n t i v e p a r t i s a degree o f f u l f i l l m e n t s c a l e . I t  i n d i c a t e s how w e l l t h e c h e c k p o i n t has been met.  In o r d e r f o r an  evaluatum t o r e c e i v e a h i g h r a t i n g on a p a r t i c u l a r c h e c k p o i n t t h e r e must be good e v i d e n c e o f good p e r f o r m a n c e ; and i t i n d i c a t e s good performance  i . e . the data i s complete,  by t h e evaluatum.  A low s c o r e on a  c h e c k p o i n t i m p l i e s t h a t e i t h e r good evidencef o r good p e r f o r m a n c e o r both i s l a c k i n g . " I t does n o t r e q u i r e , f o r example, t h a t t h e r e i s good e v i d e n c e o f bad performance, f o r o t h e r w i s e p r o d u c t s which t u r n e d i n no d a t a would do b e t t e r t h a n t h o s e t h a t were known t o f a r e b a d l y . " ( S c r i v e n , 1974b, p. 22)  3.33 The C h e c k l i s t f o r D e t e r m i n i n g Good  Achievement  3.331 C h e c k p o i n t 1: MARKET ( D i s s e r n i n a b i l i t y ) T h i s i s n o t an economic c h e c k p o i n t .  It requires that,  given worthwhile goals, there a r e dissemination plans that ensure consumers f o r t h e e d u c a t i o n a l p r o d u c t .  I t would c o n s i d e r t h e c l a r i t y ,  f e a s i b i l i t y and i n g e n u i t y ( f o r u t i l i z i n g a v a i l a b l e r e s o u r c e s ) o f t h e implementation  plans.  To f u l f i l l t h i s c h e c k p o i n t t o a h i g h d e g r e e Program B must show t h a t i t has s p e c i f i c , f e a s i b l e p l a n s t o p r o v i d e t h e s t u d e n t s w i t h ( f o l l o w i n g Broudy's a)  (1965) c r i t e r i a ) :  The h i s t o r i c a l , p h i l o s o p h i c a l , p s y c h o l o g i c a l and s o c i e t a l  foundations o f the s p e c i a l t i e s .  55  b)  The p r o f e s s i o n a l c o n t e n t o f t h e s p e c i a l t i e s over and above  that taken i n previous courses. c)  L a b o r a t o r y e x e r c i s e s , c l i n i c a l e x e r c i s e s and i n t e r n s h i p s .  d)  Methods o f p r o c e d u r e f o l l o w e d i n p r a c t i s e , p r o f e s s i o n a l  t a s k s ( g r a d i n g , m a r k i n g , making t e s t s ) , and methods o f d e a l i n g w i t h t h e a p p a r a t u s and m a t e r i a l s o f t e a c h i n g . In a d d i t i o n , t h e r e must be s t u d e n t s who want t o e n r o l l i n Program B and p r o c e d u r e s t o e n r o l l them. teach the courses.  T h e r e must be i n s t r u c t o r s a b l e to  A s c h o o l must be a y a i l a b l e w i t h a s t a f f w i l l i n g t o  t a k e on Program B; p l u s a m y r i a d o f minor d e t a i l s such as books and m a t e r i a l s b e i n g p r e s e n t , f i l m s , a c c e s s to l i b r a r i e s , a r e a s f o r s t u d y , e t c . , which, c u m u l a t i v e l y , can haye c o n s i d e r a b l e impact on t h e s u c c e s s of a  program. Summary y e r s i o n o f Market  Methodological Evidence Dissemination plan: clarity feasibility ingenuity economy Size Importance Other  Checkpoint.  Substantive Fulfillment characteristics V e r y l a r g e and/or v e r y i m p o r t a n t market w i l l be r e a c h e d L a r g e and/or i m p o r t a n t market w i l l be reached S i g n i f i c a n t market w i l l p r o b a b l y be reached P o s s i b l e , but not p r o b a b l e , t h a t a s i g n i f i c a n t market w i l l be r e a c h e d Inadequate e v i d e n c e to s u g g e s t t h a t a s i g n i f i c a n t market w i l l be r e a c h e d  3.332 C h e c k p o i n t 2: TRUE FIELD TRIALS. How t y p i c a l a r e t h e p e o p l e , t h e s e t t i n g , t h e time  rating 4 3 2 1 0  56  a l l o t t e d to t h e f i n a l v e r s i o n o f t h e program? " I t i s y e r y t e m p t i n g t o t h i n k one c a n e x t r a p o l a t e from f i e l d t r i a l s w i t h v o l u n t e e r s c h o o l s . . . but t h i s has f r e q u e n t l y proved unsound . . . i n a c t u a l p r a c t i s e , d e a d l i n e s , overcommitment, and u n d e r f i n a n c i n g combine t o r e n d e r a l m o s t a l l p r o d u c t s d e f i c i e n t on t h i s c h e c k p o i n t . " ( S c r i v e n , 1974b, p. 11) The p a r a l l e l i n s t a t i s t i c s i s t h e c o n c e p t o f g e n e r a l i z a b i 1 i t y . I f Program B i s t o o b t a i n a h i g h s c o r e on t h i s c h e c k p o i n t , t h e n i t must be-shown t h a t t h e s c h o o l , i n w h i c h t h e program was h e l d , was not p a r t i c u l a r l y a t y p i c a l .  T h i s i s t r u e r e g a r d l e s s o f whether t h e  s c h o o l was e x c e p t i o n a l l y good o r e x c e p t i o n a l l y bad.  I f i t was  e x c e p t i o n a l l y good i t i s o b v i o u s t h a t t h e program p r o b a b l y r e c e i v e d more o f i t s s h a r e o f h e l p t o be a good program.  I f i t was e x c e p t i o n -  a l l y bad, t h e n t h e t e m p t a t i o n i s to s a y : " I f i t can work under t h o s e t e r r i b l e c o n d i t i o n s , i t c a n work anywhere!"  But t h a t i s n o t n e c e s s a r i l y  the c a s e , u n l e s s i t has been shown t h a t i t c a n work anywhere. The s a f e s t b e t i s t o r u n t h e program i n an "average" s c h o o l and make a d j u s t m e n t s t o t h e program's d e v e l o p e d framework i f i t s u b s e q u e n t l y ends up i n a v e r y bad o r v e r y good s i t u a t i o n . Not o n l y s h o u l d t h e s c h o o l s e t t i n g be " a v e r a g e " , but so s h o u l d t h e u n i v e r s i t y s t a f f and t h e s t u d e n t t e a c h e r s .  If portability of  Program B i s one o f i t s a i m s , t h e n t h e t r i a l v e r s i o n s s h o u l d n o t be composed o f e x c e p t i o n a l l y good o r bad s t a f f o r e x c e p t i o n a l l y good o r bad s t u d e n t t e a c h e r s .  The a r g u m e n t s . a g a i n s t  e x c e p t i o n a l l y good o r bad  s t a f f and s t u d e n t s p a r a l l e l t h o s e g i v e n i n t h e p r e c e e d i n g  paragraph  57 for the school. Summary v e r s i o n o f TRUE FIELD TRIALS C h e c k p o i n t Methodological Evidence Final version? Typical user? Typical aid? Typical setting? T y p i c a l time-frame?  Substantive Fulfillment Characteristics  Rating  Perfectly typical Minor d i f f e r e n c e s Reasonable b e t f o r g e n e r a l i z a t i o n S e r i o u s weakness Relevance unclear  Comment: Checks s h o u l d be made on a l l t h e e v i d e n c e s o u r c e s .  4 3 2 1 0 This will  p r o v i d e a good base f o r t h e r a t i n g g i v e n . 3.333 C h e c k p o i n t 3: TRUE. CONSUMER This checkpoint o u t l i n e s gains expected o f yarious groups (consumers) i n v o l v e d i n Program B. The t y p e s o f g a i n s w i l l u s u a l l y depend on t h e p o i n t o f view taken i n t h e e v a l u a t i o n . D e c i d i n g on t h e data r e q u i r e s a c l e a r u n d e r s t a n d i n g o f t h e e v a l u a t i v e f u n c t i o n and " . . .  which a u d i e n c e s i t i s a d d r e s s e d t o , commissioned  by a n d - - r e g a r d l e s s o f t h e s e two c o n s i d e r a t i o n s — r e s p o n s i b l e t o . " ( S c r i v e n , 1974b; p. 1 2 ) . The d i f f e r e n t groups o f consumers may be i n t e r e s t e d i n d i f f e r e n t aspects o f the p r o j e c t , and, i n order to p r o v i d e d a t a f o r a l l t h e r e l e v a n t g r o u p s , each a s p e c t s h o u l d be examined. o f view.  T h i s i s e v a l u a t i n g t h e program from t h e p r u d e n t i a l p o i n t T h a t i s , each consumer o f Program B would i n e f f e c t be  a s k i n g "What b e n e f i c i a l e f f e c t s does t h i s program have on me?" To f u l f i l l t h e r e q u i r e m e n t s o f t h i s c h e c k p o i n t , t h e e v a l u a t o r  58  would d e t e r m i n e t h e g o a l s , w i t h r e s p e c t t o t h e evaluatum, o f each consumer.  He would t h e n g a t h e r d a t a on t h e s e g o a l s and f e e d them  back t o t h e consumer f o r h i s a p p r a i s a l o f t h e b e n e f i c i a l a n d / o r harmful e f f e c t s ( w i t h r e s p e c t t o t h e consumer's g o a l s ) . The e v a l u a t o r , however, must have o v e r r i d i n g judgement s i n c e he i s t h e one who t a k e s responsibility f o r the conclusions.  I t i s i m p o r t a n t t h a t t h e d a t a be  o b j e c t i v e l y g a t h e r e d so t h a t t h e consumer c a n make a v e r i f i a b l e judgement. Though t h e e v a l u a t o r t r i e s t o p r o v i d e t h e n e c e s s a r y d a t a , i t i s o b v i o u s l y n o t p o s s i b l e t o p r o v i d e " B e s t Buy" i n f o r m a t i o n i f t h e e v a l u a t i o n i s " O v e r a l l " ( s e e T a b l e I ) . L i m i t a t i o n s o f t h i s s o r t must be p o i n t e d o u t e a r l y i n e v a l u a t i o n , and r e v i s e d up o r down as t h e e v a l uation proceeds. The d i f f e r e n t consumers i n t h i s s t u d y a r e : a)  The Dean o f E d u c a t i o n and h i s r e p r e s e n t a t i v e s .  b)  The U.B.C. s t a f f i n v o l v e d i n Program B.  c)  The S t u d e n t t e a c h e r s i n Program B.  d)  The P r i n c i p a l o f t h e s c h o o l i n w h i c h Program B o p e r a t e d .  e)  The T e a c h e r s i n t h e above s c h o o l .  f)  The p u p i l s i n t h e above s c h o o l .  g)  The School Board.  h)  F u t u r e consumers o f t h e program.  The most i m p o r t a n t consumers f o r t h i s s t a g e o f t h e e v a l u a t i o n a r e t h e s t u d e n t t e a c h e r s i n Program B, and U.B.C. S t a f f , and u l t i m a t e l y  59 t h e p u p i l s i n t h e s c h o o l b e i n g exposed t o t h e s t u d e n t t e a c h e r s . Summary v e r s i o n o f TRUE CONSUMER c h e c k p o i n t . Methodological Evidence District? Principal? Teacher? Student? Taxpayer? Other?  Substantive Fulfillmentcharacteristics F u l l d a t a on a l l F a i r data on a l l Good d a t a on t h e Weak d a t a on t h e Only s p e c u l a t i o n "consumers"  Rating  r e l e v a n t "consumers" r e l e v a n t "consumers" most i m p o r t a n t "consumers" most i m p o r t a n t "consumers" about most i m p o r t a n t  4 3 2 1 0  3.334 C h e c k p o i n t 4:. LONG TERM This checkpoint o u t l i n e s follow-up plans.  These plans  a r e p a r t i c u l a r l y i m p o r t a n t when t h e p r o j e c t b e i n g e v a l u a t e d c a n be e x p e c t e d t o have l o n g range e f f e c t s — a s i s t h e d e s i r a b l e c a s e f o r most e d u c a t i o n a l p r o j e c t s . The f o l l o w - u p c a n a l s o i d e n t i f y any good o r bad s i d e e f f e c t s which may t a k e c o n s i d e r a b l e time t o s u r f a c e . Summary, v e r s i o n o f LQNGTERM, c h e c k p o i n t Methodological Evidence Week t o month later Month t o y e a r later Year t o few years later On-job o r l i f e space sample  Substantive Fulfillment characteristics Good d i r e c t e v i d e n c e about t h e e f f e c t s a t t i m e s needed. Some d i r e c t e v i d e n c e about t h e e f f e c t s a t times needed. Follow-up gives reasonable support t o a c o n c l u s i o n about e f f e c t s when rieeded Follow-up o r other data suggests a conclu s i o n about e f f e c t s when needed. U s e l e s s o r no f o l l o w - u p ; no o t h e r grounds for i n f e r r i n g long-term e f f e c t s .  Rating 4 3 2 1 0  3.335 C h e c k p o i n t 5: SIDE EFFECTS T h i s broad c h e c k p o i n t p r o v i d e s a g e n e r a l p a t t e r n f o r  60 a f r e e r a n g i n g s e a r c h f o r any good o r bad program e f f e c t s on t h e c o n sumers.  I t i s p a r t i c u l a r l y i m p o r t a n t t h a t the e v a l u a t o r makes h i s  s e a r c h under many d i f f e r e n t p o i n t s o f view. S c r i v e n (1967, pp. 77-80) has g i v e n examples o f t h e forms  these  s i d e - e f f e c t s may t a k e i n a new c u r r i c u l u m p r o j e c t . A p p l y i n g t h i s to t h e consumers o f Program B, t h e n examples o f s i d e e f f e c t s f o r each group a r e : 1)  The U.B.C. S t a f f a)  I n c r e a s e o r d e c r e a s e i n s t a t u s as a r e s u l t o f b e i n g c o n n e c t e d w i t h Program B. ( P r o m o t i o n ,  b)  Demotion).  Problems o r b e n e f i t s o f t r a v e l l i n g between t h e u n i v e r s i t y and t h e s c h o o l h o s t i n g Program B.  c)  The b e n e f i t s o r problems o f s p e n d i n g a l o t o f time on t h e Program; such a s : f a t i g u e , b e i n g a b l e t o d e v e l o p h i s / h e r own i d e a s , l o s s o f time f o r r e s e a r c h , e t c .  2)  The a)  Student-Teachers. I n c r e a s e o r d e c r e a s e i n s t a t u s as a r e s u l t o f b e i n g c o n n e c t e d w i t h Program B.  b)  Problems o r b e n e f i t s o f b e i n g removed from t h e U n i v e r s i t y s e t t i n g ( e . g . the l i b r a r y , o t h e r p r o f e s s o r s , s t u d e n t s i n other courses, etc.)  c)  F a t i g u e o r e n t h u s i a s m due t o h a v i n g to t e a c h p u p i l s i n t h e h o s t s c h o o l , as w e l l as h a v i n g to do academic a t t h e same t i m e .  work  61  d)  P r o b l e m s and b e n e f i t s (exchanging  e)  ideas,  being crowded, etc.).  P r o b l e m s and b e n e f i t s  of  The H o s t S c h o o l ' s  teachers  (see  help or to  c)  Opportunity  d)  Emotional  no q u i e t a r e a s ,  in close  encour-  contact with  the  2d).  for  professional  pleasures  (having a  development.  and p r o b l e m s  i n the s c h o o l ,  student  status.  having student  staff-room,  and t h e i r  teachers  classroom.  Principal Increase  in administrative  duties  cooperating with the U n i v e r s i t y b)  Increase or decrease  c)  Fewer o r g r e a t e r  The P u p i l s a)  together  hinder).  Increased or decreased  always  and w o r k i n g  Teachers;  b)  a)  being  I n c r e a s e d o r d e c r e a s e d work l o a d teacher to  The  studying,  aging the despondant,  host s c h o o l ' s  a)  of  of  the  in  (allocating  staff  and s t u d e n t s ,  etc.)  status.  numbers o f d i s c i p l i n e  problems.  Host-School  Did they f i n d the f r e q u e n t changing from to r e g u l a r  classrooms,  teacher disturbing  student-teacher  i n any way, o r  beneficial  i n any way. b)  Could the p u p i l s  o b t a i n more i n d i v i d u a l  h e l p as a r e s u l t  of there being teachers,  and U . B . C .  i n the  staff  school.  attention  and  student-teacher  62 6)  The Dean and h i s R e p r e s e n t a t i v e s a)  Does t h e program p r e s e n t problems o r adyantages i n t i m e t a b l i n g , a l l o c a t i n g s t a f f and rooms, and f i n a n c i n g .  b)  Has t h e Dean's o f f i c e r e c e i v e d a c c o l a d e s and/or  abuse  as a r e s u l t o f t h e Program f u n c t i o n i n g . 7)  The S c h o o l Board a)  See 6 ( a ) .  b)  Has t h e s c h o o l board o f f i c e r e c e i v e d a c c o l a d e s and/or abuse as a r e s u l t o f t h e program f u n c t i o n i n g under t h e Board's  jurisdiction.  The above a r e examples o f p o s s i b l e s i d e e f f e c t s . t o be an e x h a u s t i v e l i s t .  I t i s not meant  Side e f f e c t s d i f f e r from process e f f e c t s  i n t h a t t h e y a r e u n a n t i c i p a t e d and not p a r t o f t h e t r e a t m e n t .  I f the  side e f f e c t s are p a r t of the treatment r a t h e r than a r e s u l t o f i t then they are process e f f e c t s . Summary v e r s i o n o f SIDE EFFECTS C h e c k p o i n t . Methodological Evidence Comprehensive search? Skilled? Independent? During/End/ Later  Substantive Fulfillment characteristics Meets a l l r e q u i r e m e n t s w e l l ; s i d e e f f e c t s overall positive. G e n e r a l l y good; s i d e e f f e c t s o v e r a l l p o s i t i v e . Barely acceptable; side effects overall p o s i t i v e or n e u t r a l . Some s t u d y made, but i n c o m p l e t e . No w o r t h w h i l e s t u d y .  Comments: The e v a l u a t o r s h o u l d use a l l t h e E v i d e n c e S o u r c e s .  Rating 4 3 2 1 0 A high  r a t i n g on t h i s c h e c k p o i n t i n d i c a t e s t h a t the s i d e e f f e c t s a r e good and t h a t t h e q u a l i t y o f i n f o r m a t i o n on the s i d e e f f e c t s i s good.  63 3.336 C h e c k p o i n t 6: PROCESS The p r o c e s s c h e c k p o i n t i s m u l t i f a c e t e d .  I t i s used t o  substantiate or invalidate descriptions of the product, causal claims i n v o l v e d i n t h e e v a l u a t i o n ( t h a t t h e t r e a t m e n t c a u s e d t h e g a i n s ) and/or i t may b e a r on moral q u e s t i o n s ( e . g . t h e i n t r i n s i c a l l y o r i n h e r e n t l y n e g a t i v e d i m e n s i o n s o f c r u e l t y , i n j u s t i c e and u n h a p p i n e s s , as w e l l as t h e i r o p p o s i t e s ) . The p a y - o f f emphasis i s o f p r i m a r y i m p o r t a n c e ( S c r i v e n , 1974b, p. 16) To f u l f i l l t h i s c h e c k p o i n t t o a h i g h d e g r e e r e q u i r e s t h r e e t h i n g s , they a r e : a)  Congruency: The program as d e s c r i b e d ( s e e S e c t i o n 3.11 "The  evaluatum") and t h e i m p l e m e n t a t i o n p l a n s f o r t h e program ( s e e S e c t i o n 3.21, S t a n d a r d 2: MARKET) must be c o n g r u e n t w i t h what a c t u a l l y happened. b)  C a u s a t i o n : I f t h e Program p r o p o n e n t s s a y t h a t Program B  caused some e f f e c t Y, t h e n t h e r e must be a s e a r c h t o s e e i f Y i s present.  T h i s i s s i m p l y a check f o r t h e p r e s e n c e o f Y, i t i s n o t an  e x p l a n a t i o n f o r t h e cause o f Y-  I f Y i s n o t p r e s e n t , t h e n t h e Program's  p r o p o n e n t s c l a i m t h a t Program B caused Y i s n o t c o r r e c t . p r e s e n t t h e n t h e i r c l a i m may be c o r r e c t .  If Y is  The next c h e c k p o i n t ,  CAUSATION, d e t e r m i n e s i f o r i f n o t i t was Program B which c a u s e d Y. c) observed.  Justice:  The moral d i m e n s i o n s o f t h e program must be  F o r example t h e s e must be a s e a r c h f o r any i n j u s t i c e ,  u n h a p p i n e s s , c r u e l t y (and t h e i r o p p o s i t e s ) which i s p a r t o f t h e  64 p r o g r a m — a s opposed t o b e i n g a s i d e - e f f e c t o f t h e program  (Scriven,  1974b, p. 1 6 ) . Summary, V e r s i o n o f PROCESS C h e c k p o i n t . Methodological Evidence  Substantive Fulfillment characteristics  Rating  D e s c r i p t i v e congruence check? C a u s a l c l u e s check? Instrument v a l i d i t y ? Judge/observer r e l i a b i l i t y ?  Passes with f l y i n g c o l o r s . Appears s a t i s f a c t o r y . Reasonable r i s k . Significant omission(s). Inadequate  4 3 2 1 0  Comment: A l l o f t h e E v i d e n c e S o u r c e s s h o u l d be used by t h e e v a l u a t o r , r a t e d s e p a r a t e l y and t h e n combined i n t o an o v e r a l l p e r f o r m a n c e r a t i n g f o r PROCESS.  3.337 C h e c k p o i n t 7: CAUSATION. Whereas t h e PROCESS c h e c k p o i n t was a p r e s e n c e c h e c k f o r a p a r t i c u l a r c l a i m e d e f f e c t (E) o f Program B; CAUSATION a t t e m p t s to d e t e r m i n e i f i t was r e a l l y Program B which caused E. The p r o b l e m o f c a u s a t i o n i s e x t r e m e l y complex.  As K e r l i n g e r (1973,  p. 393) wrote: "... t h e s t u d y o f cause and c a u s a t i o n i s an e n d l e s s maze . . . w e a g r e e t h a t c a u s a l l a w s . c a n n o t be d e m o n s t r a t e d e m p i r i c a l l y , but we a r e e q u i v o c a l a b o u t t h i n k i n g c a u s a l l y . T h e r e i s l i t t l e doubt t h a t s c i e n t i s t s do t h i n k c a u s a l l y and t h a t when t h e y t a l k o f a r e l a t i o n between "p" and "q" t h e y hope and b e l i e v e t h a t "p" causes " q " , : S c r i v e n (1966b, p. 454) w r o t e : "... Can we n o t p r o c e e d f u r t h e r and d e f i n e " p o s s i b l e c a u s e s " i n terms o f some c o m b i n a t i o n o f n e c e s s a r y and s u f f i c i e n t c o n d i t i o n s , t h e s e b e i n g i n t e r p r e t e d as s i m p l e r e g u l a r i t y n o t i o n s ? The answer a p p e a r s t o be t h a t we c a n n o t . "  65 With t h e l i m i t a t i o n t h a t when t h e word " c a u s e " i s used, i t means " p o s s i b l e cause", S c r i v e n p r o v i d e s the f o l l o w i n g useful "developmental sequence" w i t h t h e comment (1966b, p. 455) " [ i t ] does n o t e s t a b l i s h t h e common i d e a t h a t l a t e r members a r e s i m p l y complex c o m b i n a t i o n s o f t h e e a r l i e r ones." 1)  B a s i c E x p e r i m e n t a l , Case:  I f whenever and however C i s  p r o d u c e d , E o c c u r s ; and E never o c c u r s u n l e s s C i s p r o d u c e d t h e n C caused E. C i s p r o d u c e d a t random i n t e r v a l s . 2)  B a s i c O b s e r v a t i o n Case:  C i s o b s e r v e d t o o c c u r on v a r i o u s  o c c a s i o n s and i s accompanied o r f o l l o w e d by E, and E n e v e r o c c u r s on o t h e r o c c a s i o n s . C i s t h e c a u s e o f E i f we c o u l d r e d u c e t h e s i t u a t i o n t o a b a s i c e x p e r i m e n t a l c a s e , i . e . i f we c o u l d randomly c o n t r o l t h e a p p e a r a n c e o f C.  I f t h i s i s not p o s s i b l e then the problem i s t o  e l i m i n a t e t h e p o s s i b i l i t y t h a t some o t h e r f a c t o r X i s c a u s i n g both C and E. 3)  Compound C a u s e s :  Not o n l y C b u t a l s o D i s needed t o b r i n g  a b o u t E. S c r i v e n r e f e r s t o C and D as c a u s a l f a c t o r s o r c o - c a u s e s o f E, 4)  M u l t i p l e Causes: " I f C and D a r e each s u f f i c i e n t t o b r i n g  about E, and n o t h i n g e l s e i s , t h e n w h i c h e v e r o c c u r s i s t h e c a u s e . I f b o t h o c c u r , one o f them may n o t have had any e f f e c t on t h i s o c c a s i o n , a p o s s i b i l i t y which we check by e x a m i n i n g t h e s i t u a t i o n f o r t h e p r e s ence o f known i n t e r m e d i a t e l i n k s w h i c h c h a r a c t e r i z e t h e modus o p e r a n d i 6 f C and D, i . e . any s e t s o f c o n d i t i o n s "C| o r C,, o r . . ." ( o r "D^ o r D o r . . .") which a r e n e c e s s a r y f o r C ( o r D) t o a c t as t h e cause o f ?  66 E.  T h i s t e s t does not a p p l y where no such l i n k s a r e known, and s i n c e  t h a t i t i s n o t l o g i c a l l y n e c e s s a r y t h a t t h e r e be any (C and E may be a d j a c e n t l i n k s i n t h e c h a i n , o r d i f f e r o n l y from a c e r t a i n d e s c r i p t i v e standpoint, or represent "action at a distance"), the t e s t i s not p a r t o f t h e meaning, ..."  ( S c r i v e n , 1966b, p. 4 5 5 - 5 6 ) .  It i s " M u l t i p l e Causes" which is.df, p a r t i c u l a r . i n t e r e s t t o ' t h i s thesis.  In S-GRADING, t h e a b s e n c e o f c o n t r o l g r o u p s and t h e i n a b i l i t y  o f a f o r m a t i v e e v a l u a t o r t o randomly a s s i g n p a r t i c u l a r t r e a t m e n t s " c " e l i m i n a t e proving the "Basic Experimental Case" and "Compound C a u s e s . "  Case", t h e " B a s i c  However, t h e modus o p e r a n d i  Observation  (MO) methods  b r i e f l y d e s c r i b e d by S c r i v e n i n " M u l t i p l e C a u s e s " and e l a b o r a t e d by him i n E v a l u a t i o n P e r s p e c t i v e s and P r o c e d u r e s  ( S c r i v e n , 1974b, p. 68-84),  p r o v i d e s a u s e f u l method o f d e t e r m i n i n g a c a u s a l i n f e r e n c e p a t t e r n . As he p o i n t o u t , t h e MO method i s r o u t i n e l y used by a n t h r o p o l o g i s t s and h i s t o r i a n s .  ( S c r i v e n , 1974b, p. 6 8 ) .  B r i e f l y , t h e p r o b l e m i s t o d e t e r m i n e what caused E.  The method  of determining t h i s i s : Step 1: A, B, C, D . . . c a n sometimes c a u s e E. Step 2: N o t h i n g e l s e i s known t o c a u s e " E " . Step 3: B b u t n o t A, C o r D was p r e s e n t . Step 4: The MO o f B ( i . e . b-j, b , b 2  3  . . ., b .), w h i c h i s h i g h l y  d i s t i n c t i v e , was p r e s e n t , then Step 5: B p r o b a b l y caused " E " . ( S c r i v e n , 1974b, p. 7 3 ) . For example, i f a s t u d e n t t e a c h e r uses a p a r t i c u l a r method o f d i s c i p l i n e then p o s s i b l e s o u r c e s f o r t h a t method a r e ; h i s u p b r i n g i n g  67 ( p a r e n t s , home, s c h o o l , e t c . ] , h t s s p o n s o r t e a c h e r i n . t h e s c h o o l , h i s own r e a d i n g i n h i s e d u c a t i o n Cor o t h e r ] c l a s s e s , h t s c o n t a c t w t t t i o t h e r s t u d e n t t e a c h e r s , h t s U n i v e r s i t y a d v t s o r , a n d ' r u l e s o f t h e s c h o o l he i s i n . I f he was o b s e r v e d t o c o n s i s t e n t l y use t h e r u l e s o f t h e s c h o o l ( t h e e f f e c t ] then a s e a r c h c o u l d be made f o r why he d i d use t h e s e r u l e s of d i s c i p l i n e .  I f i t were found t h a t t h e p r i n c i p a l had c a r e f u l l y  d i s c u s s e d t h e r u l e s of d i s c i p l i n e used i n h i s s c h o o l , t h a t he had c i r c u l a t e d a w r i t t e n l i s t of t h e s e r u l e s , t h a t t h e s p o n s o r t e a c h e r had made a p o i n t of t e l l i n g t h e s t u d e n t t e a c h e r about t h e r u l e s and t h e s t u d e n t t e a c h e r s a i d t h a t he used t h o s e r u l e s b e c a u s e t h e y were t h e s c h o o l r u l e s , then p r o b a b l y i t was t h e r u l e s of t h e s c h o o l which determined  ( i . e . was t h e cause of)  t h e d i s c i p l i n e t e c h n i q u e s used  by t h e s t u d e n t t e a c h e r . T h i s s e a r c h f o r t h e complete MO o b v i o u s l y would become more comp l i c a t e d if i t was found t h a t t h e d i s c i p l i n e r u l e s of t h e s c h o o l were t h e same as t h o s e espoused by t h e s t u d e n t t e a c h e r ' s u n i v e r s i t y advisor, Summary V e r s i o n of CAUSATION Methodological Evidence  Checkpoint.  Substantive Fulfillment Characteristics  Randomized e x p e r i m e n t a l d e s i g n ? Quasivexperimental design? Ex p o s t f a c t o ? Modus Operandi method? A priori interpretation of c o r r e l a t i o n a l data? Comment; A randomized evidence, preference.  Rating  Impeccable Good b e t P l a u s i b l e bet Weak b e t Hopeless bet  experimental design .is the best source o f  The o t h e r methods a r e ranked i n d e s c e n d i n g o r d e r o f  4 3 2 1 0  68 3.338 C h e c k p o i n t 8: STATISTICAL SIGNIFICANCE. T h i s i s t h e s t a n d a r d t h a t d e t e r m i n e s i f t h e measurements t a k e n on t h e p r o j e c t were a p p r o p r i a t e , and i f t h e y were a n a l y s e d a t an a c c e p t a b l e l e v e l o f s t a t i s t i c a l s i g n i f i c a n c e . S c r i v e n (1974b), p. 15) says " T h i s r e q u i r e s no g r e a t s o p h i s t i c a t i o n , and i t i s f r e q u e n t l y t h e o n l y work o f s o p h i s t i c a t i o n i n an e v a l u a t i o n d e s i g n . " In t h e next p a r a g r a p h he comments, ". . . i t i s a l l t o o e a s i l y o b t a i n e d w i t h o u t t h e r e s u l t s h a v i n g any e d u c a t i o n a l s i g n i f i c a n c e . . ." In o t h e r words, s i m p l y r e p o r t i n g s o p h i s t i c a t e d s t a t i s t i c a l t e s t r e s u l t s , does not mean an e v a l u a t i o n has been c a r r i e d o u t . Summary V e r s i o n o f STATISTICAL SIGNIFICANCE C h e c k p o i n t . Methodological Evidence  Substantive Fulfillment characteristics  Rating  Appropriate analysis? Flawless a n a l y s i s , astronomical Appropriate significance significance level? High s i g n i f i c a n c e , w e l l - t e s t e d Reasonably s i g n i f i c a n t Marginal s i g n i f i c a n c e Not shown to be s i g n i f i c a n t  4 2 1 0  3  To d e t e r m i n e an o v e r a l l r a t i n g f o r t h e s t a n d a r d f o r goal a c h i e v e ment i t i s proposed t h a t t h e i n d i v i d u a l r a t i n g s f o r each c h e c k p o i n t be added t o g e t h e r and a v e r a g e d . f o r goal achievement.  T h i s a v e r a g e i s then used as t h e r a t i n g  I t i s impossible f o r a product to r e c e i v e a high  r a t i n g i n t h i s way, and not be f u l f i l l i n g t h e w o r t h w h i l e g o a l s .  Each  c h e c k p o i n t has been r a t e d i n terms o f i t s own r e l e v a n c e t o t h e g o a l s and on t h e b a s i s o f good d a t a , ( i . e . good e v i d e n c e o f good  performance),  t h e r e f o r e t h e sum o f t h e c h e c k p o i n t r a t i n g s w i l l r e f l e c t t h i s .  69  The a c t u a l r a t i n g p r o c e d u r e f o r each c h e c k p o i n t and f o r t h e combined r a t i n g o f o v e r a l l worth w i l l be d i s c u s s e d l a t e r i n t h i s chapter. F i g u r e 3 has been p r e p a r e d t o summarize t h e p r e c e d i n g d i s c u s s i o n . T h i s F i g u r e i n c l u d e s t h e s t a n d a r d o f COSTS. F i g u r e 3. STANDARD  The S t a n d a r d s and T h e i r R a t i n g s  CONSIDERATIONS  RATING FOR THE STANDARD  1) The worth To be w o r t h w h i l e , t h e \ - ' Maximum p r i o r i t y , a d e s p a r a t e of the goals^should: need, e x t r e m e l y w o r t h w h i l e Program - e n a b l e one t o f u l f i l l Goals. higher goals, G r e a t i m p o r t a n c e , v e r y worth-not c o n f l i c t with \ another p r i n c i p l e of Probably worthwhile goals . . 2 our way o f l i f e , Possibly worthwhile goals . . 1 -lead to b e n e f i c i a l No good e v i d e n c e o f the g o a l s consequences, - n o t w a r r a n t h a v i n g an e x c e p t i o n made t o t h e goals i n this s i t u a t i o n . 2} The degree of goal achievement.  3) COSTS (not i n cluded in this thesis)  -MARKET -TRUE FIELD TRIALS -TRUE CONSUMERS -LONG TERM -SIDE EFFECTS -PROCESS -CAUSATION -STATISTICAL SIGNIFICANCE  -Comprehensive c o s t analysis - E x p e r t judgement o f costs -Independent judgement of c o s t s - C o s t s f o r al1 competitors  Completely f u l f i l l s the goals 4 F u l f i l l s almost a l l o f the F u l f i l l s the most i m p o r t a n t aspects of the goals . . . 2 F u l f i l l s some but not a l l of the important aspects Does n o t appear t o be d i r e c t e d to f u l f i l l i n g the B r e a k t h r o u g h f o r comparable S i g n i f i c a n t l y lower t h a n comparable p r o d u c t s . . . . R e a s o n a b l e f o r comparable  3  P r o b a b l y h i g h f o r compara b l e p r o d u c t s , o r somewhat i n c o m p l e t e d a t a ... 1 A p p a r e n t l y e x c e s s i v e f o r comparable products of data  70 I f t h e e v a l u a t o r was g o i n g t o examine c o s t s , o r a i m a t an e v a l u a t i v e c o n c l u s i o n o f "commendatory" o r h i g h e r ( s e e T a b l e 1 ) , then c o s t s t a n d a r d s would have t o be i n c l u d e d .  The term " c o s t " i s e x t r e m e l y  broad,  and i n c l u d e s n o t o n l y d o l l a r c o s t s , b u t p s y c h o l o g i c a l c o s t , l o s t o p p o r t u n i t y c o s t , s o c i a l c o s t s , e t c . ( S c r i v e n , 1974b, pp. 19-21; H a l l e r , E . J . , 1974).  In t h i s e v a l u a t i o n c o s t was n o t a m a j o r c o n s i d e r a t i o n a s a s e p a r -  a t e s t a n d a r d and w i l l n o t be i n c l u d e d i n subsequent  discussions.  The f i n a l comment i n t h i s s e c t i o n has t o do w i t h t h e t y p e o f standards.  In S e c t i o n 2.40 o f t h i s t h e s i s , s t a n d a r d s were i d e n t i f i e d  as b e i n g o f i n t r i n s i c o r e x t r i n s i c v a l u e . . A c c o r d i n g t o t h e d e f i n i t i o n s o f t h e s e s t a n d a r d s ( s e e S e c . 2.40) a p r o c e s s o r p r o d u c t cannot have i n t r i n s i c v a l u e ; i t c a n o n l y produce f e e l i n g s and e x p e r i e n c e s w h i c h may have i n t r i n s i c v a l u e .  Both t h e w o r t h o f t h e g o a l s f o r Program B  and t h e d e g r e e o f g o a l a c h i e v e m e n t a r e s t a n d a r d s o f e x t r i n s i c v a l u e . In t h i s i n v e s t i g a t i o n , goal worth i s a s t a n d a r d o f i n s t r u m e n t a l  value  because i t l e a d s t o t h e a c c o m p l i s h m e n t o f h i g h e r l e v e l g o a l s o r needs ( i . e . b e i n g a good t e a c h e r ) ; and goal a c h i e v e m e n t i s a s t a n d a r d o f i n s t r u m e n t a l v a l u e b e c a u s e i t i s a rank o r grade o f t h e program's effectiveness i n f u l f i l l i n g the goals.  3.40 S p e c i f i c a t i o n o f t h e C l a s s o f Comparison i n g r a d i n g . As s t a t e d i n c h a p t e r I I , s e c . 2.34 and 2.60 t h e c l a s s o f comparison  s h o u l d be some i d e a l form o f t h e program ( o r t h e b e s t o f a l l  r e a l o r imagined s i m i l a r programs).  The " i d e a l i z e d program" r e f e r r e d t o  i n Chapter I I i s n o t o n l y what.the c l i e n t hopes t o s e e , but a l s o  71  what he s h o u l d hope to s e e .  In o t h e r words, t h e e v a l u a t o r c a n n o t  s i m p l y a c c e p t t h e c l i e n t ' s e x p e c t a t i o n s o f t h e program as t h e s t a n d a r d s by which the c h a r a c t e r i s t i c s o f the program a r e j u d g e d .  This  i s p a r t i c u l a r l y t r u e i n e d u c a t i o n , where t h e program d e v e l o p e r ' s g o a l s s h o u l d be c o n g r u e n t w i t h t h o s e o f an e n l i g h t e n e d p u b l i c . T h e r e f o r e i t i s t h e d e c i s i o n o f t h e e v a l u a t o r as to the c r i t e r i a d e f i n i n g the standards.  He must be a b l e to j u s t i f y h i s r a t i n g s o f  c r i t e r i a and s t a n d a r d s t o h i s c l i e n t .  The e v a l u a t o r can r e a l i s t i c a l l y  d e f i n e the acceptable l e v e l s through expert o p i n i o n , l o g i c a l a n a l y s i s and/or through e m p i r i c a l s t u d i e s by him o r o t h e r s . program i s compared t o p r e d e t e r m i n e d  In o t h e r words t h e  standards of e x c e l l e n c e (Stake,  1967, p. 18) based on t h e c h a r a c t e r i s t i c s o f t h e b e s t o f a l l r e a l o r imagined s i m i l a r programs. 3.50 D e t e r m i n i n g t h e Good and Bad C h a r a c t e r i s t i c s o f t h e Evaluatum  in  S-Grading 3.51 G a t h e r i n g Data on t h e Goal Achievement C h a r a c t e r i s t i c s Table 2 represents a calendar of events i n the e v a l u a t i o n o f Program B.  I t shows t h e d a t e o f t h e e v e n t , t h e i n s t r u m e n t o r t e c h n i q u e  used ( i n c l u d i n g v e r b a l and w r i t t e n p r e s e n t a t i o n s by t h e e v a l u a t o r ) , t h e group b e i n g q u e s t i o n e d o r b e i n g a d d r e s s e d and t h e c h e c k p o i n t s b e i n g investigated.  The i n s t r u m e n t s used, and t h e q u e s t i o n e d asked i n t h e  i n t e r v i e w s , e t c . , a r e appended to t h e t h e s i s .  T a b l e 2. Calendar/'df Data C o l 1 e c t i n g E v e n t s  72  • Date (1975)  I n s t r u m e n t o r T e c h n i q u e Group b e i n g or Presentation Questioned o r Receiving  Checkpoints Being Investigated  Jan 15  Q u e s t i o n n a i r e (App A)'  U.B.C. S t a f f  T r u e Consumer  Jan 31  I n t e r v i e w (App.; B). , /  P r i n c i p a l and Vice-Principal  T r u e Consumer Side E f f e c t s Process True F i e l d T r i a l s Market  Feb 3  Likert Attitude Scale (App.C) )  Student Teachers  Process Comparisons T r u e Consumer  Feb 4  R e a c t i o n form!:. (App D')/.;. S t u d e n t T e a c h e r s  F r e e Ranging S e a r c h an open ended questionnaire  Feb 7  Staff; teaching performance ( f o r m A) (App<£) questionnaire  Student Teachers  Process  Feb 12  W r i t t e n s u b m i s s i o n from the t e a c h e r s  the e v a l u a t o r and U.B.C. s t a f f  Side Effects Process T r u e Consumer  Feb 19  Verbal Presentation  U.B.C. S t a f f  T r u e Consumer Process  Feb 27  Q u e s t i o n n a i r e on A l t e r n a t e Placement Appendix^F;„A  Student Teachers  Process Side Effects  Feb 28  Verbal P r e s e n t a t i o n  U.B.C. S t a f f  T r u e Consumer  Feb 28  Likert Attitude Scale (App £)•-)•,  Student Teachers  See Feb. 3.  U.B.C, S t a f f Teachers Vice-Principal Principal  Process Side E f f e c t s  March 4  E!v a luMoir-e e h an ^eu^me e t a ^n9..:BG.t&e'&nfU5&clci'ei £h-. . 'S t a f f ;ia n de t e a c he r ss a n d a= v e r b a l • p r e s e n t a t i on. :  March 6  R e a c t i o n form (App D)') •Student t e a c h e r s  See Feb. 4.  March 7  S t a f f teaching perform- Student teachers ance [Form B) [Appendix G)']) Q u e s t i o n n a i r e .  See F e b . 7.  73 T a b l e 2 (Cont.) Date (1975)  Instrument or Technique or Presentation  Group b e i n g Questioned or Receiving  Checkpoints Being Investigated  March 20  I n t e r v i e w (App H)  Student Teachers  True F i e l d t r i a l s T r u e Consumer Comparisons Side Effects Process-Congruency, Justice  March 25  I n t e r v i e w (App H)  Student Teachers  See March 20  March 27  Verbal presentation  S t u d e n t T e a c h e r s T r u e Consumer  March 27  Q u e s t i o n n a i r e (App I)  Student Teachers  April 2  I n t e r v i e w (App I)  S t u d e n t T e a c h e r s See March 20  April 8  O b s e r v a t i o n o f Studentt e a c h i n g (App J )  6 Student Teachers  April 9  Observation of Student- 4 Student teaching Teachers  See A p r i l 8  A p r i l 10  Observation of Studentteaching  2 Student Teachers  See A p r i l 8  A p r i l 10  Q u e s t i o n n a i r e (App K)  Teachers  Side Effects  A p r i l 11  Observation of Student Teaching  2 Student Teachers  See A p r i l 8  A p r i l 13  I n t e r v i e w (App L)  Program D i r e c t o r Need Market Process True F i e l d T r i a l s T r u e Consumer  A p r i l 15  Q u e s t i o n n a i r e (App  A p r i l 29  I n t e r v i e w (App  N)  M)  Process-congruency  Process Causation-Modus O p e r a n d i method  School's p u p i l s : grades 5 & 6  T r u e Consumer Process Side Effects  University Director  Need Side Effects Process T r u e Consumer True F i e l d t r i a l s  Field  74 T a b l e 2 (Cont.) Date (1975)  Instrument o r Technique or Presentation  Group b e i n g Questioned o r Receiving  Checkpoints Being Investigated  A p r i l 29  I n t e r v i e w (App 0)  Director of Elementary Education  .Side E f f e c t s Process  A p r i l 30  I n t e r v i e w (App 0)  Director of Side E f f e c t s S t u d e n t T e a c h i ng P r o c e s s -  May 2  I n t e r v i e w (App 0)  A s s o c i a t e Dean  Side Effects Process  End o f Academic y e a r : e v a l u a t i o n - - c o n t i n u e s w i t h d i f f e r e n t s t u d e n t teachers, different school. S e p t . 19 Q u e s t i o n n a i r e (App P)  Student Teachers Side E f f e c t s  S e p t . 25 Q u e s t i o n n a i r e (App Q)  U.B.C. S t a f f  Oct. 3  Q u e s t i o n n a i r e (App S)  Student Teachers Process - evaluat i o n o f U.B.C. Staff teaching performance.  O c t . 24  Q u e s t i o n n a i r e (App R)  Teachers  Side Effects Process T r u e Consumer  O c t . 31  I n t e r v i e w (App T)  School Board  Side Effects T r u e Consumer Process  Process (descript i o n of courses)  In a d d i t i o n t o t h e i n f o r m a t i o n c o l l e c t e d as i n d i c a t e d i n T a b l e . 3, L  the a u t h o r a l s o a t t e n d e d Program B's d i r e c t o r s ' s t a f f m e e t i n g s on the f o l l o w i n g dates: J a n u a r y 15, 22 F e b r u a r y 12, 19, 26  75 March 5, 12, 20 April  10  A t t e n d i n g t h e s e meetings p r o v i d e d i n f o r m a t i o n o n ^ h e c h e c k p o i n t s o f s i d e - e f f e c t s , p r o c e s s , market, and t r u e  consumers.  O t h e r d a t a c o l l e c t e d were census d a t a (1971) f o r t h e a r e a s e r v e d by t h e s c h o o l , and t h e o r i g i n a l a p p l i c a t i o n forms o f t h e student teachers.  These d a t a h e l p e d i n d e t e r m i n i n g a r a t i n g f o r t h e  c h e c k p o i n t o f time f i e l d t r i a l s .  3.52 D e t e r m i n i n g i f t h e C h a r a c t e r i s t i c s o f t h e Program a r e Good o r Bad,  3.521 Rating, t h e Worth o f t h e G o a l s The c h a r a c t e r i s t i c s o f t h e g o a l s a r e examined o f t h e i r worth.  i n terms  T h e r e f o r e i n t h i s s e c t i o n arguments w i l l be p r e s e n t e d  o u t l i n i n g t h e p r o c e d u r e s used i n d e t e r m i n i n g a r a t i n g f o r t h e worth o f t h e g o a l f o r t r a i n i n g p e o p l e to be good t e a c h e r s .  A complete  v a l i d a t i o n o f t h e r a t i n g g i v e n a t t h e end o f t h e arguments f o r t h e need f o r t e a c h e r t r a i n i n g has not been documented h e r e .  The  c o n c l u s i o n s r e a c h e d can be j u d g e d by t h e i r r e a s o n a b l e n e s s and not by t h e i r completeness.  T h i s i s the t y p e o f a p p r o a c h one would use f o r  his c l i e n t i n evaluation. Program B has s t a t e d t h a t i t i s a t t e m p t i n g t o c o n t r i b u t e t o t h e t r a i n i n g o f good t e a c h e r s .  Perhaps i t seems t r i t e t o ask a t t h i s  p o i n t "do we need to t r a i n and e d u c a t e p e o p l e t o be good t e a c h e r s ? "  76  but t h e q u e s t i o n w i l l be posed t o m a i n t a i n c o n t i n u i t y .  The o t h e r s i d e  o f t h e q u e s t i o n i s " o r can any e n l i g h t e n e d but u n t r a i n e d p e r s o n s t e p i n t o a c l a s s r o o m and t e a c h ? "  I f the q u e s t i o n of needing teachers a t  a l l s h o u l d a r i s e , t h e n i t would be n e c e s s a r y to p r o c e e d t h r o u g h a comp l e t e v a l i d a t i o n u n t i l e v e r y o n e was g e n e r a l l y a g r e e d t h a t t e a c h e r s were needed.  T h e r e i s no p o i n t i n a s k i n g "Do we need t o t r a i n p e o p l e  to be good t e a c h e r s ? " i f t h e r e i s w i d e s p r e a d d i s a g r e e m e n t about n e e d i n g teachers at a l l . Assuming t h a t i t i s g e n e r a l l y a g r e e d t h a t we need good t e a c h e r s t h e n we must a d d r e s s t h e problem o f somehow e s t a b l i s h i n g i f t r a i n i n g p e o p l e t o be good t e a c h e r s i s a w o r t h w h i l e g o a l . The b e s t way t o do t h i s would be e m p i r i c a l l y i . e . randomly a s s i g n i n g a group o f p r o s p e c t i v e t e a c h e r s t o s c h o o l s where t h e y i m m e d i a t e l y b e g i n t e a c h i n g . A f t e r a few y e a r s compare them t o a n o t h e r group who began t e a c h i n g a t t h e same time but who had been t h r o u g h a program o f t e a c h e r t r a i n i n g . L a c k i n g t h i s , however, one c o u l d l o o k ( c a u t i o u s l y ) a t p r e s e n t p r a c t i c e and e n l i g h t e n e d p r e s e n t t h o u g h t on t h e s u b j e c t . P r e s e n t p r a c t i c e i n d i c a t e s t h a t p e o p l e must be t r a i n e d t o be good t e a c h e r s . P r e s e n t t h o u g h t s u g g e s t s t h e same t h i n g . As Broudy wrote: " T h a t p u b l i c s c h o o l i n s t r u c t i o n can be manned by b r i g h t and d e v o t e d amateurs i s t h e i l l u s i o n which m i s g u i d e s many o f t h e c r i t i c s of t e a c h e r p r e p a r a t i o n . The s a l v a t i o n o f our s o c i e t y w i l l depend on t h e a b i l i t y o f e d u c a t i o n to e x p l o i t t h e b l e s s i n g s o f t e c h n o l o g y i n b e h a l f o f what makes l i f e w o r t h w h i l e , v i z . , t h e p o s s i b i l i t y o f h i g h grade i n d i v i d u a l e x p e r i e n c e i n something c a l l e d 'the good l i f e ' . T h a t s o l v i n g t h e s e problems w i l l  77  r e q u i r e l e s s than a g e n e r a t i o n o f p r o f e s s i o n a l e d u c a t o r s t r a i n e d and educated i n f u l l y developed p r o f e s s i o n a l s c h o o l s , I f i n d impossible to b e l i e v e . " (Broudy, 1965, p. 415) Though t h e r e a r e no s p e c i f i c f o r m u l a s t h e r e a r e g e n e r a l r u l e s , o u t l i n e d i n s e c t i o n 3.21 which w i l l v a l i d a t e t h e worth o f o u r g o a l s w i t h as much a c c u r a c y and o b j e c t i v i t y as most measurements i n e d u c a t i o n . In making o u r judgement i t i s c o n v e n i e n t t o r a t e t h e worth o f t h e g o a l s on t h e f o l l o w i n g f i v e p o i n t s c a l e . Maximum p r i o r i t y , a d e s p e r a t e need, e x t r e m e l y worthwhile goals .  4  Great importance, very worthwhile goals  3  Probably worthwhile goals  2  Possibly worthwhile goals  1  No good e v i d e n c e o f t h e g o a l s h a v i n g any worth  0  The e v i d e n c e s u g g e s t s t h a t t h e r e i s more than j u s t " P r o b a b l y w o r t h w h i l e g o a l s " i n a program t o t r a i n p e o p l e t o be good t e a c h e r s . However, as many s c h o o l s have been o p e r a t i n g w i t h some u n t r a i n e d b u t f u n c t i o n a l t e a c h e r s on t h e i r s t a f f s i t would be d i f f i c u l t t o g i v e a r a t i n g o f "Maximum p r i o r i t y , . . .".  Therefore the standard o f goal  worth f o r a program t o t r a i n p e o p l e t o be good t e a c h e r s i s g i v e n a r a t i n g o f " G r e a t Importance"  and a r a t i n g v a l u e o f 3.  N o t i c e t h a t t h e worth o f t h e g o a l s , and i t s r a t i n g , a r e i n d e p e n d e n t o f t h e program b e i n g e v a l u a t e d , i n t h e sense t h a t t h e g o a l s haye v a r y i n g d e g r e e s o f worth r e g a r d l e s s o f whether t h e r e i s a program devised to achieve those goals.  78  3.522 Goal a c h i e v e m e n t and i t s r a t i n g F o r m a t i v e e v a l u a t i o n has t h e r e s p o n s i b i l i t y both o f p r o v i d i n g v a l i d i n f o r m a t i o n f o r program improvement and o f r a i s i n g q u e s t i o n s a b o u t t h e m e r i t o f t h e program a t p a r t i c u l a r p o i n t s i n t i m e ( S c r i v e n , 1967, p. 4 1 ) . In t h e e v a l u a t i o n o f Program B i n f o r m a t i o n was c o l l e c t e d , v a l i d a t e d t h r o u g h o b s e r v a t i o n and f u r t h e r q u e s t i o n i n g , summarized, and t h e n f e d back t o t h e program p a r t i c i p a n t s so t h a t changes c o u l d be made. The changes would e i t h e r a t t e m p t t o e l i m i n a t e problems o r a t t e m p t t o i n c o r p o r a t e a d v a n t a g e s o f t h e program.  However,  even though t h e program was s t i l l i n t h e f o r m a t i v e s t a g e , an o y e r a l l e s t i m a t i o n o f w o r t h s h o u l d be made. T h i s c o u l d be done a t n a t u r a l b r e a k s i n t h e program ( e . g . at C h r i s t m a s o r d u r i n g t h e summer h o l i d a y s of e d u c a t i o n a l p r o g r a m s ) .  The o v e r a l l e v a l u a t i o n would r e f l e c t t h e  s t a t e o f t h e program a t t h a t p o i n t i n t i m e , a n d , t h e r e f o r e , so would the  r a t i n g on each o f t h e goal a c h i e v e m e n t c h e c k p o i n t s . In o t h e r words i n f o r m a t i v e e v a l u a t i o n t h e e v a l u a t o r would c y c l e  t h r o u g h t h e c h e c k p o i n t s a number o f t i m e s . Each t i m e t h r o u g h he would d e l e t e p r e v i o u s problems t h a t had been e l i m i n a t e d t h r o u g h changes i n the  program, checkmark p o s s i b l e c o n t r i b u t i o n s t o t h e a c h i e v e m e n t o f t h e  g o a l s , i d e n t i f y new problems and c o n t r i b u t i o n s , c h e c k t o s e e i f previously identified contributive features are s t i l l present ( i f s t i l l of u s e ) and so on. Then, when he made h i s judgement o f t h e o v e r a l l w o r t h , he would c o n s i d e r t h e s t a t e o f each c h e c k p o i n t , i n terms o f t h e q u a l i t y o f d a t a ( m e t h o d o l o g i c a l ) and t h e q u a l i t y o f p e r f o r m a n c e on t h e data (substantive) to determine a r a t i n g .  79 To d e t e r m i n e t h e o v e r a l l r a t i n g f o r goal a c h i e v e m e n t a s i m p l e average i s proposed w i t h t h e c o n d i t i o n s t h a t t h e r a t i n g s h o u l d n o t be c a l c u l a t e d i f t h e program i s immoral o r u n j u s t . The above s t e p s a r e summarized i n F i g u r e 4.  F i g u r e 4: D e t e r m i n i n g a R a t i n g f o r Goal A c h i e v e m e n t . SteD 1 ^  C l a s s i f y each b i t o f i n f o r m a t i o n a c c o r d i n g t o t h e checkpoint i t f a l l s under.  SteD 2  Check t o s e e i f t h e i n f o r m a t i o n i s t r u e , t h r o u g h f u r t h e r o b s e r v a t i o n and q u e s t i o n i n g .  . Step 3  Feed t h e i n f o r m a t i o n back t o t h e a p p r o p r i a t e consumer so t h a t he c o u l d change o r i n c o r p o r a t e I * „ .. the c h a r a c t e r i s t i c s i d e n t i f i e d . J importing  Step 4  Repeat s t e p s 1-3 a s many t i m e s as n e c e s s a r y  Step 5  a) Summarizea t h e np r e s e n t s t a t e o f t h e quali l i t y "] q a 1ty fiance P ' \ •v E v a l u a t i o n k i n d i c a t e d by t h e d a t a . ° b) Rate each c h e c k p o i n t (0-4)  SteD 6  a) Check t o s e e i f t h e program i s immoral o r "\ u n j u s t . I f i t i s , then g i v e t h e program a Rating ^ achievement r a t i n g o f zero. b) I f t h e program i s n o t immoral or..unjust., then Goal A c h i e v e m e n t average t h e r a t i n g s on t h e c h e c k p o i n t s . T h i s i s t h e o v e r a l l r a t i n g f o r goal a c h i e v e m e n t .  p  H  p  Data Collection  D a t  p  °Wf \J H !  f t h e  e r f o r m a n c  e  goa  The d a t a c o l l e c t e d i n t h e f o r m a t i v e e v a l u a t i o n o f Program B a r e c o n s i d e r e d c o n f i d e n t i a l and c a n n o t be r e p o r t e d here t o i l l u s t r a t e t h e f i n a l rating given to the checkpoints. S c r i v e n (1974b, p. 31) has s u g g e s t e d t h e u s e o f a " p r o d u c t  80 evaluation profile".  H i s s u g g e s t i o n i s used i n F i g u r e 5 t o i l l u s t r a t e  t h e r a t i n g s o f each g o a l a c h i e v e m e n t c h e c k p o i n t and an o v e r a l l r a t i n g o f g o a l a c h i e v e m e n t f o r Program B. ( s e e a l s o F i g u r e 2 ) .  Figure 5 Product E v a l u a t i o n P r o f i l e For Program B ( t o end o f May '75)  R A  acceptable leyel  T 1 N  G  CO  to  1—  OJ  s-  +->  cu  Sro  s: 1  CO  r— fO •r—  T3  iCD— •i— u_  2  -(->  %~  E  E cu +->  c  E  3 co  uCD  4-  <u o Os-  CU  -a •r—  o  o  3  4  <_>  CO CO  LLJ  oo  Q.  5  6  o +->  fO CO  ns  O  7  r— ro O  cu o sz rO  O •r— <4•r— •r— •r—  +J  CO  +->  ro CD H-> " r — 00 00  LU —I  •—<  CJ3  cC  o  o  8  Comments on t h e r a t i n g s 1)  M a r k e t i s i m p o r t a n t . P l a n s f o r r e a c h i n g t h e market a r e u n i q u e , f e a s i b l e and a r e e c o n o m i c a l l y v i a b l e .  D e t a i l e d program  d e s c r i p t i o n s t i l l nee ded for p o r t a b i l i t y . f,  %  2)  U.B.C. s t a f f v e r y e n t h u s i a s t i c and hard w o r k i n g , t h e y a r e n o t an average group.  T e a c h e r s seem r e a s o n a b l y a v e r a g e . S t u d e n t - t e a c h e r s  were s e l e c t e d on b a s i s o f h i g h marks.  Many immigrant c h i l d r e n  w i t h l a n g u a g e problems a t t h e s c h o o l — m a d e some s t u d e n t t e a c h i n g  81  overly d i f f i c u l t . 3)  M i s s i n g d a t a o n t h e g a i n s by t h e s c h o o l ' s p u p i l s .  4)  Informal follow-up plans, not f i n a l i z e d .  5)  Good d a t a o n s i d e e f f e c t s .  Most bad s i d e - e f f e c t s were e l i m i n a t e d  by s i m p l e a d j u s t m e n t s t o t h e program.  A number o f u n a n t i c i p a t e d  g a i n s by T e a c h e r s and S t u d e n t - t e a c h e r s . 6)  S t u d e n t - t e a c h e r s u s i n g t e c h n i q u e s l e a r n e d i n t h e i r methods courses.  Most a r e t e a c h i n g s a t i s f a c t o r i l y .  Some a r e c o n s i d e r e d  by a l l consumers t o be e x c e l l e n t . Program v e r y c o n s c i o u s o f j u s t i c e and m o r a l i t y . Some d i f f i c u l t i e s i n f u l f i l l i n g a l l t h a t was e x p e c t e d due t o s k e t c h y n a t u r e o f program d e s c r i p t i o n . 7)  Modus o p e r a n d i c h e c k o n S t u d e n t - t e a c h e r s ' t e a c h i n g b u t more d a t a needed.  8)  No s t a t i s t i c a l d a t a c o l l e c t e d .  3.60 O v e r a l l D e t e r m i n a t i o n o f Worth Each s t a n d a r d , g o a l worth and g o a l achievement, has now been r a t e d . The p r o b l e m now becomes t o make a n o v e r a l l d e t e r m i n a t i o n o f t h e worth o f t h e e d u c a t i o n a l endeavor (Program B ) . T h i s depends on two t h i n g s ( C h a p t e r I I , s e c . 2.80): a)  The d e g r e e t o which t h e evaluatum f u l f i l l s t h e s t a n d a r d s  applied to i t . b)  The r e l a t i v e p r e c e d e n c e o f t h e s t a n d a r d s .  C o n s i d e r i n g "E>" f i r s t : T h e s t a n d a r d o f Goal Worth t a k e s p r e c e d e n c e o v e r t h e s t a n d a r d o f Goal Achievement.  I f t h e g o a l s o f t h e program a r e  82  not w o r t h w h i l e , t h e n , r e g a r d l e s s o f how w e l l t h o s e g o a l s a r e b e i n g a c h i e v e d , t h e program c a n n o t r e c e i v e an o v e r a l l "good" r a t i n g . F i g u r e 6 summarizes t h i s i n t e r p r e t a t i o n o f goal worth t a k i n g p r e c e d e n c e o v e r goal  achievement. Figure 6  Comparing Goal Worth and Goal  Achievement  GOAL WORTH RATING (See F i g . 2) Not  Probably Worthwhile  Worthwhi1e G o a l s not being achieved GAR OCA A H I_ j £ V E  T „ . j Achieves N some g o a l s G L  0  The program g o a l s have not been shown t o be worthw h i l e . The program i s not f u l f i l l i n g a useful function.  M  E  (See .Fig. 2)  Extremely Worthwhile  Achieves a l l goals  The g o a l s a r e worthwhile;! but the Program has not been shown t o be achiev^ i n g the g o a l s ; o r the program has been shown to) be m o r a l l y u n a c c e p t a b l e  The g o a l s a r e worthwhile;! and the program has been shown to be a c h i e v i n g the goals i n a m o r a l l y a c c e p t a b l e manner.  I f t h e r a t i n g o f goal worth i s g r e a t e r t h a n o r equal t o "2" and t h e r a t i n g o f g o a l achievement  i s g r e a t e r t h a n o r equal t o "2;  then the Program i s a t v a r y i n g d e g r e e s o f r e a d i n e s s f o r g e n e r a l d i s t r i b u t i o n and c o u l d now undergo a summative e v a l u a t i o n . I f the r a t i n g o f goal worth i s g r e a t e r than o r equal to "2" and  83 t h e r a t i n g o f goal a c h i e v e m e n t i s l e s s than "2" t h e n t h e program needs more development b e f o r e i t i s g e n e r a l l y d i s t r i b u t e d .  The  r i g h t hand s i d e o f F i g u r e 6 ( i . e . Goal Worth has a r a t i n g g r e a t e r than or equal t o "2") i s expanded i n F i g u r e 7 t o i l l u s t r a t e the d i f f e r e n t d e s c r i p t i o n s w h i c h r e s u l t from t h e e v a l u a t i o n . I f a program has a g o a l a c h i e v e m e n t r a t i n g o f "2" o r l e s s on F i g u r e 7 then i t may be u n w o r t h w h i l e ; or t h a t i s , t h e program may have undergone an e v a l u a t i o n and found t o not be a c h i e v i n g the g o a l s ; o r t h e program may be u n w o r t h w h i l e because i t i s found t o be u s i n g m o r a l l y u n a c c e p t a b l e methods; o r i t may not have undergone an e v a l u a t i o n (or have undergone a v e r y poor e v a l u a t i o n ) i n w h i c h c a s e no judgement can be made o f i t s o v e r a l l w o r t h . I f the program has a g o a l a c h i e v e m e n t r a t i n g o f "2" o r more on Figure 7 then i t i s worthwhile  and i n c r e a s e s i n worth, as i t i s shown  to a c h i e v e more o f t h e g o a l s and as t h e g o a l s i n c r e a s e i n  importance.  Program B has a goal a c h i e v e m e n t r a t i n g o f about "2" and a g o a l w o r t h r a t i n g o f "3".  T h e r e f o r e i t c o u l d be d e s c r i b e d as a  worthwhile  program, f u l f i l l i n g i m p o r t a n t g o a l s to an a c c e p t a b l e l e v e l . t h i s statement  has t h e f o l l o w i n g r e s t r i c t i o n .  However,  The e v a l u a t i v e c o n c l u -  s i o n b e i n g aimed f o r was an " o v e r a l l e v a l u a t i v e c o n c l u s i o n " (see T a b l e 1, Chapter I I ) . C o s t d a t a were not t a k e n i n t o c o n s i d e r a t i o n , nor was program e x t e n s i v e l y compared t o o t h e r t e a c h e r t r a i n i n g programs. Therefore, rewording  t h e o v e r a l l d e t e r m i n a t i o n o f w o r t h i n terms o f  the " o v e r a l l e v a l u a t i v e c o n c l u s i o n " , t h e d e s c r i p t i o n becomes;  the  84  Figure 7:' Expansion of Figure 6 GOAL WORTH RATING J  Probably worthwhile goals but:  Very worthwhile goals but:  4  Extremely worthwhile goals but:  1)  No data on the program or 2) The data indicates the program is not achieving any part of the goals or 3) The program is morally unacceptable. Probably worthwhile goals but:  Very worthwhile goals but:  Extremely worthwhile goals but:  1)  Poor data on the program or 2) Data indicates the program is doing a poor job of achieving the goals or 3) The program has serious moral shortcomings, Probably worthwhile goals and:  Very worthwhile goals and:  Extremely worthwhile goals and:  1)  Acceptable data and 2) Data indicates the goals are being acceptably achieved. Probably worthwhile goals and: 1) 2) Probably worthwhile goals and: 1)  Very worthwhile goals and:  Extremely" worthwhile goals and:  Good data  and Data indicates the goals are almost completely being achieved by the program. Very worthwhile goals and:  Extremely worthwhile goals and:  Excellent data and 2) Data indicates the program is completely achieving the goals.  85  "Program B a c c e p t a b l y f u l f i l l e d t h e g o a l f o r p r o d u c i n g s t u d e n t t e a c h e r s , who were competent i n t e a c h i n g ( f o r t h i s l e v e l o f t h e i r t e a c h e r t r a i n i n g ) . The s t u d e n t t e a c h e r s b e n e f i t t e d from Program B; and t h e r e were no harmful e f f e c t s on t h e program p a r t i c i p a n t s . " Note t h a t t h e above d e s c r i p t i o n does n o t c l a i m t h a t Program B i s t h e b e s t program f o r t r a i n i n g t e a c h e r s a t t h e t h i r d y e a r l e v e l . To make t h a t s t a t e m e n t i t would be n e c e s s a r y t o f u l l y e v a l u a t e t h e c o m p a r a t i v e p r o g r a m s , i n c l u d i n g t h e s t a n d a r d o f c o s t , and t h e n r a n k o r d e r them and program B t o d e t e r m i n e which was b e s t . be t h e j o b o f a summative e v a l u a t o r .  T h i s would  The f o r m a t i v e e v a l u a t i o n  i l l u s t r a t e d by t h i s t h e s i s has s i m p l y p r o v i d e d Program B's d e v e l o p e r s and s u p p o r t e r s ( f i n a n c i a l and o t h e r w i s e ) w i t h a "green l i g h t " o n subsequent  development.  3.70 E x t e n s i o n o f t h e Model  3.71  Introduction C h a p t e r s Two and T h r e e have e x p l i c a t e d a l o g i c a l e v a l u a t i o n model  based on R-GRADING.  T h i s s e c t i o n "The E x t e n s i o n o f t h e Model", has been  i n c l u d e d t o show t h a t t h e model d e v e l o p e d f o r R-GRADING c a n be l o g i c a l l y and e a s i l y extended t o i n c l u d e t h e most comprehensive  evaluations, that  i s , RANKING t h e e v a l u a t a i n a c l a s s o f comparison t o d e t e r m i n e i f i t is the best  evaluatum.  Ranking i n v o l v e s g r a d i n g an evaluatum a c c o r d i n g t o c e r t a i n s t a n d a r d s and then comparing t h a t o v e r a l l r a t i n g w i t h t h e r a t i n g s o b t a i n e d by o t h e r s i m i l a r programs graded a c c o r d i n g t o t h e same  86 standards.  The e v a l u a t a a r e t h e n r a n k e d a c c o r d i n g t o how w e l l t h e y  met t h e s t a n d a r d s . T h i s t e l l s t h e c l i e n t f o r t h e e v a l u a t i o n which e v a l u a t u m i s t h e b e s t , assuming each e v a l u a t u m was e v a l u a t e d t o t h e same d e g r e e .  S i n c e each e v a l u a t u m must have gone t h r o u g h S - g r a d i n g  b e f o r e i t was ranked i n t h e c l a s s o f o t h e r r e l e v a n t e v a l u a t u m , t h e e v a l u a t i o n can a l s o i n f o r m t h e c l i e n t o f t h e o v e r a l l worth o f each evaluatum. T h e r e a r e t h r e e q u e s t i o n s which must be d i s c u s s e d f o r t h e extension o f S-grading to Ranking.  F i r s t l y , "what p r e c e d e n c e do t h e s e  s t a n d a r d s t a k e i n d e t e r m i n i n g t h e o v e r a l l worth o r m e r i t o f t h e e v a l u a t a ; " and, f i n a l l y , "how can we r a n k t h e e v a l u a t a ? " 3.72 The S t a n d a r d s t o be Used i n Ranking In C h a p t e r 3 t h e s t a n d a r d s o f Goal Worth and Goal A c h i e v e m e n t were used as s t a n d a r d s f o r S - g r a d i n g .  The l o g i c a l q u e s t i o n t h e n i s  " c o u l d we use j u s t t h o s e two s t a n d a r d s f o r R a n k i n g ? "  The answer i s  " y e s " ; but t h i s assumes t h a t t h e c o s t o f t h e p r o j e c t i s n o t o f c o n c e r n . Unfortunately cost i s usually of concern.  O b v i o u s l y i f programs X  and Y f u l f i l l e d t h e s t a n d a r d o f Goal Worth t o t h e same d e g r e e , and t h e s t a n d a r d o f Goal Achievement to t h e same d e g r e e , y e t Program X c o s t more t h a n Program Y, t h e Program Y would be a b e t t e r program.  Therefore,  though i t i s p o s s i b l e t o r a n k c o m p e t i t i v e programs o n l y on t h e b a s i s o f Goal Worth and Goal A c h i e v e m e n t , i t i s more r e a s o n a b l e t o r a n k them by a l s o c o n s i d e r i n g t h e s t a n d a r d o f C o s t .  87 The s t a n d a r d o f Extended S u p p o r t may a l s o be t a k e n i n t o c o n sideration.  However t h i s s t a n d a r d i s not as n e c e s s a r y as t h e t h r e e  p r e v i o u s l y m e n t i o n e d (Goal Worth, Goal A c h i e v e m e n t and C o s t ) . s u p p o r t and C o s t were i n c o r p o r a t e d i n t o S c r i v e n ' s (1974b) Checklist".  Extended  "Product  They w i l l now be d e s c r i b e d i n more d e t a i l .  3.721  Costs S c r i v e n (1974b, p. 21) p r o v i d e d t h r e e c r i t e r i a f o r  cost data. presented a)  The two w h i c h a r e a p p l i c a b l e t o t h i s d i s c u s s i o n a r e below. Cost d a t a must be C o m p r e h e n s i v e . T h i s means t h a t t h e e v a l u a t o r s s h o u l d i n v e s t i g a t e such c o s t s  as: c o v e r i n g m a i n t e n a n c e c o s t s * c a p i t a l c o s t s , p s y c h i c c o s t s , d o l l a r c o s t s , c o s t s o f i n - s e r v i c e u p d a t i n g o f needed h e l p e r s , and so on. S c r i v e n ( i b i d ) also suggests  that:  "A q u a l i t a t i v e c o s t - e f f e c t i v e n e s s a n a l y s i s , s h o u l d be a t t e m p t e d where p o s s i b l e , and, i f i t i s i m p o s s i b l e , then c o s t - b e n e f i t a n a l y s i s s h o u l d be done as s y s t e m a t i c a l l y as p o s s i b l e . " b)  C o s t d a t a must be v e r i f i e d . A l l the c o s t e s t i m a t e s and r e a l c o s t s s h o u l d be  v e r i f i e d , p r e f e r a b l y by an a c c o u n t a n t educational  independently  s k i l l e d in estimating costs of  products.  A summary v e r s i o n o f t h i s s t a n d a r d c o u l d be s i m i l a r t o t h e f o l l o w i n g : ( S c r i v e n , 1974b, p.  28)  88  Considerations  Rating  Comprehensive c o s t a n a l y s i s ? - B r e a k t h r o u g h f o r comparable E x p e r t judgement o f c o s t s ? products i n d e p e n d e n t judgement o f c o s t s ? " Costs f o r a l l competitors? . - Reasonable f o r comparable products - High f o r comparable p r o d u c t s o r somewhat i n c o m p l e t e d a t a - E x c e s s i v e f o r comparable products or data incomplete  4 3  2 1 0  A more e x t e n s i v e d i s c u s s i o n o f " C o s t s " has been p r o v i d e d by E.J.  H a l l e r (1974, pp. 406-450).  3.722 Extended  Support  I t i s h i g h l y d e s i r a b l e t h a t t h e r e be s y s t e m a t i c p r o c e d u r e s f o r u p d a t i n g o r u p g r a d i n g a p r o d u c t once i t i s on t h e market i n t h e l i g h t o f new i n f o r m a t i o n which c o u l d improve product.  the  As S c r i v e n (1974b, p. 21) p o i n t s o u t : ". . . t h i s i m p l i e s  the necessity f o r a systematic continuing procedure f o r c o l l e c t i n g f i e l d data." T h i s s t a n d a r d i s d e s i r a b l e , but not n e c e s s a r y .  Its presence  s h o u l d be a p l u s f o r t h e program, but i t s absence s h o u l d n o t c o u n t a g a i n s t t h e program.  I t would be o f g r e a t i m p o r t a n c e i f t h e r e was  a s t r o n g need f o r the program and t h e r e were no a v a i l a b l e a l t e r n a t i v e s . In t h a t s i t u a t i o n t h e r e s h o u l d be a c o n t i n u o u s c o l l e c t i o n o f f i e l d t r i a l data.  89 A summary v e r s i o n o f t h i s s t a n d a r d c o u l d be s i m i l a r t o t h e f o l l o w i n g : ( S c r i v e n , 1974b, p. 2 8 ) . Considerations  Rating  Postmarketing data c o l l e c t i o n ? P o s t m a r k e t i n g system f o r improvement? In-service training? Up-dating o f aids? New u s e r s and u s e r d a t a ?  -  E x c e l l e n t and comprehensive Good and f a i r l y c o m p r e h e n s i v e Minimally acceptable Weak — l e s s than a d e q u a t e N e g l i g i b l e -- a p p a r e n t l y none  4 3 2 1 0  3.73 The P r e c e d e n c e o f t h e The S t a n d a r d s i n Ranking As a r g u e d i n C h a p t e r 3 , t h e s t a n d a r d o f Goal Worth t a k e s precedence o v e r f a l l t h e other standards.  I f t h e Goals o f t h e evaluatum  are n o t w o r t h w h i l e t h e n t h e e v a l u a t u m s h o u l d n o t e x i s t .  When comparing  d i f f e r e n t e v a l u a t a , i t i s assumed t h a t t h e g o a l s a r e t h e same f o r a l l o f them. The c r i t e r i u m o f s i m i l a r g o a l s i s n e c e s s a r y i n o r d e r t o compare the e v a l u a t a . T h e r e i s no p o i n t i n comparing Program X t o Program Y i f t h e y a r e f u l f i l l i n g d i f f e r e n t g o a l s . F o r example, one c o u l d a r g u e t h a t we need methods o f t r a n s p o r t i n g p e o p l e from P o i n t A t o P o i n t B and t h e n g i v e a r a t i n g t o t h i s g o a l ; and t h e n c o n s i d e r t h e p r o p o s e d (or o p e r a t i n g ) methods o f f u l f i l l i n g t h a t g o a l . P o s s i b l e methods c o u l d be: by p r i v a t e c a r , by b u s , by t r a i n o r whatever. o f t h e method, r e m a i n s t h e same.  The g o a l , r e g a r d l e s s  What w i l l d e t e r m i n e t h e i n d i v i d u a l  method's o v e r a l l w o r t h , (assuming t h e g o a l s a r e w o r t h w h i l e ) , and hence t h e i r r a n k o r d e r , w i l l be t h e d e g r e e t o which t h e y each a c h i e v e t h e g o a l s , and t h e c o s t o f e a c h .  T h e r e f o r e , assuming t h a t t h e e v a l u a t a -  90  b e i n g compared a r e a l l t r y i n g t o f u l f i l l t h e same g o a l s , and a s s u m i n g that the goals a r e worthwhile;  then t h e s t a n d a r d s o f goal a c h i e v e m e n t  and C o s t w i l l d e t e r m i n e each program's r e l a t i v e w o r t h ( i n t h e c l a s s o f comparison) f o r the purpose o f ranking. w i l l be d i s c u s s e d  (The r o l e o f E x t e n d e d S u p p o r t  later).  I t does n o t a p p e a r t h a t a s i m p l e a v e r a g i n g o f t h e r a t i n g f o r Goal A c h i e v e m e n t and o f t h e r a t i n g f o r C o s t c a n be used t o d e t e r m i n e which evaluatum i s best.  F i g u r e 8 w i l l help c l a r i f y t h i s  F i g u r e W i l l u s t r a t e s t h e same p r o c e d u r e the o v e r a l l worth i n S-grading  statement.  t h a t was used i n d e t e r m i n i n g  u s i n g Goal Worth and Goal A c h i e v e m e n t .  The m a t r i x has been d i v i d e d i n t o f o u r q u a d r a n t s .  I t i s assumed i n t h e  f o l l o w i n g d i s c u s s i o n t h a t t h e e v a l u a t i o n was c o m p l e t e , so t h a t a low r a t i n g i s n o t due t o l a c k o f i n f o r m a t i o n on t h a t s t a n d a r d .  Each  q u a d r a n t w i l l now be d e s c r i b e d : Q u a d r a n t 1: ( h i g h c o s t , low goal a c h i e v e m e n t ) I f t h e c o s t i s h i g h and t h e e v a l u a t u m i s n o t a c h i e v i n g t h e g o a l s , then i t seems r e a s o n a b l e t o r e j e c t i t f r o m f u r t h e r c o n s i d e r a t i o n . The c o s t o f a program i s a r e l a t i v e s t a n d a r d , i . e . h i g h c o s t means t h a t an e v a l u a t u m c o s t s much more than some o t h e r e v a l u a t u m w i t h i n the a v a i l a b l e resources.  I f a l l t h e e v a l u a t a b e i n g compared f a l l i n t o  t h i s q u a d r a n t , then t h e y s h o u l d a l l be r e j e c t e d , and a new method f o r a c h i e v i n g t h e g o a l s s h o u l d be s e a r c h e d f o r . Q u a d r a n t 2: ( h i g h c o s t , g o a l s b e i n g  achieved)  I n . t h i s quadrant, t h e cost i s high, but the evaluatum i s achieving the goals.  T h e q u e s t i o n i s : "Why i s t h e e v a l u a t u m c o s t i n g  91  Figure 8. Combining the Goal Achievement and Cost Ratings Rating f o r Goal Achievement  0 R a t  0  1  2  Quadrant 1  I'  Reject evaluatum.  3  .4  Quadrant 2 Why i s the evaluatum costing so much?  g  r  Quadrant 3  o  3  s t  Quadrant 4  Why i s the evaluatum  The evaluatum i s  not achieving the goals?  acceptable.  4  so much?"  I t may be achieving the goals only because of a high  investment of resources and m a t e r i a l s .  An evaluatum"which has an  equal or greater goal achievement r a t i n g ( i . e . a r a t i n g of "2" o r more) and lower cost i s obviously superior.  I f a l l the evaluata f a l l  i n t h i s quadrant, then they should be c l o s e l y examined as to methods of c u t t i n g the c o s t s . Quadrant 3: (low cost, low goal achievement) Here the cost i s acceptable but the evaluata which f a l l i n t h i s quadrant are not achieving the goals.  The reasons an evaluatum  92 i n q u a d r a n t 3 may not be a c h i e v i n g t h e g o a l s may be e i t h e r b e c a u s e i t i s not a "good" program ( i . e . p o o r l y d e s i g n e d , e t c . ) ; o r i t may because i t has not u t i l i z e d s u f f i c i e n t r e s o u r c e s .  be  I f . a l l the evaluata  b e i n g compared f a l l i n t h i s q u a d r a n t then t h e y s h o u l d be c l o s e l y examined to d e t e r m i n e i f i t i s due t o b e i n g p o o r l y d e s i g n e d programs, o r due t o i n s u f f i c i e n t use o f r e s o u r c e s .  3.74 Ranking t h e E v a l u a t a Q u a d r a n t 4:  (Reasonable t o low c o s t , r e a s o n a b l e t o h i g h g o a l achievement)  This quadrant d e f i n e s the area o f a c c e p t a b i l i t y .  It i s only  here t h a t a s i m p l e a v e r a g e o f the r a t i n g s f o r c o s t and goal a c h i e v e m e n t s h o u l d be c a l c u l a t e d t o d e t e r m i n e t h e rank o r d e r o f the e v a l u a t a . The evaluatum w i t h t h e h i g h e s t a v e r a g e r a t i n g i s the " b e s t buy" T a b l e 1, Chapter 2 ) .  (see  I f two o r more e v a l u a t a have the same h i g h e s t  a v e r a g e r a t i n g i t i s s u g g e s t e d t h a t t h a t one w i t h the h i g h e s t g o a l a c h i e v e m e n t r a t i n g be chosen as t h e b e s t .  If there i s s t i l l a t i e  f o r the " b e s t buy" ( i . e . two o r more have the same goal a c h i e v e m e n t / r a t i n g and the same c o s t r a t i n g ) then t h a t one w i t h p l a n s f o r "extended  s u p p o r t " s h o u l d be g i v e n the h i g h e s t r a t i n g . Thus, the  s t a n d a r d o f extended s u p p o r t i s seen as a f i n a l t i e - b r e a k e r . If none o f the e v a l u a t a b e i n g ranked f a l l i n Quadrant 4 o f F i g u r e ,'8, they can s t i l l be ranked e i t h e r w i t h r e s p e c t t o c o s t s or w i t h r e s p e c t to g o a l a c h i e v e m e n t .  However, t h i s r a n k o r d e r w i l l e s t a b l i s h the b e s t  o f a bad l o t and does not mean t h e " b e s t " evaluatum evaluatum.  i s a "good"  CHAPTER IV Data Cp11ectjon_Technique_s. i n t h e A p p l i c a t i o n °f, t h e E v a l u a t i o n Model. 4.00 I n t r o d u c t i o n T h i s c h a p t e r c a t a l o g u e s t h e d a t a g a t h e r i n g . t e c h n i q u e s used i n . the a p p l i c a t i o n o f t h e e v a l u a t i o n model d e s c r i b e d i n C h a p t e r s and T h r e e .  Two  A l l o f t h e q u e s t i o n n a i r e s and o t h e r t e c h n i q u e s , used t o  g a t h e r d a t a i n the e v a l u a t i o n o f Program B, have been appended to the t h e s i s . The f i r s t p a r t o f C h a p t e r Four i l l u s t r a t e s two s t a g e s i n t h e data gathering procedures.  The f i r s t s t a g e was an o v e r a l l p o r t r a y a l  o f Program B and t h e second s t a g e was d a t a c o l l e c t i o n on s p e c i f i c a s p e c t s o f t h e Program. of Goal Achievement  These s t a g e s a r e r e l a t e d t o t h e s t a n d a r d  by T a b l e ,3.  The f i n a l p a r t o f t h e C h a p t e r  i l l u s t r a t e s how t h e c o l l e c t e d d a t a was r e c o r d e d and v a l i d a t e d .  4.10 I n i t i a l Stage of. Data C o l l e c t i o n The e v a l u a t i o n - o f Program B began a f t e r t h e program had been i n o p e r a t i o n f o r t h r e e months.  Therefore the i n i t i a l stage of  d a t a c o l l e c t i o n had two p u r p o s e s .  F i r s t l y t o d e t e r m i n e which  check-  p o i n t s i n S c r i v e n ' s E v a l u a t i o n C h e c k l i s t s h o u l d be c o n s i d e r e d f i r s t ; and s e c o n d l y , t o p r o v i d e a g e n e r a l i d e a o f t h e  b e n e f i t s - of the  Program to t h e s t u d e n t t e a c h e r s a t t h i s s t a g e i n t h e Program's development.  93  94 The f i r s t purpose was a c c o m p l i s h e d by a s k i n g each o f t h e memb e r s o f t h e development team which of t h e C h e c k p o i n t s on S c r i v e n ' s E v a l u a t i o n C h e c k l i s t s h o u l d be c o n s i d e r e d f i r s t , (see Appendix  A)  The r e s p o n s e s t o t h e C h e c k l i s t i d e n t i f i e d major a r e a s o f i n t e r e s t and c o n c e r n , from t h e p o i n t o f view o f t h e d e v e l o p e r s , and p r o v i d e d direction for setting initial priorities in data-collection. The second p u r p o s e , t h a t i s t o d e t e r m i n e t h e i n i t i a l b e n e f i t s o f t h e program, was a c c o m p l i s h e d by q u e s t i o n i n g t h r e e groups o f p e o p l e t h e s t u d e n t t e a c h e r s , t h e s c h o o l a d m i n i s t r a t o r s and t h e s c h o o l ' s pupils.  The s t u d e n t t e a c h e r s were asked t o r e s p o n d , on a f i v e p o i n t  s c a l e , t o a s e t o f a t t i t u d e s t a t e m e n t s on t h e t e a c h e r t r a i n i n g p r o cess.  The a t t i t u d e s c a l e (see Appendix C) was d e v e l o p e d u s i n g t h e  u s u a l t e c h n i q u e s ( L i k e r t , 1932).  I t was a d m i n i s t e r e d t w i c e , o v e r a  p e r i o d ofaone month, to t h e s t u d e n t t e a c h e r s i n Program B, t o t h e student teachers i n the equivalent year of the regular teacher t r a i n ing program, and t o t h e s t u d e n t t e a c h e r s i n t h e e q u i v a l e n t y e a r o f t h e t r a n s f e r t e a c h e r - t r a i n i n g program (see U.B.C. 1974-75 c a l e n d a r f o r more d e t a i l s o f t h e programs).  T h i s p r o v i d e d a means o f compar-  ing t h e a t t i t u d e s o f t h e program B s t u d e n t • t e a c h e r s w i t h t h e a t t i t u d e s i n the two comparison programs ( t h e r e g u l a r and the t r a n s f e r program). A d m i n i s t e r i n g t h e a t t i t u d e s c a l e t w i c e gave an e s t i m a t e o f s t a b i l i t y of group r e s p o n s e on the q u e s t i o n n a i r e . The s t u d e n t t e a c h e r s were a l s o asked to f i l l i n a " r e a c t i o n form" (see Appendix D).  T h i s provided the evaluator with t h e i r o v e r a l l  r a t i n g o f t h e m e r i t o f t h e program, and w i t h a g r e a t d e a l o f i n f o r m a t i o n on g e n e r a l c o m p l a i n t s and p r a i s e s .  95  The second group t o be q u e s t i o n e d was the s c h o o l a d m i n i s t r a t i o n , t h a t i s , t h e p r i n c i p a l and v i c e - p r i n c i p a l of the s c h o o l .  They were  i n t e r v i e w e d a c c o r d i n g to t h e l i s t of q u e s t i o n s g i v e n i n Appendix B. The t h i r d group to be q u e s t i o n e d was the p u p i l s of t h e s c h o o l . T h i s was a c c o m p l i s h e d  by t a l k i n g t o i n d i v i d u a l s o r small groups  d u r i n g the r e c e s s o f l u n c h b r e a k s .  The c a n d i d r e s p o n s e s of t h i s group  p r o v i d e d a g r e a t d e a l of g e n e r a l i n f o r m a t i o n , not o n l y on t h e o p e r a t i o n of Program B, as seen t h r o u g h t h e p u p i l s eyes, but a l s o on t h e general school c l i m a t e . There a r e two o b v i o u s g r o u p s o m i t t e d from t h i s i n i t i a l the t e a c h e r s i n the s c h o o l and the program d e v e l o p e r s . had been exposed t o numerous s u r v e y s a l r e a d y .  survey,  The  teachers  The School Board  and  the P r o v i n c i a l Government had both been u n u s u a l l y a c t i v e i n s e n d i n g o u t q u e s t i o n n a i r e s i n the 1974-75 y e a r .  F u r t h e r , an e x t e r n a l group  o f e v a l u a t o r s , commissioned by the F a c u l t y o f E d u c a t i o n a t U . B . C , was e v a l u a t i n g a l l t h e a l t e r n a t e programs o f f e r e d i n t e a c h e r educat i o n i n the F a c u l t y .  The l a t t e r group o f e v a l u a t o r s had asked t h e  t e a c h e r s t o f i l l i n an e x t e n s i v e q u e s t i o n n a i r e on Program B j u s t p r i o r t o the i n t e r n a l e v a l u a t i o n b e g i n n i n g .  T h e r e f o r e , i t was  c o n s i d e r e d prudent to d e l a y g i v i n g any q u e s t i o n n a i r e s t o t h e t e a c h e r s a t t h e b e g i n n i n g and i n s t e a d to r e l y on i n f o r m a l " c h a t s " with individual  teachers.  The program d e v e l o p e r s were a l s o o m i t t e d from t h i s i n i t i a l data gathering because of t h e i r frankness i n t h e i r weekly T h e s e m e e t i n g s were a t t e n d e d by the e v a l u a t o r .  meetings.  96  T h i s o v e r v i e w o f Program B was v a l u a b l e f o r a number o f r e a s o n s . I t p r o v i d e d g e n e r a l d a t a on t h e f e e l i n g s o f a l l t h e consumers toward the program; i t s u g g e s t e d major a r e a s o f c o n c e r n o r n e g l e c t f o r more d e t a i l e d i n v e s t i g a t i o n ; i t a l l o w e d t h e e v a l u a t o r t o i n t r o d u c e hims e l f and o u t l i n e the g e n e r a l scheme o f e v a l u a t i o n t o most o f the consumer groups; and i t p r o v i d e d g e n e r a l feedback to a l l t h e g r o u p s i n v o l v e d , p a r t i c u l a r l y t o t h e program d e v e l o p e r s . T h i s i n i t i a l s t a g e o f d a t a c o l l e c t i o n c o u l d a l s o be c a l l e d a ' p o r t r a y a l ' stage of data c o l l e c t i o n .  I t i s p o r t r a y a l t h a t Stake  (1972a) a d v o c a t e d when he w r o t e : "In the more o r d i n a r y and modest s i t u a t i o n ... we s h o u l d l i m i t o u r e v a l u a t i o n aims t o what we can do and t o what t h e c l i e n t needs most. What many c l i e n t s need i s a c r e d i b l e , t h o r o u g h r e p r e s e n t a t i o n o f what t h e program i s , i n c l u d i n g i n f o r m a t i o n about who l i k e s what about i t . " P o r t r a y a l i s u s e f u l (see p r e v i o u s paragraph)  but i s o f t e n o n l y a  s t e p toward an e v a l u a t i o n . ( F o r an example o f t h i s a p p r o a c h Gleadow ( 1 9 7 5 ) ) .  see  T h e r e must be a second s t a g e o f d a t a c o l l e c t i o n  i n o r d e r t o make c o n c l u s i o n s o f the w o r t h o f t h e program.  An  e s t i m a t e o f worth may be p o s s i b l e u s i n g p o r t r a y a l o r i n i t i a l  data  c o l l e c t i o n , but i t would be a guess and e x t r e m e l y d i f f i c u l t i f not i m p o s s i b l e to j u s t i f y .  4.20 Second Stage o f Data C o l l e c t i o n A f t e r t h i s i n i t i a l , ' p o r t r a y a l ' stage of data c o l l e c t i o n the d a t a g a t h e r i n g was d i r e c t e d to s p e c i f i c a r e a s o f the Program.  Each  97 o f t h e t e c h n i q u e s used w i l l be now b r i e f l y d i s c u s s e d . The t e c h n i q u e s have been c a t e g o r i z e d i n T a b l e 3 i n t h i s C h a p t e r .  4.21  E v a l u a t i o n o f U.B.C. Instructors,'-.Teaching. P e r f o r m a n c e S c r i v e n (1973) has condensed many o f t h e c o r r e l a t e s o f  good t e a c h i n g performance  i n t o a short l i s t o f primary i n d i c a t o r s .  I t was t h e s e p r i m a r y i n d i c a t o r s t h a t p r o v i d e d t h e s o u r c e o f many o f t h e q u e s t i o n s asked on Form A (Appendix E) and l a t e r i n t h e program, Form B (Appendix G ) . T h e s e two forms were f i l l e d o u t by t h e s t u d e n t t e a c h e r s d i r e c t l y onto IBM mark c a r d s f o r ease o f a n a l y s i s .  Seven  s t u d e n t - t e a c h e r s were randomly a s s i g n e d t o t h e e v a l u a t i o n o f two i n s t r u c t o r s w h i c h were randomly p a i r e d f o r each s t u d e n t - t e a c h e r , o t h e r words each i n s t r u c t o r was e v a l u a t e d by seven Subsequently,  li  student-teachers.  i t was f e l t t h a t t h i s method was t o o time consuming.  T h e r e f o r e , t h e q u e s t i o n n a i r e was r e v i s e d t o t h e form i l l u s t r a t e d i n Appendix Q.  In t h i s f i n a l form, t h e s t u d e n t - t e a c h e r gaye an o v e r a l l  r a t i n g t o t h e t e a c h i n g performance  o f t h e i n s t r u c t o r s and i n d i c a t e d  i f t h e r e were any i n s t r u c t o r s who were e x c e p t i o n s ( e i t h e r b e t t e r t h a n or worse than t h e r e s t o f t h e g r o u p ) t o t h e o v e r a l l r a t i n g .  This  s i m p l i f i e d t h e f i l l i n g o u t o f t h e q u e s t i o n n a i r e , and t h e c o l l e c t i o n and i n t e r p r e t a t i o n o f d a t a , as w e l l as i d e n t i f i e d e x c e p t i o n a l instructors.  The q u e s t i o n n a i r e s a l s o i n c l u d e an o v e r a l l e v a l u a t i o n o f  t h e t e a c h i n g performance  o f each i n s t r u c t o r . S c r i v e n (1973, p. 13)  wrote about t h i s o v e r a l l judgment o f m e r i t :  98  "Even though we do n o t know e x a c t l y what i t s r e l i a b i l i t y i s , a t l e a s t ( a ) t h e r e s u l t s cannot s o e a s i l y be w r o n g l y i n t e r p r e t e d and (b) t h e chances t h a t i t ' s n e g a t i v e l y c o r r e l a t e d w i t h good t e a c h i n g a r e p r o b a b l y lower than w i t h any o t h e r s i n g l e q u e s t i o n ( o r c o m b i n a t i o n o f a few q u e s t i o n s ) . " T h i s r e a s o n i n g c a n a l s o be a p p l i e d t o o t h e r a r e a s o f c o n c e r n ; as a r e s u l t an o v e r a l l judgement was asked f o r . o n a number o f o t h e r a s p e c t s o f t h e program.  The e v a l u a t i o n o f t h e U.B.C. i n s t r u c t o r s '  t e a c h i n g performance was a p r o c e s s  check.  4.22 Q u e s t i o n n a i r e t o S t u d e n t t e a c h e r s on t h e i r A l t e r n a t e Placement  (Appendix F)  D u r i n g t h e c o u r s e o f Program B, t h e s t u d e n t t e a c h e r s were p l a c e d i n o t h e r s c h o o l s f o r a two week p r a c t i c u m . s c h o o l was chosen by t h e s t u d e n t t e a c h e r .  The t y p e o f  This q u e s t i o n n a i r e pro-  v i d e d a check on t h e p e r c e i v e d s u c c e s s o f t h a t e x p e r i e n c e .  4.23 I n t e r v i e w o f S t u d e n t T T e a c h e r s  (Appendix H)  T h r e e randomly chosen s t u d e n t t e a c h e r s were i n t e r v i e w e d o v e r t h e space o f a few weeks. o n e - h a l f hour.  Each i n t e r v i e w t o o k  approximately  The i n t e r v i e w s were r e c o r d e d on t a p e w i t h t h e p e r -  mission of the interviewee.  4.24 F i n a l q u e s t i o n n a i r e f o r t h e S t u d e n t T e a c h e r s (Appendix I) The s o u r c e o f a l l t h e q u e s t i o n s on t h i s q u e s t i o n n a i r e was t h e o r i g i n a l d e s c r i p t i o n s o f t h e program ( s e e C h a p t e r 3-Sec. 3.11  99  "The E v a l u a t u m " ) .  I t s purpose was t o o b t a i n a f i n a l s u b j e c t i y e  c o n g r u e n c y c h e c k between what was s a i d was g o i n g t o o c c u r , and what a c t u a l l y o c c u r r e d .  4.25 O b s e r v a t i o n o f S t u d e n t T e a c h i n g F i f t e e n o f t h e s t u d e n t t e a c h e r s were o b s e r v e d for a t l e a s t one l e s s o n e a c h , by t h e e v a l u a t o r , d u r i n g t h e i r l a s t c l a s s r o o m teaching experience.  T h e r e were two a r e a s o f c o n c e r n ,  Firstly a  p r o c e s s check on c l a s s r o o m j u s t i c e was needed and s e c o n d l y an i.  • •  attempt to connect observed student teacher behaviors i n t h e c l a s s room and t h e i r Program B e x p e r i e n c e , t h a t i s , a c a u s a l check u s i n g t h e "modus o p e r a n d i " method ( s e e C h a p t e r 3, s e c t i o n 3.337) was attempted. C l a s s r o o m j u s t i c e d a t a was c o l l e c t e d by d i r e c t o b s e r v a t i o n , w i t h p a r t i c u l a r a t t e n t i o n b e i n g p a i d t o t h e s t u d e n t t e a c h e r s ' methods of d i s c i p l i n e .  I t was f e l t t h a t d i s c i p l i n e t e c h n i q u e s would be  good i n d i c a t o r s o f t h e f a i r n e s s w i t h w h i c h t h e s t u d e n t t e a c h e r s ' : d e a l t w i t h t h e p u p i l s . T h e r e f o r e , each time a p u p i l o r p u p i l s were d i s c i p l i n e d , a note was made on t h e r e a s o n f o r t h e d i s c i p l i n i n g , t h e t y p e o f d i s c i p l i n e , t e c h n i q u e used, and whether t h e d i s c i p l i n e t e c h n i q u e was e f f e c t i v e .  The d i s c i p l i n e t e c h n i q u e s were c a t e g o r i z e d  beforehand (see Appendix J ) . A f t e r the l e s s o n , t h e student teacher was a s k e d by t h e e v a l u a t o r why c e r t a i n t e c h n i q u e s were used i n particular situations.  The p u p i l s were a s k e d i n a q u e s t i o n n a i r e i f  t h e y f e l t t h e y were f a i r l y t r e a t e d by t h e s t u d e n t t e a c h e r s  (Appendix  100  M q u e s t i o n s 3 and 7) t o p r o v i d e a n o t h e r check o f c l a s s r o o m j u s t i c e . The modus o p e r a n d i check was p e r f o r m e d by r e c o r d i n g a l l t h e t e c h n i q u e s used by t h e s t u d e n t t e a c h e r and t h e n i n t e r v i e w i n g t h e s t u d e n t t e a c h e r a f t e r t h e l e s s o n t o d e t e r m i n e where he/she g o t t h e i d e a f o r a p a r t i c u l a r technique.(Was  i t h i s / h e r own i d e a ? Was i t a  v a r i a t i o n o f an i d e a from a p r o f e s s o r ? a t e a c h e r ? a t e x t b o o k ? .is t h e s p o n s o r t e a c h e r ' s i d e a ?  Was  Was t h e t e c h n i q u e f r e e l y used o r  was t h e s t u d e n t t e a c h e r t o l d t o u s e t h a t t e c h n i q u e ? ) . G e n e r a l l y , t h e number o f d i f f e r e n t t e c h n i q u e s used i n a l e s s o n was q u i t e l i m i t e d , so t h a t r e c o r d i n g them and i n t e r v i e w i n g t h e s t u d e n t t e a c h e r a f t e r the l e s s o n was n o t a t i m e consuming o r onerous t a s k .  The r e s u l t o f  t h i s was an i n d i c a t i o n o f t h e c o n t r i b u t i o n Program B made i n t r a n s f e r r i n g classroom techniques to the student teachers. E v a l u a t i n g t h e e f f e c t i v e n e s s o f t h e s t u d e n t t e a c h e r s was an area o f p a r t i c u l a r d i f f i c u l t y .  T h e r e were no i n s t r u m e n t s d e v e l o p e d  d u r i n g t h e p e r i o d o f e v a l u a t i o n d e s c r i b e d by t h i s t h e s i s , though i n i t i a l work on t h e i n s t r u m e n t s was begun i n t h i s p e r i o d .  However  methods have now been d e v e l o p e d and a r e b e i n g f i e l d t e s t e d ( a s t h i s t h e s i s i s b e i n g w r i t t e n ) i n t h e second y e a r o f Program B.  For the  sake o f c o m p l e t e n e s s , t h e p r o p o s a l and i n s t r u m e n t s b e i n g used a r e appended t o t h i s t h e s i s a s Appendix U ) .  4.26 S i d e E f f e c t s o f Program B ( T e a c h e r ' s Q u e s t i o n n a i r e ) (Appendix K) As n o t e d i n t h e c o v e r i n g l e t t e r t o t h i s q u e s t i o n n a i r e , t h i s  101  was t h e o n l y q u e s t i o n n a i r e g i v e n t o t h e t e a c h e r s .  I t s p u r p o s e was  to d e t e r m i n e t h e p o s i t i v e and n e g a t i v e s i d e e f f e c t s as p e r c e i v e d by t h e t e a c h e r s .  4.27 I n t e r v i e w with. Program B's D i r e c t o r (Appendix L ) T h i s i n t e r v i e w was tape r e c o r d e d w i t h t h e p e r m i s s i o n o f the interviewee.  I t p r o v i d e d i n f o r m a t i o n on t h e i n i t i a l  o f t h e program, and t h e r e a s o n s f o r i t s i n i t i a t i o n .  planning  It;.was a l s o  used t o d e t e r m i n e t h e v a l i d i t y o f t h e f i e l d t r i a l d a t a a l r e a d y collected.  4.28 Q u e s t i o n n a i r e f o r . School P u p i l s (Appendix M) T h i s q u e s t i o n n a i r e was g i v e n t o two g r a d e f i v e and two grade s i x c l a s s e s .  Each q u e s t i o n was r e a d t o t h e p u p i l s .  I f they  were n o t c l e a r on t h e meaning o f a q u e s t i o n t h e y were asked t o a s k for c l a r i f i c a t i o n .  I t p r o v i d e d i n f o r m a t i o n on t h e t e a c h i n g e f f e c t -  i v e n e s s o f t h e s t u d e n t t e a c h e r s , as p e r c e i v e d by t h e p u p i l s .  4.29 Interviewjwitjx U..B.C.. A d m i n i s t r a t o r s , (Appendix N) The F i e l d D i r e c t o r o f S t u d e n t T e a c h i n g , t h e D i r e c t o r o f Elementary  E d , t h e D i r e c t o r o f S t u d e n t T e a c h i n g ( E l e m e n t a r y ) , and  the A s s o c i a t e Dean were a l l i n t e r v i e w e d ( s e p a r a t e l y ) and each was asked t h e f o u r q u e s t i o n s w r i t t e n i n A p p e n d i x N.  In t h e i n t e r v i e w s and q u e s t i o n n a i r e s , u s u a l l y more than one  102  o f t h e c h e c k p o i n t s o f goal a c h i e v e m e n t was a d d r e s s e d . t h i s , T a b l e 3 has been p r e p a r e d .  To i l l u s t r a t e  The s t a n d a r d o f goal w o r t h , and a  c a t e g o r y o f " e v a l u a t i v e c o n c l u s i o n " have a l s o been added t o t h e table.  The " e v a l u a t i v e c o n c l u s i o n " c a t e g o r y r e f e r s t o a q u e s t i o n  w h i c h asks t h e r e s p o n d a n t t o make an o v e r a l l judgement o f some a s p e c t o f t h e program.  The numbers under each c a t e g o r y o f t h e C h e c k l i s t  i n T a b l e 3, r e f e r t o t h e q u e s t i o n s o f t h e i n t e r v i e w o r on t h e q u e s t i o n n a i r e which were o b t a i n i n g i n f o r m a t i o n a p p l i c a b l e t o t h a t category.  4.210 Other I n f o r m a t i o n  Sources  T a b l e 3 s u g g e s t s t h a t t h e r e a r e a r e a s w h i c h were n e g l e c t e d by t h e i n t e r v i e w s and t h e q u e s t i o n n a i r e s .  Some o f t h e s e ,  such a s l o n g term, s t a t i s t i c a l s i g n i f i c a n c e and c r i t i c a l  comparisons  were n e g l e c t e d because o f t h e c h a r a c t e r o f t h i s e v a l u a t i o n .  As  s t a t e d e a r l i e r i n t h i s t h e s i s , t h i s s t u d y was a f o r m a t i v e e v a l u a t i o n i n S-GRADING.  T h e r e f o r e comparison  g r o u p s were n o t o f c o n c e r n  except f o r e s t a b l i s h i n g a r e f e r a n t point f o r t h e a t t i t u d e q u e s t i o n n a i r e (Appendix C ) . T h e r e were no l o n g term p l a n s b e i n g d e v e l o p e d a t t h e time o f t h e e v a l u a t i o n so t h e r e was no d a t a t o c o l l e c t on t h i s c h e c k p o i n t ( n o t e t h a t i t a l s o r e c e i v e d a v e r y low r a t i n g o n t h e p r o f i l e (Chapt. I l l S e c t i o n 3 . 5 2 2 ) ) .  No measures which c o u l d have  s t a t i s t i c a l p r o c e d u r e s m e a n i n g f u l l y a p p l i e d t o them were t a k e n , except f o r t h e a t t i t u d e q u e s t i o n n a i r e administered t o t h e student t e a c h e r s ; t h e r e f o r e t h e c h e c k p o i n t o f s t a t i s t i c a l s i g n i f i c a n c e was  TABLE 3 ApDendix ' No. , A  Questionnaire  Interview  Respondant  Goal Worth  Developers  X  B  Data Sources for the Goal Achievement Checkpoints*  X  Market  Field Trials  True Long- Side Effects Consumer term  Causation 7  3  5  6  6,7  4,5  2,3  1  School Admin.  Process  Statistical Evaluative Significance Conclusions  Yes  C  X  Student teach.  All  D  X  Student teach.  2  E  X  Student teach.  1-14  page 2  F  X  Student teach.  1,5-9  10,11  G  X  Student teach.  1-12  Student teach.  20  Observation  3,5,6, - -•, 8,9,16, * ° ' 18,19 7  n  7  n  y c  17  1,2,15,20 13  9,10  Student teach.  I J  13  K  School teach.  L  Program Direc.  M  School pupils U.B.C. Administrators  1-4 2,12  1,3,4, 5,6,13  5,6,7, 10,11  8,10 5,9  5,6,14, 16,17  1 * The numbers under each checkpoint refer to the question numbers on the questionnaires noted for each appendix.  page 2  4,9,10,11 12,15,20 1-8,11,12 All  Student teach.  1  14 All  5 1-4,6,7, 8,10,11, 12,13,15, 18,19,20  21  2,3  i—  1  o  104  given a zero r a t i n g . The o t h e r c h e c k p o i n t s w i t h a p p a r e n t l y s m a l l amounts o f d a t a a r e market and t r u e f i e l d t r i a l s .  B u t t h a t i s because t h e d a t a f o r t h e s e  c h e c k p o i n t s were c o l l e c t e d by means o t h e r than q u e s t i o n n a i r e o r f o r m a l i n t e r v i e w s . .Most o f t h e i n f o r m a t i o n f o r market ( t h a t i s , t h e p l a n s f o r i m p l e m e n t a t i o n o f t h e program) was c o l l e c t e d from t h e o r i g i n a l r e c r u i t m e n t m a t e r i a l s s e n t t o t h e s t u d e n t t e a c h e r s , and t h r o u g h t h e weekly meet4ngs~wfl'thtthe> p r o g r a m ^ d e v e l . o p e r s ^ d u r i n g p l a n n i n g was r e v i s e d and/or p l a n n e d .  which t h e program  Much o f t h e t r u e f i e l d t r i a l  i n f o r m a t i o n was o b t a i n e d through ( a ) examining census d a t a f o r t h e a r e a s e r v e d by t h e s c h o o l and comparing t h i s w i t h t h e r e s t n o f t h e c i t y ; (b) examining t h e t r a n s c r i p t s o f marks and o r i g i n a l a p p l i c a t i o n forms o f t h e s t u d e n t t e a c h e r s who a p p l i e d and were a c c e p t e d , a s w e l l as t h o s e who a p p l i e d and were r e j e c t e d ; and ( c ) a s k i n g t h e a d m i n i s t r a t o r s i n t h e f a c u l t y o f e d u c a t i o n t o compare t h e a b i l i t y o f Program B's p r o f e s s o r s as a group, t o t h e r e s t o f t h e f a c u l t y . The c h e c k p o i n t o f t r u e consumers was u n s a t i s f a c t o r i l y  developed.  Gains o r l o s s e s o f t h e s t u d e n t t e a c h e r s were measured t h r o u g h  their  marks on t h e i r t e a c h i n g a b i l i t y and c o u r s e s , and t h r o u g h t h e i r a t t i tudes and o v e r a l l e v a l u a t i o n s o f t h e program.  However, s c h o o l p u p i l  g a i n s o r l o s s e s , t e a c h e r g a i n s o r l o s s e s , U.B.C. p r o f e s s o r s ' g a i n s o r l o s s e s , school a d m i n i s t r a t i o n gains or l o s s e s , U n i v e r s i t y gains or l o s s e s , and School Board g a i n s o r l o s s e s were n o t e s t a b l i s h e d i n sufficient detail.  T h e r e was some e v i d e n c e on t h i s c h e c k p o i n t f o r a l l  o f t h e s e groups however i t was s c a n t y .  Only e s t i m a t e s o f s p e c i f i c  105 g a i n s o r l o s s e s c o u l d be made.  4.30 P r a c t i c a l P r o c e d u r e s f o r t h e O r g a n i z a t i o n o f Data A d i f f i c u l t and i m p o r t a n t t a s k o f t h e e v a l u a t o r i s t o o r g a n i z e and c o n d e n s e t h e d a t a he c o l l e c t s .  The f o l l o w i n g p r o c e d u r e i s  p r e s e n t e d as a f e a s i b l e and w o r k a b l e method d e v e l o p e d f o r t h e t y p e o f e v a l u a t i o n model d i s c u s s e d i n t h i s t h e s i s .  F o r each  checkpoint  (market, t r u e consumer, e t c . ) a c h a r t i s p r e p a r e d as i l l u s t r a t e d i n F i g u r e 9"below:  F i g u r e <9. Format f o r O r g a n i z a t i o n o f Data Checkpoint: Date  Data Source  Information  V a l i d a t i o n Check  Each o f t h e headings i n F i g u r e S w i l l now be d i s c u s s e d : Date: The d a t e t h e i n f o r m a t i o n was c o l l e c t e d .  Comment  106  Data S o u r c e :  The d a t a s o u r c e would i n d i c a t e t h e p e r s o n ( s )  s u p p l i e d t h e d a t a ; the t e c h n i q u e used t o c o l l e c t t h e d a t a  who (e.g.;  q u e s t i o n n a i r e , i n t e r v i e w , i n f o r m a l d i s c u s s i o n , e t c . ) ; and t h e q u e s t i o n asked o r responded to ( i d e n t i f i e d by number i f i t was on a q u e s t i o n n a i r e o r i n a s t r u c t u r e d i n t e r v i e w ) . Information:  An a c c u r a t e summary s t a t e m e n t o f t h e i n f o r m a t i o n  c o l l e c t e d , o r , i f the i n f o r m a t i o n c a n n o t be summarized, a r e f e r e n c e t o the raw d a t a s h o u l d be p r e s e n t e d .  There  should  a l s o be a judgement on t h e p a r t o f the e v a l u a t o r as t o whether t h e i n f o r m a t i o n i s p o s i t i v e , n e g a t i v e o r n e u t r a l (equal amounts, o f p o s i t i v e n e s s and n e g a t i v e n e s s ) w i t h r e s p e c t t o a p a r t i c u l a r c h e c k p o i n t i n Need F u l f i l l m e n t . V a l i d a t i o n Check: I t i s i m p o r t a n t t h a t t h e r e be a c h e c k to d e t e r m i n e i f t h e i n f o r m a t i o n o b t a i n e d was v a l i d .  This usually  means f u r t h e r q u e s t i o n i n g o f program p a r t i c i p a n t s or o b s e r v e r s , a n d / o r o b s e r v a t i o n by t h e e v a l u a t o r . Comment:  T h i s c o u l d be a comment by t h e e v a l u a t o r about what  s h o u l d be done w i t h the i n f o r m a t i o n and whether or not what s h o u l d be done has been done.  For example, i f t h e i n f o r m a t i o n  was n e g a t i v e t h e n t h e e v a l u a t o r m i g h t make a n o t e t h a t t h a t a s p e c t o f the program s h o u l d be r e p o r t e d t o the program d e v e l o p e r s and the problem s h o u l d be e l i m i n a t e d . to see i f i t was e l i m i n a t e d .  L a t e r , he would check  I f the program was changed so as  to e l i m i n a t e t h e problem t h e n the e v a l u a t o r i n h i s f i n a l r e p o r t would use the i n f o r m a t i o n as a c a v e a t t o f u t u r e program u s e r s ,  107 but he would not g i v e a lower r a t i n g t o t h e c h e c k p o i n t  because  of i t .  An example (Figure!©)) i s now o f f e r e d t o c l a r i f y t h e above description.  A l l the data i s f i c t i t i o u s . F i g u r e 1®. Example o f Format, f o r O r g a n i z a t i o n o f Data Checkpoint: Side E f f e c t s  Date  Data Source  Information  V a l i d a t i o n Check  Comment  Feb. 15  Student teachers, Questionn a i r e 1, Quest i o n 3. "Were t h e r e any unexpected p o s i t i v e o r negative expediences i n this program."  12 o f 20 s t u dent t e a c h e r s indicated that the teachers asked them t o participate in Parent-teacher n i g h t . The student-teachers thought the experience v e r y worthwhile (judged positive) ( s e e summary of a l l the comments on t h i s , on page 125 o f d a t a notebooks)  a) The t e a c h e r s were asked f o r their.opinions of this p a r t i c i p a t i o n by t h e student-teachers. They i n d i c a t e d they thought t h a t i t was v a l u a b l e f o r t h e S.T.'s t o p a r t i c i p a t e - no negative f e e l i n g .  Report t h i s to the program d i r e c t o r s , (done Feb. 20) Decision was made to implement this in future programs.  b) Some o f t h e p a r e n t s who came to the parentteacher i n t e r views were quest i o n e d . They i n d i c a t e d they d i d not f i n d t h e presence of the student-teachers distracting.  108 The a d v a n t a g e o f t h i s approach t o r e c o r d i n g t h e d a t a i s t w o - f o l d . F i r s t l y , i t tremendously  s i m p l i f i e s the condensing o f information  when i t comes time t o make a f i n a l r e p o r t ; and s e c o n d l y , i t c l e a r l y o u t l i n e s a l l t h e d a t a w h i c h went i n t o g i v i n g a r a t i n g t o a p a r t i c u l a r checkpoint.  T h i s i s e x t r e m e l y i m p o r t a n t , f o r t h e e v a l u a t o r must be  a b l e t o v e r i f y each c o n c l u s i o n , recommendation and r a t i n g he makes. T h i s i s r e l a t i v e l y easy t o do u s i n g t h e above f o r m a t .  4.40 C o n f i d e n t i a l i t y o f Data If the data sources a r e people o r t h e i r records then s t r i c t c o n f i d e n t i a l i t y must be m a i n t a i n e d .  A l l o f the people involved i n  an e v a l u a t i o n must be a s s u r e d , and b e l i e v e t h a t a s s u r a n c e , t h a t t h e i n f o r m a t i o n c o l l e c t e d w i l l remain anonymous as t o t h e i n d i v i d u a l . In o t h e r words t h e e v a l u a t o r c o u l d i d e n t i f y t h e s o u r c e o f i n f o r m a t i o n as t h e t e a c h e r s , b u t n o t i n d i c a t e what i n d i v i d u a l t e a c h e r s had t o s a y . T h i s can d e c r e a s e t h e power o f s t a t i s t i c a l t e c h n i q u e s which r e l y on p r e - p o s t t e s t i n g o r c o r r e l a t i o n measures; b u t t h e l o s s i n s t a t i s t i c a l power due t o m a i n t a i n i n g anonymity was t h o u g h t t o be j u s t i f i e d by t h e gain i n candid information.  CHAPTER V Summary and  Conclusions  5.00 Summary T h i s s t u d y has e x p l o r e d the p o s s i b i l i t y o f d e v i s i n g a method o f e v a l u a t i o n , based on t h e l o g i c a l method o f Paul W. T a y l o r (1961) i n combination  w i t h the more p r a c t i c a l a p p r o a c h o f S c r i v e n (1974b) which  c o u l d be used i n the e v a l u a t i o n o f a t e a c h e r t r a i n i n g program.  The  goal o f the s t u d y was t o b r i d g e the gap between the t h e o r e t i c a l c o n c e p t s o f T a y l o r (1961) and S c r i v e n , and the p r a c t i c a l problems o f e v a l u a t i n g an e d u c a t i o n a l  program.  A fundamental a s s u m p t i o n o f t h i s s t u d y was t h a t e v a l u a t i o n i s a l o g i c a l procedure  r e g a r d l e s s o f what i s b e i n g e v a l u a t e d .  ". . . i t  i s the l o g i c a l method which a r a t i o n a l person would f o l l o w i f he were t r y i n g t o corne to a c a r e f u l r e f l e c t i v e d e c i s i o n about the v a l u e of something."  ( T a y l o r , 1961, p. 4 ) .  The method o f e v a l u a t i o n  includes adopting a set of standards w i t h i n a p a r t i c u l a r point of view, the o p e r a t i o n a l c l a r i f i c a t i o n o f t h o s e s t a n d a r d s , s p e c i f y i n g a c l a s s o f c o m p a r i s o n , g a t h e r i n g d a t a on the c h a r a c t e r i s t i c s o f the e v a l u a t u m , and d e d u c i n g from the d a t a g a t h e r e d and the  standards  chosen the degree to which the e v a l u a t u m f u l f i l l s the s t a n d a r d s .  The  p r o d u c t o f an e v a l u a t i o n i s a c o n c l u s i o n t h a t the e v a l u a t u m has a c e r t a i n v a l u e , worth o f m e r i t .  That c o n c l u s i o n depends on the r e l a t i v e  p r e c e d e n c e o f the s t a n d a r d s , and the d e g r e e to w h i c h the e v a l u a t u m  109  no  f u l f i l l s the standards. The s t a n d a r d s used i n t h i s s t u d y were: (1) The s t a n d a r d o f g o a l worth and (2) t h e s t a n d a r d o f goal a c h i e v e m e n t .  These s t a n d a r d s were  used f o r b r o a d l y c a t e g o r i z i n g t h e two main components o f any good e d u c a t i o n a l program, ( i . e . something w o r t h w h i l e i s b e i n g t r a n s m i t t e d i n a m o r a l l y a c c e p t a b l e manner).  The s t a n d a r d s were used i n t h e  e v a l u a t i o n o f a t e a c h e r t r a i n i n g program (Program B) from t h e educat i o n a l , i n s t i t u t i o n a l , moral and l e g a l p o i n t s o f view. The methodology d e v e l o p e d i n t h e s t u d y , was i l l u s t r a t e d by applying i t t o the formative evaluation o f a school-based, t r a i n i n g program.  teacher  The e v a l u a t i o n was an example o f S-GRAUING.  That  i s , t h e t e a c h e r t r a i n i n g program was n o t d i r e c t l y compared t o o t h e r , e x i s t i n g , - s i m i l a r programs; r a t h e r , i t was graded a c c o r d i n g t o t h e 0  degree t o which i t f u l f i l l e d a s e t o f a p p r o p r i a t e s t a n d a r d s . When GRADING, t h e e v a l u a t o r d e t e r m i n e s a s e t o f c r i t e r i a o r f u l f i l l m e n t c h a r a c t e r i s t i c s which o p e r a t i o n a l l y c l a r i f y t h e s t a n d a r d s used i n t h e e v a l u a t i o n . The c r i t e r i a a r e d e r i v e d from an i d e a l  form  o f t h e program b e i n g e v a l u a t e d , and w i l l t h e r e f o r e d i f f e r from one type o f educational product to the next.  In t h i s study, the c r i t e r i a  used were t h o s e s u g g e s t e d by Broudy i n h i s paper on t h e p r o f e s s i o n a l p r e p a r a t i o n o f t e a c h e r s . (Broudy, 1965).  The t e a c h e r t r a i n i n g  program (Program B) was e v a l u a t e d by comparing  both t h e g o a l s o f  Program B and t h e method and degree o f a c h i e v i n g t h o s e g o a l s t o Broudy's c r i t e r i a ; and, on t h e b a s i s o f t h a t c o m p a r i s o n , j u d g i n g whether t h e  <  ••~\  111  program's g o a l s were w o r t h w h i l e and b e i n g a c h i e v e d i n a m o r a l l y a c c e p t a b l e manner. I f the c l i e n t f o r the e v a l u a t i o n q u e s t i o n s t h e f i n a l judgement o f w o r t h , then the e v a l u a t o r would f i r s t show t h a t the program met the c r i t e r i a t o the d e g r e e i n d i c a t e d by t h a t f i n a l judgement o f w o r t h . I f the c l i e n t q u e s t i o n n e d the c r i t e r i a used i n d e f i n i n g the s t a n d a r d s , then t h e e v a l u a t o r must show t h a t t h o s e s t a n d a r d s were a p p r o p r i a t e f o r t h e program b e i n g e v a l u a t e d , and t h a t meeting t h e c r i t e r i a used to o p e r a t i o n a l i z e t h e s t a n d a r d s i n d i c a t e d m e r i t o r i o u s  performance.  F i g u r e ] i T c l a r i f i e s the e v a l u a t i o n p r o c e d u r e s d e v e l o p e d i n t h i s study.  More i n f o r m a t i o n can be o b t a i n e d by r e f e r r i n g t o t h e s e c t i o n s o f  t h e t h e s i s n o t e d i n each s t e p o f t h e F i g u r e . c a u t i o u s l y i n t e r p r e t e d as a temporal sequence.  F i g u r e T/b s h o u l d be In p r a c t i s e , the e v a l u a t o r  may e n t e r t h e program a f t e r the g o a l s have been e s t a b l i s h e d and a f t e r t h e program has begun o p e r a t i o n .  In t h a t c a s e , he would.have t o g a t h e r  data on t h e g o a l s and the goal achievement  c r i t e r i a a t the same t i m e .  S e c t i o n 3.70 showed how the e v a l u a t i o n p r o c e d u r e s d e v e l o p e d f o r S-GRADING c o u l d be l o g i c a l l y extended f o r RANKING. i n c l u d e d t o summarize t h a t e x t e n s i o n .  Appendix H h a s been  112 F i g u r e 11 S t e p s i n S-GRADING i n FORMATIVE EVALUATION  Determine t h e e v a l u a t u m and t h e p o i n t s o f view t o be t a k e n i n t h e E v a l u a t i o n . ( S e c . 3.11, S e c . 3.12)  E s t a b l i s h the goals o f the e v a l uatum ( S e c . 3.31)  * 1  <  Determine, the wor t h o f t h e e v a l uatum's g o a l s . (S e c . 3.521)Are the goals worthwhile? (Sec. 3.521)  YES  NO  Are other.goals pursued?  YES  NO  E s t a b l i s h the goal a c h i e v e m e n t c r i t e r i a . ( S e c . 3.'32%)  F u r t h e r e v a l u a t i o n would be pointless. "  Determine t h e " d e g r e e t o which the e v a l u a t u m i s a c h i e v i n g t h e g o a l s . ( S e c . 3.522) .  Is t h e e v a l u a t u m a c h i e v i n g t h e g o a l s ? ( S e c . 3.522) YES Determine the,,overal 1 worth o f the e v a l u a t u m ( S e c . 3.60)  being  Revise the evaluatum.F  1  •NO-  Is t h e - e v a l u a t u m a c h i e v i n g •(other g o a l s ? lis i t p o s s i b l e t o r e v i s e (the evaluatum?  YES  E v a l u a t u m has w o r t h w h i l e g o a l s p u t i s not achieving those goals, S T O p  C o n c l u s i o n : X was t h e t r e a t m e n t , X r e s u l t e d i n the a c h i e v e m e n t o f t h e w o r t h w h i l e g o a l s Y, t h e s u b j e c t s ( S ' s ) d e s i r e d , e n j o y e d o r were b e n e f i t t e d by Y. STOP  113 5.10  Conclusions A g e n e r a l : ' c o n c l u s i o n o f t h i s study i s t h a t T a y l o r ' s (1961) a n a l y s i s  o f the method and p r o d u c t o f e v a l u a t i o n can p r o v i d e a framework f o r the e v a l u a t i o n o f an e d u c a t i o n a l program.  More s p e c i f i c a l l y when T a y l o r ' s  a n a l y s i s i s combined w i t h S c r i v e n ' s e x t e n s i v e work and c h e c k l i s t , a f e a s i b l e model o f e v a l u a t i o n r e s u l t s , w h i c h r e a d i l y p r o d u c e s a d e f e n s i b l e , o v e r a l l e s t i m a t i o n o f w o r t h f o r an e d u c a t i o n a l  product.  T a b l e 4 has been p r e p a r e d t o p o i n t o u t how t h e model o f e v a l u a t i o n developed  i n t h i s t h e s i s d i f f e r s from the more p o p u l a r models o f educa-  tional evaluation.  On the l e f t o f t h e t a b l e a r e t h e l i m i t a t i o n s o f some  o f the p r e s e n t e d u c a t i o n a l e v a l u a t i o n models (see S e c t i o n 2.10).  On  the r i g h t o f the t a b l e are the c o n t r i b u t i o n s o f t h i s t h e s i s toward overcoming those l i m i t a t i o n s .  Table  4.  Comparison o f E v a l u a t i o n  L i m i t a t i o n s o f some p r e s e n t models o f e v a l u a t i o n (see Sec. 2.10). 1)  Procedures.  Contributions of t h i s thesis toward o v e r c o m i n g the l i m i t a t i o n s .  Tyler's"Model a) Focuses on the p r o d u c t not on the p r o c e s s o f evaluation.  and  a) Both p r o d u c t and p r o c e s s a r e o f c o n c e r n i n the e v a l u a t i o n (see ' p a r t i c u l a r l y Sec. 3.336, p. 6 3 ) .  b) F o c u s e s d i r e c t l y on measurea b l e g o a l s and o b j e c t i v e s but makes no d i r e c t j u d g e ments on the worth o f t h o s e goals  b) D e t e r m i n i n g the worth o f the g o a l s i s one o f the most i m p o r t a n t s t e p s i n the e v a l u a t i o n . I f a program has w o r t h l e s s g o a l s then i t has no m e r i t (see Sec. 3.21, p. 45, 3.60, p. 81) i  114  Table 4 (Cont.) 2)  Accreditation. a) Does n o t c o n n e c t the o b s e r v e d c h a r a c t e r i s t i c s o f the p r o gram to t h e i r consequences on the program p a r t i c i p a n t s .  3)  S t u f f l e b e a m ' s CIPP Model. a) P r o v i d e s d a t a t o d e c i s i o n makers but makes no d e c i s i o n on the worth o f t h a t d a t a or on the worth o f t h e product being evaluated.  4)  Stake's  a) Both the q u a l i t y o f d a t a and the o v e r a l l worth o f t h e p r o gram a r e j u d g e d . (Sec. 3.32, p. 52, 3.60, p. 81)  Model.  a ) ^ I t i s n o t c l e a r how t h e worth o f the program g o a l s is determined. 5)  a) A l l d a t a g a t h e r e d i s j u d g e d as t o whether i t i s c o n t r i b u t i n g to the a c h i e v e m e n t o f the w o r t h w h i l e g o a l s . ( S e c . 3.521, p. 75; Sec. 3.522, p. 7 8 ) .  a) See l b above and S e c t i o n 3.521, p. 75.  A l l Models. a) T h e r e i s no c l e a r l y d e f i n e d method o f combining and w e i g h t i n g the d a t a i n o r d e r to a r r i v e a t an o v e r a l l det e r m i n a t i o n o f worth f o r t h e program.  a) A c l e a r , l o g i c a l system o f w e i g h t i n g t h e d a t a o f the c r i t e r i a and c o m b i n i n g t h e s e to d e t e r m i n e t h e o v e r a l l worth has been p r e s e n t e d ( S e c t i o n s 3.522, p. 78, Sec. 3.60, p. 81.)  b) The models do not have a s p e c i f i c check on t h e moral a c c e p t a b i l i t y o f the program.  b) A check o f t h e moral a c c e p t a b i l i t y o f t h e program i s an i n t e g r a l p a r t o f the e v a l u a t i o n . I f a program i s immoral, then t h a t alone i s s u f f i c i e n t reason to make the program u n a c c e p t a b l e . ( S e c . 3.22, Sec. 3.60)  115 S c r i v e n ' s r e c e n t c h e c k l i s t f o r e v a l u a t i n g p r o d u c t s and p r o p o s a l s ( S c r i v e n 1974b) p r o v i d e d many o f t h e c h e c k p o i n t s used i n o p e r a t i o n a l i z i n g t h e s t a n d a r d o f Goal A c h i e v e m e n t (Sec. 3.33).  However, two  changes were made i n S c r i v e n ' s C h e c k l i s t t o make i t more c o n g r u e n t T a y l o r ' s (1961) a n a l y s i s o f t h e p r o c e s s o f e v a l u a t i o n .  with  F i r s t l y , the  C h e c k p o i n t "Need" was r e c o n c e p t u a l i z e d and g i v e n s e p a r a t e s t a t u s as a standard i n the evaluation. worth".  As a s t a n d a r d i t was reworded t o "goal  I t was reworded because "need" has a m u l t i p l i c i t y o f meanings  ( e . g . , as an o b j e c t i v e ; a n e c e s s i t y , a d e f i c i e n c y , an o b l i g a t i o n (Komisar, 1 9 6 1 ) ) , and u s i n g any s i n g l e meaning may be t o o r e s t r i c t i v e .  A g o a l , on  the o t h e r hand, c a n be any i n t e n d e d outcome, t h e w o r t h o f w h i c h c a n be determined  by a p p l y i n g t h e s t r a t e g y o u t l i n e d i n S e c t i o n 3.21.  Goal  w o r t h was g i v e n s e p a r a t e s t a t u s a s a s t a n d a r d i n t h e e v a l u a t i o n f o r two r e a s o n s : (1) To u n d e r l i n e t h e importance o f d e t e r m i n i n g w h e t h e r t h e g o a l s o f a program a r e w o r t h w h i l e ,  (2) T o c l a r i f y t h e c o n t r i b u -  t i o n o f "goal w o r t h " i n t h e o v e r a l l m e r i t r a t i n g o f t h e program. Secondly, i n t h i s t h e s i s the Checkpoint o f " C r i t i c a l Comparisons" was d e l e t e d from S c r i v e n ' s C h e c k l i s t . S c r i v e n e x p l a i n s , " C r i t i c a l Comparisons" i n two ways.  F i r s t l y , i t i s used t o d e t e r m i n e t h e  a p p r o p r i a t e n e s s o f t h e h y p o t h e t i c a l o r i d e a l form o f t h e program used i n S - G r a d i n g ; and s e c o n d l y i t i s an a c t u a l comparison  o f how t h e  program b e i n g e v a l u a t e d compares w i t h o t h e r , s i m i l a r programs.  That i s ,  how i t would be RANKED. The second e x p l a n a t i o n o f " C r i t i c a l C o m p a r i s o n s " i s a p r o c e d u r a l t e c h n i q u e , a n d , though i t r e s u l t s i n a judgement o f t h e r e l a t i v e  116 v a l u e o f a program, i t does n o t c o n t r i b u t e d i r e c t l y t o t h e worth o f the program.  T h e r e f o r e i t s h o u l d n o t be i n c l u d e d i n t h e C h e c k l i s t .  For example, a poor program, when compared t o o t h e r s i m i l a r p r o g r a m s , may be t h e b e s t o f t h e l o t ; b u t t h a t judgement does n o t make i t b e t t e r t h a n i t was b e f o r e i t was compared t o t h e o t h e r programs. The f i r s t e x p l a n a t i o n o f " C r i t i c a l C o m p a r i s o n " , t h a t i t i s an i n d i c a t i o n o f t h e a p p r o p r i a t e n e s s o f t h e i d e a l f o r m o f the; program used i n S - G r a d i n g , * - i ^ a n r e v a l u a t i o n o f t h e e v a l u a t i o n .  The i d e a l  form o f t h e program c h o s e n f o r S - G r a d i n g w i l l d e t e r m i n e t h e c r i t e r i a used i n e s t a b l i s h i n g t h e s t a n d a r d s . I f t h e c r i t e r i a were so e a s y t o a t t a i n t h a t even a p o o r program c o u l d g e t h i g h marks, t h e n t h a t f a u l t r e f l e c t s on t h e e v a l u a t i o n n o t on t h e program b e i n g e v a l u a t e d .  The  realism o f the c r i t e r i a i s r e f l e c t e d i n the d e f e n s i b i 1 i t y o f the ideal to b o t h t h e c l i e n t and t h e e d u c a t i o n a l  community.  The p a r a l l e l i n s t a t i s t i c s , t o t h e p r e c e e d i n g p a r a g r a p h , i s a d i s c u s s i o n o f TYPE I a n d TYPE I I e r r o r s .  Suppose we worded t h e o u t -  comes o f an e v a l u a t i o n i n t h e f o l l o w i n g s t a t i s t i c a l manner: H : The program i s a ppor program. G  H-|: The program i s a good program. I f o u r c r i t e r i a a r e so e a s y t o a t t a i n t h a t we r e j e c t H i n 0  f a v o u r o f H-| when we s h o u l d have a c c e p t e d H , t h e n a k i n d o f TYPE I 0  e r r o r has been committed.  I f , on t h e o t h e r hand, we a r e so s e v e r e d i n  our c h o i c e o f c r i t e r i a f o r t h e e v a l u a t i o n t h a t we a c c e p t H when we 0  s h o u l d have r e j e c t e d i t , t h e n a k i n d o f TYPE I I e r r o r has been committed.  117  T h e r e f o r e , as i n s t a t i s t i c s , the c h o i c e o f c r i t e r i a f o r an e v a l u a t i o n i s an a t t e m p t t o m i n i m i z e b o t h TYPE I and TYPE II e r r o r s . As s u c h , t h e c h e c k p o i n t o f C r i t i c a l Comparisons can be used as a method o f s c r u t i n i z i n g t h e c h o i c e o f c r i t e r i a , but does n o t d i r e c t l y c o n t r i b u t e t o t h e worth o f t h e program b e i n g e v a l u a t e d . F o r t h i s r e a s o n and t h e r e a s o n p r e v i o u s l y g i v e n , " C r i t i c a l C o m p a r i s o n s " was o m i t t e d f r o m t h e c h e c k l i s t o u t l i n e d i n Sec. 3.33. The e f f e c t o f t h e e v a l u a t o r e n t e r i n g a program i s an i m p o r t a n t c o n c e r n which was n o t f u l l y i n v e s t i g a t e d i n t h i s t h e s i s .  However, i t  was a p p a r e n t t h a t some p r o b l e m s , which would have r e m a i n e d as i n t e r n a l g r u m b l i n g s and been a c c e p t e d by t h e program p a r t i c i p a n t s w i t h o u t comment, r a p i d l y g a i n e d more s t a t u r e t h a n t h e y d e s e r v e d , because t h e r e was an e v a l u a t o r p r e s e n t who r e c o r d e d them.  The a p p e a r a n c e o f t h e e v a l u a t o r  tended t o make many uncommitted p a r t i c i p a n t s o f t h e program more c r i t i c a l , and many committed p a r t i c i p a n t s more d e f e n s i v e . A l t h o u g h t h e methodology o f e v a l u a t i o n developed,; by t h i s t h e s i s was o n l y a p p l i e d t o a t e a c h e r t r a i n i n g program, i t would a p p e a r t h a t i t c o u l d e a s i l y be e x t e n d e d t o t h e e v a l u a t i o n o f o t h e r e d u c a t i o n a l programs.  The two s t a n d a r d s u s e d , goal w o r t h and goal a c h i e v e m e n t ,  a r e s t a n d a r d s which a p p l y t o a l l e d u c a t i o n a l programs.  The g e n e r a l  e x p l i c a t i o n o f e v a l u a t i o n i n C h a p t e r Two can be a p p l i e d t o t h e e v a l u a t i o n o f any p r o d u c t . The f o r m o f t h e e v a l u a t i o n i l l u s t r a t e d by t h i s t h e s i s was S-GRADING. However, as shown i n Sec. 3.70, t h e e x t e n s i o n t o more c o m p r e h e n s i v e  118 evaluations  i n RANKING i s l o g i c a l l y a p p a r e n t .  F i n a l l y , the methodology d e v e l o p e d i s seen as p r o v i d i n g a l i n k between t h e t h e o r e t i c a l views o f e v a l u a t i o n and t h e i r a p p l i c a t i o n to a p a r t i c u l a r e v a l u a t i o n  study.  118a  REFERENCES A i k e n , W.M. 1942. B r o t h e r s , N.Y.,  The S t o r y o f the E i g h t Y e a r S t u d y . N.Y.  Harper &  -ATkijn, M i , K l e i n , S., F e r n s t e r m a c h e r , G. 1968. The C e n t e r s Changing E v a l u a t i o n Model. E v a l u a t i o n Comment, O c t o b e r , 1968. Baker, Eva. 1974. " F o r m a t i v e E v a l u a t i o n o f I n s t r u c t i o n " , E v a l u a t i o n i n E d u c a t i o n , ed. W.J. Popham.''McCutchan P u b l i s h i n g Corp. B e r k e l e y , C a l i f o r n i a . 1974. pp. 635-584. Bloom, B.S. 1969. "Some T h e o r e t i c a l I s s u e s R e l a t i n g t o E d u c a t i o n a l E v a l u a t i o n " , E d u c a t i o n a l E v a l u a t i o n : New R o l e s , New Means, ed. R. T y l e r . The 68th Annual Yearbook o f the NSSE. U n i v e r s i t y o f Chicago P r e s s , C h i c a g o , 111. 1969. pp. 26-50. Broudy, Harry S. of Teachers." 415.  1965. " C r i t e r i a f o r the P r o f e s s i o n a l P r e p a r a t i o n J o u r n a l o f T e a c h e r E d u c a t i o n . l!6)(4), 1965. pp. 408-  D a n i e l s , LeRoi B. 1971. The J u s t i f i c a t i o n o f C u r r i c u l a . A p a p e r p r e p a r e d f o r d i s c u s s i o n a t the AERA 1971 Annual m e e t i n g i n New York C i t y ( s e s s i o n B 14). Evans, J.W. 1974. " E v a l u a t i n g E d u c a t i o n a l Programs - Are We G e t t i n g Anywhere?" E d u c a t i o n a l R e s e a r c h e r , V o l . 3, No. 8, S e p t . , 1974. pp. 7-12. F r a n k e n a , W.K. 1973. E t h i c s . New J e r s e y . 2nd E d i t i o n .  P r e n t i c e - H a l l , Inc.  Englewood C l i f f s ,  G l a s s , G.V. 1969. The Growth o f E v a l u a t i o n M e t h o d o l o g y . of Educational Research, U n i v e r s i t y of Colorado.  Laboratory  G l a s s , G.V. 1975. "A Paradox about E x c e l l e n c e o f S c h o o l s and the People i n Them." E d u c a t i o n a l "Researcher. March 1975, pp. 9-13. Gleadow, Norman E. 1975. "The M o b e r l y T o t s Summer S c h o o l Program 1975" Research Report ( i n P r i n t ) . E v a l u a t i o n and R e s e a r c h , E d u c a t i o n S e r v i c e s Group, Vancouver School B o a r d , 1595 West 10th Ave. V a n c o u v e r , B.C. Canada Guba, Egon G. 1969. "The F a i l u r e o f E d u c a t i o n a l E v a l u a t i o n " E d u c a t i o n a l T e c h n o l o g y , May 1969. pp. 29-38.  118b  Hall, 1966. "The A p p l i e d S o c i o l o g i s t and O r g a n i z a t i o n a l S o c i o l o g y " i n S o c i o l o g y i n A c t i o n . Ed. A.B. S h o s t a k . Dorsey Press. H a l l e r , E . J . 1974. "Cost A n a l y s i s f o r E d u c a t i o n a l Program E v a l u a t i o n , " E v a l u a t i o n i n E d u c a t i o n . Ed. W.J. Popham. McCutchan P u b l i s h i n g C o r p o r a t i o n , B e r k e l e y , C a l i f o r n i a , pp. 406-449. Hayes, Samuel P. ( J r . ) 1969. E v a l u a t i n g Development P r o j e c t s . ( 5 t h E d i t i o n ) U n i t e d N a t i o n s E d u c a t i o n a l , S c i e n t i f i c and C u l t u r a l O r g a n i z a t i o n . P l a c e de F o n t e r o y , P a r i s . H e m p h i l l , John K. 1969. "The R e l a t i o n s h i p between Research and E v a l u a t i o n S t u d i e s " , E d u c a t i o n a l E v a l u a t i o n , New R o l e s , New Means. Ed. R. T y l e r . The 68th Annual Yearbook o f the NSSE. U n i v e r s i t y o f C h i c a g o P r e s s , C h i c a g o , 111. pp. 189-220. House, E r n e s t R. 1973. School E v a l u a t i o n : The P o l i t i c s and McCutchen P u b l i s h i n g C o r p o r a t i o n . C a l i f o r n i a , U.S.A.  Process.  K e r l i n g e r , F r e d N. 1973. F o u n d a t i o n s o f B e h a v i o r a l R e s e a r c h . H o l t , R i n e h a r t and W i n s t o n , I n c . T o r o n t o , Ont.  2nd  Ed.  Komisar, B. Paul 1961. Needs and the N e e d s - C u r r i c u l u m . Language and C o n c e p t s i n E d u c a t i o n . S m i t h , B.O. and E n n i s , R.H~ (Ed's) pp. 24-43. Rand M c N a l l e y & Co., C h i c a g o . L i k e r t , R. 1932. "The method o f C o n s t r u c t i n g an A t t i t u d e S c a l e " A r c h i v e s o f P s y c h o l o g y , 140? pp. 44-53. M e t f e s s e l , N.S., M i c h a e l , W.B. 1967. "A Paradigm I n v o l v i n g M u l t i p l e C r i t e r i o n Measures f o r E v a l u a t i n g the E f f e c t i v e n e s s o f School Programs" E d u c a t i o n a l and P s y c h o l o g i c a l Measurement. 27. p. 931943. P e t e r s , R.S. London.  1966.  E t h i c s and E d u c a t i o n .  George A l l e n & Unwin L t d .  Popham, W.J. 1969. " E d u c a t i o n a l Needs A s s e s s m e n t . " In C u r r i c u l u m T h e o r y Network Monograph-Supplement C u r r i c u l u m E v a l u a t i o n : P o t e n t i a l i t y and R e a l i t y Ed. J o e l W e i s s . The O n t a r i o I n s t i t u t e f o r S t u d i e s i n E d u c a t i o n . 1972. p. 22-32. Popham, W.J. 1972. An E v a l u a t i o n Guidebook. The I n s t r u c t i o n a l O b j e c t i v e s Exchange, Lo.scAngeles, C a l i f o r n i a . Popham, W.J. cations.  1974. E d i t o r . E v a l u a t i o n i n E d u c a t i o n : C u r r e n t A p p l i McCutchan P u b l i s h i n g C o r p . B e r k e l e y , C a l i f o r n i a .  118c  P r o v u s , M. 1969. " E v a l u a t i o n o f Ongoing Programs i n the P u b l i c School System". E d u c a t i o n a l . E v a l u a t i o n : New R o l e s , New Means. Ed. Paul T y l e r P a r t I I : The 68th Yearbook o f the NSSE. U n i v e r s i t y o f Chicago P r e s s , C h i c a g o , 111, pp. 242-283. P r o v u s , M. 1970. " E v a l u a t i o n o r R e s e a r c h , R e s e a r c h o r E v a l u a t i o n ? " E d u c a t i o n a l T e c h n o l o g y , Aug. 1970. pp. 50-54. P r o v u s , M. 1972. " E v a l u a t i o n as P u b l i c P o l i c y " , C u r r i c u l u m Theory Network. Monograph Supplement C u r r i c u l u m E v a l u a t i o n : P o t e n t i a l i t y and R e a l i t y . Ed. J o e l W e i s s . The O n t a r i o I n s t i t u t e f o r S t u d i e s i n E d u c a t i o n . 1972. Pp. 33-44. R o s s i , P. 1966. "Boobytraps and P i t f a l l s i n the E v a l u a t i o n o f S o c i a l A c t i o n Programs", P r o c e e d i n g s : Annual Meeting o f the A m e r i c a n S t a t i s t i c a l A s s o c i a t i o n , W a s h i n g t o n , D.C. ^'j^ S c r i v e n , M. 1966a. V a l u e C l a i m s i n the S o c i a l S c i e n c e . Bounder, C o l o r a d o , S o c i a l S c i e n c e E d u c a t i o n C o n s o r t i u m P u b l i c a t i o n No. 123. S c r i v e n , M. 1966b. "Causes, C o n n e c t i o n s and C o n d i t i o n s i n H i s t o r y , " P h i l o s o p h y o f E d u c a t i o n a l R e s e a r c h . Ed. H.S. Broudy, R.H. E n n i s , L . I . Krinerman. John W i l e y & Sons, Inc. T o r o n t o 1973. pp. 439458. S c r i v e n , M. 1967. "The Methodology o f E v a l u a t i o n " . P e r s p e c t i v e s o f C u r r i c u l u m E v a l u a t i o n . AERA Monograph s e r i e s on C u r r i c u l u m Development No. 1, 1967. Chicago 111. Radn M c N a l l y . pp. 39-83. S c r i v e n , M. 1972. " E v a l u a t i o n : Noble P r o f e s s i o n and P e d e s t r i a n P r a c t i s e . " C u r r i c u l u m E v a l u a t i o n P o t e n t i a l . A C u r r i c u l u m Theory Network Monograph. T o r o n t o : O n t a r i o I n s t i t u t e f o r S t u d i e s i n E d u c a t i o n , pp. 132-139. S c r i v e n , M. 1973. "The E v a l u a t i o n o f T e a c h i n g a t B e r k e l e y " paper ( n o t a f i n a l v e r s i o n ) . 10/10/73.  Draft  S c r i v e n , M. 1974a. "The E v a l u a t i o n o f E d u c a t i o n a l G o a l s , I n s t r u c t i o n a l P r o c e d u r e s and Outcomes, o r , the Human Cometh", Program Development i n E d u c a t i o n , Ed. B l a n e y , Horsego and M c i n t o s h . ( C e n t e r f o r Cont i n u i n g E d u c a t i o n , U n i v e r s i t y o f B.C. 1974. V a n c o u v e r , B.C. Canada), pp. 134-162. S c r i v e n , M. 1974b. " E v a l u a t i o n P e r s p e c t i v e s and P r o c e d u r e s " , E v a l u a t i o n i n E d u c a t i o n : C u r r e n t A p p l i c a t i o n s . Ed. W.J. Popham. McCutchan P u b l i s h i n g C o r p o r a t i o n , B e r k e l e y , C a l i f o r n i a , pp. 1-94.  118d  S j o r g e n , Douglas D. 1970. "Measurement T e c h n i q u e s i n E v a l u a t i o n " Review o f E d u c a t i o n a l R e s e a r c h . 40. No. 2. A p r i l 1970. pp. 301-320. S t a k e , R.E. 1967. "The Contenance o f E d u c a t i o n a l E v a l u a t i o n " C o l l e g e Record. 68. pp. 523-540.  Teachers  S t a k e , R?71971. An . E v a l u a t i o n o f TCITY, t h e Twin C i t y I n s t i t u t e f o r T a l e n t e d Youth. S t a k e , R. 1972a. An A p p r o a c h t o t h e E v a l u a t i o n o f I n s t r u c t i o n a l Programs. Paper d e l i v e r e d a t AERA Annual Meeting A p r i l 4, 1972. i n Chicago (ERIC ED064350). S t a k e , R.E. 1972b. " R e s p o n s i v e unpublished manuscript.  Evaluation".  University of Illinois,  S t a k e , R.E. 1972c. "An A p p r o a c h t o t h e E v a l u a t i o n o f I n s t r u c t i o n a l Programs" A paper d e l i v e r e d a t t h e AERA Annual m e e t i n g , A p r i l 4, 1972. i n C h i c a g o . (ERIC a c c e s s no: ED 064350) S t u f f l e b e a m , D.L. 1968. "Toward a S c i e n c e o f E d u c a t i o n a l E v a l u a t i o n " E d u c a t i o n a l T e c h n o l o g y , J u l y 30, 1968. pp. 5-12. T a y l o r , P.W. 1961. N o r m a t i v e D i s c o u r s e . Englewood C l i f f s , N . J .  Prentice-Hall, Inc.  T a y l o r , P.A. and M a g u i r e , T.0. 1966. "A T h e o r e t i c a l E v a l u a t i o n Model" Manitoba J o u r n a l o f E d u c a t i o n R e s e a r c h . 1966, ]_, pp. 12-17. T y l e r , R. 1942. B a s i c P r i n c i p l e s o f C u r r i c u l u m and I n s t r u c t i o n . U n i v e r s i t y o f Chicago Press, Chicago. 34?r 1  T y l e r , R.W. 1969. E d i t o r . E d u c a t i o n a l E v a l u a t i o n : New R o l e s , New Means. P a r t I I o f t h e 68th Annual Yearbook o f t h e NSSE. U n i v e r s i t y o f C h i c a g o P r e s s . C h i c a g o , 111. ; U.S. Dept. o f H e a l t h , E d u c a t i o n and W e l f a r e . ' 1959. A c c r e d i t a t i o n i n H i g h e r E d u c a t i o n . U.S. Government P r i n t i n g O f f i c e , W a s h i n g t o n , D.C. Welch, Wayne W. 1969. " C u r r i c u l u m E v a l u a t i o n " . Review o f E d u c a t i o n a l R e s e a r c h . V o l . 3 9 , No. 4, O c t . 1969. pp. 429-443. Welch, Wayne W. 1972. " E v a l u a t i o n o f t h e PSNS c o u r s e : I : Design and Implementation" J o u r n a l o f R e s e a r c h i n S c i e n c e T e a c h i n g 9^ pp. 139145.  118e Worthen, B. 1968. "Toward a Taxonomy o f E v a l u a t i o n D e s i g n s " . c a t i o n a l T e c h n o l o g y , Aug.,1'5, p. 3-9.  Edu-  Worthen and S a n d e r s . 1973. E d i t o r s . E d u c a t i o n a l E v a l u a t i o n : T h e o r y and P r a c t i c e . C h a r l e s A. J o n e s P u b l i s h i n g . C o . W o r t h i n g t o n , O h i o .  Yt 119  Appendix A Q u e s t i o n n a i r e f o r the Program D e v e l o p e r s Subject: Evaluation P r i o r i t i e s  1  120 Program B EVALUATION PRIORITIES  The persons involved in Program B should help determine the direction and emphasis of an internal evaluation. Therefore, you are asked to indicate on the scale, following each of the topics in the checklist, the Importance and emphasis the topic should receive, This is not a test or attitude survey. The results will be used to help determine initially, which topics should be investigated, the depth of the investigation and the sequence of investigation, 1)  MARKET Is there an implementation plan for ensuring a market for the project? Possible considerations should be: 1. Portability of the program. 2. Clarity, feasibility, ingenuity and economy of the implementation plan. 3. Size of the market for Program B. 4. Importance of the market for Program B. 5. Communications between Program B and potential consumers. Importance of evaluating Market (check one) 4 Extremely important: a comprehensive check should be made. 3  Important: check as thoroughly as you can.  2 Definitely do some checking, 1 Check only i f there is time. _  0 Don't waste time on i t ,  Any comments?  2)  TRUE FIELD TRIALS (performance) How typical is the performance of the Program B program in /s'.v'Yschool of the final, portable version. For example: 1. Are the student-teachers in Program B typical of their group? 2. Are the U.B.C. staff members a representative sample?  121 3.  Is the School representative?  4.  Are any major changes seen for the final version?  Importance of evaluating True Field Trials: (check one) 4 Extremely important: a comprehensive check should be made. 3 Important: check as thoroughly as you can. 2 Definitely so some checking, 1  Check only i f there is time,  0 Don't waste time on i t . Any comments?  3.  TRUE CONSUMER (performance) How do significant consumer groups gauge the performance of Program B. Examples of significant consumers would be: 1. The student-teachers, 2. The U.B.C. staff. 3. The*Lr.'\- 3st/;5» school teachers and principal, 4. The students in the school. 5. Vancouver School Board. ,  i  Importance of evaluating True Consumers: (check one) 4 Extremely important: a comprehensive check should be made. 3  Important: check as thoroughly as you can.  2 Definitely do some checking. 1  Check only i f there is time.  0 Don't waste time on i t . Any comments?  122 CRUCIAL COMPARISONS  [performance)  HOY/ d o e s P r o g r a m B p e r f o r m a s c o m p a r e d t o o t h e r t e a c h e r t r a i n i n g programs a t U . B . C ? I s P r o g r a m B a c h i e v i n g t h e same e x p e c t a t i o n s as o t h e r t e a c h e r - t r a i n i n g p r o g r a m s a t U . B . C ? A c r i t i c a l c o m p e t i t o r m i g h t be t h e t r a i n i n g program. Importance of e v a l u a t i n g 4 —  Any  Extremely  important:  comparisons:  a comprehensive  3 ^Important:  c h e c k as t h o r o u g h l y  2  Definitely  do some  1  Check o n l y i f  0  D o n ' t w a s t e t i m e on  teachers'  (check  one)  c h e c k s h o u l d be made.  as y o u  can;  checking.  there  is  time.  it.  comments?  S I D E EFFECTS Are there  (performance)  any u n a n t i c i p a t e d  This  c h e c k p o i n t would  1. 2. 3.  Any u n a n t i c i p a t e d Any u n a n t i c i p a t e d Any u n a n t i c i p a t e d staff. Any u n a n t i c i p a t e d the U . B . C staff.  4.  crucial  regular 3rd year  side effects  look f o r  (for  program?  example):  benefits benefits benefits  or problems o r problems o r problems  to the s t u d e n t - t e a c h e r s . to the s t u d e n t s . t o t h e s c h o o l and i t s  benefits  or problems  to the U n i v e r s i t y  Importance o f e v a l u a t i n g  Side  important:  Effects:  4  Extremely  3  Important:icheefe-asethoroughly  2  Definitely  do some c h e c k i n g .  1  Check o n l y  if  0  D o n ' t w a s t e t i m e on  Should t h i s P r o g r a m B?  to the  there  be d o n e : d u r i n g  (check  a comprehensive  Is  and  one) c h e c k s h o u l d be made.  as y o u  can.  time.  it. ; a t t h e end o f  ;  after  123 Any comments?  PROCESS  (performance)  Is t h e Program B program a c h i e v i n g what t t s e t out t o do? Is t h e r e any i n j u s t i c e ; any u n h a p p i n e s s ; any c r u e l t y o r t h e i r c o n v e r s e s which a r e p a r t o f t h e program? Examples o f t h t s c h e c k p o i n t a r e : 1. Are t h e s t u d e n t s a t the s c h o o l b e i n g t a u g h t e f f e c t i v e l y by i . . stheestudent^teachers? 2. Are t h e s t u d e n t - t e a c h e r s b e i n g t a u g h t e f f e c t i v e l y by t h e U.B.C. s t a f f ? 3. Is t h e r e any c r u e l t y , unhappiness o r I n j u s t i c e i n t h e program? 4. Is t h e r e the c o n v e r s e o f "3" i n the program? 5. Are t h e r e any moral i m p l i c a t i o n s t o c o n s i d e r ? 6. Are o b s e r v a t i o n and d e s c r i p t i o n c o n g r u e n t ? Importance 4 3 2 1 0  o f e v a l u a t i n g P r o c e s s : (check  one)  E x t r e m e l y i m p o r t a n t : a comprehensive check s h o u l d be made. Important: check as t h o r o u g h l y as you can. D e f i n i t e l y do some c h e c k i n g . Check o n l y i f t h e r e i s time. Don't waste time on i t .  Any comments?  CAUSATION  (performance)  Are the o b s e r v e d and r e p o r t e d outcomes a t t r i b u t a b l e t o Program B? F o r example: 1, Have the c l a s s r o o m t e c h n i q u e s used by t h e s t u d e n t - t e a c h e r s been l e a r n e d t h r o u g h Program B? 2, Are the b e h a v i o r s i d e n t i f i e d under " P r o c e s s " t h e r e s u l t o f the  Program B program?  124  Importance o f e v a l u a t i n g C a u s a t i o n : [check, one) 4 3 2 1 0  E x t r e m e l y i m p o r t a n t : a comprehensive check s h o u l d be made. Important: check as t h o r o u g h l y as you can. D e f i n i t e l y so some c h e c k i n g . Check o n l y i f t h e r e t s t i m e . Don't waste t i m e on i t .  Any comments?  8,  (a)  The t o p i c you a r e p e r s o n a l l y most I n t e r e s t e d i n i s number  (b)  The t o p i c you c o n s i d e r most n e c e s s a r y t o the e v a l u a t i o n o f Program B i s number .  Any comments?  Appendix B I n t e r v i e w o f School  Administration  Questions  asked;  126  1)  What needs do y o u s e e Program B f u l f i l l i n g ?  2)  Is Program B d o i n g t h e t h i n g s y o u e x p e c t e d i t t o do? I f Yes? What a r e t h o s e t h i n g s ? I f NO? What i s b e i n g o m i t t e d ?  3)  A r e y o u c l e a r about y o u r r o l e i n Program B?  4)  A r e t h e r e any u n a n t i c i p a t e d p o s i t i v e e f f e c t s o f t h e program so f a r ? I f y e s , what a r e t h e y ?  5)  A r e t h e r e any u n a n t i c i p a t e d n e g a t i v e e f f e c t s o f t h e program so f a r ? I f y e s , what a r e t h e y ?  6)  What a r e y o u r g e n e r a l i m p r e s s i o n s about t h e U.B.C. s t a f f and t h e i r f u n c t i o n i n g i n t h e program?  7)  What a r e y o u r g e n e r a l i m p r e s s i o n s about t h e s t u d e n t t e a c h e r s ?  8)  Any o t h e r comments?  127  Appendix C L i k e r t A t t i t u d e S c a l e : to measure g e n e r a l a t t i t u d e s toward t h e t e a c h e r t r a i n i n g p r o c e s s o f t h e s t u d e n t . t e a c h e r s .  128 This is not a test. Consider each item carefully before answering, Your answers will remain anonymous. Record all your answers on the computer answer card (depicted below) i  i i ! 1 i 1J ! 00 1  *-* w*  O  3 6 3  r»  rt  o  rt  ii  c c  r»  l •0  i  o  '0  0  BLi n  § § § §  r  gC  3  :  P  5  6  6  rt ;o  0  § g 3 n 8 6 8 o C t? r0  3  i» r»  19  rt  A  § § L 5 IS § u o rt 5  ij  r.  ^  L»  rt r> p p 7 '7 <7 |7 u L. o U  rt  I B g f S <3 i« s § P S I? 1? 1? o 5 § 6 !? 7  p? p?  2  a  12 is  & i 7 ; « •? >e> it  5;  g g r: C §? i/ o  8  <-  rt  !*3  •2  ! ii 3  fi  I R  7 '2  a rt Irt  ... ! o  3  i * ts l i i? iz n to n z> zr. u 2~  i i ii! (i! fi ii e'ii8;3i i rt (i:BIB Sis  ;1 - :1  1  G | G?  , G  i  •-'  i  J  O  O  O  1  2«,27.:<>,"f  I (~> IrtI  J  rt ( rt  iii;  r^ic  ij'?  vANSWER FIELD.?  Q i i3! *3; p. 1 G  1  1  G C  I  I a  qP.!I Pp. :•i!U  , i; I ij  G  rt I  I G' ;  ^ n I • S i  c 1 ' a i M ; 3 3 | 3-1 ! 35 ! 35 | 3 ? | 33 | 33 i 40 : 41 i « I 4] j « | 4b j «6 j 47 : SS ! <3  rt B|B B BIGlBiBiB B B B s u a B B B Q 3 nifi:i' § § § g g ggg L11 i l § Crt C g g g g rt gg 8 8 S 8 8 8 j1 5 rt U i 8 rt I o |3 8 r B rt irtj Li i i • | u  3  j u  i  r> ,  :5  p  '•' i I-' L. G :  G | o  [5  UNIVEHS.Tr C "  o  -rt• rt 13 ,3 0 . 0  ; o I iJ • I* i u |  3  3  3  i2 -2 :2  (5 ! L C? i n I rt  rt i i ! •6 '6  Io 1  •I  :5  I n  3  :  rt  6RIT;SH COLUMBIA  f;  ;  rt  ?2  rt • rt ' rt 2 . 2 ; 2 ^ ^ , ^  >2  o  u  P3  3  rj  I ,  G  4  £  n  ; ,C .5  d  s  5 '5  5 ; ,5 , --,  Do not put any marks in the section called "identification number", or in the section called "answer card number". Use the answer field to record your answers. There are 50 columns for answers, but you will only use the first 18 columns. For each item in the folic ./ing form there is a corresponding 5-choice column. Record your response to each item by blackening the appropriate bubble in the column corresponding to the item. See the following example: EXAMPLE: 1.  EDUCATION IS FUN.  1 - Strongly disagree 2 - Disagree 3 - Undvicided Answer  -»-4 - Agree 5 - Strongly agree  BLACKEN bubble 4 in column 1 as shown below: Item 1  I  0) USE PENCIL ONLY!  Answer (4)  129  RESPONSE CODE 1  SD  2  D  Disagree with item,  3  U  Undecided.  4  A  Agree with item.  5  SA  S t r o n g l y disagree with item.  S t r o n g l y agree w i t h i t e m .  PLEASE USE PENCIL ONLY: Note:  TEACHER TRAINING r e f e r s t o y o u r whole program o f c o u r s e s , classes, teaching practice, etc. i . e . i t includes a l l a s p e c t s o f y o u r program. STUDENT TEACHING r e f e r s t o your classroom teaching practice only.  THE STATEMENTS: PLEASE RESPOND TO THESE ON YOUR COMPUTER ANSWER CARD, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. 11. 12. 13. 14. 15. 16. 17. 18.  I HOPE TO KEEP IN TOUCH WITH MY PRESENT CLASSMATES IN THE FUTURE. U.B.C, HAS MADE A SIGNIFICANT CONTRIBUTION TO TEACHING IN B.C. STUDENT TEACHING ISN'T WORTH THE EFFORT. UNIVERSITY IS AN ENJOYABLE EXPERIENCE. I LOOK FORWARD TO MY STUDENT TEACHING EXPERIENCE, GENERALLY SPEAKING, MY EDUCATION COURSES ARE USELESS. MY U.B.C. INSTRUCTORS ARE FINE PEOPLE. TEACHING IS AN ENJOYABLE EXPERIENCE. MY CLASSMATES ARE A VALUABLE SOURCE OF IDEAS FOR TEACHING. MY SPONSOR TEACHERS HAVE HELPED ME DEVELOP MANY USEFUL TEACHING SKILLS. TEACHING SCHOOL IS A VERY IMPORTANT JOB. TEACHER TRAINING IS BORING. AS A WHOLE, MY EDUCATION COURSES ARE INTERESTING. MY SPONSOR TEACHERS ARE GENERALLY ENTHUSIASTIC ABOUT TEACHING. TEACHER TRAINING IS IMPORTANT. MY EDUCATION COURSES ARE NOT HELPING ME TO BECOME A GOOD TEACHER. TEACHER TRAINING IS AN ENJOYABLE EXPERIENCE. MY SPONSOR TEACHERS HAVE A STRONG INFLUENCE ON MY TEACHING.  Appendix D R e a c t i o n Form  131  REACTION FORM  Date C i r c l e t h e number below w h i c h b e s t r e p r e s e n t s y o u r f e e l i n g s about Program B a t t h i s time. 1 very negative  2  3  4 neutral  5  6  7 very positive  B r i e f l y d e s c r i b e any i n c i d e n t s t h a t y o u saw as p o s i t i v e , and any t h a t you saw as n e g a t i v e r e l a t i n g t o y o u r c u r r e n t r e a c t i o n t o Program B.  A n y t h i n g e l s e you'd l i k e t o mention?  132  Appendix E E v a l u a t i o n o f UBC i n s t r u c t o r s ' t e a c h i n g p e r f o r m a n c e . Form A.  form A RESPONSE CODE Strongly disagree with item.  1  SD  2.  D  Disagree with  3.  U  Undecided,  4,  A  Agree with  5.  SA  item.  item.  S t r o n g l y agree w i t h  item.  COURSE  INSTRUCTOR_  IDENTIFICATION No;  COURSE  INSTRUCTOR__  IDENTIFICATION No..  1) 2) 3) 4) 5) 6) 7) 8) 9) 10) 11)  T h i s i n s t r u c t o r g i v e s u n s a t i s f a c t o r y answers t o s t u d e n t s ' questions. T h i s i n s t r u c t o r o f t e n misses t h e p o i n t o f a q u e s t i o n asked by a student. T h i s i n s t r u c t o r seldom v a r i e s h i s method o f i n s t r u c t i o n . This i n s t r u c t o r ' s presentations i n class are poorly organized. This i n s t r u c t o r i s very careful not to offend h i s students w i t h h i s remarks, This i n s t r u c t o r i s impatient with h i s students, The l e v e l o f d i f f i c u l t y o f t h i s c o u r s e i s about r i g h t . The i n s t r u c t o r seldom c h e c k s t o s e e i f s t u d e n t s u n d e r s t a n d what he i s t e a c h i n g . This i n s t r u c t o r c l e a r l y explains d i f f i c u l t ideas. T h i s i n s t r u c t o r uses humour e f f e c t i v e l y i n h i s t e a c h i n g . T h i s i n s t r u c t o r seldom engages h i s s t u d e n t s i n c l a s s discussions.  12)  The r e a d i n g m a t e r i a l s i n t h i s c o u r s e ( t e x t , o t h e r s o u r c e s ) a r e of l i t t l e use.  13)  I d e v e l o p e d my a b i l i t y t o marshal o r i d e n t i f y main p o i n t s o r c e n t r a l i s s u e s i n t e a c h i n g from t h i s c o u r s e . T h i s c o u r s e i s o f d e f i n i t e v a l u e f o r a p e r s o n who w i s h e s t o become a t e a c h e r .  14')  I f y o u have any comments then p l e a s e w r i t e them on t h e back o f t h i s page w i t h t h e name o f t h e i n s t r u c t o r y o u r comments p e r t a i n t o .  134 OVERALL EVALUATION OF INSTRUCTORS 1, p r o f e s s o r A 2. p r o f e s s o r B 3. p r o f e s s o r C 4. p r o f e s s o r D 5. p r o f e s s o r ^ E \ 6 . 1  p  ,0 -0 |0o U  r» II  g  !*(»•  L»  O  I  u  O  o  p  f*>  U  f»  1  ip np  ir  0  ~  <  „ I ,J F i2: ? i I 1  « s>  professor F. o  i O'  1  j  iR!  !  e8  p  w  C |&|i p3 n '  ,11 1  ;•> 1 4  0 p.  l>  IJ  j  5 ^8 g " ^'". i";§ g p.  p.  g  *—  § §  PC  O  0 5 5 1?  p  p  p  >  r? ? P P p R  b  U  O  <  t» 0  C15 § PL?o §°~BC s  !8  8  9  !3  P  ••j! ;5  9 1-3 ;9 !•}  p  .'2  g|ej§B  V-  !  2;: ?* 3r  JI  .1 i a : !i  11  O  I u  1  w  p  | P  p'i$!S  p  P i  0  P  P  ;2  2  (5 i I ;4 L.  !  JiJid  U N I V £ H 5 . T r C" ERITiSM COLUMBIA  i ; I i5!  1  !  !  3  3  i3!i5il5  i  p r; n b ii u  li  P2  r  3  ii  •S p  p.  D  3  £ j '5 i -5  p1  4 ? ; 4 3 < 44 | 4 i I 4 5 ; 4 7 • 52 ,  p  1n i p 0 Li  i p 0 I! U g g g gg g o g g C g" g g S gg gg P^ g  IU ] li  p p I p :2 2 I 2 o 0j a  I P  s  S 5 r ir p b p I p ip  i !4C  p  '5!  2 . I 33 i 34 ! 35 j 36 ; 3 ; i 33 j 33 ( 4U : 41  (] I h 0 | 0U  T  12 i -  i  •! P i 5 j Q i p. i  Q ; ;.4  !5 ! £ ! 5 ' ;5  a ;  2;.; £ 7 .  F  1  n ?s  J£  }i § i ^ANSWER P I E L D J ! 3 j  6  1  [3  ! !i  ts »r I!  b' ! C1 p.: a !  ;  13 )«•; 15-  p :2  P3  IT  Pi  b  Ps Ps p |5 5  P  4  fj f; ^2 2  8 3  •P5 !5  p  ii «_»  c  o4 -^1  :5  PLEASE RATE EACH INSTRUCTOR IN THE APPROPRIATE COLUMN (ILLUSTRATED ABOVE).  How good a j o b i s the i n s t r u c t o r d o i n g i n g e t t i n g a c r o s s t o you the knowledge, u n d e r s t a n d i n g , s k i l l s and a t t i t u d e s t h a t appear t o be t h e g o a l s o f t h e c o u r s e ? (In a n s w e r i n g , t r y not t o be u n d u l y i n f l u e n c e d by y o u r l i k i n g o r d i s l i k e f o r t h e i n s t r u c t o r a s a p e r s o n ; o r b y y o u r f e e l i n g s about t h e c o u r s e c o n t e n t , i f t h a t wasn't something which t h e i n s t r u c t o r c o u l d choose f r e e l y . ) P l e a s e use t h e f o l l o w i n g s c a l e f o r y o u r r e s p o n s e s on t h e computer 1)  O u t s t a n d i n g l y bad  2)  Below a v e r a g e  3)  Average  4)  Above a v e r a g e  5)  O u t s t a n d i n g l y good  card:  ij  p 2 p. r;  c  .5  135  Appendix F Q u e s t i o n n a i r e t o s t u d e n t - t e a c h e r s on t h e i r a l t e r n a t e p l a c e m e n t .  136 PROGRAM B EVALUATION  This questionnaire deals with your most recent experience in alternate placements. 1,  During your alternate placement were you given the placement you requested? YES NO  2,  Which of the following was your placement? OPEN AREA  ;  FREE SCHOOL FAMILY GROUPING OTHER [PLEASE SPECIFY) 1  3,  Was the placement more demanding or less demanding of your time andtenergy than you had anticipated it would be? MORE DEMANDING LESS DEMANDING WHAT I EXPECTED UNDECIDED COMMENTS:  4.  Did you find your experience beneficial?  YES  No  UNDECIDED Please comment:  5.  Were your sponsor teachers expecting you? YES UNDECIDED Comments:  NO  137  D i d you have a c l e a r i d e a o f y o u r t e a c h i n g d u t i e s - b e f o r e y o u r a l t e r n a t e e x p e r i e n c e began? YES NO Comments:  D i d y o u r s p o n s o r t e a c h e r have a c l e a r i d e a o f y o u r t e a c h i n g d u t i e s when you a r r i v e d : YES NO Comments:  G e n e r a l l y s p e a k i n g , d i d y o u a c t u a l l y do what y o u had e x p e c t e d t o do i n y o u r a l t e r n a t e e x p e r i e n c e ? YES NO Clf "NO" p l e a s e comment) Comments:  D i d y o u f e e l y o u were s u p e r v i s e d o f t e n enought by: a, YOUR SPONSOR TEACHER b. YOUR U.B.C. ADVISOR Comments:  YES  NO  UNDECIDED  YES  NO  UNDECIDED  Do you t h i n k t h i s a l t e r n a t e e x p e r i e n c e s h o u l d be c o n t i n u e d i n f u t u r e Program B's? YES, WITH CHANGES YES, WITHOUT CHANGES NO, IT SHOULD BE DISCONTINUED  10.  138  (Continued),  I f y o u checked "YES, WITH CHANGES", what changes would y o u s u g g e s t ?  I f y o u checked "NO, IT SHOULD BE DISCONTINUED", would y o u p l e a s e I n d i c a t e why y o u f e e l t h a t way?  11.  C o n s i d e r i n g e v e r y t h i n g , what i s y o u r o v e r a l l e v a l u a t i o n o f t h e a l t e r n a t e placement? EXTREMELY VALUABLE VALUABLE HAD SOME VALUE _^  HAD LITTLE VALUE A COMPLETE. WASTE OF TIME  12,  Any o t h e r comments?  Appendix Q E v a l u a t i o n o f UBC  Instructors-  teaching performance Form B  140 FORM B 1  SD  Strongly disagree with item.  2  D  Disagree with item.  3  U  Undecided.  4  A  Agree w i t h i t e m .  5  SA  S t r o n g l y agree w i t h i t e m ,  COURSE:  INSTRUCTOR:  IDENTIFICATION NO.  COURSE:  INSTRUCTOR:  IDENTIFICATION NO.  1)  The i n s t r u c t o r appears i n t e r e s t e d i n t e a c h i n g t h i s c o u r s e ,  2)  O v e r a l l , t h i s i s a good c o u r s e ,  3)  The p l a n f o r t h i s c o u r s e t s c l e a r l y e v i d e n t .  4)  The i n s t r u c t o r a l l o w s a few s t u d e n t s t o m o n o p o l i z e c l a s s discussions.  5)  The i n s t r u c t o r i s seldom a v a i l a b l e f o r a s s i s t a n c e o u t s i d e o f c l a s s time.  6)  The i n s t r u c t o r o f t e n s t i m u l a t e s s t u d e n t s t o u n d e r t a k e a d d i t i o n a l work i n t h e c o u r s e on t h e i r own,  7)  The i n s t r u c t o r seldom asks q u e s t i o n s which s t i m u l a t e d i s c u s s i o n .  8)  The i n s t r u c t o r seems competent course.  9)  The i n s t r u c t o r f r e q u e n t l y p o i n t s o u t how d i f f e r e n t p a r t s o f t h e course are r e l a t e d .  10)  I c a n r e c o g n i z e good and bad t e a c h i n g t e c h n i q u e s because o f t h i s course.  i n the subject matter o f t h i s  0  11)  T h i s c o u r s e i n c r e a s e d my awareness o f i s s u e s r e l a t e d t o t h e subject matter.  12)  T h i s c o u r s e i s o f d e f i n i t e v a l u e f o r a p e r s o n who w i s h e s t o become a teacher.  I f you have any comments then p l e a s e w r i t e them on t h e back o f t h i s page w i t h t h e name o f t h e i n s t r u c t o r y o u r comments p e r t a i n t o .  141 O v e r a l l E v a l u a t i o n o f I n s t r u c t o r s ( s e e Appendix £ ) 1 Professor 2 Professor 3 Professor 4 Professor 5 Professor etc. A B O D E  PLEASE RATE EACH INSTRUCTOR IN THE APPROPRIATE COLUMN ABOVE)  (ILLUSTRATED  How good a j o b i s t h e i n s t r u c t o r d o i n g i n g e t t i n g a c r o s s t o y o u t h e knowledge, u n d e r s t a n d i n g , s k i l l s and a t t i t u d e s t h a t a p p e a r t o be the g o a l s o f t h e c o u r s e ? ( I n a n s w e r i n g , t r y n o t t o be u n d u l y i n f l u enced by y o u r l i k i n g o r d i s l i k e f o r t h e I n s t r u c t o r as a p e r s o n ; o r by y o u r f e e l i n g s about t h e c o u r s e c o n t e n t , i f t h a t wasn't something which t h e i n s t r u c t o r c o u l d choose f r e e l y . ) P l e a s e use t h e f o l l o w i n g s c a l e f o r y o u r r e s p o n s e s on t h e computer" card: 1. 2. 3. 4. 5.  O u t s t a n d i n g l y bad. Below a v e r a g e . Average. Above a v e r a g e . O u t s t a n d i n g l y good,.  Appendix H Interview of student; teachers ;  143 Interview Questions 1)  D i d you have any s e r i o u s problems: a)  In f i n d i n g a p l a c e t o l i v e ?  b)  With t r a n s p o r t a t i o n t o t h e  c)  With t h e p u p i l s you  d)  With t h e UBC p r o f e s s o r s ?  e)  With t h e t e a c h e r s  f)  With y o u r f e l l o w s t u d e n t - t e a c h e r s ?  g)  Anything else?  school?  taught?  2)  Was Program B more c o s t l y t o you than a 3 r d y e a r r e g u l a r program a t t h e U n i v e r s i t y would have been?  3)  Would i t m a t t e r t o you where t h e s c h o o l was l o c a t e d i n terms o f c o s t ? T r a n s p o r t a t i o n ? Type o f s t u d e n t s ?  4)  How d i d you come t o know about Program B?  5)  Why d i d you a p p l y t o Program B?  6)  Why do you want t o be a t e a c h e r ?  7)  D i d Program B meet y o u r e x p e c t a t i o n s ?  8)  Were t h e f o l l o w i n g p h y s i c a l f a c i l i t i e s a t t h e s c h o o l  9)  a)  Washrooms?  b)  Lunchrooms, s t a f f r o o m s ?  c)  A r e a s t o r e l a x , smoke, t a l k ?  d)  Work and p r e p a r a t i o n a r e a s ?  adequate:  D i d you have s u f f i c i e n t a c c e s s t o books and s u p p l i e s f o r : a)  Your u n i v e r s i t y c o u r s e work?  b)  Your t e a c h i n g ?  10)  D i d you f e e l t h a t t h e work l o a d ( c o u r s e s , c l a s s r o o m t e a c h i n g , s u p e r v i s i o n , e t c . ) was e x c e s s i v e ?  11)  Do you t h i n k t h a t h a v i n g methods c o u r s e s a t t h e s c h o o l i s an advantage o v e r h a v i n g them a t the u n i v e r s i t y ? Why?  12)  144  O v e r a l l , were t h e method c o u r s e s u s e f u l ? -- How were t h e y u s e f u l o r why were t h e y n o t u s e f u l ?  13)  What do t h e g o a l s o f Program B seem t o be? — Do you t h i n k i t has succeeded i n r e a c h i n g t h e s e g o a l s ? — On what e v i d e n c e a r e you b a s i n g y o u r o p i n i o n ?  14)  Would you recommend Program B as i t i s t o o t h e r p o t e n t i a l s t u d e n t - t e a c h e r s ? Why o r why n o t ?  15)  Would you t a k e Program B a g a i n i f you were back i n Semptember o f 1974? Why o r why not?  16)  I s Program B e s p e c i a l l y s u i t e d f o r c e r t a i n t y p e s o f s t u d e n t t e a c h e r s ? What t y p e ? v.. •  17)  Do you t h i n k t h e y e a r f o l l o w i n g t h i s w i l l be: ~ valuable?  18)  -- i n t e r e s t i n g ? D i d any i n d i v i d u a l s have a l a r g e n e g a t i v e o r p o s i t i v e e f f e c t on the d i r e c t i o n t h e program took?  19)  Was t h i s s c h o o l i d e a l f o r Program B?  20)  C o u l d t h e r e have been more s t u d e n t - t e a c h e r i n p u t i n t o t h e program? I f s o , when o r where? I f n o t , a r e you s a t i s f i e d w i t h t h e amount o f i n p u t you had?  Why o r why not?  Appendix  I  Fina'l Q u e s t i o n n a i r e f o r S t u d e n t Teache  146 Program B E v a l u a t t o n Did Program B l i n k t h e methods c o u r s e s t o a c t u a l t e a c h i n g ? frequently  never  Did t h e program r e l a t e t h e s t u d y o f d e v e l o p m e n t a l p s y c h o l o g y t o immediate e x p e r i e n c e s ? frequently  never  Did t h e f o u n d a t i o n c o u r s e f o c u s on s c h o o l based i s s u e s and problems? frequently  never  Did t h e p s y c h o l o g y component i n c r e a s e y o u r s e n s i t i v i t y t o t h e needs and r e q u i r e m e n t s o f y o u r s t u d e n t s ? Yes  No _  Undecided  .  D i d t h e f o u n d a t i o n s component e n a b l e y o u t o make c l e a r , w e l l founded d e c i s i o n s as t o what g o a l s a t e a c h e r ought t o p u r s u e ? Yes  No  ^_ U n d e c i d e d  Did t h e f o u n d a t i o n s component e n a b l e y o u t o promote t h e moral development o f your students? Yes  No  Undecided  Did t h e f o u n d a t i o n s component g i v e y o u a c l e a r u n d e r s t a n d i n g o f the n a t u r e o f t h e t a s k o f d e v e l o p i n g t h e a b i l i t y t o t h i n k ? Yes  No  Undecided  Did you have s u f f i c i e n t o p p o r t u n i t y t o t r y o u t y o u r own i d e a s i n class? Yes  No  Undecided  Do y o u f e e l t h e U.B.C. p r o f e s s o r s a r e e n t h u s i a s t i c about Program B's a p p r o a c h t o t e a c h e r - t r a i n i n g ? Yes  No  Undecided  Do y o u f e e l t h e t e a c h e r s a t ( s c h o o l name)are e n t h u s i a s t i c about Program B's a p p r o a c h t o t e a c h e r - t r a i n i n g ? n  : .-:Yes'' . '•; - N o ' r . ^ , Undecided"-.:  :  ; ; :  147 11.  Do y o u t h i n k program B has been f l e x i b l e ? Yes  12.  No ^  Undecided  Were y o u a b l e t o make as much u s e o f t h e U.B.C. l i b r a r y a s y o u wanted t o ? Yes  14.  ^_ U n d e c i d e d _^  Do you t h i n k Program B has been v a r i e d ? Yes  13.  No  No  Undecided  C o n s i d e r i n g e v e r y t h i n g , what i s y o u r o v e r a l l e v a l u a t i o n o f Program B? EXCELLENT  GOOD  O.K.  POOR  A WASTE OF TIME  148  Appendix J D i s c i p l i n e techniques  149 DISCIPLINE TECHNIQUES 1.  Simple c o n t r o l a.  A l o o k o r a frown.  b.  Shake o f head o r o t h e r m i l d p h y s i c a l g e s t u r e .  c.  Mild verbal reproof.  d.  Movement t o t r o u b l e a r e a .  e.  Shouting at student ( s ) .  2.  Individual conference with p u p i l .  3.  P a y i n g f o r , r e b u i l d i n g o r r e p l a c i n g damaged i t e m ,  4.  Loss o f some p r i v e l e g e .  5.  Detention a f t e r school.  6.  D i s m i s s a l from c l a s s . a.  Sent t o o f f i c e .  b.  Sent t o h a l l w a y .  c.  Other area.  7.  I s o l a t i o n i n the classroom, i . e . sent t o a corner, e t c .  8.  P u n i s h i n g a whole group o r c l a s s .  9.  E x t r a work o r t a s k s .  10.  Enforced apologies.  11.  Physical contact ~ shaking, s t r i k i n g , f o r c e f u l l y d i r e c t i n g .  12.  T h r e a t o f any o f t h e above.  13.  Other.  Appendix K S i d e E f f e c t s o f Program B ( T e a c h e r s ' Q u e s t i o n n a i r e )  151  Program B's  Evaluation  Dear Teacher, During the past few months of Program B an internal as you know, has been carried out.  evaluation,  Knowing how loaded down with  paper-work and forms you are, I have not bothered you with any questionnaires or opionnaires — until now. the l a s t questionnaire that I w i l l give  This is the f i r s t and  to you.  Its purpose is to  attempt to establish i f there have been any unanticipated benefits, and/or problems which have resulted from the operation of Program B in your school. side-effects)  These unanticipated aspects of the program (or  are extremely important.  Will you please put your responses to this questionnaire in the envelope provided for them in the o f f i c e .  Please do not put your  name on your response. Thank you, Norman Gleadow  152 SIDE EFFECTS OF PROGRAM B T h i s form i s t o d e t e r m i n e i f t h e r e a r e any u n a n t i c i p a t e d e f f e c t s o f Program B. These e f f e c t s a r e t h e b e n e f i t s a n d / o r t h e problems w h i c h were n o t f o r s e e n when t h e p r o g r a m was i n i t i a t e d . P l e a s e do n o t p u t y o u r name on t h i s form so as t o m a i n t a i n y o u r anonymity. 1.  A r e y o u aware o f any UNANTICIPATED BENEFITS o r any UNANTICIPATED PROBLEMS o f Program B t o t h e STUDENT-TEACHERS? UNANTICIPATED BENEFITS  2.  UNANTICIPATED PROBLEMS  A r e y o u aware o f any UNANTICIPATED BENEFITS o r UNANTICIPATED PROBLEMS o f Program B t o t h e SCHOOL'S STUDENTS? UNANTICIPATED BENEFITS  UNANTICIPATED PROBLEMS  I f y o u have any a d d i t i o n a l comments p l e a s e p u t them on t h e b a c k .  3.  153 A r e y o u aware o f any UNANTICIPATED BENEFITS o r any UNANTICIPATED PROBLEMS o f Program B t o t h e SCHOOL'S TEACHERS? UNANTICIPATED BENEFITS  4.  A r e y o u aware o f any UNANTICIPATED BENEFITS o r any UNANTICIPATED PROBLEMS o f Program B t o t h e U.B.C. PROFESSORS? UNANTICIPATED BENEFITS  5.  UNANTICIPATED PROBLEMS  UNANTICIPATED PROBLEMS  A r e y o u aware o f a n y t h i n g y o u w o u l d c o n s i d e r i n j u s t , i m m o r a l , o r c a u s i n g s e r i o u s u n h a p p i n e s s as a r e s u l t o f Program B?  Appendix L I n t e r v i e w w i t h Program B's D i r e c t o r  155 Questions Asked. 1)  How d i d t h e i d e a o f Program B o r i g i n a t e ? When? impetus f o r Program B.  What was t h e  2)  What d i d Program B hope t o do t h a t wasn't b e i n g done by t h e t h i r d year regular course a t U.B.C?  3)  Why were t h e t h i r d y e a r t r a n s f e r s t u d e n t s chosen?  4)  How were p e r s p e c t i v e s t u d e n t - t e a c h e r s n o t i f i e d o f Program B?  5)  How were t h e s t u d e n t - t e a c h e r s f i n a l l y s e l e c t e d f o r Program B? How many were t u r n e d down? Why?  6)  How d i d t h e UBC s t a f f f o r Program B g e t c h o s e n ?  7)  How was t h e h o s t s c h o o l chosen?  8)  What c o n n e c t i o n has t h e s c h o o l b o a r d w i t h t h e program?  9)  Has t h e r e been any u n s o l i c i t e d , w r i t t e n f e e d - b a c k on t h e s t u d e n t teachers;': performance?  10)  Do y o u t h i n k t h e average t h i r d y e a r s t u d e n t c o u l d o p e r a t e w e l l i n Program B?  11)  What do y o u t h i n k c o n s t i t u t e s t h e i d e a l s t u d e n t - t e a c h e r f o r Program B?  12)  Do y o u know o f any r e s e a r c h r e s u l t s on programs s i m i l a r t o Program B?  13)  Do y o u t h i n k i t would be f e a s i b l e t o have a l l t e a c h e r - t r a i n i n g done v i a Program B? Why o r why not?  14)  What i s t h e most e x p e n s i v e p a r t o f Program B as compared t o t h e r e g u l a r t h i r d y e a r Program?  15)  What a r e some o t h e r expenses w h i c h a r e a s s o c i a t e d w i t h Program B?  16)  What do y o u t h i n k a r e t h e g r e a t e s t s t r e n g t h s and weaknesses o f Program B?  156  Appendix M Q u e s t i o n n a i r e f o r School P u p i l s .  157 Your Grade QUESTIONNAIRE FOR STUDENTS We would l i k e you to answer some q u e s t i o n s about the s t u d e n t - t e a c h e r s who have t a u g h t you o v e r t h e p a s t s c h o o l y e a r . P l e a s e do NOT put y o u r name on t h i s paper. I f you w i s h t o make any w r i t t e n comments, p l e a s e do so i n the spaces under each q u e s t i o n . 1.  D i d the s t u d e n t - t e a c h e r s t e a c h too f a s t ?  YES  NO  2.  D i d the s t u d e n t - t e a c h e r s t e a c h t o o slow?  YES  NO  3.  D i d t h e y g i v e you f a i r t e s t s and exams?  YES  NO  4.  D i d you u n d e r s t a n d what they were t r y i n g t o t e a c h you i n c l a s s ? YES NO  5.  D i d any o f them e v e r h e l p you w i t h s c h o o l work o u t s i d e o f c l a s s ? YES  6.  C o u l d t h e s t u d e n t - t e a c h e r s c o n t r o l t h e k i d s who d i d n ' t behave? YES  7.  NO  NO  Were t h e y e v e r u n f a i r t o you?  YES  NO  I f you answered y e s to t h i s q u e s t i o n , would you p l e a s e t e l l us how the s t u d e n t - t e a c h e r s were u n f a i r ?  8.  D i d you f i n d them i n t e r e s t i n g to l i s t e n t o ? YES  9.  D i d you f i n d i t easy to l e a r n t h i n g s from them? YES  10.  NO  NO  Were the s t u d e n t - t e a c h e r s ' l e s s o n s i n t e r e s t i n g ? YES  NO  11.  YES 12.  158  D i d t h e y g i v e y o u good answers t o y o u r q u e s t i o n s ? NO  D i d y o u f i n d i t easy t o g e t t h e s t u d e n t - t e a c h e r s ' a t t e n t i o n i n class? YES  NO  13.  D i d t h e y l e a r n y o u r name?  14.  D i d y o u f i n d them easy t o t a l k t o ?  15.  D i d y o u e v e r t a l k t o any o f t h e s t u d e n t - t e a c h e r s i n t h e classroom? YES  16.  NO  NO  NO  NO  P l a c e a check-mark b e s i d e t h o s e s u b j e c t s w h i c h a s t u d e n t - t e a c h e r taught you ART ENGLISH OR LANGUAGE ARTS ( r e a d i n g , s p e l l i n g , MATH o r ARITHMETIC writing) SCIENCE P.E. SOCIAL STUDIES MUSIC OTHER (WHAT IS?IT?  19.  YES  D i d y o u f i n d s w i t c h i n g b a c k - a n d - f o r t h from y o u r r e g u l a r t e a c h e r to t h e s t u d e n t - t e a c h e r s c o n f u s i n g ? YES  18.  NO  D i d y o u e v e r t a l k t o any o f t h e s t u d e n t - t e a c h e r s o u t s i d e o f t h e classroom? YES  17.  YES  )  Which o f t h e above s u b j e c t s do y o u t h i n k t h e s t u d e n t - t e a c h e r s taught the best? ' .  159 Which o f t h e above s u b j e c t s do y o u t h i n k t h e s t u d e n t - t e a c h e r s taught the worst? ' . D i d t h e s t u d e n t - t e a c h e r s do a good j o b o f t e a c h i n g you? YES  NO  NOT SURE  Appendix N I n t e r v i e w w i t h UBC  Administrators.  161 Q u e s t i o n s Asked. 1)  What rieed do y o u s e e Program B f u l f i l l i n g ?  2)  Have y o u had any s p e c i f i c f e e d b a c k on Program B's p a r t i c i p a n t s ? I f s o , what?  3)  What do you s e e as t h e s t r o n g and weak p o i n t s o f Program B?  4)  What i s y o u r own judgment as t o t h e o v e r a l l p e r f o r m a n c e o f Program B?  5)  Would y o u d e s c r i b e t h e U.B.C. p r o f e s s o r s i n v o l v e d i n Program B as a n / a v e r a g e g r o u p , a p o o r group o r an above a v e r a g e group?  Appendix 0 Student Teachers' Questionnaire (Side E f f e c t s )  163  SIDE EFFECTS - STUDENTS  1.  P l e a s e i n d i c a t e your o v e r a l l r e g i s t r a t i o n procedures.  e v a l u a t i o n of t h e  E x c e l l e n t , c o u l d n ' t be smoother  /  /  Good, h a r d l y any d i f f i c u l t i e s  /  /  Acceptable,  /  /  /  /  /  /  Not  t h e r e were a few problems  good, t h e r e were many problems  Terrible,  complete  confusion  Are t h e r e any good and/or bad p o i n t s about t h e r e g i s t r a t i o n p r o c e d u r e you would l i k e t o comment on? I f s o , p l e a s e use t h e space below.  2.  (a)  Please  i n d i c a t e your method of t r a n s p o r t a t i o n t o S c h o o l from your p l a c e of . r e s i d e n c e .  Walking  J_  (or running)  Bicycle  l_  Motorcycle  l_  Your own c a r  ]_  In a c a r p o o l  l_  i  164  /  Bus Other (b)  (Please  /  specify)  About how l o n g does i t t a k e you to get t o from your p l a c e o f r e s i d e n c e ? L e s s than 15 m i n u t e s  School  /  Between 15 -and 30 m i n u t e s Between 30 and 45 m i n u t e s Between 45 and 60 m i n u t e s  /  More than an hour  (c)  Are you e x p e r i e n c i n g any problems w i t h your p r e s e n t t r a n s p o r t a t i o n arrangements from your p l a c e of r e s i d e n c e to -. - N. S c h o o l ? Yes  /  /  No  /  /  I f " y e s " p l e a s e i n d i c a t e what t h o s e problems a r e .  Were you g i v e n an adequate o r i e n t a t i o n No (a)  (b)  /  /  Yes /  /  to the school? Not Sure /  /  I f "No" p l e a s e i n d i c a t e how i t was i n a d e q u a t e . I f "Not Sure" p l e a s e i n d i c a t e , i n g e n e r a l terms, why you f e e l uneasy about t h e o r i e n t a t i o n .  Do you have adequate work space i n Yes /  7  No /  School?  7  Comments  Are  t h e washroom f a c i l i t i e s  Yes /  /  adequate?  No /  /  Comments  Are arrangements f o r e a t i n g l u n c h , d r i n k i n g c o f f e e , smoking, e t c . a c c e p t a b l e t o you? Yes /  7  Comments  No /  7  166  7.  Are t h e r e any problems or i r r i t a t i o n s which are " i n the bud" and should be d e a l t w i t h now b e f o r e they got worst?  Yes /  I f "Yes"  8.  No /  /  please o u t l i n e those  /  problems.  I s t h e r e a n y t h i n g e l s e you would l i k e to comment P l e a s e do i n the space p r o v i d e d below.  /  on?  Appendix P U.B.C. S t a f f Q u e s t i o n n a i r e on C o u r s e s  168  Instructors' Questionnaire A  Course:  Instructor  Course w i l l  be g i v e n i n F a l l Term  /  Hours o f i n s t r u c t i o n p e r week  /  S p r i n g Term  hrs.  /  •  1.  Recommended t e x t b o o k  2.  I s (are) the textbook(s) a v a i l a b l e at the u n i v e r s i t y Bookstore? Yes  3.  /  No  /  /  (s) i f any  Comments:  Which p a r t ( s ) o f t h e t e x t b o o k ( s ) w i l l All  ?  /  /  7;  or  /  you be  emphasizing?  169  How w i l l you be measuring the s t u d e n t ' s p r o g r e s s c o u r s e ( p l e a s e check a l l t h a t a p p l y ) ? Comments: F i n a l exam ( w r i t t e n ) /  /  F i n a l exam ( o r a l )  l_  /  Mid term exam Other exams (or t e s t s )  /  /  /  (Please give / frequency  i n your  approximate )  Observation  (Please i n d i c a t e the type of o b s e r v a t i o n a l s e t t i n g you w i l l be a r r a n g i n g ) .  Short w r i t t e n assignments  (one page or l e s s )  l_ /  Longer w r i t t e n a s s i g n m e n t s ( s h o r t e s s a y s )  j_  /  Major p r o j e c t s  /  /  Other ( p l e a s e s p e c i f y )  Any  other  comments?  Appendix  Q  S t u d e n t - t e a c h e r s ' Q u e s t i o n n a i r e on E v a l u a t i o n o f U.B.C. I n s t r u c t o r s ' T e a c h i n g Performance.  171  EVALUATION OF UBC STAFF TEACHING PERFORMANCE  In t h i s p a r t i c u l a r q u e s t i o n n a i r e , you a r e being asked t o r a t e the , UBC s t a f f , as a group, on a number of t e a c h i n g criteria. I f t h e r e a r e any i n d i v i d u a l s t a f f members who you f e e l do n o t f i t i n t o a r a t i n g f o r t h e r e s t of t h e group then p l e a s e i n d i c a t e , under t h e heading of e x c e p t i o n s , who those i n d i v i d u a l s are and why they do n o t f i t i n t o your o v e r a l l r a t i n g .  1.  How f r e q u e n t l y do UBC s t a f f members come l a t e to c l a s s without an adequate e x p l a n a t i o n ?  /  /  /  /  /  very frequently  exceptions?  /  /  /  /  sometimes  No  /  /or  Yes _/  /  never  /:  please  indicate  who the e x c e p t i o n s a r e and why they a r e e x c e p t i o n s .  172  2.  How w e l l i s t h e s t a f f s t i c k i n g to what you were l e d t o expect i n .their courses?  l—l  l_l  l_l  very well  exceptions? who  3.  LJ  LJ  sometimes deviate  No /  /or  Yes ]_  very poorly  / :  please  indicate  the e x c e p t i o n s a r e and why they a r e e x c e p t i o n s .  How much o p p o r t u n i t y do you get to ask q u e s t i o n s i n c l a s s ?  1—1  l_l  l_l  ample opportunity  exceptions? who  LJ  occasional opportunity  No /  /  or  Yes /  LJ insufficient opportunity  / :  please  indicate  the e x c e p t i o n s a r e and why they a r e e x c e p t i o n s .  173  4.  How w e l l o r g a n i z e d a r e t h e UBC s t a f f member's p r e s e n t a t i o n s in class.  /  /  /  /  /  very well organized  exceptions?  /  /  /  /  O.K.  No / /  or  Yes  /  very poorly organized  /  I  :  please  indicate  who t h e e x c e p t i o n s a r e , and why they a r e e x c e p t i o n s .  5.  Do t h e UBC s t a f f members know t h e i r s u b j e c t m a t e r i a l ?  Yes  /  /  exceptions?  No  No  /  /  /  /  or  Don't Know  Yes  /  /:  please  /  /  indicate  who t h e e x c e p t i o n s a r e , and why they a r e e x c e p t i o n s .  174  6.  How  u s e f u l a r e the assignments you have had  L—l  L_l  l_l  LJ  very u s e f u l  exceptions?  LJ useless  No  / /  what the e x c e p t i o n s  7.  so f a r ?  or  Yes  a r e , and why  ]_  / :  they a r e  please indicate  exceptions.  How worthwhile have you found the textbooks you bought and are p r e s e n t l y using? ( P l e a s e take i n t o c o n s i d e r a t i o n the c o s t of the books, the assignments ( r e a d i n g s , problems, e t c . ) you have had u s i n g the textbook, the r e a d i n g l e v e l , the c o n t e n t , etc.)  l—l  LJ  LJ  very worthwhile exceptions?  LJ  O.K. No  / /  what the e x c e p t i o n s  or  LJ worthless  Yes  a r e , and why  ]_  / :  they a r e  please indicate exceptions.  175  8.  Are t h e UBC s t a f f member r e a d i l y a v a i l a b l e when you need them?  Yes.  /  /  exceptions?  No  No / /  /  /  or  Don't Know  Yes  / /  :  please  /  /  indicate  who t h e e x c e p t i o n s a r e , and why they a r e e x c e p t i o n s .  What i s your o v e r a l l e v a l u a t i o n of t h e j o b t h e UBC s t a f f members a r e doing i n g e t t i n g a c r o s s t o you t h e knowledge, u n d e r s t a n d i n g , s k i l l s and a t t i t u d e s t h a t appear t o be t h e g o a l s of t h e courses? (In answering, t r y n o t t o be unduly i n f l u e n c e d by your l i k i n g o r d i s l i k i n g f o r t h e i n s t r u c t o r s as people; o r by your f e e l i n g s about t h e c o u r s e s ' content i f t h a t i s n ' t something t h e i n s t r u c t o r s c o u l d choose f r e e l y ) O u t s t a n d i n g l y good Above average Average Below Average O u t s t a n d i n g l y bad  exceptions?  No  / / or  Yes  / / / / /  / / / / /  /  / :  please  indicate  who t h e e x c e p t i o n s a r e , and why they a r e e x c e p t i o n s .  176  10.  .If you have any  other comments, p l e a s e do  so:  Appendix R Teachers' Questionnaire  2  178  Dear  Teacher,  This  questionnaire  how  has  naire  looks  questions you  wish  more  to  When  add box  you  the  ciated  your  under  you  slot  could  for  a  number  than  simple  school.  The  checkmark; please  this  do  questionnaire,  i n your  office.  them  within  about  question-  i s .  The  however, so  in  Yours  truly,  please  I t would a  time.  Norman  egr  questions  i f  the  question.  complete  your  of  i t actually  answers,  each  complete  i f you  Thank  just  to  mail  a  operating^your  formidable  require  "comments"  in  been  asks  Gleadow Evaluator  week.  put be  i t  appre-  179  TEACHER'S  PLEASE THE  I N D I C A T E YOUR  APPROPRIATE  THIS  FORM.  BOX.  QUESTIONNAIRE  ANSWER  BY P U T T I N G  P L E A S E DO NOT  I F YOU WISH TO MAKE  ANY  A CHECKMARK I N  PUT YOUR NAME COMMENTS,  ON  P L E A S E DO  SO  I N THE BOX  PROVIDED.  1.  The s t u d e n t - t e a c h e r s have been t a k i n g a number o f U n i v e r s i t y courses a t your s c h o o l . P l e a s e p l a c e checkmarks b e s i d e those u n i v e r s i t y courses f o r which you have r e c e i v e d course OUTLINES; Music  Math  Science  Social Studies  Psychology  '  Other _____ (Please s p e c i f y )  P.E. Language Arts  Comments:  2.  Do you f i n d t h e m a t e r i a l you r e c e i v e about t h e courses b e i n g taught t h e student t e a c h e r s o f any v a l u e ? Not a p p l i c a b l e  /  /,  Yes  /__/,.  No  Haven't had time t o read i t / /  Comments:  /  /  4 180  3.  A r e you b e i n g kept aware o f t h e s t u d e n t - t e a c h e r s ' c o u r s e ASSIGNMENTS? Yes,  f o r a l l courses  /  /  Yes,  f o r most c o u r s e s  /  /  Yes,  f o r about h a l f the c o u r s e s  /  /  Yes, but f o r l e s s than h a l f the c o u r s e s  /  /  No, have not r e c e i v e d any i n f o r m a t i o n of t h i s type  /  /  Comments:  4.  I n your o p i n i o n , have the U.B.C. p r o f e s s o r s a d e q u a t e l y c o n s u l t e d w i t h you w i t h r e s p e c t t o : (a)  Placement o f s t u d e n t - t e a c h e r s w i t h sponsor Yes  /  /  No  /  /  Not a p p l i c a b l e  teachers? /  /  Comments:  (b)  Teaching time f o r t h e s t u d e n t - t e a c h e r working w i t h you? Yes  /  Comments:  /  No l_  /  Not a p p l i c a b l e  /  /  5  181  (c)  A p p r o p r i a t e t e a c h i n g s t r a t e g i e s (methods, c l a s s r o o m management, m a t e r i a l to c o v e r , e t c . ) f o r your student teacher? Yes  /  /  No  /  /  Not a p p l i c a b l e  /  /  Comments:  (d)  The e v a l u a t i o n o f your s t u d e n t - t e a c h e r s ' t e a c h i n g effectiveness. Yes  /  /  No  /  /  Not a p p l i c a b l e  /  /  Comments:  5.  A r e you c l e a r about your r o l e i n the ( P l e a s e check one o f the f o l l o w i n g : )  . program?  No, I'm t o t a l l y confused about what I'm expected I t h i n k I know what I'm expected s t i l l some areas o f c o n f u s i o n .  t o do.  /  /  /  /  /  t o do b u t t h e r e a r e  Yes, I have a v e r y c l e a r i d e a o f what I am expected to do.  Comments:  /  6  182  6.  G e n e r a l l y , how would you r a t e t h e communication between the teachers . . and the U.B.C. s t a f f .  /__/ Excellent  /_/ Good  LJ  LJ  " Fair  Poor  I  I  Terrible  Comments:  7.  Are t h e s t u d e n t - t e a c h e r s /  /  /  /  Always  courteous /  /  /  /  Sometimes  /  /  Never  Comments:  8.  A r e the s t u d e n t - t e a c h e r s a p p r o p r i a t e l y dressed?  / / Always  Comments:  //  // Sometimes  LJ  LJ Never  7 183  Do.you f e e l the s t u d e n t - t e a c h e r s a r e s e p e r a t l n g themselves from the o p e r a t i o n of the s c h o o l ( i . e . s t a y i n g i n t h e i r room too much, not mixing s u f f i c i e n t l y w i t h the t e a c h e r s , e t c . ) Yes  10.  /  /  No  /  Undecided  /  Do you approve of the s t u d e n t - t e a c h e r s ' b e h a v i o r c o n t a c t s w i t h the p u p i l s i n your school? approve  /  /  disapprove  /  /  / /  i n a l l their  undecided  I  f  I f you d i s a p p r o v e of t h e i r b e h a v i o r , would you please'' i n d i c a t e your reasons.  11.  Are y o u . s a t i s f i e d teaching? Yes  /  /  w i t h the s t u d e n t - t e a c h e r s ' p r e p a r a t i o n f o r  No  /  /  undecided  / /  I f you a r e not s a t i s f i e d w i t h t h e i r p r e p a r a t i o n , p l e a s e i n d i c a t e how they c o u l d prepare b e t t e r .  8  184  12.  G e n e r a l l y , how would you r a t e the o v e r a l l r a p p o r t the s t u d e n t - t e a c h e r s and the teachers? /  /  /  excellent  13.  /  /  good  /  fair  /  between  /  poor  /__/ non-existent  Do you f e e l t h a t your p u p i l s a r e g e t t i n g b e h i n d i n t h e i r work because of the program? Yes  /  /  No  /  Undecided  /  /  /  I f you answered " y e s " would you p l e a s e i n d i c a t e the s e v e r i t y of the problem.  14.  Have you found Program B too t i r i n g ? Yes  /  /  No  /  /  Not Sure  I f y e s , i n what way has i t been too t i r i n g ?  /  /  9 185  15.  Do you have adequate time to do your own marking, e t c . Yes  /  /  No  /  /  lesson planning,  Not  Sure  /  /  Didn't attend  /  Comments:  16.  D i d you  f i n d the Communications Workshop u s e f u l ?  Yes  /  /  No Undecided  17.  /  /  /  What i s your o v e r a l l e v a l u a t i o n of the program as i t has operated i n your s c h o o l f o r the p a s t month? Excellent  /  /  Good  /  /  O.K.  /  /  Poor  /  /  /  /  " Terrible 18.  /  /  Are t h e r e any minor or major problems you f e e l s h o u l d taken c a r e of now b e f o r e they get any worse? Yes  /  /  No  /  /  be  10  186  18.  continued  I f you answered "Yes" p l e a s e i n d i c a t e what these problems are.  19.  Are t h e r e any s p e c i f i c p o s i t i v e a s p e c t s of Program B y o u have n o t i c e d i n the p a s t month?  Yes  /  /  No  /  /  I f you answered " y e s " w i l l you p l e a s e i n d i c a t e what p o s i t i v e aspects are.  20.  I f you have any o t h e r comments, p l e a s e do so.  those  187  Appendix S Interview with D i r e c t o r o f Elementary a t S c h o o l Board O f f i c e s .  Education  188 Q u e s t i o n s Asked. (I)  Have you r e c e i v e d any i n f o r m a t i o n on b e n e f i c i a l e f f e c t s o f the program from: P a r e n t s ? S t u d e n t s ? T e a c h e r s ? P r i n c i p a l s ? Administrators?  2)  Have you r e c e i v e d any i n f o r m a t i o n on harmful e f f e c t s o f t h e program from: P a r e n t s ? S t u d e n t s ? T e a c h e r s ? P r i n c i p a l s ? Administrators?  3)  Does t h e program c o s t the s c h o o l D i s t r i c t money? manpower? Is i t worth t h e c o s t ( i f any)?  4)  Has i t i n c r e a s e d y o u r w o r k l o a d u n a c c e p t a b l y ?  5) ,q\  Do you s e e any need f o r t h i s a p p r o a c h t o t e a c h e r - t r a i n i n g ? o r why n o t ?  6)  What a s p e c t o f the program a r e y o u most i n t e r e s t e d i n ?  7)  Do you have any p l a n s f o r e v a l u a t i n g the program? I f n o t , why n o t ?  8)  What i s y o u r o v e r a l l i m p r e s s i o n o f Program B as i t has o p e r a t e d to d a t e ?  9)  Any o t h e r comments?  v-  time?  Why  I f s o , what?  189  Appendix I f E v a l u a t i o n o f the S t u d e n t T e a c h e r s E f f e c t i v e n e s s i n Program B (a P r o p o s a l s u b m i t t e d t o and a c c e p t e d by t h e t e a c h e r s h o s t i n g Program B)  190 P r o p o s a l f o r the E v a l u a t i o n of the Student  Teachers  E f f e c t i v e n e s s i n the Program  Research  has not y e t e s t a b l i s h e d u s a b l e c o n n e c t i o n s between observed  teacher b e h a v i o r s  i n the c l a s s r o o m and  e d u c a t i o n a l g a i n s by the p u p i l s .  Class-  room o b s e r v a t i o n a l instruments a r e l e g i t i m a t e t o o l s f o r o b t a i n i n g an a c c u r a t e account  of what i s t a k i n g p l a c e i n the c l a s s r o o m but they do n o t n e c e s s a r i l y  i d e n t i f y v a r i a b l e s which can be used  to judge e f f e c t i v e t e a c h e r s .  d e s c r i p t i o n s o f a t e a c h e r ' s classroom t e c h n i q u e a r e o n l y secondary of teaching e f f e c t i v e n e s s .  For example, one  c l a s s r o o m w i t h w e l l developed another  t e a c h e r may  t e a c h e r may  At b e s t indications  run a very  "tight"  systems of management and r e g u l a t i o n , w h i l e  have a v e r y " l o o s e " classroom.  The  two  t e a c h e r s would have  d i f f e r e n t p r o f i l e s on any o b s e r v a t i o n a l instrument which d e s c r i b e d the t e a c h i n g a c t , y e t both might be d e s c r i b e d as b e i n g "good" t e a c h e r s . I f we  a r e to e v a l u a t e t e a c h i n g e f f e c t i v e n e s s , then we must go beyond  d e s c r i p t i o n s of the t e a c h i n g a c t and s e a r c h f o r primary teaching.  T h i s i s not meant to suggest  i v e i n d i c a t o r s of good t e a c h i n g . to a s m a l l number o f secondary  i n d i c a t o r s o f "good"  t h a t t h e r e a r e no secondary- o r d e s c r i p t -  Some weight i n the e v a l u a t i o n can be  given  i n d i c a t o r s chosen on a common s e n s i c a l , o r moral  b a s i s , and which can be expected  to a t l e a s t weakly c o r r e l a t e w i t h good t e a c h -  ing.  T h e r e f o r e , a major f e a t u r e o f the e v a l u a t i o n p l a n i s extreme  parsimony i n the d a t a - g a t h e r i n g , , combined w i t t u extreme c a r e i n i d e n t i f y i n g use to which the d a t a w i l l be put. validity.  Parsimony i s advocated  the  f o r the sake o f  Only d a t a which can be demonstrated to be connected w i t h t e a c h i n g  m e r i t s h o u l d be c o l l e c t e d and  then o n l y when t h e r e i s a l e g i t i m a t e use f o r i t .  -2-  191 The major uses o f the e v a l u a t i o n o f s t u d e n t - t e a c h i n g i n c l u d e : 1.  E v a l u a t i o n f o r self-improvement.  2.  E v a l u a t i o n to a s s i s t t h e sponsor t e a c h e r and U.B.C. a d v i s e r w i t h the p r o c e s s o f improving the s t u d e n t t e a c h e r ' s t e a c h i n g performance.  3.  E v a l u a t i o n to a d v i s e o t h e r p o s s i b l e consumers, such as p u p i l s d e s i r i n g teacher a s s i s t a n c e i n other school activities.  4.  E v a l u a t i o n f o r p e r s o n n e l d e c i s i o n s by s c h o o l and U.B.C. staffs.  5.  E v a l u a t i o n by e x t e r n a l audiences s u c c e s s o f the Project.  D e s c r i p t i o n o f the E v a l u a t i o n Instruments these p r o p o s a l ) I.  i n t e r e s t e d i n , e.g.,  (These instruments a r e a t t a c h e d to  Primary I n d i c a t o r s (1)  O v e r a l l E v a l u a t i o n o f Teaching. T h i s i s a s i m p l e r a t i n g on a f i v e p o i n t s c a l e , r a n g i n g from  out-  s t a n d i n g l y good t o o u t s t a n d i n g l y bad o f the s t u d e n t t e a c h e r ' s t e a c h i n g ability.  T h i s s i n g l e r a t i n g has two main advantages.  Firstly,  i t cannot  be e a s i l y m i s i n t e r p r e t e d , and s e c o n d l y , t h e chances t h a t a r a t i n g i s n o t r e l a t e d to an i n d i c a t i o n o f t e a c h i n g a b i l i t y  a r e extremely  remote.  I d e a l l y t h i s r a t i n g s h a l l be g i v e n by the p u p i l s b e i n g taught by the s t u d e n t t e a c h e r .  However, i n an elementary  special d i f f i c u l t i e s , particularly may n o t f u l l y understand  school, this  presents  i n t h e lower grades where the p u p i l s  what i s expected o f them.  I n t h i s s i t u a t i o n , we  may have to r e l y on r a t i n g s g i v e n t h e s t u d e n t t e a c h e r by t h e sponsor ers of  and the U.B.C. a d v i s o r s . the t e a c h i n g o f the student  S i n c e these people a r e n o t primary  teach-  consumers  t e a c h e r , then the o v e r a l l e v a l u a t i v e r a t i n g  must be based on as many c l a s s r o o m v i s i t s as p o s s i b l e - c e r t a i n l y n o t on the b a s i s o f a s i n g l e  visit.  192 (2)  Pupil  Gains  A t e a c h e r ' s main f u n c t i o n i s t o meet t h e e d u c a t i o n a l needs o f his/her pupils.  I n o t h e r words, as a r e s u l t o f the t e a c h i n g a c t , t h e  p u p i l s s h o u l d have gained some combination o f knowledge, s k i l l , i t i o n or attitude.  dispos-  I t i s i n c o n c e i v a b l e to c a l l someone a good t e a c h e r  who's p u p i l s do n o t " l e a r n " .  T h e r e f o r e , i t i s proposed  t h a t the s t u d e n t  t e a c h e r s u s e some method o f d e t e r m i n i n g whether t h e p u p i l s have l e a r n e d a n y t h i n g w h i l e exposed to the s t u d e n t t e a c h e r ' s t e a c h i n g .  To g i v e t h i s  some e x p e r i m e n t a l v a l i d i t y a s i m p l e p r e - t e s t , p o s t - t e s t procedure i s suggested.  That i s , i n c o n s u l t a t i o n w i t h t h e sponsor  be made up on a p a r t i c u l a r u n i t (Dr.  the  p r o v i d e the techniques needed f o r t e s t  T h i s t e s t w i l l be g i v e n to t h e p u p i l s b e f o r e t h e u n i t i s  taught and a f t e r the u n i t it  will  t h e s t u d e n t t e a c h e r w i l l be t e a c h i n g .  Ed. 311 course w i l l  construction.)  teacher, a test  i s taught.  Using s i m p l e s t a t i s t i c a l  techniques  can be determined whether o r n o t t h e r e Was p u p i l g a i n as measured by test.  Secondary I n d i c a t o r s 1)  Minimal  teaching o b l i g a t i o n s  Minimal  t e a c h i n g o b l i g a t i o n s a r e d u t i e s or minimum requirements f o r  .teachers.  F u l f i l l i n g these d u t i e s does n o t n o r m a l l y count f o r t h e t e a c h e r ,  but n o t f u l f i l l i n g minimal  them does account a g a i n s t t h e t e a c h e r .  Typically  these  o b l i g a t i o n s a r e b e i n g on time f o r t h e c l a s s he/she i s to teach  (except when i l l e t c . ) , t e s t i n g and marking p u p i l s f a i r l y , knowing t h e s u b j e c t matter o f what he/she i s t e a c h i n g , e t c . the s t u d e n t t e a c h e r i s f u l f i l l i n g  The b e s t judges o f whether  these minimal o b l i g a t i o n s a r e t h e primary  consumers o f h i s t e a c h i n g , the p u p i l s .  However, as f o r the o v e r a l l  e v a l u a t i o n , t h e r e s p o n s i b i l i t y f o r d e t e r m i n i n g i f t h e minimal  obligations  -4193 a r e b e i n g f u l f i l l e d may have to f a l l on the sponsor  t e a c h e r s and U.B.C.  advisors. 2)  Out-of-Class Contributions In the  The  program, the student  t e a c h e r s a r e p a r t o f the s c h o o l .  s c h o o l i s not simply a g e o g r a p h i c a l l o c a t i o n , d i f f e r e n t from t h a t of  the U n i v e r s i t y , where c l a s s e s a r e h e l d .  Therefore, the student-teachers  should do more than teach and a t t e n d c l a s s e s , and they s h o u l d teach o t h e r than by b e i n g i n a classroom.  T h e i r p u b l i c and p r o f e s s i o n a l s e r v i c e to  the s c h o o l , s t a f f and p u p i l s through be acknowledged.  extra-curricular activities  The s c h o o l a d m i n i s t r a t i o n , t h e sponsor  U.B.C. a d v i s o r s would probably be the b e s t judges  should  t e a c h e r and the  o f the degree o f o u t -  of-class contributions. 3)  D e s c r i p t i o n o f Teaching " D e s c r i p t i o n o f Teaching"  i s perhaps a misnomer f o r t h i s s e c t i o n .  The  t e a c h i n g a c t i s n o t t o be described*, i n s t e a d , t h e r e i s a check made f o r u n d e s i r a b l e extremes i n t h e student extremes a r e common s e n s i c a l .  teachers' teaching s t y l e .  For example, i t i s u n l i k e l y  These  t h a t a good  t e a c h e r would c o n s t a n t l y use o r a l o r w r i t t e n language which was too d i f f i c u l t f o r the p u p i l s to understand;  o r i t i s u n l i k e l y t h a t a good  teacher would be c o n s i s t e n t l y d i s o r g a n i z e d . a b l e to have the p u p i l s respond not p o s s i b l e , then t h e sponsor  t o t h i s c h e c k l i s t ; however, i f t h a t i s t e a c h e r s , and U.B.C. a d v i s o r s s h o u l d  the c h e c k l i s t on the b a s i s o f many classroom 4)  As b e f o r e , i t would be d e s i r -  complete  visits.  Performance i n U n i v e r s i t y Courses. The  student  t e a c h e r s a r e a l s o s t u d e n t s o f the U n i v e r s i t y .  they have c e r t a i n standards o r n o t a student  Therefore,  to achieve i n t h e i r u n i v e r s i t y courses.  t e a c h e r has met the standards  Whether  f o r a p a r t i c u l a r course  will  -5-  194 be determined by the p r o f e s s o r t e a c h i n g t h a t c o u r s e . i n d i c a t o r s o f t h e student necessary  teachers  These marks a r e n o t  t e a c h i n g e f f e c t i v e n e s s , b u t they a r e  f o r e s t a b l i s h i n g an o v e r a l l e v a l u a t i o n o f t h e s t u d e n t  teachers'  performance i n the program.  The E v a l u a t i o n Procedures Each sponsor t e a c h e r and each U.B.C. a d v i s o r w i l l for  each s t u d e n t  t e a c h e r under h i s / h e r s u p e r v i s i o n .  receive a booklet  The b o o k l e t w i l l be organ-  i z e d as f o l l o w s : COVER -  Student t e a c h e r s ' name, grade he/she i s t e a c h i n g , Sponsor Teachers' o r U.B.C. A d v i s o r ' s name.  Page 6 - O v e r a l l e v a l u a t i o n o f t e a c h i n g r a t i n g s c a l e . Page 5 - Minimal t e a c h i n g o b l i g a t i o n s c h e c k l i s t Page 4 - O u t - o f - C l a s s Page 1,2 a n d 3 -  Contributions  D e s c r i p t i o n of teaching  checklist.  The measurement o f p u p i l gains w i l l be handled themselves.  by t h e student  teachers  The grades o b t a i n e d on the U.B.C. c o u r s e s w i l l be c o l l e c t e d by  the e v a l u a t o r when those marks a r e a v a i l a b l e . made on the b a s i s of as many c l a s s r o o m  The v a r i o u s r a t i n g s w i l l be  o b s e r v a t i o n s as p o s s i b l e by the Sponsor  Teacher and the U.B.C. a d v i s o r . A l l o f t h i s data w i l l be e n t e r e d on the s t u d e n t e v a l u a t i o n form. and  teacher's  teaching  T h i s t e a c h i n g e v a l u a t i o n form (T-form) w i l l be an a v e r a g i n g  condensing o f the above i n f o r m a t i o n from the sponsor t e a c h e r , U.B.'C.  a d v i s o r and ( i f p o s s i b l e ) the p u p i l s .  -6-  Inputs  t o T-Form  195  Method o f O b t a i n i n g  Inputs  1.  P u p i l estimate of teaching merit if possible. Otherwise, sponsor t e a c h e r and U.B.C. a d v i s o r overa l l evaluation of teaching.  1.  S i n g l e q u e s t i o n c a l l i n g f o r an evaluation of the teaching s k i l l on a simple A-F s c a l e ( s p e c i a l c o n s i d e r a t i o n f o r K-3).  2.  E s t i m a t e s o f classroom j u s t i c e , e.g., a p p r o p r i a t e q u i z z e s and materials, f a i r hearing f o r complaints, e t c . *  2.  Program o f adequate random v i s i t s to y i e l d r e l i a b l e judgments, p l u s d e t a i l e d study o f m a t e r i a l s , q u i z z e s , e t c . V i s i t s by s c h o o l s t a f f and U.B.C. s t a f f .  3.  Non-classroom c o n t r i b u t i o n to t e a c h i n g .  3.  School s t a f f , peers, and p o s s i b l y pupils.  4.  L e a r n i n g gains due to student t e a c h e r .  4.  C o n t r o l l e d study u s i n g p r e - and p o s t - t e s t s e t c . , done by s t u d e n t teacher w i t h a d v i c e o f e v a l u a t i o n specialist.  5.  Course-work i n u n i v e r s i t y courses r e l a t e d to t e a c h i n g .  5.  U.B.C. s t a f f .  * I f i t i s r e a l l y n e c e s s a r y to do s o , s c h o o l and U.B.C. s t a f f of t e a c h i n g s t y l e o f s t u d e n t t e a c h e r .  could estimate merit  Each o f these i n p u t s c o n t r i b u t e s something i n t h e d i r e c t i o n o f good t e a c h i n g . The o p t i m a l model f o r w e i g h t i n g  these i n p u t s i s equal w e i g h t i n g .  T h i s means  t h a t i n a summative e v a l u a t i o n o f the student t e a c h e r each i n p u t has equal weight. Any  e v a l u a t i o n o f a student t e a c h e r by the p u p i l s  be made a v a i l a b l e t o the s t u d e n t t e a c h e r .  ( i f performed) s h o u l d  Anonymity o f the p u p i l s should be  assured. H o p e f u l l y , i n the y e a r s to come, a b e t t e r way to e v a l u a t e t e a c h i n g w i l l come about.  B u t , meanwhile, l e t ' s n o t u s e something worse.  196  EVALUATION OF STUDENT TEACHERS  STUDENT TEACHER NAME DATE  ( p l e n s e put y o u r numa i n t h e a p p r o p r i a t e s p a c e ) SPONSOR TBACKER U.B.C. ADVISOR  ( p l e a s e i n d i c a t e whon t h e s t u d e n t eerved,  t e a c h e r wao ob-  and what s u b j e c t and grade he/she  &ate o f observation  Subject taught  being  was t e n c h i n e )  grac e  197  Plaa'-e r».te each o f th«  following  ctancrii't j tii<n' o f the  t u i o l . t r i i t e a c h i n g on t h e s c a l e b e u i d e each SA o a t r o n g l y B j i r e a  stu'1.-;it  question.  I e x c e p t i o n a l l y good performance )  A = ftj=;re3 ( s c c e p t a b l e  performance)  U » undecided D = disagree  (performance n o t a t Rn  acceptable  level) SD = s t r o n g l y d i s a g r e e  ( e x c e p t i o n a l l y poor  per-  formance ) 1. The s t u d e n t  taacher presentd  o f t h e l e s s o n a t un a p p r o p r i a t e  the content SA  pace,  A  U  D  SD  comments:  2. The s t u d e n t ented  teacher's  lessons are pres-  i n a well organised  comments:  3. The s t u d e n t  —  teacher's  E n g l i s h are appropriate e l he/she i s t e a c h i n g .  S3  SA  manner, —  o r a l and w r i t t e n f o r t h e grpde  levSA  A  U  SD  198  4.  The s t u d e n t  t e e c h e r has an e f f e c t i v e  fair  system o f c l a s s r o o m comments:  control  5. The s t u d e n t t e a c h e r t e a c h e s of  and  (discipline).  I SA  A  U  D  SD  SA  A  U  D  SD  SA  A  U  D  SD  SA  A  U  D  SD  SA  A  U  D  SD  the content  t h e l e s s o n a t an approx^riate l e v e l o f  difficulty  f o r the grade l e v e l .  comments:  6. The s t u d e n t ual  t e a c h e r uses a u d i o - v i s -  m a t e r i a l when a p p r o p r i a t e .  comments:  7.  The s t u d e n t  t e a c h o r a c c e p t s and a c t s  on c o n s t r u c t i v e c r i t i c i s m . comments:  8. The p u p i l s r e a c t p o s i t i v e l y student  teacher.  comments:  t o the-  |  199  A  ?. The  p u p i l s appear to be l e a r n i n g  the  course u v t t e r i a l b e i n g t a u g h t t h e n by  the  otudar,t t e e c h e r ,  SA  W i l l you p l e a s o i;ive the e v i d e n c e used to determine i n no. 9.  10.  the r a t i n g  giveni  P l e a s e comment on any o t h e r t r a i t s which you  are c o n t r i b u t i n g teaching  t o , o r d e c r e a a i n e the s t u d e n t  effectiveness.  A  you  think  teacher's  TJ  D  SD  200  OUT-GF-CLASr: CCI'TRI3UTIC?I 1. Kns t h e s t u d e n t  teacher c o n t r i b u t e d to the organ-  i z a t i o n and/or o p e r a t i o n o f s c h o o l s p o r t s ?  never  seldonu , sometimes frequently  2. Han t h e s t u d e n t  don't know  very frequently  teacher contributed to or p a r t i c -  i p a t e d on s c h o o l committees?  |  [  never  |  seldom ^  3. Han t h e s t u d e n t  |  [  j  [ — [  sometimes frequently  t e a c h e r spent  very frequently  don' t know  o u t - o f - c l a s s time ( f o r  example, l u n c h time o r a f t e r s c h o o l ) h e l p i n g p u p i l s in  t h o i r s c h o o l work?  tzz  •  never  seldom>-: sometimes frequently  4. Kan t h e s t u d e n t fort  tzi  EZI •  t e a c h e r c o n t r i b u t e d any time o r e f -  t o o r g a n i s e d s o c i a l events  as p l a y s , s k i t s ,  [ZH never  •  assemblies,  tZ3  i n t h e s c h o o l , such  pageants,  •  seldom*-. sometimes frequently  5. Has t h e s t u d e n t  don't know  very frequently  teacher volunteered  etc.?  LZI  very frequently  ' CJ don't know  f o r or participated  i n any e x t r a - c u r r i c u l s r a c t i v i t i e s n o t c o v e r e d by t h e above f o u r q u e s t i o n s ? nature  o f the a c t i v i t y ,  I f so ple-.se b r i e f l y n o t e t h e and t h e amount o f p a r t i c i p a t i o n .  201  MINIMAL STUDKHT TEACHEH OBLIGATIONS X. Th.i s t u d e n t t e a c h e r spends an adequate amount o f time coriulting  rfith  •  the sponsor  teacher.  •  yes  r.o  2. T h j s t u d e n t  teacher f a i l e d  . • undecided t o appear^on o c c a s i o n s  udequ-.'.te e x p l a n a t i o n s ; o r was f r e a u e h t l y  •  no  without  late.  •  '•  ye3  very  undecided  3. The s t u d e n t t e a c h e r g i v e s t h e p u p i l e enough o p p c r t u n i t i e o to a3k q u j H t i o n a .  • • yea  A. Ihe s t u d e n t pupils'  no  O undecided  teacher gives s a t i s f a c t o r y  questions.  • • yes  no  responses  t o the  • undecided  5. The s t u d e n t t e a c h e r seems s e r i o u s l y l a c k i n g i n knowledge of t h e s u b j e c t n a t t e r he/she i s t e a c h i n g .  • • yeB  no  •  undecided  6. Ihe s t u d e n t t e a c h e r i s a v a i l a b l e to t h e p u p i l s , when needed, no q u e s t i o n s o r e x t r a h e lupn.d e c i d e d f o r o u t - yes of-class 7. The t e s t s and marking o f t h e s t u d e n t t o a c b e r eeem f a i r .  • • yes  no  •  undecided  6. F u r t h u r comments on t h e above r a t i n g s , o r f u r t h u r i n f o r mation? .  202  OVERALL EVALUATION" CF TEACHING Will job  the B t u d e n t  knowledge, seem  y o u please  give  teacher  overall  i e doing  understanding,  t o be a p p r o p r i a t e  your  skills,  and a t t i t u d e s  wering  t r y n o t t o be i n f l u e n c e d b y y o u r  liking  f o r the B t i i d e n t  Outstandingly Good  Above Average  Average  Comments t o c l a r i f y t h e above  o f the  i n c o m m u n i c a t i n g the  f o r these, grade  teachar  rating  as a  that  l e v e l s . I n ansliking  or dis-  person.  Below Average  rating?.  CD  Outstandingly Bad  Appendix U Summary o f E v a l u a t i o n S t e p s i n RANKING.  204  Steps i n RANKING. (See S e c . 3.70 o f . t h i s t h e s i s )  D e t e r m i n e t h e e v a l u a t a and the p o i n t s o f v i e w t o be taken i n the e v a l u a t i o n . z  Establish the goals of the evaluata.  Are t h e eva uata a t t e m p t i n g t o a c h i e v e t h e same g o a l s ;  NO  Eliminate those e v a l u a t a which are attempting to a c h i e v e d i f ferent goals.  NO  F u r t h e r evaluat i o n would be pointless.  YES Are t h e goa s worthwhi Further evaluation would be p o i n t l e s s A l l t h e programs s h o u l d be r e v i s e d .  YES Determine the degree t o which each e v a l u a t u m i s a c h i e v i n g the goals.  NO Are t h e e v a l u a t a attempting to yes achieve other goals? NO A r e some o f t h e evaluata achieving the goals to an a c c e p t a b l e degree?  NO  Are a l l t h e e v a l u a t a achieving] the g o a l s t o an a c c e p t a b l e degree? YES  YES E l i m i n a t e those proDetermine the r e l a t i v e costs grams n o t a c h i e v i n g of the evaluata achieving the the g o a l s from f u r t h e r g o a l s . consideration.  -*-(go t o next page)  205  (from p r e v i o u s page) A r e some o f t h e evaluata being NO operated a t acceptable levels? NO  YES.  Rank o r d e r t h e e v a l u a t a on t h e b a s i s o f goal achievement and point out the shortcomings o f t h e e v a l u a t a i n terms o f t h e i r e x c e s s i v e costs.'  1  Are t h e e v a l u a t a b e i n g o p e r a t e d at acceptable cost levels. YES Average t h e r a t i n g s f o r goal achievement and c o s t f o r each evaluatum o p e r a t i n g a t acceptable cost. A r e t h e r e any t i e s f o r t h e highest average r a t i n g ? NO •  Evaluatum w i t h h i g h e s t a v e r a g e r a t i n g i s t h e " B e s t Buy".!  YES  .1 P i c k t h e e v a l uatum w i t h t h e h i g h e s t goal achievement r a t i n g as t h e " B e s t Buv" 1  Is t h e r e s t i l l a tie? YES The evaluatum with the best p l a n s f o r extended s u p p o r t i s t h e b e s t buy, If there i s still a tie then the t i e d evaluata are o f equal w o r t h . Note:  " B e s t Buy" means t h a t t h e evaluatum, X, o f f e r s t h e b e s t performance f o r l e s s c o s t t h a n .the e v a l u a t a t o which i t i s b e i n g compared.  1  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0093721/manifest

Comment

Related Items