UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Program evaluation in context : models of negotiated knowledge production and use Levitan, Alberta Potter 1983

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.

Item Metadata

Download

Media
831-UBC_1983_A1 L49.pdf [ 13.95MB ]
Metadata
JSON: 831-1.0095877.json
JSON-LD: 831-1.0095877-ld.json
RDF/XML (Pretty): 831-1.0095877-rdf.xml
RDF/JSON: 831-1.0095877-rdf.json
Turtle: 831-1.0095877-turtle.txt
N-Triples: 831-1.0095877-rdf-ntriples.txt
Original Record: 831-1.0095877-source.json
Full Text
831-1.0095877-fulltext.txt
Citation
831-1.0095877.ris

Full Text

PROGRAM EVALUATION IN CONTEXT: MODELS OF NEGOTIATED KNOWLEDGE PRODUCTION AND' USE by A l b e r t a P o t t e r L e v i t a n B.A. A n t i o c h Col l e g e 1959 M.A. U n i v e r s i t y o f M i c h i g a n 1961 A D i s s e r t a t i o n S u b m i t t e d i n P a r t i a l F u l f i l m e n t o f the Requirements f o r the Degree o f Do c t o r i n P h i l o s o p h y i n The F a c u l t y o f Graduate S t u d i e s (Department o f S o c i o l o g y ) We a c c e p t t h i s d i s s e r t a t i o n as c o n f o r m i n g t o t h e r e q u i r e d s t a n d a r d The U n i v e r s i t y o f B r i t i s h Columbia February- 19S3 (c) A l b e r t a P o t t e r L e v i t a n , 1983 In p r e s e n t i n g t h i s t h e s i s i n p a r t i a l f u l f i l m e n t of the requirements f o r an advanced degree a t the U n i v e r s i t y o f B r i t i s h Columbia, I agree t h a t the L i b r a r y s h a l l make i t f r e e l y a v a i l a b l e f o r r e f e r e n c e and study. I f u r t h e r agree t h a t p e r m i s s i o n f o r e x t e n s i v e copying of t h i s t h e s i s f o r s c h o l a r l y purposes may be granted by the head of my department or by h i s o r her r e p r e s e n t a t i v e s . I t i s understood t h a t copying or p u b l i c a t i o n of t h i s t h e s i s f o r f i n a n c i a l g a i n s h a l l not be allowed without my w r i t t e n p e r m i s s i o n . Department of The U n i v e r s i t y of B r i t i s h Columbia 1956 Main Mall Vancouver, Canada V6T 1Y3 Date DE-6 (3/81) i i . ABSTRACT Recent growth and c o - o p t a t i o n o f s o c i a l reform programs i n t o the s t r u c t u r e o f the S t a t e , and p a r a l l e l development o f p u b l i c p o l i c y m a k i n g , have p r e c i p i t a t e d c l o s e r l i n k a g e s between s o c i a l r e s e a r c h and the p o l i c y i n t e r v e n t i o n p r o c e s s . Program e v a l u a t i o n r e f e r s to a v a r i e t y o f d e s c r i p t i v e and a n a l y t i c a l s t u d i e s o f program p r o c e s s and/or program impact. Two models o f program e v a l u a t i o n r e s e a r c h are r e l i e d upon i n d e s i g n and implemen-t a t i o n o f e v a l u a t i o n s t u d i e s : ( l ) a c o n v e n t i o n a l model d e r i v e d from a p o s i t i v i s t paradigm o f s o c i a l r e s e a r c h and (2) an a l t e r n a t i v e model e v o l v e d from an i n t e r p r e t i v e paradigm. C r i t i c a l review o f these models suggests t h e i r complementarity f o r comprehensive e v a l u a t i o n s t u d i e s , but emphasizes the e x t e n t to which they m i n i m i z e the s i g n i f i c a n c e o f l a r g e r p o l i t i c a l / economic c o n t e x t o f program development i n shaping e v a l u a t i o n p r o c e s s e s . The purpose o f t h i s d i s s e r t a t i o n has been t o develop a w i d e r s e l e c t i o n o f e v a l u a t i o n r e s e a r c h models which s p e c i f i c a l l y take i n t o account c o n s t r u c -t i o n o f the r e s e a r c h " p r o d u c t " and c h a r a c t e r i s t i c s o f the l a r g e r s t r u c t u r a l c o n t e x t in which such p r o d u c t s a r e d e s i g n e d to be used. The t h e o r e t i c a l s t r a t e g y r e l i e s on a s p e c t s o f S t r a u s s ' n e g o t i a t e d o r d e r paradigm and approaches to p o l i c y r e s e a r c h taken by Rein and Wiseman, and i n v o l v e s an e f f o r t to r e l a t e more s t a b l e s t r u c t u r a l c h a r a c t e r i s t i c s o f the s o c i a l , p o l i t i c a l , economic and o r g a n i z a t i o n a l c o n t e x t o f reform programs to a s e r i e s o f s i x b a s i c areas o f n e g o t i a t i o n in the e v a l u a t i o n p r o c e s s . These i n c l u d e : ( l ) d e l i n e a t i o n o f major a c t o r s ; (2) o r g a n i z a t i o n a l placement o f program e v a l u a t i o n work; (3) c h o i c e o f g e n e r a l r e s e a r c h s t r a t e g y ; (k) s e l e c -t i o n o f a p p r o p r i a t e r e s e a r c h model and methodology; (5) c o n s t r u c t i o n and c o n t e n t o f r e s e a r c h r e p o r t s ; and (6) p l a n n i n g f o r r e s e a r c h u t i l i z a t i o n . T h i s framework p r o v i d e d the t h e o r e t i c a l p e r s p e c t i v e f o r d e s c r i p t i o n and a n a l y s i s o f f o u r case s t u d i e s : two i n hou s i n g p o l i c y , one i n p r i v a t e s o c i a l s e r v i c e d e l i v e r y and one i n d e l i v e r y o f l e g a l e d u c a t i o n s e r v i c e s . C o n c l u s i o n s from case s t u d i e s , and o t h e r r e s e a r c h s u g g e s t e d f o u r models o f n e g o t i a t e d knowledge. New models i n c l u d e elements o f p o s i t i v i s t and i n t e r p r e t i v e models but a r e de s i g n e d around the s t r u c t u r a l c o n t e x t o f program p l a n n i n g and implementation and f o c u s d i r e c t l y on t h e s i x b a s i c a r e a s o f n e g o t i a t i o n . T h i s expanded r e p e r t o i r e o f models o f n e g o t i a t e d knowledge p r o d u c t i o n and use have been l a b e l l e d E x p e r i m e n t a l , M a n a g e r i a l , C o l l a b o r a t i v e , and T r a n s f o r m a t i v e . TABLE OF CONTENTS Li s t o f F i g u r e s v i i Acknowledgments : v i i i P r e f a c e i x CHAPTER 1 SOCIAL REFORM, POLICY RESEARCH AND THE PROGRAM EVALUATION PROCESS 1 1.1 S o c i a l Reform and P o l i c y R e search 1 The P o l i c y S c i e n c e P e r s p e c t i v e 3 The P o l i c y A n a l y s i s P e r s p e c t i v e 6 Imp1icat ions 11 1.2 Program E v a l u a t i o n as P o l i c y R e search 12 Issues in the Theory and P r a c t i c e o f Program E v a l u a t i o n 13 The Program E v a l u a t i o n E n t e r p r i s e . . . . 16 1.3 The Research T o p i c : Approach, S o u r c e s , Methods and Models . 20 The Research Problem 20 T h e o r e t i c a l Sources 23 Methodology 26 Models o f N e g o t i a t e d Knowledge P r o d u c t i o n and Use . . . 31 CHAPTER 2 CONTEMPORARY MODELS OF PROGRAM EVALUATION: PARADIGMS AND LIMITATIONS 36 2.1 I n t r o d u c t i o n 36 2.2 The A g r i c u l t u r a l - B o t a n y Paradigm and C o n v e n t i o n a l E v a l u a t i o n s 37 2.3 The S o c i a l A n t h r o p o l o g y Paradigm and I l l u m i n a t i v e E v a l u a t i o n s 1^ 2.k L i m i t a t i o n s and C r i t i c i s m s from W i t h i n Each Paradigm . . . 7^ The I n t e r p r e t i v e C r i t i q u e o f the P o s i t i v i s t Model . . . 8^ The P o s i t i v i s t C r i t i q u e o f the I n t e r p r e t i v e Model . . . 52 2.5 I m p l i c a t i o n s 55 CHAPTER 3 THE NEGOTIATED ORDER PARADIGM AND AN ANALYSIS OF PROGRAM EVALUATION CONTEXTS 61 3.1. I n t r o d u c t i o n 61 3.2 Overview o f S t r a u s s on N e g o t i a t i o n 63 N e g o t i a t i o n and S o c i a l Order . 63 Background t o the N e g o t i a t e d Order Paradigm 65 Key Concepts i n the Paradigm 66 S t r a u s s ' Use o f the Paradigm i n Case Study A n a l y s i s . . 69 The S t r a u s s Paradigm: Summary and I m p l i c a t i o n s 71 3.3 The Proposed A n a l y t i c a l Framework f o r P o l i c y Research S t u d i e s 75 Background t o the Framework 75 A n a l y z i n g S t r u c t u r a l C o n t e x t 79 A n a l y z i n g B a s i c Areas o f N e g o t i a t i o n . . . . 83 I m p l i c a t i o n s f o r Case Study A n a l y s i s and Model Development 99 V CHAPTER 4 CHANGING STRUCTURAL CONTEXT AND PATTERNS OF COOPERATION AND ANTAGONISM 101 4.1 The S t r u c t u r a l C o n t e x t : The P o l i c y S h i f t from "Slum C l e a r a n c e " t o "Neighbourhood and Housing R e h a b i l i t a t i o n " . 102 P o l i c y I s s u e s : P r o d u c t i o n v s . D i s t r i b u t i o n o f Housing . 103 The E v o l u t i o n o f R e h a b i l i t a t i o n P o l i c y 106 k.2 Case: S t a g e II E v a l u a t i o n o f the S t r a t h c o n a R e h a b i l i t a t i o n P r o j e c t 108 Background t o t h e Stage II E v a l u a t i o n I l l Phase I: P r e - C o n t r a c t N e g o t i a t i o n s 112 Phase 2: The E v a l u a t i o n R e s e a r c h 115 Phase 3: The E v a l u a t i o n R e p o r t 120 k.3 The SRP Stage II E v a l u a t i o n : A n a l y s i s o f the R e l a t i o n s Between S t r u c t u r a l C o n t e x t , N e g o t i a t i o n C o n t e x t , and Pol i c y Consequences 129 B a s i c A r e a s o f N e g o t i a t i o n 131 Summary and I m p l i c a t i o n s f o r Knowledge P r o d u c t i o n and Use 138 k.k Case: E v a l u a t i o n o f t h e Neighbourhood Improvement Program D e l i v e r y System i n B r i t i s h C olumbia 142 The S t r u c t u r a l C o n t e x t o f the NIP/RRAP Program . . . . . 142 NIP and RRAP: C i t i z e n E v a l u a t i o n 1 ^ The B r i t i s h C o l u m b i a E v a l u a t i o n : A B r i e f C h r o n o l o g y . . 146 The E v a l u a t i o n P r o c e s s : M e t h o d o l o g i c a l C o n f l i c t . . . . 150 An Ethnography o f C o n f l i c t : The D e t a i l s o f I n t e r a c t i o n . 158 The F i e l d w o r k P r o c e s s : An E x e r c i s e i n Re s e a r c h Team C o n f l i c t 159 O r g a n i z a t i o n o f t h e F i n a l Report and The A c t i o n P l a n M e e t i n g 1& 2 4.5 The NIP/RRAP E v a l u a t i o n : A Study i n A m b i g u i t y and Ant a g o n i s m 163 B a s i c A r e a s o f N e g o t i a t i o n 164 Summary and I m p l i c a t i o n s f o r Knowledge P r o d u c t i o n and Use 169 CHAPTER 5 PROGRAM SCALE, LEVELS OF INVOLVEMENT AND THE SUBSEQUENT USES OF EVALUATION STUDIES IN ONGOING PROGRAMS 176 5.1 Case: An Assessment o f t h e Vancouver J e w i s h F a m i l y S e r v i c e Agency 177 The S t r u c t u r a l C o n t e x t : P u b l i c / P r i v a t e S o c i a l S e r v i c e , Community O r g a n i z a t i o n , and P a t t e r n s o f C h a r i t a b l e C o n t r i b u t i o n s '78 P r e l i m i n a r y N e g o t i a t i o n s : The O u t s i d e E v a l u a t o r s as Determined " O u t s i d e r s " 182 The R e s e a r c h P r o c e s s : M e t h o d o l o g y , Short-Term Consequences and Problems o f D i s t r i b u t i o n 1^3 5.2 The JFSA A s s e s s m e n t : L e v e l s o f Involvement, E v a l u a t i o n Use, and the Legacy o f an Unchanged S t r u c t u r a l C o n t e x t 1^7 B a s i c A r e a s o f N e g o t i a t i o n 188 Ongoing Problems and the Use/Misuse o f the Report . . . 192 Summary and I m p l i c a t i o n s f o r Knowledge P r o d u c t i o n and Use . . . . '95 v i 5.3 Case: L e g a l S e r v i c e s S o c i e t y S c h o o l s Program E v a l u a t i o n R e p o r t 1980 198 The S t r u c t u r a l C o n t e x t : S t a b i l i t y and S h i f t s 198 The S c h o o l s Program E v a l u a t i o n : A B r i e f C h r o n o l o g y . . . 203 D e f i n i n g the P r o c e s s : The D e c i s i o n , Type o f E v a l u a t i o n and C o n s u l t a n t ' s R o l e 203 The I n t e r n a l E v a l u a t i o n : E v a l u a t i o n Needs and Methodology 207 I m p l i c a t i o n s o f the I n t e r n a l E v a l u a t i o n and I t s Methodology 210 Subsequent A s p e c t s o f Program Change 212 A s s e m b l i n g t h e E v a l u a t i o n R e p o r t 214 5.4 The LSS S c h o o l s Program E v a l u a t i o n : The Locus o f E v a l u a t i o n Ownership and the P r o c e s s o f O r g a n i z a t i o n a l Change . . . . 217 B a s i c A r e a s o f N e g o t i a t i o n 217 The Impact o f S t r u c t u r a l C o n t e x t on Long-Term Change . . 219 O v e r v i e w and I m p l i c a t i o n s f o r Knowledge P r o d u c t i o n and Use 221 CHAPTER 6 MODELS OF NEGOTIATION KNOWLEDGE PRODUCTION AND USE: SOURCES AND DEVELOPMENT . . . , 226 6.1 The T h e o r e t i c a l A p p r o a c h and R e l e v a n t Debate 227 The A n a l y t i c a l : S t r a t e g y 227 The C h o i c e o f A n a l y t i c a l Framework and R e l e v a n t Debate . 229 6.2 O v e r v i e w o f the Case S t u d i e s : I m p l i c a t i o n s f o r Knowledge P r o d u c t i on and Use 234 S t u d i e s i n Hou s i n g P o l i c y . . 234 The SRP S t a g e II E v a l u a t i o n : C o n c l u s i o n s 235 The NIP/RRAP E v a l u a t i o n : C o n c l u s i o n s 236 I m p l i c a t i o n s f o r Knowledge P r o d u c t i o n and Use 238 S t u d i e s i n P r i v a t e S o c i a l S e r v i c e s and Le g a l E d u c a t i o n . 240 The JFSA E v a l u a t i o n : C o n c l u s i o n s and I m p l i c a t i o n s . . . 240 LSS S c h o o l s P r o j e c t E v a l u a t i o n : C o n c l u s i o n s and Impl i c a t i o n s 243 Summary o f Four E v a l u a t i o n Case S t u d i e s 246 6.3 P r e l i m i n a r y F o r m u l a t i o n s : Models o f N e g o t i a t e d Knowledge . 250 Background t o the Models 250 The E x p e r i m e n t a l Model 255 The M a n a g e r i a l Model 257 The C o l l a b o r a t i v e Model 259 The T r a n s f o r m a t i v e Model 260 6.4 P o s s i b l e D i r e c t i o n s f o r F u t u r e R e s e a r c h 263 BIBLIOGRAPHY . . . . . 268 APPENDICES 287 v i i LIST OF FIGURES FIGURE 1: Card w i t h L i s t of S i x E l i g i b i l i t y C r i t e r i a 155 FIGURE 2: O rgan i za t i on Chart of LSC 1978/1979 201 FIGURE 3: O rgan i za t i on Chart of LSS 1980 202 FIGURE k: Summary of Four Eva lua t i on Case Stud ies 2^7 v i i i ACKNOWLEDGMENTS I would l i k e to thank the C h a i r p e r s o n o f my academic committee, Dr. E l v i W h i t t a k e r , f o r her a s s i s t a n c e as I s t r u g g l e d through s e v e r a l a n a l y t i c a l approaches b e f o r e c o n s t r u c t i n g the p a r t i c u l a r documentary r e a l i t y p r e s e n t e d here. I have a l s o b e n e f i t t e d from Dr. Robert Ratner's c r i t i c a l d i s c u s s i o n s o f the i n t e r n a l l o g i c o f the work and c l a r i f i c a t i o n o f the c o n n e c t i o n s between each s t a g e in the a n a l y s i s . Dr. Roy Turner and Dr. N e i l Guppy, from v e r y d i f f e r e n t p e r s p e c t i v e s w i t h i n s o c i o l o g y , l e d me to look more c l o s e l y a t the o b j e c t i o n s to the n e g o t i a t e d o r d e r paradigm, and marshal my arguments more s y s t e m a t i c a l l y . I want to c o n g r a t u l a t e my husband Seymour and daughter Rachel , who have endured and s u r v i v e d w i t h few c o m p l a i n t s and e n d l e s s s u p p o r t . To Diana Raynor and The r e s a MacFarlane, my t y p i s t s , I owe much g r a t i t u d e f o r t h e i r w i l l i n g n e s s to cope w i t h the i n e v i t a b l e changes and d e t a i l s o f each d r a f t . i x PREFACE There a r e two c h a r a c t e r i s t i c s o f t h i s r e s e a r c h which c o n t r i b u t e to i t s uniqueness as a study o f program e v a l u a t i o n as a form o f p o l i c y r e s e a r c h . F i r s t , t h i s work uses the methods and t e c h n i q u e s o f s o c i o l o g y and s o c i a l a n t h r o p o l o g y to examine the a c t i v i t i e s o f s o c i a l s c i e n t i s t s doing a type o f f i e l d r e s e a r c h . Thus, t h i s d e s c r i p t i o n and a n a l y s i s emphasizes the d e t a i l s o f program e v a l u a t i o n work from the p e r s p e c t i v e o f the e v a l u a t o r s as a c t o r s in v a r i o u s s o c i a l s e t t i n g s r a t h e r than e x p e r t s who do t h e i r work o u t s i d e the f l o w o f day-to-day s o c i a l a c t i o n . In a d d i t i o n , the case s t u d i e s d i s c u s s e d r e p r e s e n t a secondary a n a l y s i s by me o f program e v a l u a t i o n s t u d i e s in which I was a major p a r t i c i p a n t . That i s , I am examining from a f r e s h v i e w p o i n t f i e l d d a t a and r e s e a r c h p r o d u c t s t h a t were generated by me, in c o o p e r a t i o n w i t h o t h e r r e s e a r c h e r s , f o r use i n f o u r d i f f e r e n t p o l i c y s e t t i n g s . It i s a b i t d i f f i c u l t to s p e c i f y a l l the methods and s o u r c e s employed in p r o d u c i n g t h i s d i s s e r t a t i o n . It i s e s s e n t i a l ! t h a t readers understand t h a t d u r i n g more than t h r e e - f o u r t h s o f my work. 1i f e I have been employed in non-academic s e t t i ngs>_ do i ng a p p l i e d and/or a c t i o n r e s e a r c h . These j o b s have i n v o l v e d me in a wide v a r i e t y o f communities ( g e o g r a p h i c , e t h n i c , economic) and were g e n e r a l l y connected w i t h one o r another type of " s o c i a l r e f o r m " program. The a c t u a l prog rams'in which I have been employed ranged q u i t e w i d e l y in scope and c o n t e n t . They i n c l u d e : a s s i s t i n g in an e x p e r i m e n t a l program f o r h i g h I.Q. p r e - a d o l e s c e n t d e l i n q u e n t s i n Chicago; p l a n n i n g f o r new i n t e g r a t e d school d i s t r i c t s i n P r o v i d e n c e , Rhode I s l a n d ; e v a l u a t i n g the f i r s t y e a r ' s r e s u l t s o f O p e r a t i o n HeadStart in M a s s a c h u s e t t s ; working as an urban p l a n n e r and then c o a u t h o r i n g a manual on the use o f s u r v e y r e s e a r c h methods as a t e c h n i q u e f o r improving the p r o c e s s o f comprehensive urban p l a n n i n g ( f o r the X Department o f Housing and Urban Development i n the U n i t e d S t a t e s ) ; seven y e a r s c o n t i n u e d involvement (as Vancouver F i e l d S t a f f f o r the Company o f Young Canadians) i n the o r g a n i z a t i o n and development o f a s e r i e s o f Vancouver s e l f -h e l p groups, e.g., Women's H e a l t h C o l l e c t i v e , Mental P a t i e n t s ' A s s o c i a t i o n , CCEC C r e d i t Union and s e v e r a l g e o g r a p h i c a l l y based community groups. In the most r e c e n t phase o f my work l i f e I have been s e l f - e m p l o y e d as a p l a n n i n g c o n s u l t a n t and the coaut h o r o f s e v e r a l program e v a l u a t i o n s , i n c l u d i n g the f o u r which a r e used as case s t u d i e s in t h i s d i s s e r t a t i o n . One o t h e r unusual c o n s u l t i n g c o n t r a c t i n v o l v e d me in the p r o d u c t i o n o f a market f e a s i b i l i t y s tudy f o r the Vancouver N a t i v e V i l l a g e Development S o c i e t y , a B r i t i s h Columbia N a t i v e Indian group i n t e r e s t e d i n the development o f an e d u c a t i o n a l and e n t e r t a i n m e n t complex in the Vancouver harbour a r e a . T h i s employment has r e q u i r e d t h a t I work in a v a r i e t y o f o c c u p a t i o n a l r o l e s : (1) as an " o r g a n i z e r " o f demands f o r s o c i a l r eform, (2) as a " d e s i g n e r " of s o c i a l r e f o r m programs f o r s p e c i f i c s o c i a l / p o l i t i c a l c o n s t i t u e n c i e s , (3) as an "implementer" o f o t h e r people's reform programs, and (4) as the "p r o d u c e r o f e v a l u a t i o n s " o f my own and o t h e r p e o p l e ' s programs. In a l l f o u r t ypes o f r o l e s I was r e q u i r e d t o do e v a l u a t i v e work r e g a r d i n g the programs themselves and the perso n n e l employed by them. It i s important to p o i n t out t h a t the approach I take when working " i n the f i e l d " has some r e l a t i o n s h i p to the methods I have used in i n t e r p r e t i n g my f i e l d e x p e r i e n c e s (and many o t h e r t y p e s o f r e s o u r c e s ) f o r the a n a l y s i s p r e s e n t e d h e r e . Without becoming too p h i l o s o p h i c a l , l e t me say t h a t in my work as r e s e a r c h e r / o r g a n i z e r / e v a l u a t o r I f o l l o w a .kind o f " n a t u r a l i s t i c b e h a v i o u r i s m " . In t h i s v e r s i o n o f the r e s e a r c h a c t one i s i n v o l v e d i n " t h e s t u d i e d commitment t o a c t i v e l y e n t e r the worlds o f n a t i v e p e o p l e and to render those worlds u n d e r s t a n d a b l e from the s t a n d p o i n t o f a t h e o r y t h a t i s grounded in the b e h a v i o u r s , 1anguage, d e f i n ? t i o n s , a t t i t u des, and f e e l i ngs o f those s t u d i e d " (Denzin 1971)- I v e r y much a p p r e c i a t e Denzin's x i e x p a n s i o n o f the term " n a t i v e p e r s o n . " He s a y s , i n a f o o t n o t e , t h a t " n a t i v e p e r s o n " i s " i n t r o d u c e d here as a g e n e r i c term which c o v e r s a l l people s t u d i e d by the s o c i o l o g i s t . They may be e x p e r i m e n t a l s u b j e c t s , i n t e r v i e w e e s , f r i e n d s , c o l l e a g u e s , and n e i g h b o u r s . " In t h i s s ense, then, much o f my work i s f i r m l y based in the c o n c e p t s o f everyday l i f e , day-to-day a c t i v i t i e s , e t c . , as the s o u r c e o f t h e o r e t i c a l i n s i g h t s and the i n i t i a l f o r m u l a t i o n o f e v a l u a t i v e c o n s t r u c t s . But, a l o n g w i t h Denzin, I have always been prep a r e d t o use (and o f t e n do employ) the f u l l range o f q u a n t i t a t i v e t e c h n i q u e s in doing e v a l u a t i v e r e s e a r c h . The p a r t i c u l a r c o m b i n a t i o n o f methods depends on t h e i r a p p r o p r i a t e -ness to the r e q u i r e m e n t s o f a s p e c i f i c r e s e a r c h problem. C e r t a i n l y , t h e r e a r e s t r o n g elements o f t h a t type o f n a t u r a l i s t i c b e h a v i o u r i s m i n v o l v e d i n my e f f o r t s t o produce t h i s d i s s e r t a t i o n . F i r s t o f a l l , the r e s e a r c h t o p i c and methods r e p o r t e d here d i d not s t a r t w i t h my r e c e i v i n g e x t e n s i v e t r a i n i n g i n the p r e v a i l i n g t h e o r e t i c a l n o t i o n s about how e v a l u a t i o n r e s e a r c h i s / s h o u l d be a c c o m p l i s h e d . It was o n l y a f t e r t h r e e y e a r s in the f i e l d t h a t I began to read t h e abundant l i t e r a t u r e on program e v a l u a t i o n (as a r t and s c i e n c e ) . Up t o then I had r e l i e d on s e v e r a l y e a r s o f g raduate t r a i n i n g in s o c i o l o g i c a l r e s e a r c h and my work in urban p l a n n i n g and community o r g a n i z a t i o n as the b a s i s f o r d o i n g program e v a l u a t i o n s . During the i n i t i a l y e a r s t h a t I became a p a r t n e r in Ptarmigan P l a n n i n g A s s o c i a t e s , s e l f - e m p l o y e d as a program e v a l u a t o r , I was s e n s i t i v e t o the r i s k s and c o m p l e x i t i e s o f the work. As a r e s u l t o f p r e v i o u s work and t r a i n i n g I was c o n s t a n t l y aware o f the v a r i a t i o n s i n the a c t o r s , i s s u e s and s e t t i n g s around which e v a l u a t i o n r e s e a r c h was b e i n g done. Each program p r e s e n t e d a new e x p e r i e n c e i n working out d i f f e r e n t arrangements r e g a r d i n g the p l a n n i n g , implementation and w r i t e up o f the r e s e a r c h and recommendations. C e r t a i n fundamental problems i n a c c o m p l i s h i n g the r e q u i r e d e v a l u a t i v e work were d e f i n e d in a l t e r n a t i v e ways in each r e s e a r c h s e t t i n g . These i n c l u d e d problems r e g a r d i n g x i i s t r u c t u r a l arrangement i n w h i c h t h e work would be done; t h e c o n c e p t u a l p e r s p e c -t i v e from w h i c h the e v a l u a t i o n p r o c e s s would be d e v e l o p e d ; the m e t h o d o l o g i c a l s t r a t e g i e s w h i c h would be employed; the format f o r document p r o d u c t i o n , p u b l i -c a t i o n and d i s t r i b u t i o n ; and the e x t e n t t o w h i c h the e v a l u a t o r would be engaged i n p o l i c y f o r m u l a t i o n and i m p l e m e n t a t i o n . A l l t h i s e n t a i l e d a p r o c e s s o f s t a t i n g , d i s c u s s i n g and t r y i n g t o r e s o l v e a wide range o f d i f f e r e n c e s among the a c t i v e p a r t i c i p a n t s (managers, s t a f f , p o l i t i c i a n s , l e a d e r s o f community g r o u p s , government r e p r e s e n t a t i v e s , c oncerned c l i e n t s and t h e e v a l u a t o r o r e v a l u a t i o n team) i n the e v a l u a t i o n a c t i v i t i e s . D u r i n g the c o u r s e o f w o r k i n g on f o u r program e v a l u a t i o n s I came t o see t h i s p r o c e s s as a v a r i a n t o f what i s t r a d i t i o n a l l y d e f i n e d as n e g o t i a t i o n . I t s h o u l d be u n d e r s t o o d t h a t , i n c o n t r a s t t o my emphasis on " n e g o t i a t e d " a r r a n g e m e n t s , s e v e r a l o f the more p o p u l a r t e x t s and handbooks on program e v a l u a t i o n ( R o s s i and W i l l i a m s 1972; Boruch and R i e c k e n 1975; G u t t e n t a g and S t r u e n i n g 1975) tended t o g i v e a f a i r l y r i g i d , e x p e r i m e n t a l l y based p r e s c r i p t i o n r e g a r d i n g "good" e v a l u a t i o n r e s e a r c h . T h e i r models l e f t o u t many o f the e s s e n t i a l a c t o r s i n t h e e v a l u a t i o n p r o c e s s . The v a r i e d p a t t e r n s o f i n t e r a c t i o n among s p o n s o r s , r e s e a r c h e r s , managers, s t a f f , p a r t i c i p a n t s ( c l i e n t s , p a t i e n t s ) , p o l i t i c i a n s and p o l i c y m a k e r s were o f t e n subsumed w i t h i n a r e s i d u a l c a t e g o r y c a l l e d " r e s e a r c h c o m p l i c a t i o n s / b a r r i e r s . " These c o m p l e x i t i e s o f economic/ s o c i a l / p o l i t i c a l i n t e r a c t i o n i n the f i e l d were o f t e n c o n s t r u e d as o b s t a c l e s to a c h i e v i n g one's main g o a l : a r i g o r o u s , o b j e c t i v e e v a l u a t i o n o f a s p e c i f i c a l l y d e s c r i b a b l e program i n s o c i a l r e f o r m . A f t e r b r i e f d i s c u s s i o n s o f s t r a t e g i e s d e s i g n e d t o overcome such c o m p l i c a t i o n s , many o f the t e x t s gave t h e i r main a t t e n t i o n t o a d e t a i l e d e x p l i c a t i o n o f s c i e n t i f i c s t a n d a r d s w h i c h must be met i n o r d e r t o produce an o b j e c t i v e e v a l u a t i o n . Such books o f t e n c o n c l u d e d by o u t l i n i n g what one u n f o r t u n a t e l y had t o s e t t l e f o r i n p l a c e o f r i g o r o u s , o b j e c t i v e e v a l u a t i o n d e s i g n s . These c o n t r a s t i n g c o n c e p t u a l i z a t i o n s o f the x i i i e x p e r i e n c e and meaning o f the program e v a l u a t i o n p r o c e s s . ( i . e . , my own and the v e r s i o n in the c o n v e n t i o n a l t e x t ) were the s o u r c e o f my concern w i t h documenting a c t u a l v a r i a t i o n s in d o i n g t h i s p a r t i c u l a r form o f s o c i a l r e s e a r c h . 1 L o o k i n g back, we can see t h a t time and a g a i n p o l i t i c a l p a s s i o n has been a d r i v i n g s p i r i t b e h i nd t h e c a l l f o r r a t i o n a l a n a l y s i s . On t h e one hand, r a t i o n a l i s t s a d v o c a t e d b r i n g i n g t o s o c i a l a f f a i r s t h e n e u t r a l i t y and detachment t h a t have s e r v e d n a t u r a l s c i e n c e and t e c h n o l o g y so w e l l . And, on t h e o t h e r hand, the v e r y c a l l f o r v a l u e - f r e e i n q u i r y was p o l i t i c a l l y m o t i v a t e d , gen-e r a l l y by d i s c o n t e n t . The s a v a n t s o f the E n l i g h t e n m e n t were f i g h t i n g t r a d i t i o n s e n t r e n c h e d i n r o y a l and e c c l e s i a s t i c a l power. The P r o g r e s s i v e s wanted s o c i a l i n i t i a t i v e s t o w h i c h t h e l e g i s l a - . t o r s and mayors o f t h e time were i n d i f f e r e n t . The p o l i c y makers of the 196O1s sought t o use a n a l y t i c a l methods t o break up b u r e a u c r a t i c hardpan so t h a t new programs c o u l d t a k e r o o t (Cronbach 1980:35). CHAPTER SOCIAL REFORM, POLICY RESEARCH AND THE PROGRAM EVALUATION PROCESS 1.1 S o c i a l Reform and P o l i c y R esearch S o c i a l r e f o r m programs have a long and r e s p e c t a b l e h i s t o r y i n Western l i b e r a l d e m o c r a t i c t r a d i t i o n . Ever s i n c e t h e b e g i n n i n g o f t h e n i n e t e e n t h c e n t u r y t h e r e have been s i g n i f i c a n t programs t o s t u d y and improve s o c i a l con-d i t i o n s ( e . g . , s a n i t a t i o n , h o u s i n g , h e a l t h c a r e ) p a r t i c u l a r l y among the urban poor. Many i n d i v i d u a l s and o r g a n i z a t i o n s were c o n c e r n e d w i t h the s y s t e m a t i c p l a n n i n g and i m p l e m e n t a t i o n o f the "reform 1* 1 o f t h e problems o f urban s o c i e t y . One c l a s s i c example was t h e work o f Sydney and B e a t r i c e Webb and o t h e r mem-bers o f t h e F a b i a n S o c i e t y i n E n g l a n d . T h e i r s o c i a l and economic r e s e a r c h was d e s i g n e d t o p r o v i d e a s y s t e m a t i c knowledge base f o r improvements t o t h e hou s i n g and w e l f a r e programs i n B r i t a i n . In r e c e n t decades, i n both Europe and N o r t h A m e r i c a , t h e s i z e and scope o f the programs have i n c r e a s e d q u i t e d r a m a t i c a l l y . At t h e same t i m e , the i n i t i a t i v e f o r s o c i a l r e f o r m has been c o - o p t e d i n t o t h e p o 1 i t i c a 1 / e c o n o m i c 2 i n s t i t u t i o n s o f t h e S t a t e . In t h e U.S. new p r o j e c t s f o r s o c i a l r e f o r m o r i g i n a l l y funded and promoted by t h e Ford F o u n d a t i o n and the P r e s i d e n t ' s Committee on J u v e n i l e D e l i n q u e n c y ( d u r i n g 1960-1964), were u l t i m a t e l y i n c o r -p o r a t e d i n t o t h e American "War on P o v e r t y " ( M a r r i s and Rein 1972). S i m i l a r i n n o v a t i v e programs a f f e c t i n g h o u s i n g , n e i g h b o u r h o o d s , employment and h e a l t h c a r e were i n i t i a t e d i n Canada w i t h h i g h hopes f o r t h e s o l u t i o n o f b a s i c urban problems (Dennis and F i s h 1972; Draper 1971). When t h e s e reform p r o -grams d i d not f u l f i l l t h e i r p u b l i c i z e d e x p e c t a t i o n s , a l l t y p e s o f r e f o r m programs were s u b j e c t t o much c l o s e r s c r u t i n y than i n e a r l i e r decades. Decreases i n economic growth r a t e s , r a p i d i n f l a t i o n , and l i m i t e d s o u r c e s f o r i n c r e a s i n g government revenues encouraged the p o l i c y makers' demands f o r s y s t e m a t i c e v a l u a t i o n o f r e f o r m programs funded by a l l l e v e l s o f government. As a consequence, more " s c i e n t i f i c " p l a n n i n g , m o n i t o r i n g and e v a l u a t i o n o f t h e s e programs has become i n s t i t u t i o n a l i z e d as p a r t o f the o v e r a l l p o l i c y i n t e r v e n t i o n and program f u n d i n g p r o c e s s . The i n i t i a t i v e f o r c o - o p t i n g s o c i a l reform programs i n t o t h e s t r u c t u r e o f t h e S t a t e was p a r t o f an o v e r a l l p r o c e s s o f government e x p a n s i o n and r e q u i r e d p a r a l l e l development o f an a r e a o f government a c t i v i t y c a l l e d p u b l i c p o l i c y m a k i n g . The n o t i o n o f " p u b l i c p o l i c y " has been i n use f o r t he l a s t two c e n t u r i e s a t l e a s t (Cronbach 1980:25) , but i t seems t o have been r e i f i e d as an e n t i t y , i n s t i t u t i o n o r o b j e c t o f s c i e n t i f i c a n a l y s i s i n t h e l a s t twenty-f i v e y e a r s . S t a r t i n g from an i n i t i a l base i n s o c i o l o g y and p o l i t i c a l s c i e n c e t h e r e have been two major p e r s p e c t i v e s employed i n t h e e f f o r t t o s c r u t i n i z e t h e p o t e n t i a l c o n t r i b u t i o n o f s o c i a l s c i e n c e t h e o r y and knowledge t o t h e form u l a t i o n o f p u b l i c p o l i c y . The two p e r s p e c t i v e s may be l a b e l l e d " p o l i c y s c i e n c e " and " p o l i c y a n a l y s i s . " 3 The P o l i c y S c i e n c e P e r s p e c t i v e The p o l i c y s c i e n c e p e r s p e c t i v e d i r e c t s s o c i a l s c i e n c e r e s e a r c h toward r a t i o n a l i z i n g the p r o c e s s o f making p u b l i c p o l i c y ( L a z a r s f e l d 1975). E t z i o n i has r e f e r r e d to p o l i c y s c i e n c e as an "in-between d i s c i p l i n e " (1971:11) , d e s i g n e d to b r i d g e the gap between a b s t r a c t theory and p o l i c y recommendations. Thus, p o l i c y s c i e n c e f u r n i s h e s " p r a c t i c a l knowledge t h a t has no a n a l y t i c foun-d a t i o n but i s u s e f u l f o r moderating o r complementing a n a l y t i c knowledge" ( E t z i o n i 1978:4) . The emphasis in p o l i c y s c i e n c e i s to " p r e p a r e a l t e r n a t i v e r a t i o n a l e s and to pry p o l i c y m a k e r s l o o s e from t h e i r a n t i q u a t e d assumptions" (1978:2) . From these comments i t becomes e v i d e n t t h a t one o f the main assump-t i o n s o f p o l i c y s c i e n c e i s a s e p a r a t i o n o f the v a l u e component o f s o c i a l r e s e a r c h from the f a c t f i n d i n g a c t i v i t i e s o f the s o c i a l r e s e a r c h e r . From the p o l i c y s c i e n c e p e r s p e c t i v e t h i s o f t e n takes the form o f a sharp d i s t i n c t i o n between ends and means, between the p r o c e s s o f s e t t i n g p o l i c y g o a l s and the r e s e a r c h a c t i v i t i e s e n t a i l e d in the p l a n n i n g , implementation and e v a l u a t i o n o f those p o l i c i e s as they are...interpreted in s p e c i f i c p u b l i c l y funded programs. P o l i c y s c i e n c e i s c r i t i c a l o f the l i m i t e d r e s e a r c h c a p a b i l i t i e s o f most p o l i c y -makers and t h e i r s t a f f s , a r g u i n g t h a t the usual t a s k f o r c e s o r commissions " f r e q u e n t l y l a c k the f a c i l i t i e s , e x p e r i e n c e , and c o n t i n u i t y - - i n s h o r t , the s t r u c t u r a l p r e r e q u i s i t e s f o r c o n d u c t i n g h i g h q u a l i t y r e s e a r c h " (1978:3) . What seems n e c e s s a r y a r e more f a c i l i t i e s on the domestic f r o n t ( l i k e RAND C o r p o r a t i o n working f o r the armed s e r v i c e s ) in which p o l i c y r e s e a r c h i s i n s t i t u t i o n a l i z e d . As a consequence o f these c o n c e r n s , major a t t e n t i o n has been g i v e n to p o l i c y r e s e a r c h , which i s b r o a d l y d e f i n e d as " d e c i s i o n o r i e n t e d r e s e a r c h " (Coleman 1972:2) . :More p a r t i c u l a r l y p o l i c y r e s e a r c h i s "concerned w i t h mapping a l t e r n a t i v e approaches and w i t h s p e c i f y i n g p o t e n t i a l d i f f e r e n c e s i n the i n t e n t i o n , e f f e c t , and c o s t o f v a r i o u s programs" ( E t z i o n i 1978:1) . The work o f p o l i c y r e s e a r c h i s seen as a d i s t i n c t i v e e n t e r p r i s e w i t h purposes t h a t 4 differ from both pure and applied social science research. In principle policy research is defined as being more c r i t i c a l than applied research (Etzioni 1971:9) , while being less formalized, less analytic than pure research (Etzioni 1978:4) . Following the approach taken by Coleman and Etz ioni , an effort has been made to outline specific characteristics of policy research. Finsterbusch and Hamilton (1978:88) offer the following: 1. the research problem originates in the policy arena; 2. the audience for the research results is a group of po l i t i ca l actors and interested parties, usually with conflicting interests; 3. the research must examine the consequences of a policy for a l l interested parties; 4. the research is constrained by the timing of decisions; 5. the policy researcher must invest a significant amount of time and energy in communicating with policymakers and interested parties. Examples of policy studies include social impact assessments, studies for environmental impact statements, reports to various policy oriented commissions and program evaluations. In this context policy research is seen as offering a set of rational pro-cedures and results which can be used in policy formulation as well as the planning, implementation and evaluation of public programs. "The deviation from functional rationality in social science research begins with the fai lure to clearly define the overall research objective and the system of instrumental subgoals" (Finsterbusch and Hamilton 1978:89). Although goal setting is seen as a continuous, iterative process, there is great stress on goal definition and the lega l /po l i t i ca l context within which the research results wil l be received and processed. However, not a l l those concerned with policy science, and policy research in particular, take such a strong posi t iv ist l ine. In their analysis of two studies of Community Mental Health Centers, Siegel and Doty (1978:144) suggest 5 t h a t p o l i c y r e s e a r c h s t u d i e s can v a r y c o n s i d e r a b l y i n t h e i r emphasis on g o a l s and purposes. T h e i r two case s t u d i e s d i f f e r e d s i g n i f i c a n t l y , both i n t h e i r r e s e a r c h o r i e n t a t i o n and s t r u c t u r a l r e l a t i o n s h i p to the program being s t u d i e d . The "management ana 1ysis", conducted i n t e r n a l l y by a knowledge u n i t which was an i n t e g r a l p a r t o f a p o l i c y m a k i n g body, took purpose and implementation s t r a -tegy l a r g e l y as a " g i v e n " . The "advocacy r e s e a r c h " , produced by independent o u t s i d e r s c l a i m i n g to r e p r e s e n t the i n t e r e s t s o f the community not the program, q u e s t i o n e d both program purposes and implementation s t r a t e g y (1978:140). In a n a l y z i n g these two p i e c e s o f p o l i c y r e s e a r c h S i e g e l and Doty b r i n g the p o l i c y s c i e n c e approach c l o s e r to t h a t o f p o l i c y a n a l y s i s . That i s , t h e i r a n a l y s i s f o c u s e s on the b a l a n c e o f " i n t r i n s i c " (paradigm m o d i f i c a t i o n ) to " e x t r i n s i c " (paradigm s h i f t ) c r i t i c i s m o f the p o l i c y and p r a c t i c e s o f the Community Mental H e a l t h C e n t e r s p r e s e n t e d in each o f the two case s t u d i e s . However, the a u t h o r s c o n t i n u e to be i n t e r e s t e d i n the g o a l a n a l y s i s developed i n each s t u d y , c r i t i -c i z i n g the p r o c e d u r e s and l i m i t a t i o n s o f the m e t h o d o l o g i e s employed i n both. E t z i o n i s t r e s s e s the c r i t i c a l o r i e n t a t i o n of p o l i c y r e s e a r c h , but sees no i n t r i n s i c d i f f i c u l t i e s in u s i n g the time, t r a i n i n g , books and computers o f s o c i a l r e s e a r c h t o s a t i s f y the i n f o r m a t i o n needs o f a l l i n t e r e s t e d p a r t i e s engaged in the p o l i c y p r o c e s s . In commenting on the most n o t o r i o u s phrase w r i t t e n i n t o the 1964 a n t i - p o v e r t y l e g i s l a t i o n , "maximum f e a s i b l e p a r t i c i p a t i o n of the poor", he i s e x t r e m e l y c r i t i c a l o f the l a c k o f p r e l i m i n a r y i n v e s t i g a t i o n o f what such p a r t i c i p a t i o n e n t a i l e d , how i t was t o be e f f e c t e d , and what the consequences were l i k e l y to be. Thus, he urges t h a t : If w e l l planned p o l i c y r e s e a r c h had been done b e f o r e the e n a b l i n g l e g i s l a t i o n was passed, the best methods o f s t u d y i n g the problems and a l t e r n a t i v e s o l u t i o n s would have been d e l i n e a t e d . The r e s e a r -c h e r s , not n e c e s s a r i l y a c c e p t i n g the e x p l i c i t and i m p l i c i t g o a l s of the sponsors o f the l e g i s l a t i o n , would have e x p l o r e d the under-l y i n g assumptions, examined the p o s s i b l e consequences o f d i f f e r e n t a l t e r n a t i v e s , and e v a l u a t e d the e f f e c t s o f s m a l l - s c a l e demonstra-t i o n p r o j e c t s (1978:2) . 6 The approach o f p o l i c y s c i e n c e , as a d i s c i p l i n e t h a t f a l l s between the a n a l y t i c s c i e n c e s and p o l i c y recommendations, "the way med i c i n e comes between p h y s i o l o g y and pharmacy" (1978:4), would produce r e s e a r c h r e s u l t s and " d i a g n o s e s " t h a t c o u l d be e f f e c t i v e l y communicated to p o l i c y m a k e r s and o t h e r i n t e r e s t groups. T h e r e f o r e , "knowledge makers who c a r e s h o u l d make their"" work r e l e v a n t t o the c i t i z e n s a t l a r g e and the p o l i c y makers of the i n s t i t u -t i o n s and movements which respond to t h e i r t r u e a l l i e s " (1978:5)• In t h i s knowledge p r o d u c t i o n framework t h e r e i s a t a c i t assumption t h a t p o l i c y r e s e a r c h i s e s s e n t i a l l y c r i t i c a l but s i m u l t a n e o u s l y " v a l u e - f r e e " , i . e . , t h a t i t can s e r v e a l l p o t e n t i a l users e q u a l l y . The f a c t s e s t a b l i s h e d by p o l i c y r e s e a r c h a r e s e p a r a t e from the v a l u e component embedded i n the goa l s e t t i n g a c t i v i t i e s of the p o l i c y p r o c e s s . The P o l i c y A n a l y s i s P e r s p e c t i v e G e n e r a l l y the " p o l i c y a n a l y s i s " approach to the p o s s i b l e r e l a t i o n s h i p between s o c i a l s c i e n c e and p u b l i c p o l i c y has been concerned w i t h i s s u e s and problems c l o s e r t o the c e n t e r o f p o l i t i c a l and s o c i a l t h e o r y . T h i s p e r s p e c t i v e can be seen in the work o f P e t e r M a r r i s and M a r t i n Rein (197,4) which documented both p o l i c y and program dilemmas which e v o l v e d i n the community a c t i o n and p o v e r t y programs o f the 1960's. In h i s ess a y s on s o c i a l s c i e n c e and p u b l i c p o l i c y , Rein (1976) s u g g e s t s t h a t t h e study o f s o c i a l p o l i c y i s " b a s i c a l l y con-cerned w i t h the range o f human needs and the s o c i a l i n s t i t u t i o n s c r e a t e d to meet them" (1976:20). But he emphasizes t h a t we have no adequate d e f i n i t i o n o f "need", and t h e r e i s c o n f u s i o n about the d i s t i n c t i o n between "need", " p r e f e r e n c e " , and " s o c i a l problem". Furthermore, we f i n d t h a t the i n s t i t u t i o n a l arrangements de s i g n e d t o meet these needs appear to vary s i g n i f i c a n t l y , a c c o r d i n g to the h i s t o r i c a l , c u l t u r a l and p o l i t i c a l s e t t i n g in which they have been de v e l o p e d . Thus, the p o l i c y " t r u t h s " o f one p l a c e o r one decade may be c h a l l e n g e d in 7 a n o t h e r . In Canada, f o r example, p o l i c y m a k e r s i n P a r l i a m e n t and at C e n t r a l Mortgage and Housing C o r p o r a t i o n were committed t o the p r i n c i p l e t h a t the "urban b l i g h t " o f the i n n e r c i t i e s had to be removed through e x t e n s i v e d e m o l i -t i o n of s u b s t a n d a r d housing to meet the need f o r adequate, s a f e and s a n i t a r y housing f o r a l l Canadians. The s o c i a l , c u l t u r a l and economic f a l l o u t from t h i s p o l i c y was d i s p r o p o r t i o n a t e l y heavy f o r low income and e l d e r l y urban r e s i d e n t s . These t a r g e t groups were s u b s e q u e n t l y housed in dense, h i g h - r i s e p r o j e c t s which had t h e i r own n e g a t i v e impacts, i . e . a l i e n a t i o n , i n c r e a s e d l e v e l s o f c r ime, v i o l e n c e and mental i l l n e s s ( F r i e d 1967; Dennis and F i s h 1972; C l a i r m o n t and M a g i l l 1974). U l t i m a t e l y , in the 1970's t h e r e was another p o l i c y s h i f t and programs f o r neighbourhood improvement and housing r e h a b i l i t a t i o n were funded as the a p p r o p r i a t e i n s t i t u t i o n a l arrangement f o r a s s i s t i n g the poor, e l d e r l y and o t h e r s on f i x e d incomes to meet t h e i r b a s i c need f o r s a f e , c l e a n and s e c u r e s h e l t e r . T h i s example i l l u s t r a t e s Rein's view t h a t " i d e a s are embed-ded in p r a c t i c e ; the r e l a t i o n s h i p between means and ends o f s o c i a l p o l i c y i s not s e t t l e d , nor i s i t e v e r l i k e l y t o be f i n a l l y r e s o l v e d " (1976:21). E v o l v i n g s o c i a l o b j e c t i v e s a r e c e n t r a l to the development of s o c i a l p o l i c y ; t h e r e f o r e p o l i c y a n a l y s i s n e c e s s a r i l y f o r c e s us t o c a r e f u l l y examine the m u l t i p l e , ambi-guous and c o n f l i c t i n g o b j e c t i v e s o f any s p e c i f i c p o l i c y . S i m i l a r l y Rein attempts to deal d i r e c t l y w i t h the f a c t - v a l u e dilemma and o f f e r s a view o f p o l i c y a n a l y s i s as n e c e s s a r i l y i n v o l v i n g the i n t e r p r e t a t i o n o f b e l i e f s and r e s e a r c h paradigms. He i s c r i t i c a l o f the p o s i t i v i s t s e p a r a t i o n o f f a c t s , t h e o r i e s and v a l u e s and d e l i n e a t e s h i s argument as to why p o l i c y con-f l i c t cannot be r e s o l v e d through r e l i a n c e on p o l i c y - r e l e v a n t s o c i a l s c i e n c e . S i n c e t h i s argument assumes t h a t f a c t s and v a l u e s a r e always i n t e g r a t e d and cannot be i s o l a t e d from one a n o t h e r (1976:79), he i n s i s t s t h a t we must become aware o f the i d e o l o g i c a l p e r s p e c t i v e s t h a t a r e t y p i c a l l y encountered in the a n a l y s i s of p u b l i c p o l i c y . These " v a l u e s c r e e n s " s e r v e t o " s c r e e n in and out c e r t a i n e v i d e n c e " and p r o v i d e a framework f o r i n t e r p r e t a t i o n (1976:80). Rein 8 s u g g e s t s t h r e e major v a l u e s c r e e n s which can o r i e n t the p o l i c y a n a l y s t to d i f -f e r e n t causes and means o f r e s o l v i n g contemporary problems. These i n c l u d e : the p o l i c y a n a l y s t from the LIBERAL p o l i t i c a l t r a d i t i o n who w i l l f i n d t h a t the i n s t i t u t i o n s and o r g a n i z a t i o n s i n s o c i e t y must be reformed; the CONSERVATIVE who c a l l s a t t e n t i o n t o the f a i l u r e o f people to cope and f i n d s p eople a c c o u n t -a b l e f o r t h e i r own f a i l u r e s and achievements; and the RADICAL who sees fundamen-t a l change i n h i b i t e d by the d i s t r i b u t i o n o f e s t a b l i s h e d power, so t h a t important reform i s p o s s i b l e o n l y through r e v o l u t i o n (1976:84). Not a l l i d e o l o g i e s emphasize one o f these themes s p e c i f i c a l l y and most p o l i c y a n a l y s t s r e c o g n i z e some m e r i t in each view, so t h a t a s y n t h e s i s can be o r g a n i z e d . N e v e r t h e l e s s , Rein a r g u e s , one i s f o r c e d however r e l u c t a n t l y t o make c h o i c e s , s i n c e each o f f e r s a competing framework o f thought: A l l these i d e o l o g i e s d i r e c t our a t t e n t i o n to s p e c i f i c f a c t s , and they a l s o h e l p t o o r g a n i z e them. In t h i s sense a l l r e s e a r c h i s b i a s e d : i t works w i t h i n an i n t e r p r e t i v e framework e n a b l i n g the a n a l y s t to make p o l i c y - r e l e v a n t sense o f the f a c t s t h a t he •• o b s e r v e s (ibid :84 ) . As i n d i c a t e d p r e v i o u s l y , the p o l i c y s c i e n c e approach seeks to r a t i o n a l i z e the p o l i c y p r o c e s s through the u t i l i z a t i o n o f r e l e v a n t s o c i a l s c i e n c e t e c h -n i q u e s , employing concepts t h a t reduce the need f o r t r a n s i t i o n and t r a n s l a t i o n , and knowledge t h a t i s , in general', "not f a r removed from the observed w o r l d and t h a t e x p e r i e n c e d by the p o l i c y m a k e r " ( E t z i o n i 1978:4). I.'n c o n t r a s t , Rein sees major o b s t a c l e s which appear to f r u s t r a t e the p o t e n t i a l c o n t r i b u t i o n o f s o c i a l s c i e n c e t h e o r y and knowledge to p o l i c y f o r m u l a t i o n . There i s , f i r s t o f a l l , what he sees as an i n h e r e n t c o n f l i c t i n the p o l i t i c a l d e c i s i o n - m a k i n g p r o c e s s , i . e . , " d i f f e r e n t p e o p l e want d i f f e r e n t t h i n g s and make use o f the p o l i t i c a l pro-cess to s a t i s f y t h e i r r e s p e c t i v e i n t e r e s t s o r ' d i s i n t e r e s t e d ' a s p i r a t i o n s " (Rein 1976:98). Second, even i f t h e r e i s no immediate c l a s h of i n t e r e s t s , implementation of r e s e a r c h f i n d i n g s may s t i l l be a problem because o f the com-p l e x i t y o f t h e s i t u a t i o n f o r which the p o l i c y i s proposed (1976:101). For 9 example, Canada Manpower and Immigration funds t r a i n i n g programs to b r i n g mar-g i n a l workers (young, p o o r l y educated, e x - a l c o h o l i c s , n a t i v e s , women) i n t o the labour f o r c e . S o c i a l r e s e a r c h e r s (Enoch and L e v i t a n 1981) e v a l u a t e a woodwork t r a i n i n g program as being r e l a t i v e l y e f f e c t i v e i n p r e p a r i n g s e l e c t e d i n d i v i d u a l s f o r employment in k i t c h e n - c a b i n e t - m a k i n g b u s i n e s s e s and f o r e n t r y i n t o p r e -a p p r e n t i c e s h i p programs run by the B.C. C a r p e n t e r s ' Union.. However, the unem-ployment r a t e among f u l l y q u a l i f i e d journeyman c a r p e n t e r s i s c l o s e t o 20%. How r e l e v a n t i s t h a t program e v a l u a t i o n t o q u e s t i o n s o f manpower p o l i c y in a depressed employment market which must be faced by ma r g i n a l and f u l l y q u a l i f i e d workers a l i k e ? F i n a l l y , Rein argues, in a p l u r a l i s t i c system w i t h competing i n t e r p r e t i v e frameworks, s o c i a l s c i e n c e f i n d i n g s a r e seldom so c o n c l u s i v e t h a t they permit a f i r m c h o i c e - - n e g a t i v e f i n d i n g s c o r r o b o r a t e a l r e a d y known c r i t i -c i s m s , w h i l e p o s i t i v e f i n d i n g s can be shown t o be too " c o s t l y " , o r the product o f c a r e f u l s e l e c t i o n o f c a n d i d a t e s l i k e l y to " s u c c e e d " (1976:106). Y e t , d e s p i t e these o b s t a c l e s , Rein takes an o p t i m i s t i c view, s u g g e s t i n g s e v e r a l r e s e a r c h s t r a t e g i e s through which e m p i r i c a l f i n d i n g s may c o n t r i b u t e to the development o f p o l i c y . One s t r a t e g y , which i s l a b e l l e d a CONSENSUAL APPROACH, can be seen as the p o s i t i o n most commonly taken by s o c i a l r e s e a r c h e r s who f o l -low the p o l i c y s c i e n c e approach to p o l i c y r e s e a r c h advocated by E t z i o n i . In t h i s c o n t e x t government c o n t r a c t s f o r the i n f o r m a t i o n i t wants, e.g. the "management review" o f Community Mental H e a l t h C e n t e r s done by the U.S. Government A c c o u n t i n g O f f i c e ; or i n t e r m e d i a r i e s b r i n g t o g e t h e r knowledge produ-c e r s and knowledge u s e r s , e.g., Westinghouse C o r p o r a t i o n brought t o g e t h e r a group o f c o n s u l t a n t s in e d u c a t i o n a l t e s t i n g , p s y c h o l o g y and s o c i o l o g y t o pro-v i d e e v a l u a t i o n s t u d i e s o f P r o j e c t H e a d s t a r t f o r the Department o f H e a l t h , E d u c a t i o n S W e l f a r e . Another s t r a t e g y , which Rein has r e f e r r e d t o as a CONTENTIOUS APPROACH, may be taken in s e t t i n g s where r e s e a r c h a c t s as a MORAL WITNESS f o r the f a i l u r e o f s o c i e t y t o honour i t s commitments, e.g., t h e 10 S t a g e II E v a l u a t i o n o f t h e S t r a t h c o n a R e h a b i l i t a t i o n P r o j e c t f u l l y documents s e r i o u s p r o c e d u r a l o b s t a c l e s t o c i t i z e n p a r t i c i p a t i o n i n the h o u s i n g r e h a b i l i -t a t i o n component and o v e r s p e n d i n g on an e x c e s s i v e l y l a r g e c a p a c i t y sewer and w a t e r system. S i m i l a r l y t h e c o n t e n t i o u s approach i s r e f l e c t e d when t h e r e s e a r c h e r a c t s as a .'SOC J A L. -.CRITIC under c i r c u m s t a n c e s where government has e s t a b l i s h e d no commitments t o be honoured, e.g. the v a r i o u s r e s e a r c h s t u d i e s done by women's groups and a d d i c t i o n f o u n d a t i o n s r e g a r d i n g women's use of minor.' t r a n q u i l l i z e r s and a l c o h o l , p r e p a r e d as p a r t o f a f i v e y e a r e f f o r t t o o b t a i n f u n d i n g f o r n e c e s s a r y e d u c a t i o n and t r e a t m e n t programs. The t h i r d r e s e a r c h s t r a t e g y i n w h i c h e m p i r i c a l f i n d i n g s c o n t r i b u t e t o t h e development o f p o l i c y Rein d e f i n e s as a PARADIGM-CHALLENGING APPROACH. T h i s s t r a t e g y r e q u i r e s the r e s e a r c h e r t o a c t i n d e p e n d e n t l y o f t h e e s t a b l i s h e d framework, t o expose i t s fundamental weaknesses and pr o p o s e a l t e r n a t i v e p r i n c i p l e s o f i n t e r v e n t i o n , w h i l e assuming t h a t p o l i t i c a l s u p p o r t f o r the a s s a u l t w i l l f o l l o w a t a l a t e r t i m e (1976:125). There a r e l i m i t e d examples o f t h e l a t t e r a p p r o a c h ; however, a s p e c t s o f K l e i b e r and L i g h t ' s (1978) d e s c r i p t i v e and e v a l u a t i v e s t u d y o f t h e Vancouver Women's H e a l t h C o l l e c t i v e exposes c e r t a i n fundamental weaknesses i n t h e m e d i c a l model o f h e a l t h c a r e d e l i v e r y t o women and proposes e x t e n s i o n o f t h e s e l f - h e l p model t o major a r e a s o f h e a l t h c a r e f o r women. It i s i m p o r t a n t t o note t h a t " w h i l e r e s e a r c h i s t y p i c a l l y u n d e r t a k e n w i t h i n one o r a n o t h e r o f t h e s e frames o f r e f e r e n c e , i t i s not uncommon f o r a s i n g l e s t u d y t o s h i f t from one s t r a t e g i c l e v e l t o a n o t h e r t o accommodate the v a r i e d m o t i v e s o f t h e r e s e a r c h e r " (1.976:125). T h i s s h i f t i n g between l e v e l s becomes e v i d e n t i n any d e t a i l e d a n a l y s i s o f e m p i r i c a l r e s e a r c h w h i c h hopes t o be u s e f u l i n t he development o f p o l i c y . F o r example, t h e e v a l u a t i o n o f Community Mental H e a l t h C e n t e r s i n t h e U.S. done by a t a s k f o r c e o f the C e n t e r f o r t h e Study o f R e s p o n s i v e Law ("Nader':s R a i d e r s " ) employed a r a t h e r t r a d i t i o n a l c o n s e n s u a l a p proach t o t h e d e s i g n o f t h e e v a 1 u a t i v e r e s e a r c h . The a u t h o r s kept t o o f f i -c i a l documents i n t h e i r d e s c r i p t i o n and a n a l y s i s o f program g o a l s , d i d l i t t l e 11 o r no a n a l y s i s o f a c t u a l program d e l i v e r y and never q u e s t i o n e d t a c i t d e f i n i t i o n s o f "community". However, i n t h e i r f i n a l a p p r a i s a l o f the program they produced p a r a d i g m - c h a l l e n g i n g recommendations r e g a r d i n g t h e d i r e c t i o n s f o r a c t i o n w h i c h s h o u l d be f o l l o w e d i n new mental h e a l t h s e r v i c e d e l i v e r y ( S i e g e l and Doty 1978: 146). W i t h t h e s e p r o v i s o s r e g a r d i n g o b s t a c l e s l i k e l y t o f r u s t r a t e p o t e n t i a l r e s e a r c h use, and t h e problems o f s h i f t i n g s t r a t e g y l e v e l s , R e i n o u t l i n e s t h r e e main a r e a s o f s t u d y which seem t o be r e l e v a n t t o d e v e l o p i n g a v i a b l e s o c i a l p o l i c y paradigm. These a r e a s i n c l u d e : 1. documenting the d i s p a r i t i e s between e x i s t i n g i d e a l s and e s t a b -l i s h e d r e a l i t i e s by e m p i r i c a l s t u d y and, as a member of s o c i e t y , u s i n g one's i n t u i t i v e sense t o a s s i s t i n i d e n t i f y i n g w h i c h d i s p a r i t i e s may be most a c c e p t a b l e p o l i t i c a l l y (1976:127); 2. s t u d y i n g t h e o r i e s o f p o l i c y i n t e r v e n t i o n i n an a c t i o n frame-work ( i d e o l o g i c a l l y and e m p i r i c a l l y l i n k e d t o r e a l s o c i a l p r o c e s s e s ) d e s i g n e d t o a s s i s t i n r e d u c i n g t h e d i s p a r i t y between i d e a l s and r e a l i t y (1976:130); and 3- e v a l u a t i n g s p e c i f i c programs which t r a n s l a t e a b r o a d l y d e f i n e d t h e o r y o f i n t e r v e n t i o n i n t o c o n c r e t e a c t i o n s (1976:133). I m p l i c a t i o n s A l t h o u g h t h e p o l i c y s c i e n c e and p o l i c y a n a l y s i s approaches d i f f e r i n t h e i r u n d e r l y i n g e p i s t e m o l o g i e s and t h e o r e t i c a l s t a t e m e n t s about the p o t e n t i a l uses o f p o l i c y r e s e a r c h , t h e r e a r e s e v e r a l a r e a s i n which they tend t o c o i n c i d e . F i r s t , both r e c o g n i z e t h e d i r e c t and immediate l i n k a g e s t h a t must be f o r g e d between th e p o l i c y r e s e a r c h e r and t h e v a r i o u s a c t o r s i n p u b l i c s e t t i n g s t o w h i c h she must r e l a t e . Second, an a c c e p t a n c e o f c o n t r o v e r s y i s e x p l i c i t i n b o t h a p p r o a c h e s , a l t h o u g h t h e r e i s d i s a g r e e m e n t about the p o s s i b i l i t i e s f o r r e s o l v i n g such c o n f l i c t t h r o u g h r e s e a r c h and a n a l y s i s . T h i r d , a l t h o u g h t h e l e a d i n g f i g u r e s i n t h e two camps s t r e s s t h e c r i t i c a l n a t u r e of p o l i c y r e s e a r c h , t h e r e i s an o v e r a l l sense o f o p t i m i s m conveyed by both r e g a r d i n g t h e u l t i m a t e 12 u s e f u l n e s s o f s o c i a l s c i e n c e r e s e a r c h i n t h e p o l i c y p r o c e s s . And, f i n a l l y , b o t h approaches d e f i n e t h e e v a l u a t i o n o f government-funded programs c o n c e r n e d w i t h s o c i a l r e f o r m as a key a r e a o f p o l i c y r e s e a r c h . The main t h e o r e t i c a l and e m p i r i c a l work o f t h i s d i s s e r t a t i o n f o c u s e s on the program e v a l u a t i o n p r o c e s s as a form o f p o l i c y r e s e a r c h . However, a t v a r i o u s p o i n t s where i t appears n e c e s s a r y t o t h e a n a l y s i s , t h i s s t u d y w i l l t o u c h on q u e s t i o n s o f p o l i c y i d e a l s : v s . t h e r e a l i t i e s o f l e g i s l a t i o n , i m p l e m e n t a t i o n and f u n d i n g , and e x p l o r e the t h e o r i e s o f p o l i c y i n t e r v e n t i o n w h i c h u n d e r l i e the g o a l s and o r g a n i z a t i o n a l a rrangements o f a p a r t i c u l a r program. 1.2 Program E v a l u a t i o n as P o l i c y Research Program e v a l u a t i o n as a form o f p o l i c y r e s e a r c h can r e f e r t o a wide v a r i e t y o f d e s c r i p t i v e and a n a l y t i c a l s t u d i e s o f program p r o c e s s ( f o r m a t i v e e v a l u a t i o n s ) o r program impact (summative e v a l u a t i o n s ) , o r some type o f comprehensive p r o -gram assessment w h i c h i n c l u d e s both modes o f e v a l u a t i o n s t u d y . Cronbach o f f e r s the f o l l o w i n g d e f i n i t i o n s o f " e v a l u a t i o n " and "program" w h i c h a r e s u f f i c i e n t l y g e n e r a l as t o a l l o w a range o f s p e c i f i c s t u d i e s t o be c o v e r e d : By the term EVALUATION, we mean s y s t e m a t i c e x a m i n a t i o n o f e v e n t s o c c u r r i n g i n and consequent on a contemporary program--an exami-n a t i o n conducted t o a s s i s t i n i m p r o v i n g t h i s program and o t h e r programs ha v i n g t h e same g e n e r a l purpose. By the term PROGRAM, we mean a s t a n d i n g arrangement t h a t p r o v i d e s f o r a s o c i a l s e r v i c e (1980:14). F o l l o w i n g Cronbach's p o s i t i o n ( 1 9 8 0 : 1 7 ) , I do not s t r e s s q u a n t i t a t i v e -s t a t i s t i c a l methods, o r " g o a l a t t a i n m e n t " , o r the i n t e n t t o j u d g e a program as good o r bad i n any d e f i n i t i o n o f what c o n s t i t u t e s a program e v a l u a t i o n s t u d y . F u r t h e r , program e v a l u a t i o n s may sometimes be concerned w i t h an e s t a b l i s h e d program t h a t i s o n g o i n g , sometimes w i t h a program t h a t had a l i m i t e d time frame and has a l r e a d y been t e r m i n a t e d , and i n o t h e r i n s t a n c e s w i t h a p l a n t h a t c o u l d 13 be e s t a b l i s h e d i f found s u i t a b l e . The main elements t h a t program e v a l u a t i o n s t u d i e s appear to have i n common a r e t h a t they are t i t l e d "program e v a l u a t i o n " and a r e g e n e r a l l y d e s i g n e d to be used by a c l i e n t or c l i e n t s ( o t h e r than the program e v a l u a t o r h e r s e l f ) who have some v e s t e d i n t e r e s t in the program in quest i o n . O b v i o u s l y , I want to emphasize the d i v e r s i t y o f p o l i c y r e s e a r c h work t h a t goes under the l a b e l program e v a l u a t i o n . But, what i s j u s t as important,!! want to p r e s e n t an a n a l y s i s of program e v a l u a t i o n as a p r o c e s s t h a t i n v o l v e s the n o t i o n of " j u d g i n g m e r i t " (Weiss 1972:1) , where m e r i t i t s e l f i s d e f i n e d in d i v e r s e ways depending on the v a l u e p o s i t i o n of the judge and the p r ocedures s e l e c t e d f o r judgement. Issues in the Theory and P r a c t i c e o f Program E v a l u a t i o n The recent l i t e r a t u r e on the use o f s o c i a l r e s e a r c h i n p u b l i c p o l i c y m a k i n g (Rein 1976; Weiss 1977; E t z i o n i 1978) and program e v a l u a t i o n (Guttentag and S t r u e n i n g 1975; F r a n k l i n and T h r a s h e r 1976; Cronbach 1980) s u g g e s t s t h a t thoroughgoing e v a l u a t i o n o f an e x i s t i n g program or e a r l y p l a n n i n g f o r the e v a l u -a t i o n o f a newly developed program i n v o l v e s a s e r i e s o f t h e o r e t i c a l and p r a c -t i c a l d e c i s i o n s . Such d e c i s i o n s r e q u i r e i n f o r m a t i o n about the s p e c i f i c program ( t h i s n o t i o n of "program" i n c l u d e s p r o j e c t s , a g e n c i e s or s u b - u n i t s o f an o v e r a l l n a t i o n - w i d e program) and the s o c i a 1/po 1itica 1 c o n t e x t in which the program i s o p e r a t e d . Any e f f o r t t o examine e m p i r i c a l l y the s p e c i f i c a c t i v i t i e s e n t a i l e d in c o n s t r u c t i n g a program e v a l u a t i o n and implementing i t s r e s u l t s and recom-mendations r e q u i r e s some a n a l y s i s o f the g e n e r a l i s s u e s around which such d e c i s i o n - m a k i n g tends to take p l a c e . These q u e s t i o n s a r e d e a l t w i t h in some d e t a i l In Chapter 3 ( s e c t i o n 3 -3 ) , but I want to o u t l i n e them here b r i e f l y s i n c e they become important i n d e f i n i n g the t o p i c o f t h i s d i s s e r t a t i o n . 14 The f i r s t i s s u e which may become the focus o f d i s c u s s i o n i n the program e v a l u a t i o n p r o c e s s i s the d e l i n e a t i o n o f a l l the major a c t o r s whose i n t e r e s t s w i l l be r e p r e s e n t e d throughout the e v a l u a t i o n study. The f o r m a l i t y ( i n c o n t r a c t u a l terms) o r i n f o r m a l i t y (as o b s e r v e r s and/or c r i t i c s ) o f t h a t rep-r e s e n t a t i o n , and the scope and i n t e n s i t y o f c o n t r o l o v e r subsequent phases o f the e v a l u a t i o n p r o c e s s may become c o n t r o v e r s i a l q u e s t i o n s r i g h t a t the s t a r t , and grow o r fade as the work c o n t i n u e s . A second, c l o s e l y l i n k e d i s s u e , i s the q u e s t i o n o f the o r g a n i z a t i o n a l placement o f the program e v a l u a t i o n r e s e a r c h . D e c i s i o n s must be made by key a c t o r s whether the work w i l l be done w i t h i n the program s e t t i n g (by " i n - h o u s e " r e s e a r c h e r s ) , w i l l be d i r e c t e d by program man-agement us<ing o u t s i d e c o n s u l t a n t s , o r be commissioned r e s e a r c h done by a f i r m o f program e v a l u a t o r s s e l e c t e d through a p u b l i c p r o c e s s ( r e q u e s t s f o r p r o p o s a l s , i n t e r v i e w s , h i r i n g committee, e t c . ) . In l a r g e s c a l e and/or f e d e r a l l y funded programs the p u b l i c p r o c e s s may be r e q u i r e d , but the s p e c i f i c d e t a i l s in each case may vary s i g n i f i c a n t l y depending upon p r e - e x i s t i n g i m p l i c i t agreements between program f u n d e r s , managers, and l i s t s o f p r e f e r r e d c o n s u l t a n t s . The t h i r d i s s u e , the g e n e r a l r e s e a r c h s t r a t e g y to be f o l l o w e d , and the f o u r t h , the s p e c i f i c model and methodology t o be used a l s o have o b v i o u s con-n e c t i o n s . In a n a l y z i n g q u e s t i o n s o f r e s e a r c h s t r a t e g y , we must understand t h a t f r e q u e n t l y t h i s l i e s so f a r below t h e s u r f a c e of d i s c u s s i o n t h a t i t never emerges as an i s s u e . However, t h e r e are s e t t i n g s in which an awareness o f the " v a l u e s c r e e n s " o f the major a c t o r s , and o v e r t c o n s i d e r a t i o n o f the consequences o f s e l e c t i n g the c o n s e n s u a l , c o n t e n t i o u s o r p a r a d i g m - c h a l l e n g i n g approach become the focus o f s e r i o u s a n a l y s i s w i t h i n the e v a l u a t i o n team and between tha t team and the funders o f the e v a l u a t i o n study o r program managers ( K l e i b e r and L i g h t 1978). More o f t e n , d i f f e r e n c e s emerge around q u e s t i o n s c o n c e r n i n g the r e s e a r c h model and/or s p e c i f i c methodology to be employed in the e v a l u a t i o n s t u d y . These f r e q u e n t l y f o c u s on the methods used in the r e s e a r c h d e s i g n , the. 15 p o s i t i v i s t s e x p e c t i n g formal e x p e r i m e n t a t i o n and s t a t i s t i c a l measures, the e t h n o g r a p h e r s emphasizing the n e c e s s i t y f o r o b s e r v a t i o n a l t e c h n i q u e s , and those t r a i n e d in the t r a d i t i o n o f s o c i a l a n a l y s i s (Rowbottom 1977) s e e k i n g some ba l a n c e o f the two approaches l i k e l y to i n c r e a s e the use o f the r e s e a r c h f i nd i ngs. The f i f t h i s s u e around which t h e r e i s o f t e n e x t e n s i v e d i s c u s s i o n and nego-t i a t i o n i n p o l i c y r e s e a r c h s t u d i e s g e n e r a l l y (and program e v a l u a t i o n s in p a r t i -c u l a r ) , i s the c o n s t r u c t i o n , c o n t e n t and t i m i n g o f the r e s e a r c h r e p o r t . Reaching agreement on the a c c e p t a b i l i t y o f the f i n a l d r a f t o f a program e v a l u -a t i o n may be s i m p l e and s t r a i g h t f o r w a r d i n some c a s e s ; whereas in o t h e r s a c c e p t a n c e (and payment) may be w i t h h e l d as a r e s u l t o f p r e v i o u s l y u n r e s o l v e d c o n f l i c t s o v e r models and methods, o r o v e r the c o n t e n t and i n t e r p r e t a t i o n o f the r e s e a r c h f i n d i n g s . O f t e n the term " c o u n t e r i n t u i t i v e " i s used i n t h i s con-t e x t and u s u a l l y i t i s a g l o s s f o r a s i t u a t i o n where the e x p e c t a t i o n s o f the p o l i c y m a k e r s / p r o g r a m funders/managers have not been r e i n f o r c e d by the f i n d i n g s o f the e v a l u a t i o n r e p o r t . In t h e s e s i t u a t i o n s i t becomes v e r y d i f f i c u l t t o f i n d some way to d i s c u s s the s i x t h , and f i n a l i s s u e , p l a n n i n g f o r r e s e a r c h u t i l i z a t i o n . In most s e t t i n g s p o l i c y r e s e a r c h e r s make some s y s t e m a t i c e f f o r t t o a s s u r e t h a t a " p o s i t i v e r e c e p t i o n " i s g i v e n t o t h e i r r e s e a r c h r e s u l t s and p o l i c y recommendations. However, where program e v a l u a t o r s have s p e c i f i c a l l y chosen a c o n t e n t i o u s o r p a r a d i g m - c h a l l e n g i n g approach t h e i r p l a n s f o r r e s e a r c h u t i l i z a t i o n may d i f f e r . In such s i t u a t i o n s the e v a l u a t i o n document i s designed as a t o o l to provoke c o n t r o v e r s y and/or educate a c o n s t i t u e n c y r e g a r d i n g i t s r i g h t s and needs. T h i s i s s u e o f p l a n n i n g f o r r e s e a r c h u t i l i z a t i o n may become the f i r s t phase o f another c y c l e o f program p l a n n i n g and change in those s e t t i n g s where the e v a l u a t i o n component i s p a r t of an o v e r a l l management c y c l e (Crowfoot 1 9 7 7 ) . 16 These s i x t o p i c s (the major a c t o r s , o r g a n i z a t i o n a l placement, r e s e a r c h s t r a t e g y , models and methods, r e s e a r c h r e p o r t and r e s e a r c h u t i l i z a t i o n ) are l a b e l l e d in s l i g h t l y d i f f e r e n t ways in the v a r i o u s major t e x t s on program e v a l u a t i o n (Caro 1971; C a r t e r and Wharf 1973; Cronbach 1980; F r a n k l i n and T h r a s h e r 1976; R o s s i , Freeman and Wright 1979)- Each i s g i v e n v a r y i n g degrees of emphasis depending upon the a u t h o r s ' v a l u e s , i d e o l o g i c a l p o s i t i o n and r e s e a r c h paradigm. However, as i n d i c a t e d i n the p r e c e d i n g d i s c u s s i o n , these r e s e a r c h q u e s t i o n s cannot s t a n d on t h e i r own i n any s o c i o l o g i c a l a n a l y -s i s o f the f a c t o r s a f f e c t i n g the d e s i g n , p r o d u c t i o n and p o t e n t i a l use o f p r o -gram e v a l u a t i o n s as c o n t r i b u t i o n s to the p o l i c y p r o c e s s . It i s n e c e s s a r y to examine the l a r g e r o r g a n i z a t i o n a l arrangements i n which p o l i c y r e s e a r c h i s conducted and to develop s y s t e m a t i c documentation of how these r e s e a r c h i s s u e s are l i n k e d t o one a n o t her in s p e c i f i c program e v a l u a t i o n s e t t i n g s . An e x t e n s i v e examination of the s o c i a l , p o l i t i c a l , economic and o r g a n i z a t i o n a l a s p e c t s o f the p o l i c y i n t e r v e n t i o n p r o c e s s i s p r e s e n t e d in Chapter 3 ( s e c t i o n 3 -3 ) . The o n l y p o i n t I want t o make here i s t h a t no model of p o l i c y r e s e a r c h can be complete w i t h o u t s y s t e m a t i c c o n s i d e r a t i o n of t h e s e a s p e c t s o f the p o l i c y p r o c e s s being i n c l u d e d as p a r t o f the s t r a t e g y of program d e s c r i p t i o n and assessment. The Program E v a l u a t i o n E n t e r p r i s e B e f o r e p r o c e e d i n g w i t h a d i s c u s s i o n of my r e s e a r c h t o p i c , I want t o p r e -sent b r i e f l y some o f the c h a r a c t e r i s t i c s o f program e v a l u a t i o n as a knowledge p r o d u c t i o n i n d u s t r y . T h i s background m a t e r i a l i s intended as one way to out-l i n e f o r readers t h e e x t e n t t o which the work o f program e v a l u a t i o n r e s e a r c h i s e m p i r i c a l l y l i n k e d t o a s p e c t s o f the l a r g e r p o l i t i c a l e c o n o m y b o t h the p u b l i c and the p r i v a t e s e c t o r s . 17 During the past decade managers o f programs in the areas o f h e a l t h , edu-c a t i o n , s o c i a l housing and w e l f a r e s e r v i c e s were f o r c e f u l l y n o t i f i e d t h a t t h e i r programs had to become more e f f i c i e n t as w e l l as becoming more p o l i t i c a l l y a c c o u n t a b l e . They had to look t o a new k i n d o f s p e c i a l i s t , the program e v a l u -a t o r , to h e l p s o l v e the problems o f d e m o n s t r a t i n g program e f f i c i e n c y and docu-menting program impact. From the p e r s p e c t i v e o f these program managers t h e r e were many reasons f o r p e r m i t t i n g and o c c a s i o n a l l y even e n c o u r a g i n g an e v a l u a -t i o n of' t h e i r programs. From the vantage p o i n t o f p o l i t i c i a n s and o t h e r p o l i c y m a k e r s t h e r e was a d i f f e r e n t , but o f t e n o v e r l a p p i n g s e t o f reasons f o r i n s t i t u t i n g a proc e s s o f e v a l u a t i o n o f a l l programs coming up f o r new and/or c o n t i n u e d f u n d i n g . The r e s u l t i n g e v a l u a t i o n s were u s u a l l y i n i t i a t e d f o r many d i f f e r e n t and, at t i m e s , even c o n f l i c t i n g p u r p o s e s ; f o r example; choosing the best among s e v e r a l a l t e r n a t i v e forms o f program development, r e - a d j u s t i n g the o r g a n i z a t i o n o f a program a l r e a d y in o p e r a t i o n , m a i n t a i n i n g q u a l i t y c o n t r o l o v e r program s e r v i c e s , f o r c i n g s u b o r d i n a t e s t o comply w i t h p o l i c i e s from above, documenting t h a t one's agency (or agency s u p e r v i s e d by one's p o l i c y committee) d e s e r v e s i t s p r e s e n t budget o r a budget i n c r e a s e , c r e a t i n g support f o r a f a v o u r i t e new p r o j e c t , c a s t i n g s u s p i c i o n on a p o l i c y s u p p o r t e d by p o l i t i c a l opponents o r a competing program, and so on. From the p e r s p e c t i v e o f the program e v a l u a t o r the range o f purposes was always g r e a t e r than any s i n g l e study c o u l d f u l f i l l , even when managers and p o l i c y m a k e r s were d i r e c t and honest about t h e i r e v a l u a t i o n agendas. The s o c i a l , p o l i t i c a l and economic c o n t e x t s o f t h i s p r o c e s s v a r i e d so e x t e n s i v e l y t h a t no i n d i v i d u a l was f u l l y equipped to cope w i t h a l l c o n t i n g e n c i e s . New p o l i c i e s and new programs at a l l t h r e e l e v e l s o f government p r e s e n t e d a complex and growing market f o r the s k i l l s o f program e v a l u a t o r s from a l l the s o c i a l s c i e n c e d i s c i p l i n e s . ' Furthermore, as t h e f u n d i n g s o u r c e s f o r s o c i a l s e r v i c e , housing and e d u c a t i o n programs were reduced, as a r e s u l t o f i n f l a t i o n and 18 i d e o l o g i c a l s h i f t s , such programs i n c r e a s i n g l y found i t n e c e s s a r y to j u s t i f y t h e i r impact and produce good "bottom l i n e " r e s u l t s . Program e v a l u a t i o n has become an e n t e r p r i s e t h a t r e q u i r e s a team o f : .r r e s e a r c h e r s and, i n the case o f l a r g e s c a l e government funded programs, the e v a l u a t i o n team r e q u i r e s s u b s t a n t i a l o r g a n i z a t i o n a l b a c k i n g from a f i r m o r r e s e a r c h i n s t i t u t e . There i s l i t t l e "hard d a t a " on program e v a l u a t i o n f i r m s i n Canada, and the main bases f o r t h i s p a r t o f the d i s c u s s i o n d e r i v e from s o u r c e s r e l e v a n t t o the U n i t e d S t a t e s (Abt 1976; Cronbach I980). In n e i t h e r c a s e , Canada nor t h e U n i t e d S t a t e s , do we have any s y s t e m a t i c i n f o r m a t i o n on the number o f f i r m s nor a comprehensive assessment o f the q u a l i t y o f the knowledge product these f i r m s a r e p a i d t o produce. In g e n e r a l , we have a sense t h a t t h e demand f o r e v a l u a t i o n s i s t i e d d i r e c t l y to l e v e l s o f government spending d u r i n g each f i s c a l y e a r , and the e x p a n s i o n / c o n t r a c t i o n c y c l e f o r i n d i -v i d u a l f i r m s can be v e r y r a p i d (Cronbach 1980). There a r e a very small number o f v e r t i c a l l y i n t e g r a t e d f i r m s t h a t o b t a i n the l a r g e budget c o n t r a c t s (e.g., Peat, Marwick and P a r t n e r s in Canada, Abt A s s o c i a t e s in the U n i t e d S t a t e s ) , and a r a p i d l y f l u c t u a t i n g number of small r e s e a r c h groups and i n d i v i d u a l s who o p e r a t e on a "piece-work" b a s i s . The few l a r g e f i r m s have f u l l - t i m e s t a f f s o f s o c i a l s c i e n t i s t s , networks o f f i e l d o b s e r v e r s and i n t e r v i e w e r s , p u b l i c r e l a t i o n s p e r s o n n e l and c o l l e c t i o n s o f s p e c i a l i z e d c o n s u l t a n t s on c a l l ( e . g ., p etroleum e n g i n e e r s , urban a r c h a e o l o g i s t s ) . S e v e r a l o f the s m a l l e r f i r m s and i n d i v i d u a l s s u b s i s t through s u b c o n t r a c t i n g t o l a r g e r f i r m s , and/or having o t h e r s o u r c e s o f s t a b l e income. T h i s i s p a r t i c u l a r l y the case f o r s o c i o l o g i s t s / a n t h r o p o l o g i s t s who have t h e i r economic base in u n i v e r s i t y o r c o l l e g e t e a c h i n g and f o r p i a n n e r s / e c o l o g i s t s who are employed by a r c h i t e c t u r a l o r e n g i n e e r i n g f i rms. One a d d i t i o n a l aspect o f the economic s t r u c t u r e s h o u l d be mentioned here, i . e . , t h e i n c r e a s i n g s c a l e o f e v a l u a t i o n s t u d i e s . It has been e s t i m a t e d t h a t 19 a t t h e b e g i n n i n g o f t h e 19701s, i n t h e U n i t e d S t a t e s , about 300 new s t u d i e s were begun each y e a r , w i t h d i r e c t f e d e r a l s u p p o r t and w i t h an average budget o f about $100,000 (Eaton I980). At t h e c l o s e o f the 1970's r e q u i r e m e n t s f o r l a r g e - s c a l e e v a l u a t i o n s had i n c r e a s e d t o a p o i n t where i t was not uncommon f o r a s i n g l e m a j o r s t u d y t o have e x p e n d i t u r e s o v e r one m i l l i o n d o l l a r s a y e a r (Cronbach 1980:39)• Both Cronbach (1980) and Eaton (I980) use t h e term "mushroomed" when r e f e r r i n g t o t h e unplanned growth o f e v a l u a t i o n a c t i v i t i e s ' i n t h e l a s t t e n y e a r s . I t has been s a i d t h a t t h e r e i s a "boom town" e x c i t e -ment i n t h e e v a l u a t i o n community. But i t appears t h a t f e d e r a l e x p e n d i t u r e s i n t h e U n i t e d S t a t e s f o r e v a l u a t i o n s t u d i e s have been n e a r l y l e v e l i n t h e l a s t few y e a r s . A s u r v e y o f p o l i c y r e s e a r c h u n d e r t a k e n by f e d e r a l government departments and a g e n c i e s d u r i n g 1975-1977 i n d i c a t e d t h a t on an o v e r a l l base o f one b i l l i o n o r j u s t s l i g h t l y more, program e v a l u a t i o n r e p r e s e n t e d between 5-2% and 5-6% o f budget o b l i g a t i o n s i n each o f t h o s e t h r e e f i s c a l y e a r s (Cronbach 1980:42). A l t h o u g h we l a c k s i m i l a r s u r v e y d a t a f o r comparable f e d e r a l g o v e r n -ment departments here i n Canada, i t appears s a f e t o assume t h a t the economic p i c t u r e f o r program e v a l u a t i o n i s r a t h e r s i m i l a r . For example, i n t h e l a s t few y e a r s t h e r e has been a marked r e d u c t i o n i n t h e amount a v a i l a b l e f o r s o c i a l h o u s i n g programs at CMHC and fewer d o l l a r s f o r i n n o v a t i v e programs at H e a l t h and W e l f a r e . The one a r e a t h a t has seen c o n t i n u e d e v a l u a t i o n and s o c i a l impact s t u d i e s i s p r i m a r y r e s o u r c e development i n t h e w e s t e r n p r o v i n c e s arid the T e r r i t o r i e s . F i n a l l y , we. s h o u l d note t h a t s m a l l e r U n i t e d S t a t e s and Canadian f i r m s c o n t i n u e t o a s s i s t l o c a l a g e n c i e s and m u n i c i p a l governments t o m o n i t o r and a s s e s s t h e i r programs. At the same t i m e , t h e l a r g e r f i r m s have begun t o p l a y an i m p o r t a n t r o l e i n t h e e v a l u a t i o n o f programs funded by t h e f e d e r a l l e v e l o f government but d e v e l o p e d f o r and implemented i n v a r i o u s p a r t s o f the C a r i b b e a n , A f r i c a and S o u t h e a s t A s i a (Weiss 1977,). 20 1.3 The Research T o p i c : Approach, S o u r c e s , Methods and Models S i n c e e v e r y program e v a l u a t i o n i s r e s p o n s i v e to d i f f e r e n t a s p e c t s o f the s i x t h e o r e t i c a l / p r a c t i c a l q u e s t i o n s and y i e l d s a d i f f e r e n t s e t o f inputs i n t o the p o l i c y p r o c e s s , t h e r e i s a danger th a t s o c i a l s c i e n t i s t s concerned w i t h making g e n e r a l i z a t i o n s about program e v a l u a t i o n as a type o f p o l i c y r e s e a r c h w i l l s t a y w i t h the dominant p o l i c y s c i e n c e paradigm. From t h i s p e r s p e c t i v e one must take t h e e x p e r i m e n t a l / p o s i t i v i s t model as a r e s e a r c h s t a n d a r d , c o n s t r u c t a r o l e f o r the e v a l u a t o r as a r e l a t i v e l y p a s s i v e t e c h -n i c i a n , and d e f i n e many o f the c o n t r a d i c t i o n s o f the e v a l u a t i o n p r o c e s s as the p r o d u c t s o f inadequate communication, f a u l t y methodology and s l o p p y s t a t i s t i c a l d e s i g n . On the o t h e r hand, t h e r e i s a second p e r s p e c t i v e a v a i l -a b l e which i n c l u d e s elements o f t h e p o l i c y s c i e n c e paradigm, but combines i t w i t h the p h i l o s o p h i c a l and p o l i t i c a l i n s i g h t s o f p o l i c y a n a l y s i s . In t h i s a l t e r n a t i v e t h e e v a l u a t o r i s seen as an a c t i v e p o l i c y a n a l y s t and i n t e r p r e t e r o f r e a l i t y , who has a number o f r e s e a r c h models and methods i n her r e p e r t o i r e , and attempts t o use a c r i t i c a l but e s s e n t i a l l y o p t i m i s t i c approach to the problems e n t a i l e d in p r o d u c i n g a program e v a l u a t i o n . G e n e r a l l y , the work o f t h i s d i s s e r t a t i o n takes i t s s t a r t i n g p o i n t i n t h e second approach t o p o l i c y r e s e a r c h and program e v a l u a t i o n . The Research Problem The g e n e r a l focus o f t h i s r e s e a r c h i n v o l v e s l o o k i n g a t the uses o f s o c i a l r e s e a r c h i n p u b l i c p o l i c y m a k i n g , and, in p a r t i c u l a r , examining the c o n s t r u c t i o n and use o f program e v a l u a t i o n s t u d i e s in a s e r i e s o f s o c i a l reform programs. In t h e cou r s e o f s e t t i n g up the r e s e a r c h d e s i g n f o r t h i s d i s s e r t a t i o n I have made c e r t a i n c o n c e p t u a l d e c i s i o n s which e n t a i l e d the use o f m u l t i p l e , t h e o r e t i c a l l y c o m p a t i b l e paradigms: the s o c i o l o g y o f •:. 21 knowledge, s o c i a l o r g a n i z a t i o n t h e o r y and a r e c e n t v e r s i o n o f the n e g o t i a t e d o r d e r paradigm ( B u r r e l 1 and Morgan 1979). I c o n s i d e r program e v a l u a t i o n s t u d i e s and o t h e r forms o f p o l i c y r e s e a r c h as s o c i a l l y produced documents which a r e open to a n a l y s i s through the use o f a s o c i o l o g y o f knowledge approach (Smith 1974b). These s o c i a l l y c o n s t r u c t e d documentary r e a l i t i e s a r e " p r o d u c t s " which a r e purchased by a d e f i n a b l e o r g a n i z a t i o n whose management wishes o r i s r e q u i r e d t o fund such a program e v a l u a t i o n study. These s p e c i f i c o r g a n i z a t i o n a l "consumers" o f the r e s e a r c h d i f f e r in t h e i r b a s i c s t r u c t u r e s , d e c i s i o n - m a k i n g o r i e n t a t i o n s , management i d e o l o g i e s and s e v e r a l o t h e r f a c t o r s to which o r g a n i z a t i o n t h e o r y i s d i r e c t l y r e l e v a n t . However, the consumers o f the study may i n c l u d e o t h e r major a c t o r s o u t s i d e t h e f u n d i n g o r g a n i z a t i o n ; thus, community r e s i d e n t s , program c l i e n t s , the media and o t h e r i n t e r e s t e d groups become members o f a l a r g e r "market" f o r program e v a l u a t i o n s t u d i e s . In o r d e r to study t h i s l a r g e r s t r u c t u r a l c o n t e x t I have used a t h i r d paradigm t o attempt t o l i n k up the p r o d u c t i o n and con-sumption a s p e c t s o f the r e s e a r c h market. It i s the n e g o t i a t e d o r d e r paradigm ( S t r a u s s 1978) t h a t I have employed as a t h e o r e t i c a l g u i d e f o r i n t e g r a t i n g i s s u e s in t h e c o n s t r u c t i o n o f t h e p o l i c y r e s e a r c h document (the program e v a l u a t i o n study) w i t h the problems o f a n a l y z i n g the o r g a n i z a t i o n a l s t r u c t u r e and o p e r a t i o n s o f t h e consumers o f such r e s e a r c h . In t h i s e f f o r t to a n a l y z e the p r o c e s s , p r o d u c t s , u s e r s and uses o f p o l i c y r e s e a r c h , I found t h a t t h e r e were c r i t i c a l l i m i t a t i o n s i n the r e s e a r c h models t h a t tend t o predominate in d i s c u s s i o n s o f program e v a l u a t i o n work. T h e r e -f o r e , I came t o t h e d e c i s i o n t h a t t h e s p e c i f i c purpose o f t h i s d i s s e r t a t i o n would be t o develop a w i d e r s e l e c t i o n o f e v a l u a t i o n r e s e a r c h models which would s p e c i f i c a l l y take i n t o account c o n s t r u c t i o n o f the r e s e a r c h " p r o d u c t " and c h a r a c t e r i s t i c s o f the l a r g e r s t r u c t u r a l c o n t e x t i n which such p r o d u c t s 22 a r e d e s i g n e d to be used. My t h e o r e t i c a l s t r a t e g y i n v o l v e s an e f f o r t t o r e l a t e t h e more s t a b l e , s t r u c t u r a l c h a r a c t e r i s t i c s o f the s o c i a l , p o l i t i c a l , economic, and o r g a n i z a t i o n a l c o n t e x t s o f reform programs to t h e s e r i e s o f s i x t h e o r e t i c a l / p r a c t i c a l i s s u e s ( o u t l i n e d in 1.2) in the program e v a l u a t i o n p r o c e s s . The expanded set o f p o l i c y r e s e a r c h models emerge from both t h e o r e t i c a l and case study s o u r c e s . The f i r s t s t e p i n t h i s p r o c e s s o f f o r m u l a t i n g new models i n v o l v e s a c r i t i c a l review o f the two major models o f program e v a l u a t i o n r e s e a r c h d e s c r i b e d in t h e contemporary l i t e r a t u r e on p o l i c y r e s e a r c h i n e d u c a t i o n . The c r i t i c a l review examines each model in terms o f i t s main assumptions and d i r e c t i v e s r e g a r d i n g r e s o l u t i o n o f the s i x t h e o r e t i c a 1 / p r a c t i c a 1 i s s u e s . Second, t a k i n g a s y m b o l i c i n t e r a c t i o n i s t p e r s p e c t i v e on t h e study o f s o c i a l o r g a n i z a t i o n and s o c i a l s t r u c t u r e (the n e g o t i a t e d o r d e r paradigm), I con-s t r u c t a framework f o r the a n a l y s i s o f program e v a l u a t i o n s t u d i e s which s p e c i f i e s components o f the s t r u c t u r a l c o n t e x t of p o l i c y r e s e a r c h and r e d e f i n e s the s i x t h e o r e t i c a l / p r a c t i c a l i s s u e s as " b a s i c areas o f n e g o t i a t i o n " . Then using t h i s framework, I d e s c r i b e and a n a l y z e the program e v a l u a t i o n p r o c e s s in two p a i r s of e v a l u a t i o n r e s e a r c h s t u d i e s . The case s t u d i e s pro-v i d e an o p p o r t u n i t y t o e x p l o r e e m p i r i c a l l y the d i f f e r i n g p a t t e r n s o f know-ledge p r o d u c t i o n and use which emerge when v a r y i n g d e c i s i o n s a r e n e g o t i a t e d i n v o l v i n g the s i x p o l i c y r e s e a r c h i s s u e s . In the f i n a l c h a p t e r I o u t l i n e and d i s c u s s s e v e r a l models o f n e g o t i a t e d knowledge p r o d u c t i o n and use suggested i n the c o u r s e o f the case study a n a l y s i s . These models are not designed t o be e i t h e r e x c l u s i v e o r e x h a u s t i v e . Rather, they a r e p r e s e n t e d i n o r d e r to i n d i c a t e (a) the range o f n e g o t i a t e d o p t i o n s which may surround each o f the major t h e o r e t i c a l / p r a c t i c a l r e s e a r c h i s s u e s , and (b) t h e degree t o which s t r u c t u r a l and n e g o t i a t i o n c o n t e x t s may 23 shape t h e p o l i c y r e s e a r c h p r o c e s s . In a d d i t i o n , these models suggest pos-s i b l e d i r e c t i o n s f o r "sampling the p o p u l a t i o n " o f program e v a l u a t i o n s t u d i e s t o d e v e l o p a d d i t i o n a l models o f n e g o t i a t e d knowledge p r o d u c t i o n and use in t h e p o l i c y r e s e a r c h p r o c e s s . T h e o r e t i c a l Sources The most f u l l y developed d i s c u s s i o n o f program e v a l u a t i o n as a form o f p o l i c y r e s e a r c h has emerged in the e v a l u a t i o n o f s c h o o l s and e d u c a t i o n pro-grams g e n e r a l l y . T h e r e f o r e t h e most e x t e n s i v e , immediate so u r c e f o r an a n a l y s i s o f e v a l u a t i o n r e s e a r c h is l o c a t e d in t h e l i t e r a t u r e on contemporary models of program e v a l u a t i o n developed and used i n p o l i c y r e s e a r c h in e d u c a t i o n . The d i s c u s s i o n p r e s e n t e d i n Chapter 2 i s intended as a c r i t i c a l o v e r v i e w o f the p r i n c i p a l models o f r e s e a r c h that e d u c a t i o n a l program e v a l u a t o r s c l a i m a r e most f r e q u e n t l y employed in doing t h e i r type o f p o l i c y r e s e a r c h . A l t h o u g h t h e s e t h e o r e t i c a l examples (or " i d e a l types") have been taken from e d u c a t i o n a l r e s e a r c h , t h e d e s c r i p t i v e and c r i t i c a l remarks a r e d i r e c t l y a p p l i c a b l e t o examples o f p o l i c y r e s e a r c h in the areas o f housing, h e a l t h c a r e , w e l f a r e s e r v i c e s , o r employment t r a i n i n g . S u b s t a n t i v e d e t a i l s o f the r e s e a r c h models may be s h i f t e d w i t h o u t f i n d i n g s i g n i f i c a n t changes being made in t h e u n d e r l y i n g paradigms i n c o r p o r a t e d i n t o the two models which are p r e s e n t e d f o r c r i t i c a l a n a l y s i s . D e s c r i p t i o n o f the a g r i c u l t u r a l - b o t a n y paradigm and c o n v e n t i o n a l program e v a l u a t i o n s t u d i e s r e l i e s on the work o f R o s s i and W i l l i a m s (1972), Boruch and Riecken (1975) and s e v e r a l a r t i c l e s in the Handbook o f E v a l u a t i o n Research e d i t e d by G u t t e n t a g and S t r u e n i n g (1975). The b a s i c framework f o r t h i s model o f program e v a l u a t i o n d e r i v e s from a p o s i t i v i s t / e x p e r i m e n t a 1 i s t paradigm o f s o c i a l s c i e n c e . The a l t e r n a t i v e view o f e v a l u a t i o n r e s e a r c h which has 2k emerged in the l a s t decade c h a l l e n g e s many o f the assumptions o f the p o s i t i v -i s t model. T h i s second view d e r i v e s from an a n t h r o p o l o g i c a l o r i n t e r p r e t i v e paradigm o f s o c i a l s c i e n c e r e s e a r c h and has been termed " i l l u m i n a t i v e " e v a H u a t i o n . The d i s c u s s i o n o f the second model r e l i e s on work by P a r l e t t and Hamilton (1976), R.E. Stake (1974), and commentary by the S t a n f o r d E v a l u a t i o n Consortium (1976). In Chapter 3 my focus s h i f t s t o t h e problem o f c o n s t r u c t i n g a t h e o r e t i c a l framework f o r the a n a l y s i s of program e v a l u a t i o n r e s e a r c h which extends beyond the l i m i t s o f t h e models p r e s e n t e d in Chapter 2 . A f t e r having reviewed two models o f how program e v a l u a t i o n s h o u l d be a c c o m p l i s h e d , I d i r e c t my a t t e n t i o n to a t h e o r e t i c a l p e r s p e c t i v e which encourages a n a l y s i s o f program e v a l u a t i o n r e s e a r c h as i t i s accomplished i n s p e c i f i c o r g a n i z a t i o n a l c o n t e x t s . The d i s c u s s i o n p r e s e n t e d in Chapter 3 r e l i e s on the t h e o r e t i c a l p e r s p e c t i v e o f s y m b o l i c i n t e r a c t i o n , and more s p e c i f i c a l l y the n e g o t i a t e d o r d e r paradigm develo p e d by Anselm S t r a u s s . The c e n t r a l i d e a i n t h i s t h e o r y o f s o c i a l o r g a n i z a t i o n i s a concept o f s o c i a l s t r u c t u r e as being " i n p r o c e s s " , and the b e l i e f t h a t " s o c i a l o r d e r s a r e always i n some sense n e g o t i a t e d o r d e r s " (1978:x?). S t r a u s s d e s c r i b e s the s o c i a l o r d e r i n terms o f the p r o c e s s o f ongoing n e g o t i a t i o n s which i s c a r r i e d on by many types o f p a r t i c i p a n t s , each w i t h d i s t i n c t i v e w o r l d views. N e g o t i a t i o n s a r e conducted s i m u l t a n e o u s l y at s e v e r a l l e v e l s i n t h e s o c i a l o r d e r , and each l e v e l can o f f e r r e s o u r c e s t o and impose c o n s t r a i n t s on o t h e r s . In d e v e l o p i n g t h e n e g o t i a t e d o r d e r paradigm S t r a u s s d e f i n e s two key terms: " s t r u c t u r a l c o n t e x t " and " n e g o t i a t i o n c o n t e x t " (1978:99)• The s t r u c t u r a l c o n t e x t r e f e r s t o s a l i e n t s t r u c t u r a l p r o p e r t i e s t h a t bear on n e g o t i a t i o n , the more s t a b l e arrangements w i t h i n which n e g o t i a t i o n s take p l a c e . The concept o f n e g o t i a t i o n c o n t e x t i s analogous to the use o f the 25 term "awareness c o n t e x t " in h i s e a r l i e r work ( G l a s e r and S t r a u s s 1964). He s p e c i f i e s t h a t t h e r e a r e d i f f e r e n t k i n d s o f n e g o t i a t i o n c o n t e x t s which p e r t a i n t o i n t e r a c t i o n among n e g o t i a t i n g p a r t i e s . These are r e l a t e d t o p e r m u t a t i o n s o f s e v e r a l p r o p e r t i e s i d e n t i f i a b l e in any n e g o t i a t i o n c o n t e x t : the n e g o t i a t o r s , t h e n a t u r e and t i m i n g o f the n e g o t i a t i o n s , the b a l a n c e of power, the n a t u r e o f the r e s p e c t i v e s t a k e s , t h e v i s i b i l i t y o f the t r a n s a c -t i o n s , the number and c o m p l e x i t y o f the i s s u e s , the c l a r i t y o f l e g i t i m a c y b o u n d a r i e s , and o p t i o n s to a v o i d i n g o r d i s c o n t i n u i n g n e g o t i a t i o n (1978:100). He remarks t h a t : A s t e a d y f o c u s on s t r u c t u r a l and n e g o t i a t i o n c o n t e x t s and on t h e i r r e s p e c t i v e p r o p e r t i e s i n c r e a s e s the l i k e l i h o o d that the a n a l y s i s o f s p e c i f i c c o u r s e s o f n e g o t i a t i o n w i l l be c a r e f u l l y l o c a t e d " w i t h i n " the l a r g e r s o c i a l s t r u c t u r e . (No r e i f i c a t i o n i s i n t e n d e d r e g a r d i n g s o c i a l s t r u c t u r e ) (1978:101). T h i s paradigm i s d i r e c t l y a p p l i c a b l e to an a n a l y s i s o f the e v a l u a t i o n r e s e a r c h p r o c e s s and uses o f the documentary p r o d u c t i n the p l a n n i n g f o r program s t a b i l i t y and/or change. Its main purpose i s to p r o v i d e a t h e o r e t -i c a l g uide f o r i n t e g r a t i n g elements o f the two o t h e r paradigms i n v o l v e d in t h i s a n a l y s i s o f the c o n s t r u c t i o n and use o f p o l i c y r e s e a r c h : the s o c i o l o g y o f knowledge approach used in examining the p r o c e s s and product o f r e s e a r c h ( t h e e v a l u a t i o n document) and the o r g a n i z a t i o n t h e o r y employed in the e f f o r t t o s p e c i f y the o r g a n i z a t i o n a l consumers o f such r e s e a r c h (program f u n d e r s , managers and s t a f f , p o l i c y m a k e r s and o t h e r i n t e r e s t e d p a r t i e s ) . Used as a means t o make t h e o r e t i c a l l y grounded o b s e r v a t i o n s , the n e g o t i a t e d o r d e r p a r a -digm helps to c h a r a c t e r i z e the n e g o t i a t i o n c o n t e x t o f a s p e c i f i c program e v a l u a t i o n in r e l a t i o n t o the o v e r a l l s t r u c t u r a l context o f the p o l i c y i n t e r -v e n t i o n approach which s e r v e d to guide program development and implementation. Second, i t o f f e r s a framework f o r a s s e s s i n g what has been o m i t t e d o r g l o s s e d o v e r in e x i s t i n g models o f program e v a l u a t i o n r e s e a r c h which tend t o f o c u s 26 on s e l e c t e d p r o p e r t i e s o f the n e g o t i a t i o n c o n t e x t and s c r e e n out elements in t h e s t r u c t u r a l c o n t e x t which may o f f e r r e s o u r c e s / c o n s t r a i n t s on e v e r y a s p e c t o f the e v a l u a t i o n p r o c e s s . F i n a l l y , i t p e r m i t s us to c o n c e p t u a l i z e each o f the s i x t h e o r e t i c a l / p r a c t i c a l i s s u e s of p o l i c y r e s e a r c h as a nego-t i a t i o n c o n t e x t f o r i n t e r a c t i o n among s e v e r a l n e g o t i a t i n g p a r t i e s . T h i s s u g g e s t s a wide v a r i e t y o f r e s e a r c h q u e s t i o n s which g e n e r a l l y go unasked in the two main models o f program e v a l u a t i o n r e s e a r c h . For example: Under what c i r c u m s t a n c e s might a l l t h e major a c t o r s i n the program e v a l u a t i o n p r o c e s s become a c t i v e l y i n v o l v e d in t h e f i e l d r e s e a r c h and p l a n n i n g f o r r e s e a r c h u t i l i z a t i o n ? Does the r e s e a r c h methodology become a s o u r c e o f c o n f l i c t among the p a r t i e s o n l y a t t h e d e s i g n stage o r can i t re-emerge as an i s s u e s u b s e q u e n t l y , e s p e c i a l l y where r e s e a r c h r e s u l t s a r e d e f i n e d as being " c o u n t e r i n t u i t i v e " by p o l i c y m a k e r s ? How might d i f f e r e n c e s in o v e r a l l r e s e a r c h s t r a t e g y a f f e c t nego-t i a t i o n s among v a r i o u s e v a l u a t i o n teams i n l a r g e s c a l e e v a l u a t i o n s t u d i e s ? Do key a c t o r s i n t h e s t r u c t u r a l c o n t e x t o f the program become i n v o l v e d in n e g o t i a t i o n s r e g a r d i n g r e s e a r c h s t r a t e g y o r sub-sequent u t i l i z a t i o n o f r e s e a r c h f i n d i n g s ? These a r e examples o f t h e k i n d s o f d e t a i l e d r e s e a r c h q u e s t i o n s which p r o v i d e c l u e s t o c o n t e x t u a l d i f f e r e n c e s in the p r o d u c t i o n and use o f p o l i c y -r e l e v a n t r e s e a r c h knowledge. The n e g o t i a t e d o r d e r paradigm o f f e r s a t h e o r e t i c a l framework which encourages c l o s e e m p i r i c a l s c r u t i n y o f both the s t r u c t u r a l and n e g o t i a t i o n c o n t e x t s o f each program e v a l u a t i o n , and c a r e f u l documentation o f the d i s t i n c t i v e w o r l d views h e l d by key p a r t i c i -pants in these n e g o t i a t i o n s . Methodology: Case S t u d i e s Based on Documents, In t e r v i e w s and F i e l d Data In Chapters k and 5 I d e s c r i b e and a n a l y z e two s e t s o f program e v a l u a t i o n case s t u d i e s in terms o f t h e i r s t r u c t u r a l c o n t e x t s , the b a s i c areas o f nego-t i a t i o n and d i f f e r i n g p a t t e r n s o f knowledge p r o d u c t i o n and use which emerged 27 in each s p e c i f i c e v a l u a t i o n c o n t e x t . The f i r s t two s e t s o f case s t u d i e s were e v a l u a t i o n s commissioned by v a r i o u s l e v e l s o f government (and i n one case a community group a l s o ) t o a s s e s s the d e l i v e r y o f s e r v i c e and o v e r a l l program impact d u r i n g a s h i f t i n Canadian housing p o l i c y . The o r i g i n a l i n t e r v e n t i o n t h e o r y had f o c u s e d on t h e goal o f slum c l e a r a n c e , w h i l e the new p o l i c y was o r i e n t e d to a goal o f upgrading the c o n d i t i o n s i n e x i s t i n g housing s t o c k and r e h a b i l i t a t i n g t h e p h y s i c a l f a c i l i t i e s and a m e n i t i e s o f l o c a l neighbourhoods. The i n i t i a l case study p r e s e n t e d i n Chapter k concerns the e v a l u a t i o n o f the S t r a t h c o n a R e h a b i l i t a t i o n P r o j e c t , a h o u s i n g and neighbour-hood r e h a b i l i t a t i o n p i l o t p r o j e c t l o c a t e d i n the S t r a t h c o n a a r e a , a neighbour-hood a d j a c e n t t o the c e n t r a l b u s i n e s s d i s t r i c t i n Vancouver. The SRP, as one o f a s e r i e s o f p i l o t s t u d i e s , was d e s i g n e d t o s e r v e as a model f o r sub-sequent n a t i o n a l l e g i s l a t i o n r e g a r d i n g housing c o n s e r v a t i o n and neighbourhood r e h a b i l i t a t i o n i n c i t i e s and towns a c r o s s Canada. The second case study d i s c u s s e d i n Chapter k concerns an e v a l u a t i o n o f the new n a t i o n a l l e g i s l a t i o n as i t was implemented i n B r i t i s h Columbia, i . e . , e v a l u a t i o n o f the Neighbourhood Improvement Program (NIP) and R e s i d e n t i a l R e h a b i l i t a t i o n A s s i s t a n c e Program (RRAP) as they were d e l i v e r e d i n s e l e c t e d urban neighbour-hoods and r u r a l towns i n B.C. A l t h o u g h the two case s t u d i e s i n v o l v e the same s u b s t a n t i v e p o l i c y a r e a ( h o u s i n g ) , o v e r l a p p i n g time frames, and a s i m i l a r r e s e a r c h team, t h e r e a r e s i g n i f i c a n t d i f f e r e n c e s in the s t r u c t u r a l and n e g o t i a t i o n c o n t e x t s o f the two e v a l u a t i o n s which emerge i n t h e i r d e s c r i p t i o n s and a n a l y s e s in Chapter k. These v a r i a t i o n s o f f e r t h e i n i t i a l i n d i c a t o r s o f the d i f f e r i n g p a t t e r n s o f knowledge p r o d u c t i o n and use in p o l i c y r e s e a r c h s tud i e s . In Chapter 5 I p r e s e n t two a d d i t i o n a l program e v a l u a t i o n case s t u d i e s in o r d e r to extend my d e s c r i p t i o n and a n a l y s i s of t h e program e v a l u a t i o n 28 p r o c e s s . The f i r s t study d e s c r i b e d and a n a l y z e d in t h a t c h a p t e r f o c u s e s on an assessment o f the Vancouver Je w i s h F a m i l y S e r v i c e Agency, a p r i v a t e s o c i a l s e r v i c e agency which i s s u p p o r t e d by f u n d i n g from the U n i t e d Way and the J e w i s h Community Fund and C o u n c i l . The second study concerns the e v a l u a t i o n o f the Legal S e r v i c e s S o c i e t y S c h o o l s Program, an ongoing program f o r the d e l i v e r y o f l e g a l e d u c a t i o n s e r v i c e s and m a t e r i a l s i n t h e B.C. s c h o o l system, supported by f u n d i n g from the P r o v i n c i a l A t t o r n e y G e n e r a l ' s M i n i s t r y but a c c o u n t a b l e a l s o t o the M i n i s t r y o f E d u c a t i o n (which p r o v i d e s i n d i r e c t f u n d i n g through r e l e a s e - t i m e f o r t e a c h e r s , some t r a n s p o r t a t i o n a s s i s t a n c e , e t c . ) . These two e v a l u a t i o n case s t u d i e s d i f f e r from each o t h e r , as w e l l as d i f f e r i n g from the cases p r e s e n t e d in Chapter k, in terms o f s e v e r a l o r g a n i z a t i o n a l / s t r u c t u r a l f a c t o r s , e.g., s u b s t a n t i v e p o l i c y a r e a , s c a l e o f o p e r a t i o n , t h e i r ongoing n a t u r e and l e n g t h o f time in e x i s t e n c e . However, d e s p i t e o v e r t d i f f e r e n c e s in t h e two programs, the case study a n a l y s i s s u g g e s t s important s i m i l a r i t i e s i n t h e s t r u c t u r a l c o n t e x t s o f each program which may have c o n t r i b u t e d t o t h e p a t t e r n o f knowledge p r o d u c t i o n and use o f the e v a l u a t i o n r e s e a r c h m a t e r i a l s . S i n c e I had an ongoing i n t e r e s t i n q u e s t i o n s r e g a r d i n g t h e c o n s t r u c t i o n o f r e s e a r c h documents and the uses o f these r e s e a r c h f i n d i n g s by program f u n d e r s , managers and p o l i c y m a k e r s , I began c o l l e c t i n g f i e l d data on each of the program e v a l u a t i o n s t u d i e s (as w e l l as o t h e r s t u d i e s done by members o f the Ptarmigan P l a n n i n g work group) at the time o f t h e i n i t i a l c o n t a c t w i t h members o f the o r g a n i z a t i o n making t h e r e s e a r c h r e q u e s t . The main method-o l o g i c a l approaches which guided the data c o l l e c t i o n were d e s i g n e d to s e r v e both the s p e c i f i c purpose o f p r o d u c i n g an e v a l u a t i o n document and the more g e n e r a l purpose o f d e s c r i b i n g and a n a l y z i n g t h e p r o c e s s o f document p r o d u c t i o n and use i n p o l i c y s e t t i n g s . Each o f the c o n t r a c t s o r l e t t e r s o f agreement 29 w i t h f u n d i n g o r g a n i z a t i o n s gave p e r m i s s i o n f o r t h e r e s e a r c h data and the r e s u l t i n g document to be used f o r academic purposes. In o r d e r to reach c o n c l u s i o n s which had a r e l a t i v e l y s t r o n g l e v e l o f c r e d i b i l i t y and a u d i t a -b i l i t y I r e l i e d on two main approaches to d a t a c o l l e c t i o n . On the one hand I emphasized the c o l l e c t i o n s o f data from s e v e r a l sources ( f i e l d n o t e s , r e c o r d s , documents, taped i n t e r v i e w s ) i n o r d e r to f o l l o w the p r i n c i p l e o f t r i a n g u l a t i o n and c o r r o b o r a t i o n o f c r i t i c a l o b s e r v a t i o n s and a c t i o n s . Second, s i n c e a l l o f the program e v a l u a t i o n s i n v o l v e d s e v e r a l o t h e r persons working i n r e s e a r c h c a p a c i t i e s I made the r e s e a r c h n e g o t i a t i o n s the t o p i c of u n s t r u c t u r e d i n t e r v i e w s w i t h team members and s e l e c t e d independent o b s e r v e r s o f the r e s e a r c h p r o c e s s . T h e r e was an e x t e n s i v e v a r i e t y o f d a t a s o u r c e s , ranging from p e r s o n a l f i e l d notes and r e c o r d s to p u b l i c p o l i c y documents. These i n c l u d e d my own t a s k / t i m e r e c o r d which i n c l u d e d a day-by-day d e s c r i p t i o n o f the s p e c i f i c r e s e a r c h t a s k s , who was i n v o l v e d in the t a s k s , and how long i t took t o a c c o m p l i s h them. T h i s r e c o r d s e r v e d as a d e t a i l e d a c t i v i t y log as w e l l as the b a s i s f o r keeping t r a c k o f r e s e a r c h c o s t s and b i l l i n g to c l i e n t s . A l s o my f i e l d notes i n c l u d e d o b s e r v a t i o n s and c o n s c i o u s attempts t o a n a l y z e t h e i n t e r a c t i o n o f t h e main a c t o r s engaged i n the e v a l u a t i o n p r o c e s s . S e v e r a l p o r t i o n s o f t h e notes set out d e t a i l s o f the t i m i n g o f a c t i v i t i e s , d e s c r i b e t h e s e t t i n g s o f s p e c i f i c types o f i n t e r a c t i o n , and t h e p a r t i e s i n v o l v e d i n s p e c i f i c o r g a n i z a t i o n a l maneuvers. There are more than one hundred hours o f taped i n t e r v i e w s w i t h the major a c t o r s i n each o f the case s t u d i e s ; most o f these are from the S t r a t h c o n a case study and t h e case study o f t h e LSS S c h o o l s Program. For the o t h e r two case s t u d i e s t h e r e a r e notes from s t r u c t u r e d i n t e r v i e w s w i t h t h e major a c t o r s ( 1 0 from the NIP/RRAP e v a l u a t i o n and 6 from the JFSA s t u d y ) . 30 In a d d i t i o n t h e r e a r e s e v e r a l hours o f taped r e c o r d s o f the meetings o f the working teams in the f o u r case study s e t t i n g . These a r e augmented by r e c o r d s o f a l l c o r r e s p o n d e n c e among the major a c t o r s i n each stu d y , program f i l e s and f i n a n c i a l r e c o r d s , and notes from u n s t r u c t u r e d d i s c u s s i o n s w i t h s e l e c t e d p o l i c y m a k e r s . T h e re a r e a s u b s t a n t i a l number (over 50) o f r e s e a r c h and p o l i c y documents r e l a t e d t o the case s t u d i e s , case study a r e a s / t a r g e t g roups, and the v a r i o u s f u n d i n g o r g a n i z a t i o n s which p r o v i d e d background m a t e r i a l f o r a n a l y z i n g the s t r u c t u r a l c o n t e x t o f t h e case s t u d i e s . A l t h o u g h memory becomes a f a c t o r i n t h i s type o f secondary a n a l y s i s t h e range o f data c o l l e c t e d w h i l e the program e v a l u a t i o n s were in p r o c e s s has p r o v i d e d t h e r e s o u r c e s t o " c r o s s - e x a m i n e " my e x p e r i e n c e o f the p a s t . In a d d i t i o n , d u r i n g t h e f i n a l y e a r o f w r i t i n g and r e - w r i t i n g the d i s s e r t a t i o n I d i d f o c u s e d i n t e r v i e w s w i t h a l l but one member o f the v a r i o u s s t u d y teams and s e v e r a l o f the main a c t o r s which d e a l t g e n e r a l l y w i t h the f i n a l c o n s t r u c -t i o n o f t h e r e s e a r c h document and i t s r e c e p t i o n and subsequent use by the c l i e n t s / c o n s u m e r s who had funded the r e s e a r c h . A g a i n , t h i s was done i n an e f f o r t to expand the "memory-base", and c o l l e c t the r e p o r t s and i n t e r p r e t a -t i o n s o f o t h e r s who had been engaged i n the program e v a l u a t i o n p r o c e s s . In c o n c l u d i n g t h i s s e c t i o n on methodology i t i s important f o r the reader to r e c o g n i z e t h a t I have used the n e g o t i a t e d o r d e r paradigm in an h e u r i s t i c f a s h i o n and do not c l a i m to be p r o d u c i n g d e s c r i p t i v e ethnography i n the s y m b o l i c i n t e r a c t i o n i s t t r a d i t i o n . My m e t h o d o l o g i c a l emphasis has been c e n t e r e d on a s t r a t e g y o f t r i a n g u l a t i o n , r e l y i n g on d a t a from documents, r e c o r d s , tapes and i n t e r v i e w s as w e l l as t h e notes from f i e l d o b s e r v a t i o n . 31 Models o f N e g o t i a t e d Knowledge P r o d u c t i o n and Use The models o f n e g o t i a t e d knowledge p r o d u c t i o n and use developed in Chapters 4 and 5 , and d e s c r i b e d more f u l l y in Chapter 6 are designed to be p r e l i m i n a r y f o r m u l a t i o n s to be e x p l o r e d more s y s t e m a t i c a l l y in f u r t h e r r e s e a r c h . In t h a t f i n a l c h a p t e r I p r e s e n t f o u r p o s s i b l e models, suggested i n the c o u r s e o f the case study a n a l y s e s , which I have l a b e l l e d e x p e r i m e n t a l , m a n a g e r i a l , c o l l a b o r -a t i v e and t r a n s f o r m a t i v e . These l a b e l s have been g i v e n p a r t l y as q u i c k d e s c r i p -t i o n s , but a l s o as r e p r e s e n t i n g some important a s p e c t o f the dominant area o f n e g o t i a t i o n in the e v a l u a t i o n p r o c e s s . For example, i n the e x p e r i m e n t a l model o f knowledge p r o d u c t i o n and use, t h e program e v a l u a t o r takes the f a m i l i a r con-se n s u a l approach to o v e r a l l r e s e a r c h s t r a t e g y , s h a r i n g w i t h the major a c t o r s ( u s u a l l y l i m i t e d t o r e p r e s e n t a t i v e s o f the f u n d i n g agency and management of the program) a p o s i t i v i s t model o f e v a l u a t i o n r e s e a r c h and r e l i a n c e on o b j e c t i v e measures o f program impact. A l l the e v a l u a t i o n r e s e a r c h i s d e s i g n e d to be accomplished by e v a l u a t o r s whose work base i s e x t e r n a l t o the program under s c r u t i n y . High t e c h n i c a l s t a n d a r d s a r e important f o r the e v a l u a t i o n r e s e a r c h e r s and c l o s e adherence t o e x p e r i m e n t a l d e s i g n a r e key g o a l s f o r program managers and s t a f f . T h e r e f o r e , the r e s u l t i n g r e s e a r c h r e p o r t i s c o n s t r u c t e d in a nego-t i a t i o n c o n t e x t where c o n f l i c t i s m i n i m i z e d and o b j e c t i v e , numerical e v a l u a t i o n f i n d i n g s a r e g i v e n g r e a t credence. The f i n a l e v a l u a t i o n document i s p r e s e n t e d to the f u n d i n g agency f o r i n t e r p r e t a t i o n and p o s s i b l e f u t u r e use i n the p o l i c y p r o c e s s . T h e re i s minimal e v a l u a t o r input i n t o subsequent phases o f p o l i c y development and program m o d i f i c a t i o n . Another p a t t e r n o f knowledge p r o d u c t i o n and use o f p o l i c y r e s e a r c h can be d e f i n e d as the manageri a 1 model. T h i s i s t h e model f r e q u e n t l y employed in managerial reviews o f ongoing programs o r program components where the e v a l u -a t i o n r e s e a r c h i s done by " i n - h o u s e " r e s e a r c h e r s and the major a c t o r s a r e a l l members o f one o r g a n i z a t i o n a l c o n t e x t (e.g. M i n i s t r y of E d u c a t i o n , C e n t r a l 32 Mortgage and H o u s i n g ) . Here to o the e v a l u a t o r ( s ) employ a consensual r e s e a r c h s t r a t e g y , where consensus i m p l i e s a c c e p t a n c e o f the t h e o r y and p r a c t i c e o f "good management" and i s the main focus o f program assessment. There i s an e x p l i c i t emphasis on the d i s c o v e r y o f the day-to-day f a c t s o f program o p e r a t i o n , c a r e f u l r e c o r d i n g o f i n d i v i d u a l accomplishments and f a i l u r e s , and a steady r e l i a n c e on whatever r e s e a r c h models and methods have been f o l l o w e d by s t a f f in assembling the data r e q u i r e d by program m o n i t o r i n g p r o c e d u r e s . T h i s does not always mean q u a n t i f i c a t i o n of e v e r y t h i n g under s c r u t i n y , but i t does mean a c c e p t i n g the c a t e g o r i e s and l a b e l s g i v e n t o data i n the ongoing p r o c e s s of program management, e.g., s u b s t a n d a r d h o u s i n g , unemployment r a t e s , s c h i z o -p h r e n i c p a t i e n t s , persons i n e l i g i b l e f o r program b e n e f i t s , e t c . There i s an emphasis on s t a t i s t i c a l or o b s e r v a t i o n a l c o n c l u s i o n s which are not " c o u n t e r -i n t u i t i v e " t o the a c c e p t e d wisdom o f the M i n i s t r y / a g e n c y , and the r e s u l t i n g e v a l u a t i o n r e p o r t may f r e q u e n t l y be used as grounds f o r f i r i n g p e r s o n n e l , e l i m i -n a t i n g a p r o j e c t / d e p a r t m e n t , o r a l t e r n a t i v e l y as a means to cover up problems in program d e l i v e r y which have been d e f i n e d d e l i b e r a t e l y as l y i n g o u t s i d e the parameters o f the management review. The t h i r d p a t t e r n developed more f u l l y in Chapter 6 has been c a l l e d the c o l l a b o r a t i v e model. T h i s model may be used i n o r g a n i z a t i o n s which are attemp-t i n g t o make g r a d u a l changes in t h e i r s t r u c t u r e and long term o p e r a t i o n . Thus, programs which have reached some type o f o r g a n i z a t i o n a l c r i s i s p o i n t in terms o f work overload/employee t u r n o v e r / r a p i d l y s h i f t i n g demands f o r program s e r v i c e tend to look f o r a way to e v a l u a t e t h e i r e x i s t i n g s i t u a t i o n and make s y s t e m a t i c improvements. In t h i s model t h e r e i s o f t e n an attempt to i n v o l v e as many a c t o r s o r r e p r e s e n t a t i v e s o f i n t e r e s t e d p a r t i e s as the e v a l u a t i o n p r o c e s s can e f f e c -t i v e l y i n c l u d e . The e v a l u a t i o n i s l o c a t e d both e x t e r n a l l y t o the program ( o f t e n through use o f s p e c i f i c c o n s u l t a n t s ) and s i m u l t a n e o u s l y as an " i n - h o u s e " p r o c e s s o f s e l f - a n a l y s i s and i n t e r n a l r e s e a r c h . Given t h a t t h e r e a r e c o n f l i c t i n g 33 t h e o r i e s of i n t e r v e n t i o n r e p r e s e n t e d in most reform programs, the e v a l u a t i o n n e g o t i a t i o n s f r e q u e n t l y c e n t e r on d e v e l o p i n g a r e s e a r c h s t r a t e g y t h a t i s m i n i -m a l l y a c c e p t a b l e t o a l l p a r t i e s . Then the e v a l u a t i o n r e s e a r c h e r s tend to make use of an a r r a y o f models and methods which w i l l be approved and understood by d i f f e r e n t components o f the program's s t a f f and s e r v i c e u s e r s . The f i n a l form o f the e v a l u a t i o n r e p o r t i s developed c o l l e c t i v e l y as a documentary review of program changes in p r o c e s s , i n c l u d i n g m i n o r i t y p o l i c y p o s i t i o n s and r e p o r t s o f program i n a d e q u a c i e s and f a i l u r e s . A l t h o u g h some form o f r e p o r t goes i n t o "document time", most r e s e a r c h r e s u l t s a r e u t i l i z e d immediately by v a r i o u s p a r t i e s t o the e v a l u a t i o n through a p r o c e s s of c o n t i n u a l d i s c u s s i o n and feedback. S i n c e t h i s model tends to focus on a form of c o l l a b o r a t i v e effort,sma11 s c a l e c o n f l i c t s can u s u a l l y be managed w i t h i n the n e g o t i a t i o n p r o c e s s . However, s i g -n i f i c a n t d i f f e r e n c e s in i n t e r v e n t i o n t h e o r y and major d i f f e r e n c e s in power or r e s o u r c e s among the p o t e n t i a l p a r t n e r s can o f t e n produce s e r i o u s o b s t a c l e s to the achievement o f long term o r g a n i z a t i o n a l changes i n program p o l i c y and/or o p e r a t i o n . The f o u r t h model o f knowledge p r o d u c t i o n and use p r e s e n t e d in Chapter 6 has been l a b e l l e d the t r a n s f o r m a t i v e model and may be the l e a s t common approach to p o l i c y r e s e a r c h p r e s e n t l y in use. In t h i s model program e v a l u a t o r s seek a c t i v e support and involvement o f those a c t o r s l e a s t l i k e l y t o be r e p r e s e n t e d in the o t h e r models, i . e . the program p a r t i c i p a n t s and s e r v i c e users/consumers. The e v a l u a t i o n i s s p e c i f i c a l l y d e s i g n e d t o be e x t e r n a l t o the program and at some c o n c e p t u a l d i s t a n c e from the t h e o r i e s and assumptions o f program funders and managers. The r e s e a r c h s t r a t e g y tends t o range from the c o n t e n t i o u s approach t o a p a r a d i g m - c h a l l e n g i n g s t a n c e which c a l l s i n t o q u e s t i o n program p o l i c y , f u n d i n g , s t r u c t u r e and o p e r a t i o n . The c h o i c e o f r e s e a r c h models and methods may v a r y , but where the a n a l y s i s r e l i e s on a p o l i t i c a l economy o r M a r x i s t approach t h e r e may be s i g n i f i c a n t r e l i a n c e on p o s i t i v i s t models and 3h and an emphasis on o b j e c t i v e m e t h o d o l o g i e s . The m e t h o d o l o g i c a l d e t a i l s may depend t o some degree on t h e p a r t i e s n e g o t i a t i n g and the c o n s t i t u e n c y to be a ddressed i n th e e v a l u a t i o n document. F r e q u e n t l y the f i n a l form o f the e v a l u -a t i o n r e p o r t i s developed by the e v a l u a t o r ( s ) and o t h e r community r e p r e s e n t a t i v e s as a document to provoke c r i t i c a l d i s c u s s i o n o f program p o l i c y , o p e r a t i o n and change. It i s s t r u c t u r e d t o emphasize the gaps between program t h e o r y / g o a l s and the l i m i t e d o b j e c t i v e s o f day-to-day program s e r v i c e . Program e v a l u a t o r s and community r e p r e s e n t a t i v e s o f t e n p l a n how r e s e a r c h r e s u l t s may be used to educate program c l i e n t s / c o n s u m e r s and t o r e p u d i a t e s the f i n d i n g s o f program e v a l u a t i o n s which have been done u s i n g a managerial or e x p e r i m e n t a l model o f knowledge p r o d u c t i o n and use. These f o u r models do not exhaust the p o s s i b T n t i e s o f u s i n g the n e g o t i a t e d o r d e r paradigm t o e x p l o r e v a r i a t i o n s in the p o l i c y r e s e a r c h p r o c e s s . Furthermore, t h e r e a r e elements of each model which may be used by v a r i o u s p a r t i e s t o the n e g o t i a t i o n s in any s p e c i f i c program e v a l u a t i o n c o n t e x t . But I have l i m i t e d my d i s c u s s i o n in Chapter 6 to these f o u r because they a r e r . r e f l e c t e d most o b v i o u s l y in the case s t u d i e s in which I have been most e x t e n -s i v e l y i n v o l v e d . In t h e c o n c l u d i n g p o r t i o n o f Chapter 6 I suggest a p l a n f o r "sampling the u n i v e r s e " o f program e v a l u a t i o n s t u d i e s to develop a d d i t i o n a l models of knowledge p r o d u c t i o n and use i n the a r e a o f p o l i c y r e s e a r c h . 35 NOTES TO,CHAPTER 1 1. It i s important t o p o i n t out t h a t program e v a l u a t i o n i s s t i l l "an open employment f r o n t i e r " f o r s o c i o l o g i s t s at a time when the academic market-p l a c e i s s h r i n k i n g q u i t e r a p i d l y . S c r u t i n y o f the " j o b o p e n i n g s " l i s t e d in t r a d i t i o n a l s o u r c e s o f employment i n f o r m a t i o n show t h a t program e v a l u -a t i o n r e p r e s e n t s a major s o u r c e o f non-academic employment f o r s o c i a l s c i e n t i s t s (Eaton I98O). 36 CHAPTER 2: CONTEMPORARY MODELS OF PROGRAM EVALUATION: PARADIGMS AND LIMITATIONS 2.1 Int roductIon T h i s c h a p t e r i s inte n d e d as an overview of the p r i n c i p a l models o f r e s e a r c h t h a t program e v a l u a t o r s c l a i m a r e most f r e q u e n t l y (and most a p p r o p r i a t e l y ) employed i n doing t h e i r work. It i s important t o emphasize t h a t d i s c u s s i o n o f these models, as " i d e a l t y p e s " , t r a n s c e n d s the enormous v a r i a t i o n s in scope, s e t t i n g , and o b j e c t i v e s t h a t a r e c h a r a c t e r i s t i c o f programs as they o p e r a t e i n the f i e l d . T h i s c r i t i c a l review o f two major models o f e v a l u a t i o n r e s e a r c h cen-t e r s on the f i e l d o f e d u c a t i o n where program e v a l u a t i o n has r e c e i v e d e x t e n s i v e a t t e n t i o n i n the development of p u b l i c p o l i c y . The d e s c r i p t i o n and a n a l y s i s o f each model i n c l u d e s d i s c u s s i o n o f i t s main assumptions and d i r e c t i v e s r e g a r d i n g r e s o l u t i o n of the s i x t h e o r e t i c a 1 / p r a c t i c a 1 i s s u e s o f p o l i c y r e s e a r c h . D e s p i t e marked d i f f e r e n c e s in the c o n t e x t in which program e v a l u a t i o n i s p r a c t i c e d , t h e r e tends to be minimal v a r i a t i o n i n the r e s e a r c h models i m p l i c i t in t h a t p r a c t i c e . A n a l y s i s o f the u n d e r l y i n g approach taken in the t e x t s on program e v a l u a t i o n (Rossi and W i l l i a m s 1972; Weiss 1972a;^Caoo•1971; . Rossi Freeman and Wright 1979) r e v e a l s a r a t h e r l i m i t e d s e t o f p r e s c r i p t i o n s f o r a c c o m p l i s h i n g the work o f e v a l u a t i o n . A l t h o u g h many we l l known examples in the l i t e r a t u r e r e f l e c t c e r t a i n i n d i v i d u a l p r e f e r e n c e s (Guttentag and Struening', 1975), t h e r e i s one plan f o r e v a l u a t i o n r e s e a r c h which has tended t o predomi-nat e , i . e . , a c o n v e n t i o n a l view which d e r i v e s b a s i c a l l y from the p o s i t i v i s t paradigm o f s o c i a l s c i e n c e . (Rossi and W i l l i a m s 1972; Boruch and'Riecken 11g75) • In the l a s t decade a n o t h e r view o f e v a l u a t i o n r e s e a r c h has emerged to c h a l l e n g e the p o s i t i v i s t s . T h i s second view d e r i v e s , n a t u r a l l y enough, from an i n t e r -p r e t i v e (sometimes r e f e r r e d to as " a n t h r o p o l o g i c a l " ) paradigm o f s o c i a l s c i e n c e ( P a r l e t t and Hamilton 1976;: R.E. Stake 1974). 37 It s h o u l d be noted t h a t the two d i s t i n c t paradigms, f r e q u e n t l y l a b e l l e d the " q u a n t i t a t i v e " and " q u a l i t a t i v e " o r i e n t a t i o n s to knowledge, have both e v o l v e d most e x t e n s i v e l y w i t h i n e d u c a t i o n a l r e s e a r c h . In an a r t i c l e which reviews and a n a l y z e s t h e r e l a t i o n s among e d u c a t i o n a l r e s e a r c h paradigms^, R i s t (1977:^3) p o i n t s out t h a t when we speak o f q u a n t i t a t i v e o r q u a l i t a t i v e metho-d o l o g i e s : We a r e , i n the f i n a l a n a l y s i s , speaking o f an i n t e r r e l a t e d s e t o f assumptions about the s o c i a l world which are p h i l o s o p h i c a l , i d e o l o g i c a l , and epistemologica1. They encompass more than simply data g a t h e r i n g t e c h n i q u e s ... the s e l e c t i o n o f a p a r t i -c u l a r methodology i s p r o f o u n d l y t h e o r e t i c a l , r e g a r d l e s s o f i t s : r e l a t i v e a v a i l a b i l i t y . Research methods r e p r e s e n t d i f f e r e n t means o f a c t i n g upon the environment. To choose one l i n e o f a c t i o n o v e r and a g a i n s t a n o t h e r i s t o have foregone o t h e r s a v a i l a b l e from a d i f f e r e n t p e r s p e c t i v e and o r i e n t a t i o n . Each method r e v e a l s p e c u l i a r elements o f s y m b o l i c r e a l i t y . And t o a c c e n t u a t e one aspect o f t h a t r e a l i t y v s . a n o t h e r i s t o i n f l u e n c e both o b s e r v a t i o n s and c o n c l u s i o n s (Denzin, 1970:298). A l l knowledge i s s o c i a l . What f o l l o w s in t h i s c h a p t e r i s a b r i e f c r i t i c a l d e s c r i p t i o n o f these models o f e v a l u a t i o n r e s e a r c h as developed in the f i e l d o f e d u c a t i o n a l program-ming. It i s intended to o u t l i n e the c o n n e c t i o n s between: (a) t h e i r under-l y i n g paradigms, and (b) t h e i r more commonly d i s c u s s e d " p r e s c r i p t i o n s " r e g a r d i n g the c o r r e c t forms o f r e s e a r c h d e s i g n and methodology. In a d d i t i o n , I p o i n t out some l i m i t a t i o n s and c r i t i c i s m s o f each approach in a format t h a t r e f l e c t s the b a s i c c o n t r o v e r s i e s which take precedence in n e g o t i a t i o n s between " i d e a l t y p e " e v a l u a t o r s committed t o each o f the two models. 2.2 The A g r i c u l t u r a l - B o t a n y Paradigm and C o n v e n t i o n a l E v a l u a t i o n s The dominant paradigm in e d u c a t i o n a l e v a l u a t i o n i s the " c l a s s i c a l " o r " a g r i c u l t u r a 1 - b o t a n y " approach t o knowledge which makes use of a h y p o t h e t i c o -d e d u c t i v e methodology d e r i v e d from the e x p e r i m e n t a l and p s y c h o m e t r i c t r a d i -t i o n s i n p s y c h o l o g y ( P a r l e t t and Hamilton 1976). The l a b e l " a g r i c u l t u r a l -38 botany" i s d e l i b e r a t e a c c o r d i n g to Par 1ett .(1972). Many o f the s t a t i s t i c a l and e x p e r i m e n t a l t e c h n i q u e s used i n e d u c a t i o n r e s e a r c h were developed o r i g i -n a l l y (e.g. by F i s h e r ) f o r use i n a g r i c u l t u r a l e x p e r i m e n t a t i o n . Most commonly t h i s model o f e v a l u a t i o n i s p r e s e n t e d as an assessment o f the e f f e c t i v e n e s s o f i n n o v a t i o n ( e.g., r e a d i n g program, c l a s s r o o m o r g a n i z a t i o n , c u r r i c u l u m ) by examining whether or not i t has reached c e r t a i n s t a n d a r d s on a predetermined 2 s e t o f c r i t e r i a . The analogy between s t u d e n t s and p l a n t crops becomes r a t h e r , o b v i o u s . That i s , s t u d e n t s a r e g i v e n p r e t e s t s ( s e e d l i n g s a r e weighed o r measured) and then s u b m i t t e d to d i f f e r e n t l e a r n i n g e x p e r i e n c e s (treatment c o n d i t i o n s ) . A f t e r a p e r i o d o f time has e l a p s e d , t h e i r a t t a i n m e n t (growth or y i e l d ) i s measured to i n d i c a t e the r e l a t i v e e f f i c i e n c y o f the methods ( f e r t i -l i z e r s ) u s e d . ( P a r l e t t and Hamilton 1976). S t u d i e s o f t h i s k i n d a r e d e s i g n e d to produce d a t a o f one p a r t i c u l a r t y p e , i . e . , " o b j e c t i v e " n u merical data t h a t permit the use o f s t a t i s t i c a l a n a l y s i s . I s o l a t e d v a r i a b l e s l i k e IQ, s o c i a l c l a s s , t e s t s c o r e s , p e r s o n a l i t y p r o f i l e s , and a t t i t u d e r a t i n g s a r e c o d i f i e d and p r o c e s s e d t o i n d i c a t e the e f f i c i e n c y o f new c u r r i c u l a , media o r methods ( L i g h t and Smith 1970). T h i s mainstream paradigm o f knowledge p r o d u c t i o n in i t s more " p r e s c r i p -t i v e " form as a model o f e v a l u a t i o n r e s e a r c h dominates much o f the massive two volumes (1400 pages) Handbook o f E v a l u a t i o n Research ( G u t t e n t a g and S t r u e n i n g 1975). In t h a t t e x t and o t h e r s (Rossi and W i l l i a m s 1972), e v a l u a t i o n i s p r e s e n t e d as b a s i c a l l y a s c i e n t i f i c o r t e c h n o l o g i c a l t a s k . The work o f the i m p a r t i a l , h i g h l y s k i l l e d i n v e s t i g a t o r i s t o p r o v i d e a M f i r m assessment" of a program o r i n s t i t u t i o n ( R o s s i , Freeman and Wright 1979:21). T h e r e f o r e the e v a l u a t o r e n t e r s the s i t u a t i o n a f t e r the i n i t i a l e vents in the l i f e o f a pro-gram have taken p l a c e , works a l o n g s i d e the program f o r a time making o b s e r v a -t i o n s and c o l l e c t i n g d a t a , and c o n c l u d e s her e f f o r t s in a r e p o r t t o an a l l -powerful d e c i s i o n maker ( u s u a l l y l o c a t e d o u t s i d e the program under c o n s i d e r a t i o n ) . 39 The p r e s c r i b e d format f o r d e s i g n and methodology in t h i s model o f e v a l u -a t i v e r e s e a r c h can be c o n c e p t u a l i z e d as a s e r i e s o f e i g h t b a s i c s t e p s (The S t a n f o r d E v a l u a t i o n Consortium 1976) : 1. Pr o p o s a l f o r S e r v i c e s : An agency o r i n d i v i d u a l proposes a new s e r v i c e o r changes in e x i s t i n g s e r v i c e s . 2. Demand f o r E v a l u a t i o n : As p a r t o f a p r o c e s s o f g a t h e r i n g sup-p o r t f o r a p r o p o s a l , or as an o r d i n a r y management t a c t i c , an eva1uation - i s r e q u i r e d . Some person or work team, i n s i d e o r o u t s i d e o f the o p e r a t i n g program, becomes the e v a l u a t o r ( E ) . Issues c o n c e r n i n g c o n t i n u a t i o n , m o d i f i c a t i o n , o r expansion o f program s e r v i c e s a r e p l a c e d on the f u t u r e agenda o f a d e c i s i o n -maker (D) who i s supposed t o use the " o b j e c t i v e f a c t s " which • w i l l be p r o v i d e d by E. 3. Program R e a l i z a t i o n : Many persons under agency s u p e r v i s i o n and a c c o r d i n g to p l a n n i n g g u i d e l i n e s t u r n the o r i g i n a l p r o p o s a l i n t o a r e a l program. Short o r long term a p p r o v a l i s r e c e i v e d and the e v a l u a t o r e n t e r s the scene j u s t as t h i s s t a g e i s com-p l e t e d . The e v a l u a t o r has had l i t t l e o r no input in the d e s i g n i n g o f the program i t s e l f . h. E Formulates Q u e s t i o n s : Once E has been f o r m a l l y h i r e d o r a s s i g n e d to the task o f e v a l u a t i o n , she takes the f i r s t conven-t i o n a l s t e p s o f p o s i t i v i s t s c i e n t i f i c method s t a t i n g the q u e s t i o n s t o be answered o r hypotheses t o be t e s t e d . Some w r i t e r s say t h a t t o f o r m u l a t e these q u e s t i o n s p r o p e r l y E must c o n s t r u c t a c o n c e p t u a l model o f program o p e r a t i o n s and e f f e c t s (Riecken and Boruch 1974). 5. D e l i v e r y : The d e c i s i o n s about r e s e a r c h s t r a t e g y must be made at t h i s p o i n t in t h e l i f e o f the program. Conceptual f a c t o r s a r e c e n t r a l t o t h i s work, but E must a l s o take i n t o account p r a c t i c a l , p o l i t i c a l and e t h i c a l f a c t o r s . There a r e two a l t e r -n a t i v e d e s i g n s t r a t e g i e s : a) E Sets Assignment Ru l e s : In some s e t t i n g s E may determine who w i l l r e c e i v e a p a r t i c u l a r treatment and who w i l l r e c e i v e e i t h e r no treatment or a c o n t r a s t i n g t r e a t m e n t . She then c o n f i n e s her a t t e n t i o n t o making sure t h a t s u b j e c t s are a s s i g n e d t o treatments a c c o r d i n g to her d e s i g n . T h i s i s s o c i a l e x p e r i m e n t a t i o n s t r i c t l y d e f i n e d . b) E Records P a r t i c i p a t i o n : F r e q u e n t l y E cannot d i c t a t e t r e a t -ment assignments. For example, she i s h i r e d o n l y a f t e r some communities have chosen t o p a r t i c i p a t e in a new E n g l i s h Language program, o r a sc h o o l system has adopted a new s e r -v i c e , o r some f a m i l i e s (not o t h e r s ) have chosen to p a r t i c i -pate in a new program f o r p r e - s c h o o l e r s . In these c i r c u m -s t a n c e s most " d e s i g n " o p t i o n s are r e s t r i c t e d . E's task i s to c o l l e c t d a t a on the program as i t stands she rec o r d s who e n t e r s each treatment group, o r who i s in each group at ko 5. b) (Continued) the p o i n t o f her a r r i v a l , and background data on a l l p a r -t i c i p a n t s . She i s f r e e t o s p e c u l a t e about program p o s s i -b i l i t i e s not i n e f f e c t and t o c o l l e c t any d a t a t h a t appears r e l e v a n t t o such p o s s i b i l i t i e s . 6. E Measures Outcomes: The next s t e p i s to measure the impact or outcome o f the program a c c o r d i n g to the predetermined c r i -t e r i a o f e f f e c t i v e n e s s . Has p a r t i c i p a t i o n in P r o j e c t H e a d s t a r t i n c r e a s e d t h e r e a d i n g and v o c a b u l a r y l e v e l s o f c h i l d r e n in grades 1 - 3? What are the d i f f e r e n c e s in achievement t e s t s c o r e s among grade 8 s t u d e n t s who take math c l a s s e s and those who f o l l o w a s e l f - p a c i n g program? E may choose to d e s i g n and/ or a d m i n i s t e r measurement instru m e n t s of her own o r may r e l y e x t e n s i v e l y on secondary s o u r c e s , r e p l i c a t i o n s o f e x i s t i n g methods. 7. E P r o c e s s e s the Data: T h i s i s a s t e p t h a t r e q u i r e s a h i g h l e v e l o f t e c h n i c a l e x p e r t i s e . While the d e s i g n and measure-ment phases n e c e s s i t a t e complex r e s e a r c h s k i l l s , the data p r o c e s s i n g phase i s the area where v i r t u o s o s k i l l w i t h computer s o f t w a r e and s t a t i s t i c a l t o o l s i s s t r o n g l y v a l u e d i n t h i s r e s e a r c h model . 8. E Reports Her C o n c l u s i o n s t o D: The work of e v a l u a t i o n has been done. If i t has been done w e l l , the d e c i s i o n maker i s p r e s e n t e d w i t h the e v i d e n c e she r e q u i r e s to proceed r a t i o n a l l y w i t h f u t u r e program o p e r a t i o n . If the program f a i l s t o produce the promised b e n e f i t s , t o an a c c e p t a b l e l e v e l , then i t is t e r m i n a t e d to save r e s o u r c e s . In g e n e r a l , the program whose b e n e f i t s have o u t -weighed i t s c o s t s i s m a i n t a i n e d o r expanded. These e i g h t s t e p s o u t l i n e a p r o c e s s f o r e v a l u a t i v e r e s e a r c h which i s con-s i s t e n t w i t h the t h e o r e t i c a l and m e t h o d o l o g i c a l requirements o f the p o s i t i v i s t paradigm. The r e s u l t i n g product i s an e v a l u a t i o n document which i s p r e s e n t e d as an o b j e c t i v e , unbiased assessment o f the program under s c r u t i n y . When handed over t o the d e c i s i o n m a k e r ( s ) , the e v a l u a t i o n r e p o r t e n t e r s "document time" (Smith 1973) where much o f what has gone i n t o t h e making o f t h a t e v a l u -a t i o n i s o b l i t e r a t e d . T h i s f u r t h e r enhances the c l a i m o f o b j e c t i v i t y and u s u a l l y makes i t i m p o s s i b l e f o r o t h e r s (e.g. t e a c h e r s , p r i n c i p a l s , p a r e n t s , members o f the media, o t h e r s o c i a l s c i e n t i s t s ) t o d i s c o v e r the assumptions and i n t e r p r e t i v e work which went i n t o p r o d u c t i o n of the e v a l u a t i o n document. F i n a l l y , i t i s d i f f i c u l t t o g r a s p the n a t u r e of the management p r a c t i c e s o f the program under e v a l u a t i o n , and f r e q u e n t l y i m p o s s i b l e t o o b t a i n any u s e f u l 41 i n d i c a t i o n o f the s t r u c t u r a l o r n e g o t i a t i o n c o n t e x t i n which the e v a l u a t i o n document was produced. T h i s i s the s i t u a t i o n one f a c e s i n r e a d i n g a program e v a l u a t i o n r e p o r t c o n s t r u c t e d in the " p u r e s t " form o f the p o s i t i v i s t t r a d i t i o n . D e s p i t e these l i m i t a t i o n s i n t h e p r e s e n t a t i o n o f i n f o r m a t i o n r e g a r d i n g the c o n t e x t o f program implementation and s e r v i c e d e l i v e r y , t h i s model o f program e v a l u a t i o n i s o f t e n p r e s e n t e d as t h e s t a n d a r d a g a i n s t which a l l work in the f i e l d s h o u l d be measured. Its s t r e n g t h s are d e f i n e d i n terms of the o b j e c t i -v i t y o f i t s methodology and the ease w i t h which one can g e n e r a l i z e from s p e c i f i c r e s u l t s i n a sample o f cases to the l a r g e u n i v e r s e o f s i m i l a r " t r e a t m e n t s " in s i m i l a r s c h o o l s and c l a s s r o o m s e t t i n g s . P a r t i c u l a r l y in the e v a l u a t i o n of h e a v i l y funded, l a r g e - s c a l e n a t i o n a l programs, the p o l i c y m a k e r s and e v a l u a t o r s tend to depend almost e x c l u s i v e l y on t h i s p o s i t i v i s t model f o r making t h e i r o v e r a l l assessment o f program impact. 2.3 The S o c i a l A n t h r o p o l o g y Paradigm and I l l u m i n a t i v e E v a l u a t i o n A g a i n r e f e r r i n g back t o the a r e a o f e d u c a t i o n a l r e s e a r c h we f i n d t h e c l e a r e s t attempt to d e velop a l t e r n a t i v e approaches t o program e v a l u a t i o n . In t h i s c o n t e x t i t i s important t o r e c o g n i z e t h a t we are not d e a l i n g w i t h a s i t u a t i o n o f p a r i t y among the v a r i o u s r e s e a r c h methodologies and models o f e v a l u a t i o n ( R i s t 1977). In e d u c a t i o n a l r e s e a r c h the a g r i c u l t u r a l - b o t a n y paradigm and q u a n t i t a t i v e methods a r e dominant. T h i s approach i s more w i d e l y p u b l i s h e d , a c c e p t e d , taught and rewarded because o f a c o n t i n u i n g b e l i e f t h a t t h i s i s the type o f r e s e a r c h t h a t c o n s t i t u t e s s c i e n t i f i c knowledge. However, c o n f l i c t i n g i n t e r v e n t i o n t h e o r i e s in e d u c a t i o n and i n c r e a s e d demands f o r c o n t r o l by t h e consumers o f e d u c a t i o n a l programs have encouraged the e v o l u t i o n o f another model o f program e v a l u a t i o n i n t h e f i e l d o f e d u c a t i o n a l r e s e a r c h . hi There a r e v a r i a t i o n s in d e t a i l s o f the a l t e r n a t i v e view o f e v a l u a t i o n r e s e a r c h ; however, t h e e p i s t e m o l o g i c a l s o u r c e l i e s i n the id e a o f an i n t e r -p r e t i v e s o c i a l s c i e n c e and a paradigm o f s o c i a l s c i e n c e which i s common to p a r t s o f s o c i o l o g y and s o c i a l a n t h r o p o l o g y (Fay 1975; Giddens 1976; P a r l e t t and Hamilton 1976). The primary concern o f t h i s paradigm i s w i t h d e s c r i p t i o n and i n t e r p r e t a t i o n r a t h e r than measurement and p r e d i c t i o n . As a model o f e v a l u -a t i v e r e s e a r c h i t takes account o f the wid e r c o n t e x t s i n which i n n o v a t i v e p r o -grams f u n c t i o n . The approach d e s c r i b e d as " I l l u m i n a t i v e E v a l u a t i o n " ( P a r l e t t and Hamilton 1976) p r o v i d e s an example^ o f one o f the major a l t e r n a t i v e s t o t r a d i t i o n a l models o f program e v a l u a t i o n . The aims o f i l l u m i n a t i v e e v a l u a t i o n a r e t o study the i n n o v a t i v e program i n terms o f ( P a r l e t t and Hami1 ton:1kh): How i t o p e r a t e s . How i t i s i n f l u e n c e d by v a r i o u s school s i t u a t i o n s i n which i t i s a p p l i e d . What those d i r e c t l y concerned reg a r d as i t s advantages and d i s a d v a n t a g e s . How students:' i n t e l l e c t u a l t a s k s and academic e x p e r i e n c e s a r e most a f f e c t e d . In the p r o c e s s o f e v a l u a t i n g a program the main purpose i s " t o d i s c o v e r and document what i t i s l i k e to be p a r t i c i p a t i n g in t h e scheme, whether as t e a c h e r o r p u p i l , and, in a d d i t i o n , t o d i s c e r n and d i s c u s s the i n n o v a t i o n ' s most s i g n i -f i c a n t f e a t u r e s , r e c u r r i n g c o n c o m i t a n t s , and c r i t i c a l p r o c e s s e s " ( P a r l e t t and Hami1 ton .1976) . The model seeks t o address and i l l u m i n a t e a complex a r r a y o f q u e s t i o n s which d i f f e r q u i t e c o n s i d e r a b l y from the " t r e a t m e n t " and "impact" q u e s t i o n s which a r e c e n t r a l t o the t r a d i t i o n a l model o f e v a l u a t i v e r e s e a r c h . It i s c l a i m e d t h a t ( P a r l e t t and Hami 1 ton 19-76) . ''Research on i n n o v a t i o n can be e n l i g h t e n i n g t o the i n n o v a t o r and to the whole academic community by c l a r i f y i n g the p r o c e s s e s o f e d u c a t i o n and by h e l p i n g the i n n o v a t o r and o t h e r i n t e r e s t e d p a r -t i e s to i d e n t i f y those p r o c e d u r e s , those elements in the educa-t i o n a l e f f o r t , which seem t o have had d e s i r a b l e r e s u l t s . ' 43 I l l u m i n a t i v e e v a l u a t i o n i s not p r e s e n t e d as a s t a n d a r d m e t h o d o l o g i c a l package but as a g e n e r a l r e s e a r c h s t r a t e g y . It aims to be both a d a p t a b l e and e c l e c t i c . The c h o i c e o f r e s e a r c h t a c t i c s f o l l o w s from d e c i s i o n s i n each s p e c i -f i c case as to the best a v a i l a b l e t e c h n i q u e s , not from a g e n e r a l i z e d r e s e a r c h d o c t r i n e . "The problem d e f i n e s the methods used, not v i c e v e r s a " ( P a r l e t t and Hami 1 ton,1976) . Given the l i m i t a t i o n s o f d i f f e r e n t methods, no method i s used e x c l u s i v e l y o r i n i s o l a t i o n - - d i f f e r e n t t e c h n i q u e s a r e combined to throw l i g h t on common problems and c r o s s - c h e c k t e n t a t i v e f i n d i n g s through a p r o c e s s o f " t r i a n g u l a t i o n " ^ . T h i s s h i f t in paradigm e n t a i l s more than an exchange o f m e t h o d o l o g i e s ; i t i n v o l v e s new s u p p o s i t i o n s , concepts and t e r m i n o l o g y . At the o u t s e t the e v a l u a t o r i s concerned w i t h f a m i l i a r i z i n g h e r s e l f w i t h the day-to-day r e a l i t y o f the s e t t i n g ( s ) she i s s t u d y i n g , f o c u s i n g in p a r t i c u l a r on the " i n s t r u c t i o n a l system" and the " l e a r n i n g m i l i e u " . These two concepts are c e n t r a l to an under-s t a n d i n g o f i l l u m i n a t i v e e v a l u a t i o n . The " i n s t r u c t i o n a l system" r e f e r s t o those f o r m a l i z e d p l a n s and statements which r e l a t e t o p a r t i c u l a r t e a c h i n g arrangements. Each o f t h ese p l a n s / s t a t e m e n t s c o n s t i t u t e s o r d e f i n e s an i n s t r u c -t i o n a l system and i n c l u d e s such elements as a s e t o f pedagogic assumptions, a new s y l l a b u s and d e t a i l s o f t e c h n i q u e s and equipment. The t r a d i t i o n a l e v a l u a t o r b u i l d s her study around i n n o v a t i o n s d e f i n e d in these formal p l a n s , e x t r a c t i n g program g o a l s , o b j e c t i v e s and outcomes. From t h e s e , in t u r n , she d e r i v e s the t e s t s , a t t i t u d e and s k i l l i n v e n t o r i e s t h a t she w i l l a d m i n i s t e r i n o r d e r to e v a l u a t e the " e f f e c t s " o f the " t r e a t m e n t " . T h i s t e c h n o l o g i c a l approach f a i l s t o r e c o g n i z e , o r i g n o r e s , t h e f a c t t h a t i n s t r u c t i o n a l systems when adopted undergo s i g n i f i c a n t m o d i f i c a t i o n by t e a c h e r s , a d m i n i s t r a t o r s and s t u d e n t s as they i n t e r p r e t and r e i n t e r p r e t the i n s t r u c t i o n a l system f o r t h e i r p a r t i c u l a r s e t t i ng. The swi tch from d i s cu s s i ng i n s t r u c t i o n a l systems in t h e i r ab s t r ac t form (as in convent iona l e v a l u a t i on research) to de s c r i b i n g the d e t a i l s of i t s implemen-t a t i o n in s p e c i f i c s e t t i n g s requ i res the second concept, the " l e a r n i n g m i l i e u " . Th is i s the soc ia1 -psycholog ica1 and mate r i a l environment in which students and teachers work together . It represents " a network or nexus of c u l t u r a l , s o c i a l , i n s t i t u t i o n a l and p sycho log i ca l v a r i a b l e s " ( P a r l e t t and Hamilton 1976). The c on f i g u r a t i on s o f each lea rn ing m i l i e u , i t s d i v e r s i t y and complex i ty , the g i n t e r a c t i o n of var ious f a c t o r s - -acknowledging a l l these i s cons idered an e s s e n t i a l p r e r e q u i s i t e f o r the se r ious study of educat iona l programs in t h i s a l t e r n a t e approach to e v a l u a t i o n . In the course of doing i l l u m i n a t i v e e v a l u a - ' t i o n the researcher makes no attempt to manipu late, con t r o l or e l i m i n a t e •-s i t u a t i o n a l v a r i a b l e s . She takes the complex scene as g iven arid her main r e s p o n s i b i l i t y i s to " un rave l i t , i s o l a t e i t s s i g n i f i c a n t f e a t u r e s , d e l i n e a t e c yc l e s of cause and e f f e c t , and comprehend r e l a t i o n s h i p s between b e l i e f s and p r a c t i c e s and between o r g a n i z a t i o n a l pat terns and the responses o f i n d i v i d u a l s " J P a r l e t t and Hamilton 1976).. C h a r a c t e r i s t i c a l l y there are three stages in the i l l u m i n a t i v e e v a l u a t i o n . These steps over lap and are f u n c t i o n a l l y i n t e r r e l a t e d , but can be presented as concep tua l l y d i s t i n c t . The i n v e s t i g a t o r s observe, i nqu i r e f u r t h e r and then seek to e x p l a i n . The obse rva t i on phase occupies a c e n t r a l p lace in i l l u m i n a t i v e e v a l u a t i o n . The i n v e s t i g a t o r bu i l d s up a continuous r e c o r d ' o f ongoing events , t r a n s a c t i o n a l and informal remarks