Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Understanding the dynamics of stakeholder participation in evaluation research : A document study Alex, Jacqueline Patricia Smit 1995

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_1995-0316.pdf [ 6.78MB ]
Metadata
JSON: 831-1.0086688.json
JSON-LD: 831-1.0086688-ld.json
RDF/XML (Pretty): 831-1.0086688-rdf.xml
RDF/JSON: 831-1.0086688-rdf.json
Turtle: 831-1.0086688-turtle.txt
N-Triples: 831-1.0086688-rdf-ntriples.txt
Original Record: 831-1.0086688-source.json
Full Text
831-1.0086688-fulltext.txt
Citation
831-1.0086688.ris

Full Text

UNDERSTANDING THE DYNAMICS OF STAKEHOLDER PARTICIPATION EVALUATION RESEARCH:  A DOCUMENT STUDY  by JACQUELINE PATRICIA SMIT ALEX B.S.W., Dordt C o l l e g e , 1987 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER IN SOCIAL WORK in THE FACULTY OF GRADUATE STUDIES (School o f S o c i a l Work)  We accept t h i s t h e s i s as conforming to the r e q u i r e d  standard  THE UNIVERSITY OF BRITISH COLUMBIA May 1995 ®Jacgueline Smit Alex, 1995  In  presenting  this  degree at the  thesis  in  University of  freely available for reference copying  of  department  this or  thesis for by  his  or  partial  fulfilment  of  the  requirements  British Columbia, I agree that the and study. I further  representatives.  an  It  is  granted  advanced  Library shall make  agree that permission for  scholarly purposes may be her  for  by the  understood  that  it  extensive  head of copying  my or  publication of this thesis for financial gain shall not be allowed without my written permission.  Department  of  QpC/AU  The University of British Columbia Vancouver, Canada  DE-6 (2/88)  1 1  ABSTRACT  Stakeholder p a r t i c i p a t i o n i n evaluation  has  become  i n c r e a s i n g l y acceptable i n evaluation  p r a c t i c e as  s t r i v e to produce r e l e v a n t  studies.  and  useful  e v a l u a t o r u n d e r s t a n d i n g of the o p e r a t i o n a l  researchers  However,  dynamics of  participation i s limited. This  exploratory  d e s c r i p t i v e study q u a l i t a t i v e l y examined  twelve documents comprising r e t r o s p e c t i v e  reviews or case  of p a r t i c i p a t o r y attempts f o r the purpose of i d e n t i f y i n g factors that contribute The  studies the  to e f f e c t i v e s t a k e h o l d e r p a r t i c i p a t i o n .  study e x p l o r e d the p a r t i c i p a t o r y dynamics, the mechanisms by  which p a r t i c i p a t i o n i s o p e r a t i o n a l i z e d , dynamics between r e s e a r c h e r and p r a c t i c e were s i g n i f i c a n t :  and  the r e l a t i o n a l  stakeholder.  Four f a c t o r s  for  the p a r t i c i p a t o r y p r o c e s s should  r e i n f o r c e the genuineness of s t a k e h o l d e r p a r t i c i p a t i o n , p a r t i c i p a t i o n i s enabled by a t t e n t i o n successful  to group p r o c e s s ,  p a r t i c i p a t i o n c u l t i v a t e s a sense of ownership,  s t a k e h o l d e r involvement c r e a t e s r e a d y - f o r - u s e c o n d i t i o n s  and among  participants. Although c o l l a b o r a t i o n was the b e n e f i t s  to e v a l u a t i o n s are  involve multiple  a complex p r o c e s s ,  significant.  E v a l u a t i o n s which  s t a k e h o l d e r s are more l i k e l y to be  l o c a l c o n t e x t s and The  shown to be  r e s u l t i n increased  v a l i d i t y and  p a r t i c i p a t i o n of s t a k e h o l d e r s i n e v a l u a t i o n  implications  has  sensitive  utilization. important  f o r the e v a l u a t o r s r o l e i n f a c i l i t a t i n g the 1  to  group  I l l  process and a t t e n d i n g t o v a r i e d  stakeholder learning  styles.  E x t e n s i v e p l a n n i n g and p r e p a r a t i o n by the r e s e a r c h e r impacts the meaningfulness o f p a r t i c i p a t i o n involvement especially  f o r stakeholders.  Beneficiary  was shown t o p r e s e n t unique c h a l l e n g e s t o e v a l u a t o r s , i n the areas o f access  to i n f o r m a t i o n and e q u i t y o f  participation. Stakeholder implications  p a r t i c i p a t i o n i n e v a l u a t i o n has tremendous  f o r s o c i a l p o l i c y and s o c i a l work p r a c t i c e .  Elements o f a u t h e n t i c p a r t i c i p a t i o n i n e v a l u a t i o n correspond t o f a c t o r s which l e g i t i m i z e p a r t i c i p a t i o n i n p o l i c y s e t t i n g s . redefining service  t h e i r r o l e s and s k i l l s ,  recipients  s o c i a l workers can enable  t o have r e a l impact i n e v a l u a t i o n s , i n p o l i c y  p l a n n i n g and program development, and i n t h e i r own plans.  By  treatment  iv TABLE OF CONTENTS Abstract  i i  Table o f Contents  iv  L i s t of Tables  vi  Acknowledgements  v i i  Introduction  1  Chapter 1 L i t e r a t u r e Review  3  Program E v a l u a t i o n Utilization  3 6  Stakeholders F o c u s i n g E v a l u a t i o n Questions Beneficiaries  9 11 13  Paradigm Approaches t o Stakeholder P a r t i c i p a t i o n P a r t i c i p a t o r y Research F e m i n i s t Research Fourth Generation Evaluation  16 16 18 19  The S t a k e h o l d e r Approach: C i t i e s i n Schools Push/Excel  20 22 24  Two C o n t r o v e r s i a l Attempts  Collaborative Evaluation E v a l u a t o r A t t i t u d e s Concerning Implementation Stakeholder P a r t i c i p a t i o n i n A c t i o n Roles and R e l a t i o n s h i p s E t h i c a l Considerations Relevance t o S o c i a l Work Chapter 2 Methodology  25 Issues 25 28 32 35 39 43  Research Design  43  Sampling Design  44  Sample D e s c r i p t i o n o f Sample Sample Composition  45 45 54  Method o f Data A n a l y s i s  56  V  Chapter 3 R e s u l t s and D i s c u s s i o n R e i n f o r c i n g the Genuineness of P a r t i c i p a t i o n I n t e r a c t i v e Communication I t e r a t i v e l y S t r u c t u r e d Feedback V i s i b l e Results Shared D e c i s i o n Making  59 60 60 66 71 73  Enabling P a r t i c i p a t i o n F a c i l i t a t i n g the E v a l u a t i o n F a c i l i t a t i n g the Group Process  78 78 83  S u c c e s s f u l P a r t i c i p a t i o n C u l t i v a t e s Ownership Hands On P a r t i c i p a t i o n Stakeholder P e r c e p t i o n s of an " I n s i d e Job" Tailor-Made E v a l u a t i o n  90 91 92 96  C r e a t i n g a Readiness f o r Use  97  Summary  101  Chapter 4 I m p l i c a t i o n s  f o r P r a c t i c e and C o n c l u s i o n s  Implications for Evaluation Practice A Strategy for Partnership O r g a n i z a t i o n a l Support The Role of the Expert E v a l u a t o r Group Process S k i l l s Responsiveness of Communication L i m i t a t i o n s of Stakeholder P a r t i c i p a t i o n i n E v a l u a t i o n E x t e n s i v e P r e p a r a t i o n and Planning C o l l a b o r a t i o n with B e n e f i c i a r i e s Inadequate R e p r e s e n t a t i o n of B e n e f i c i a r i e s Access to Information E q u i t y of P a r t i c i p a t i o n Use of Stakeholder Feedback Creating Structure V u l n e r a b i l i t y of Researcher  108 108 108 110 111 112 114 116 116 117 120 123 124 127 129 12 9  Implications for P o l i c y Legitimizing Participation Social Service Practice  131 134 137  L i m i t a t i o n s of the Study  139  Future  141  Research P o s s i b i l i t i e s  Conclusions Bibliography  143 147  vi LIST OF TABLES  Table 1  Descriptive  C h a r a c t e r i s t i c s of Sample  Table 2  Main Themes and Subthemes  53 61  vii ACKNOWLEDGMENTS I wish t o thank my t h e s i s committee, R i c h a r d S u l l i v a n and C h r i s McNiven, and e x t e r n a l reader, Mary R u s s e l l , f o r t h e i r support and v a l u a b l e i n p u t i n t h i s document. I w i l l never f o r g e t t e a c h i n g you how to use a computer, C h r i s , and I hope our chats on s o l v i n g Canada's s o c i a l w e l f a r e problems don't end here. Thanks t o John Crane f o r s p u r r i n g my i n t e r e s t i n program e v a l u a t i o n (I won't t e l l anyone about you h o l d i n g my s e l f - d i r e c t e d r e a d i n g c l a s s e s i n the f a c u l t y b a r i f you won't) . To my undergraduate mentor, F r e d DeJong, thanks f o r b e l i e v i n g i n me and f o r s t a r t i n g me on t h i s journey. I am g r a t e f u l t o my f a m i l y and f r i e n d s who have p r o v i d e d such tremendous support, encouragement, and p r a y e r s d u r i n g these p a s t years. Thanks t o Mom and Dad A l e x f o r t h e i r phone c a l l s and k i n d words o f encouragement. Kathy Smith (I s t i l l can't g e t used t o t h a t ) , my dear s i s t e r , you l e d the way many y e a r s ago. I o n l y wish I c o u l d have been the f i r s t . Thanks t o Monigue and Shan f o r h a v i n g the k i d s so I c o u l d go o f f t o s c h o o l . I hope someday I can "graduate" t o your g r a c i o u s d i s p o s i t i o n s - - s o m e t h i n g my degrees can't g i v e me. I am indebted t o my mom and dad, Wolter and N e l l y Smit, f o r s t r e s s i n g the importance o f e d u c a t i o n and h e l p i n g me pursue my dreams. Your l o v i n g support and encouragement were f a r more v a l u a b l e than p a y i n g t u i t i o n fees. I hope one day you can r e a l i z e your dream too. Daddy. I am a l s o g r a t e f u l t o the Board o f D i r e c t o r s a t the C r i s i s Pregnancy Centre o f Vancouver f o r t h e i r support and c o n f i d e n c e i n me and f o r p r o v i d i n g the f l e x i b i l i t y and time f o r me t o f i n i s h t h i s project. To my dear f r i e n d , M a r i l y n Jeske, thanks f o r b e i n g my p e r s o n a l c h e e r l e a d e r . Your words o f encouragement and f i n e e d i t i n g s k i l l s w i l l never be f o r g o t t e n - - n o r w i l l those i n d e l i b l e words: "Make t h i s a sentence." Don't f o r g e t t h a t I f i r s t saw and coveted one o f those d o c t o r a l hats a t your g r a d u a t i o n . F i n a l l y , t o my husband, Wendell, whom I l u r e d t o Vancouver on a ten-month p r e t e n s e , I know t h i s has been a l o n g and arduous journey f o r you ( j u s t w a i t ' t i l you see what's ahead!). Thanks f o r your many s p e c i a l meals w h i l e I was w r i t i n g my t h e s i s and f o r b e l i e v i n g i n me d u r i n g these p a s t f i v e y e a r s . J a c q u e l i n e Smit  Alex  1 INTRODUCTION  Program e v a l u a t i o n has i n c r e a s i n g l y acknowledged the importance o f s t a k e h o l d e r s i n the e v a l u a t i o n p r o c e s s .  In  response t o c r i t i c i s m s t h a t e v a l u a t i o n r e s e a r c h p r o v i d e s too narrow a focus, answers i n s i g n i f i c a n t q u e s t i o n s , and i s unused, e v a l u a t o r s have expanded t h e i r focus t o i n c l u d e the p a r t i c i p a t i o n of s t a k e h o l d e r groups i n the e v a l u a t i o n p r o c e s s . The l i t e r a t u r e r e v e a l s t h a t although e v a l u a t i o n r e s e a r c h e r s support  a p h i l o s o p h y which maintains  the need f o r m u l t i p l e  s t a k e h o l d e r involvement, many e v a l u a t i o n s c a t e r o n l y t o the needs and wants o f d e c i s i o n makers and/or p o l i c y makers.  Such  e v a l u a t i o n s have been c r i t i c i z e d f o r b e i n g d i v o r c e d from the l e g i t i m a t e i n t e r e s t s o f s e r v i c e p r o v i d e r s and b e n e f i c i a r i e s . Although the r a t i o n a l e f o r the i n c l u s i o n o f m u l t i p l e s t a k e h o l d e r s i s compelling, still  lacking.  c o l l a b o r a t i v e research i n p r a c t i c e i s  T h i s can be a t t r i b u t e d , i n p a r t , t o a p e r v a s i v e  l a c k o f understanding  and awareness o f the nature and dynamics of  the p a r t i c i p a t o r y process  f o r r e s e a r c h e r s who want t o i n c l u d e  stakeholders i n evaluation. s t a k e h o l d e r s continues  C o l l a b o r a t i v e research with multiple  t o p r e s e n t a tremendous c h a l l e n g e t o most  evaluation researchers. The purpose o f t h i s study i s t o i d e n t i f y the f a c t o r s t h a t c o n t r i b u t e t o the e f f e c t i v e p a r t i c i p a t i o n o f s t a k e h o l d e r s i n e v a l u a t i o n r e s e a r c h through c o n s i d e r a t i o n o f the dynamics o f p a r t i c i p a t i o n and the mechanisms by which p a r t i c i p a t i o n i s  2 operationalized.  S p e c i a l c o n s i d e r a t i o n i s g i v e n to the nature of  i n t e r a c t i o n s between r e s e a r c h e r and s t a k e h o l d e r . examination  This  p r o v i d e s e v a l u a t o r s w i t h an understanding  c e n t r a l concepts  of the  of s t a k e h o l d e r p a r t i c i p a t i o n i n c l u d i n g  p r o c e d u r a l dynamics, r e l a t i o n a l i n t e r a c t i o n s , and  the  the c o n t e x t u a l  f a c t o r s which a f f e c t the development and success of the approach. T h i s study i s guided by a s o c i a l j u s t i c e p e r s p e c t i v e toward program e v a l u a t i o n , a conceptual framework espoused by House (1976, 1980)  and r o o t e d i n Rawls'  i s a view t h a t i s concerned  (1971) theory of j u s t i c e .  It  w i t h d e m o c r a t i z i n g e v a l u a t i o n to  ensure t h a t a l l groups are i n c l u d e d and t h a t t h e i r i n t e r e s t s are r e p r e s e n t e d f a i r l y and e q u a l l y . concept  House (1993) r e v e a l s t h a t Rawls'  of e q u a l i t y of o p p o r t u n i t y or "equal r e s p e c t " empowers  p a r t i c i p a n t s to p a r t i c i p a t e e f f e c t i v e l y and m e a n i n g f u l l y i n a democratic  process.  T h i s problem i s important as c u r r e n t e v a l u a t i o n s may than s a t i s f a c t o r y . beneficiaries,  f o r the study of s o c i a l problems  be producing r e s u l t s which are  By e x c l u d i n g c e r t a i n s t a k e h o l d e r s , such  from determining  the s u b s t a n t i v e focus of  e v a l u a t i o n s , the r e l e v a n c y and v a l i d i t y of the i n f o r m a t i o n c o l l e c t e d may  be  less  compromised.  as  3 Chapter 1  LITERATURE REVIEW  Program E v a l u a t i o n  Program e v a l u a t i o n e s c a l a t e d i n the 1960's, f o l l o w i n g the surge o f new s o c i a l programs designed p r i m a r i l y f o r the disadvantaged as d e t e r r e n t s a g a i n s t the n e g a t i v e e f f e c t s o f poverty.  As programs were launched i n such areas as e d u c a t i o n ,  h e a l t h , housing, c r i m i n a l j u s t i c e and w e l f a r e , p o l i t i c i a n s and taxpayers i n s i s t e d on assessment f i n a n c i a l investments. improve  and formal a c c o u n t i n g f o r huge  I f s o c i a l programs were i n t e n d e d to  the w e l f a r e o f i n d i v i d u a l s and s o c i e t i e s ,  they wanted  assurance t h a t funds were b e i n g spent as i n t e n d e d and that they were p r o d u c i n g intended r e s u l t s  (Rubin & Babbie, 1989; Patton,  1987) . F o l l o w i n g the expansion o f s o c i a l programs, program e v a l u a t i o n developed and matured, d i s t a n c i n g i t s e l f  from  s c i e n t i f i c r e s e a r c h and becoming i t s own expert p r o f e s s i o n .  This  l i t e r a t u r e review b r i e f l y o u t l i n e s the o r i g i n s and dynamics o f program e v a l u a t i o n , d i s c u s s e s the impact and importance o f u t i l i z a t i o n i s s u e s on i t s development, and reviews the r o l e o f the s t a k e h o l d e r i n e v a l u a t i o n approaches. are  Many o f the r e f e r e n c e s  taken from l i t e r a t u r e p u b l i s h e d i n the 1980s, when the most  p r o l i f i c d i s c u s s i o n on t h i s t o p i c o c c u r r e d .  4  H i s t o r i c a l l y , e v a l u a t i o n r e s e a r c h was standards as s c i e n t i f i c r e s e a r c h .  judged by the same  However, many r e s e a r c h e r s  and  p r o f e s s i o n a l s i n the f i e l d demanded r e v i s i o n s to the c o n v e n t i o n a l formal e v a l u a t i o n , i n d i c a t i n g t h a t the i s s u e s i n e v a l u a t i n g s o c i a l programs were not b e i n g served by the c u r r e n t s c i e n t i f i c model.  As opposed to s c i e n t i f i c r e s e a r c h , e v a l u a t i o n s are used  to make d e c i s i o n s concerning ongoing funding, to determine whether changes were needed to improve a program's e f f e c t i v e n e s s or e f f i c i e n c y ,  to determine whether a program had reached i t s  intended g o a l s or o b j e c t i v e s , and/or to assess the  long-term  e f f e c t s of a program. S o c i a l researchers Guba, 1989)  (Cronbach,  1982;  Patton, 1986;  Lincoln &  promote the i d e o l o g y t h a t e v a l u a t i o n s s h o u l d p r o v i d e  maximal u s e f u l n e s s to d e c i s i o n s makers and e v a l u a t i o n consumers. They r e g a r d e v a l u a t i o n r e s e a r c h as a means to p r o v i d e a spectrum of i n f o r m a t i o n and  to r a i s e new  q u e s t i o n s and new p e r s p e c t i v e s ,  r a t h e r than simply to determine t h a t something had an  effect.  Cronbach e t a l . (1980) propose t h a t r e s e a r c h s h o u l d i n f o r m us about the s i d e e f f e c t s , not j u s t the " e f f e c t s " , and  should  p r o v i d e a more h o l i s t i c p i c t u r e of d i f f e r e n t outcomes. (1982) concurs w i t h H a s t i n g s ' was  needed was  Cronbach  (1966) e a r l i e r argument t h a t what  data t h a t c o u l d r e v e a l the "why"  of the  results.  Even proponents of the s c i e n t i f i c methodology. Cook & Campbell (1976),  acknowledge t h a t q u e s t i o n s most s u i t e d to the  scientific  b l u e p r i n t are not always q u e s t i o n s t h a t are most r e l e v a n t f o r s o c i a l research.  5 These r e s e a r c h e r s propose t h a t e v a l u a t i o n s t h a t span a broad range of q u e s t i o n s have a b e t t e r chance of b e i n g s t i m u l a t i n g d i s c u s s i o n and p r o v i d i n g d i v e r s e  relevant--by  perspectives--and  that emphasis should be p l a c e d on the r e l e v a n c y of i n f o r m a t i o n not on the form of i n q u i r y . although  conventional  significant effects, Patton  r e s e a r c h may  They i n d i c a t e t h a t  be producing  these e f f e c t s may  the  not be  statistically  substantive.  (1986) suggests t h a t e v a l u a t i o n r e s e a r c h e r s  should be  fair  and c o n s c i e n t i o u s i n t a k i n g account of m u l t i p l e p e r s p e c t i v e s , m u l t i p l e i n t e r e s t s , and m u l t i p l e  realities.  With the r e a l i z a t i o n t h a t s o c i a l program e v a l u a t i o n i s s u b s t a n t i a l l y d i f f e r e n t than b a s i c r e s e a r c h , e v a l u a t i o n i n t o i t s own  s p e c i a l i z e d p r o f e s s i o n (House, 1994)  wide range of approaches and methodologies.  and  evolved  includes a  E v a l u a t o r s produce  i n f o r m a t i o n aimed a t a c t i o n i n order to a f f e c t program d e c i s i o n making and p o l i c y d e c i s i o n s , d i s t a n c i n g themselves from traditional scientific  r e s e a r c h aimed at " t r u t h " .  Patton  (1987)  reports that: E v a l u a t i o n has gone beyond a narrow focus on whether a program has a t t a i n e d i t s g o a l s to encompass a broad a r r a y of q u e s t i o n s , purposes, approaches, models and uses....Users of e v a l u a t i o n have become more demanding and l e s s t o l e r a n t of e v a l u a t i o n s t h a t are o v e r l y academic, i r r e l e v a n t to t h e i r needs, and/or w a s t e f u l of time and r e s o u r c e s , (p. 18) D e f i n e d b r o a d l y as the "systematic  examination of events  o c c u r r i n g i n and consequent on a contemporary program" et a l . , 1980,  p. 14),  s o c i a l program e v a l u a t i o n s are designed  p r o v i d e u s e f u l and a c c u r a t e activities,  (Cronbach  i n f o r m a t i o n about a  to  program--its  c h a r a c t e r i s t i c s , dynamics, and outcomes--in order  to  6 improve the program i n q u e s t i o n or a s s i s t i n decision-making processes r e l a t e d to the program under  review.  Utilization  Many r e s e a r c h e r s develop e v a l u a t i o n s w i t h the s o l e purpose of p r o v i d i n g h i g h - q u a l i t y i n f o r m a t i o n to d e c i s i o n makers.  It i s  assumed t h a t e v a l u a t i o n r e s u l t s w i l l be used i f e v a l u a t i o n i n v e s t i g a t i o n s are shaped around the needs and wants of d e c i s i o n makers. E v a l u a t i o n s which appeal to the p r e f e r e n c e s of d e c i s i o n makers are s t a t e d to be a s s o c i a t e d w i t h h i g h e r l e v e l s of use (Cousins & Leithwood, 1986;  Patton, 1987;  Reineke,  1991).  E v a l u a t i o n u s e r s are encouraged to "be i n v o l v e d i n ways manageable f o r them, i n the p l a n n i n g and c a r r y i n g out of the evaluation. credibility 1986,  Such involvement  seems l i k e l y to ensure  and r e l e v a n c e of the r e s u l t s "  p. 360).  the  (Cousins and Leithwood,  D e c i s i o n makers and program managers are  i d e n t i f i e d as " s t a k e h o l d e r s " i n the e v a l u a t i o n , and e v a l u a t o r s i n c l u d e them as c o l l a b o r a t o r s i n the r e s e a r c h p r o c e s s because they b e l i e v e t h a t t h e i r c o n t i n u e d involvement utilization  (Patton, 1986,  Mowbray, 1988; Patton  1987;  Chelimsky,  G i l l & Zimmerman, 1990;  (1988a),  1987;  enhance Smith,  1988;  Wholey, 1983).  the p i o n e e r i n u t i l i z a t i o n - f o c u s e d r e s e a r c h ,  f i n d s t h a t t h i s approach i s both marketable advocates  will  and a c c o u n t a b l e .  t h a t e v a l u a t i o n s designed f o r d e c i s i o n makers are  He  7 making major impacts and that e v a l u a t i o n s l e v e l are appropriate  performed a t the l o c a l  f o rclient-oriented studies.  Patton"s views, however, a r e c h a l l e n g e d claim that targeting evaluations noticeably increased decisions  by c o l l e a g u e s  who  t o key d e c i s i o n makers has not  the impact o f e v a l u a t i o n s  on program  (Weiss, 1988a; Cronbach, 1982; L e v i n e & Levine,  1977).  Weiss and Cronbach argue that d e c i s i o n s a r e not u s u a l l y determined by a s i n g l e d e c i s i o n maker but by a p l u r a l i s t i c , p o l i c y - m a k i n g community.  They contend t h a t e f f o r t s by e v a l u a t o r s  to make t h e i r f i n d i n g s "user f r i e n d l y " have not r e s u l t e d i n an i n c r e a s e o f the i n f l u e n c e o f e v a l u a t i o n s  on d e c i s i o n makers.  In  f a c t , Weiss suggests t h a t s a t i s f y i n g the immediate c l i e n t may imply i r r e l e v a n c e t o the concerns o f other groups o f p o l i c y actors. Weiss questions  whether e v a l u a t o r s who a r e under p r e s s u r e to  s a t i s f y needs o f c l i e n t s o r i n t e r e s t groups and who a r e responsible  t o e v a l u a t i o n funders r a t h e r than s c i e n t i f i c  peers,  are p l a c i n g too much emphasis on the s a l e a b i l i t y o f e v a l u a t i o n r a t h e r than on i n t e g r i t y . query by s u g g e s t i n g  G i l s i n a n & Volpe  (1986) support  this  that "worry about whether o r how r e s u l t s of  our e f f o r t s g e t used by those i n decision-making p o s i t i o n s r a i s e s the i s s u e o f the e n t r e p r e n e u r i a l r o l e v e r s u s the s c i e n t i s t  role"  (p. 180). Regardless, both Weiss and Patton have been c r i t i c i z e d f o r being  too p r e o c c u p i e d w i t h d e c i s i o n makers t o the n e g l e c t o f the  consumer and the g e n e r a l p u b l i c , e l i c i t i n g s t r o n g responses from  8 many r e s e a r c h e r s (Smith & Chircup, 1987; Stake, 1983b, 1986;  Guba & L i n c o l n , 1989;  L e v i t o n & Hughes, 1981; al.,  1984).  Maclure,  House (1994, 1980)  become a p o l i t i c a l d e c i s i o n makers.  1990;  1980;  Berk & R o s s i ,  Reineke,  1991;  1976;  Zinober e t  i n d i c a t e s t h a t e v a l u a t i o n has  a c t i v i t y - - o n e designed to serve the needs of He c o n s i d e r s f o c u s i n g o n l y on the u s e r ' s  i n t e r e s t as u n j u s t i f i a b l e , unreasonable, Rossi  House, 1994,  (1976) suggest  and immoral.  Berk &  t h a t "to see o n e s e l f as a h i r e d hand and  i n c o r p o r a t e the v a l u e s of employers i n t o r e s e a r c h designs i s to a b d i c a t e one's moral and p o l i t i c a l [stakeholders]"  (p. 328).  r e s p o n s i b i l i t i e s to other  Guba & L i n c o l n  (1989) i n d i c a t e t h a t  r e s e r v i n g power and decision-making a u t h o r i t y to u s e r s i s "not o n l y m o r a l l y and e t h i c a l l y wrong but a l s o p o l i t i c a l l y n a i v e c o n c e p t u a l l y narrow"  and  (p. 15). To deny s t a k e h o l d e r s the  o p p o r t u n i t y to p a r t i c i p a t e and s t r u g g l e w i t h i s s u e s of v a l u e to them i s seen as a d i s s e r v i c e the c o n v e n t i o n a l s c i e n t i f i c  (Reineke,  1991).  Even proponents  methodology defend the concept  of  of  multiple stakeholder representation: Formal policymakers should p r o b a b l y not be the o n l y group whose i n f o r m a t i o n needs, and hence whose p o l i t i c a l i n t e r e s t s , e v a l u a t o r s should meet. Every p o l i c y d e c i s i o n has the p o t e n t i a l to impact on m u l t i p l e s t a k e h o l d e r groups, and d i s c u s s i o n s w i t h these groups o f t e n teach us t h a t they want to l e a r n d i f f e r e n t t h i n g s . (Cook, 1985, p. 44) As a r e s u l t of these i n f l u e n c e s , the concept of was  utilization  m o d i f i e d as o t h e r s t a k e h o l d e r s came to be seen as e q u a l l y  important,  e s p e c i a l l y those who  d e l i v e r e d and those who  consumed  services.  Greene (1988b) r e v e a l s " d i f f e r e n t i a t e d concepts  of  e v a l u a t i o n use which i n c l u d e i n s t r u m e n t a l ( d e c i s i o n - or a c t i o n -  9 o r i e n t e d ) , conceptual  ( e d u c a t i o n a l ) , and  persuasive)  341).  uses"  (p.  Mark & Shotland  (political  i n e v a l u a t i o n extend beyond  increased  i n c l u d e a d e s i r e to a c c u r a t e l y r e p r e s e n t  d e c i s i o n making p r o c e s s and p a r t i c i p a t i o n serves  or  (1985) i n d i c a t e that the r a t i o n a l e s f o r  i n v o l v i n g stakeholders u t i l i z a t i o n and  symbolic  a recognition that  stakeholder  as a mechanism of empowerment.  the views of m u l t i p l e stakeholders  the  Representing  i n e v a l u a t i o n has become an  accepted p r a c t i c e i n e v a l u a t i o n r e s e a r c h  (House, 1990).  Stakeholders  Stake stakeholder  (1981) r e v e a l s that M a r c i a Guttentag c o i n e d  the  to h e l p i d e n t i f y whose s i d e the e v a l u a t o r was  S t a k e h o l d e r s can be d i v i d e d i n t o three groups:  term on.  those persons or  groups f o r whom an e v a l u a t i o n r e p o r t i s to be d i r e c t e d , those persons or groups who  have a v e s t e d  impact of an e v a l u a t i o n ,  and  at r i s k by the e v a l u a t i o n .  i n t e r e s t i n the outcome or  those persons or groups t h a t are Guba and  Lincoln  these groups as agents, b e n e f i c i a r i e s , and These s t a k e h o l d e r s  represent  victims.  multiple perspectives  i n t e r e s t s , w i t h each group b r i n g i n g t h e i r own values  i n t o the e v a l u a t i o n p r o c e s s .  p o l i c y makers, p o l i t i c i a n s ,  (1981) c a t e g o r i z e  and  assumptions  These groups may  and  include  s p e c i a l i n t e r e s t groups, program  funders or managers, program s t a f f  (service providers),  labour  put  10 unions,  program c l i e n t s  (beneficiaries), prospective  clients,  f a m i l y members o f c l i e n t s , and concerned o r d i s g r u n t l e d c i t i z e n s . The s t a k e h o l d e r concept r e p r e s e n t s an a p p r e c i a t i o n t h a t each program a f f e c t s many groups, which have d i v e r g e n t and even i n c o m p a t i b l e concerns. I t realizes--and legitimizes--the d i v e r s i t y o f i n t e r e s t s a t p l a y i n the program w o r l d . I t r e c o g n i z e s the m u l t i p l e p e r s p e c t i v e s t h a t these i n t e r e s t s b r i n g t o judgement and understanding. I t takes e v a l u a t i o n down from the p e d e s t a l and p l a c e s i t i n the midst o f the fray. I t aims t o make e v a l u a t i o n a conveyer o f i n f o r m a t i o n , not a d e l i v e r e r o f t r u t h ; an a i d , n o t a judge. R e a l i z a t i o n of the l e g i t i m a c y o f competing i n t e r e s t s and the m u l t i p l i c i t y o f p e r s p e c t i v e s and w i l l i n g n e s s t o p l a c e e v a l u a t i o n a t the s e r v i c e of d i v e r s e groups a r e important i n t e l l e c t u a l advances. (Weiss, 1986, pp. 153-154) For many e v a l u a t o r s  (House, 1993, 1980, 1986b; Stake, 1983b;  S c r i v e n , 1980), the commitment t o m u l t i p l e s t a k e h o l d e r s i n e v a l u a t i o n r e s e a r c h i s not an attempt to i n c r e a s e u t i l i z a t i o n but to r e d e f i n e the decision-making  community--"to c o u n t e r a c t the  c o n c e n t r a t i o n o f expert c o n t r o l " (Smith & Chircup,  1987, p. 12).  House (1980) advances the concept o f d e m o c r a t i z a t i o n  of research  because: People should be g i v e n a c h o i c e so t h a t t h i n g s a r e n o t determined f o r them, even i n t h e i r own i n t e r e s t s , but c h o i c e should be d i s t r i b u t e d i n such a manner t h a t s o c i a l groups and s o c i a l c l a s s e s have equal o p p o r t u n i t i e s f o r making such choices. Lower s o c i a l groups should be g i v e n an o p p o r t u n i t y to determine the c h o i c e s i n t h e i r i n t e r e s t s , (p. 145) Many e v a l u a t o r s encourage s p e c i a l e f f o r t s t o b r i n g i n questions  and i n t e r e s t s of m i n o r i t y and disadvantaged  groups--  groups who a r e r e l a t i v e l y powerless o r a t r i s k o f e x p l o i t a t i o n o r disenfranchisement  (House, 1993, 1980; Guba & L i n c o l n , 1989;  Cronbach e t a l . , 1980; Cronbach, 1982).  I t i s their  contention  that the e x i s t e n c e o f a stake, i n any shape o r form, should the s t a k e h o l d e r  give  the r i g h t t o p a r t i c i p a t e and t o p r o v i d e i n p u t i n  11 the e v a l u a t i o n .  "One  of the major p o t e n t i a l c o n t r i b u t i o n s of  e v a l u a t i o n i s to h e l p those w i t h a l e g i t i m a t e i n t e r e s t i n a g i v e n i s s u e to r e c o g n i z e what i s a t stake f o r them" (Cronbach e t a l . , 1980,  p. 105) .  Focusing E v a l u a t i o n Questions  A l k i n e t a l . (1985) i n d i c a t e t h a t the p r o c e s s of framing  the  e v a l u a t i o n q u e s t i o n s i s one of the most e s s e n t i a l a s p e c t s of evaluation.  A d e c i s i o n to e v a l u a t e f o r one p a r t i c u l a r  client,  program manager or d e c i s i o n maker, i n f l u e n c e s the k i n d s of q u e s t i o n s asked  (Shadish et a l . 1991), and a d e c i s i o n to look a t  something means a d e c i s i o n not to look a t something e l s e 1986;  (Patton,  Cronbach et a l . , 1980). R o s s i & Freeman (1993) r e v e a l t h a t what i s seen as a problem  by one group may  not be p e r c e i v e d as such by o t h e r s .  r e i n f o r c e d by Crane's  (1985) a s s e r t i o n t h a t "one  This i s  group's s o c i a l  problem i s o f t e n the r e s u l t of another group's s o c i a l (p. 3 ) .  Patton  (1986) concludes  t h a t "the p r a c t i c e of e v a l u a t o r s  answering the wrong q u e s t i o n . . . i s widespread" Wallston  (p. 64).  (1981) b e l i e v e s t h a t there has been a g e n e r a l  u n d e r v a l u i n g of q u e s t i o n s i n the r e s e a r c h p r o c e s s . that "we  tend to ask q u e s t i o n s of i n t e r e s t to us,  areas important that we Although  policy"  and/or p r o b l e m a t i c to us"  She  reflects  that r e f l e c t  (p. 606), r e v e a l i n g  c o u l d run the r i s k of a s k i n g a l l the wrong q u e s t i o n s . a p a r t i c i p a t o r y model w i t h a v a r i e t y of p e r s p e c t i v e s  may  12 be d i f f i c u l t  i n c e r t a i n stages of the r e s e a r c h p r o c e s s , W a l l s t o n  c o n s i d e r s i t to be an e s s e n t i a l step i n h y p o t h e s i s g e n e r a t i o n . Cronbach and h i s a s s o c i a t e s (1980) concur t h a t the r e s p o n s i b i l i t y of the e v a l u a t o r to i n c l u d e m u l t i p l e i n t e r e s t s i s e s p e c i a l l y important when the r e s e a r c h q u e s t i o n s are b e i n g The  l i t e r a t u r e has underscored  formulated.  the importance  of s t a k e h o l d e r  p a r t i c i p a t i o n i n the process of i d e n t i f y i n g s p e c i f i c f o r e v a l u a t i o n (Gold, 1983; Cook, 1985;  L i n c o l n & Guba, 1981;  Levine & Levine, 1977;  questions  Patton,  Rubin & Babbie,  1978;  1989) .  The e v a l u a t o r has a duty to i d e n t i f y a l l audiences, to do h i s b e s t to determine what t h e i r concerns and i s s u e s are, and to honor and respond to those concerns and i s s u e s . F a i l u r e t o do so may cause s i g n i f i c a n t e v a l u a t i o n q u e s t i o n s to be o v e r l o o k e d or to be d i m i n i s h e d i n importance. Justice and f a i r n e s s r e q u i r e t h a t everyone w i t h a stake a l s o have a v o i c e . ( L i n c o l n & Guba, 1981, p. 306) A r e s p e c t f o r the d i r e c t experiences and a s p i r a t i o n s of s t a k e h o l d e r s r e q u i r e s and l e g i t i m i z e s the c o n s i d e r a t i o n of m u l t i p l e sources and types of r e s e a r c h q u e s t i o n s  (Cook, 1985).  F e m i n i s t r e s e a r c h b e l i e v e s t h a t " o b j e c t i v e " r e s e a r c h assumes that "the one a s k i n g the q u e s t i o n s knows b e t t e r than the s u b j e c t what the important q u e s t i o n s a r e " et a l . (1980) suggest  (Tomm, 1987,  p. 4) .  Cronbach  t h a t experts and d e c i s i o n makers have  " b l i n d s p o t s " and recommend t h a t both p r o s p e c t i v e and program c l i e n t s should be c o n s u l t e d a t t h i s stage.  actual  Tovar  (1989)  r e v e a l s t h a t " c o l l a b o r a t i o n i n groups can be a v e r y s u c c e s s f u l strategy f o r s e l e c t i n g relevant evaluation questions"  (p. 52).  13 Beneficiaries  The  involvement  of b e n e f i c i a r i e s as s t a k e h o l d e r s i n  e v a l u a t i o n has been g i v e n p a r t i c u l a r emphasis i n the stage of an e v a l u a t i o n .  formation  R o s s i & Freeman (1993) imply t h a t i n  some e v a l u a t i o n s the b e n e f i c i a r y would appear to have the s t r o n g e s t stake i n the outcome of a program's e v a l u a t i o n . Whitaker  (1974) proposes t h a t " i n order to i n c l u d e c r i t e r i a of  responsiveness  and e q u i t y , i t i s necessary  b e i n g served to p r o v i d e standards  to a l l o w the  people  f o r e v a l u a t i o n " (p. 760) .  Campbell has been s t a t e d to v a l u e knowledge o b t a i n e d  Even  from  b e n e f i c i a r i e s more than t h a t from s o c i a l s c i e n t i s t s because  "the  impressions of program p a r t i c i p a n t s are more l i k e l y to be grounded i n e x t e n s i v e program experience" L e v i t o n , 1991, Maclure  p.  (Shadish, Cook, &  133).  (1990) a s s e r t s t h a t not to a l l o w the p a r t i c i p a t i o n  of b e n e f i c i a r i e s i n e v a l u a t i o n "would be tantamount to b e l i e v i n g that the automobile  manufacturer i s the most a p p r o p r i a t e person  to e v a l u a t e the worth of a c a r , not the buyer or consumer" (p. 150).  The b e l i e f t h a t o u t s i d e , o b j e c t i v e r e s e a r c h e r s are i n  a b e t t e r p o s i t i o n to judge programs and e f f e c t s than the i s c o n s i d e r e d to be  client  illogical.  Most proponents of b e n e f i c i a r y involvement  i n evaluation  r e s e a r c h f e e l t h a t each s t a k e h o l d e r group should share e q u a l l y i n the e v a l u a t i o n p r o c e s s .  They request the o p p o r t u n i t y f o r a l l  groups w i t h a stake to p a r t i c i p a t e - - a r e c o g n i t i o n and  14 a p p r e c i a t i o n o f the c h a l l e n g e s stakeholding  and o p p o r t u n i t i e s f a c i n g each  group and a chance f o r g r e a t e r u n d e r s t a n d i n g and  r e s p e c t between groups w i t h d i f f e r i n g  perspectives.  As most programs a r e not e q u a l l y r e s p o n s i v e  t o the needs of  a l l people, i t has been suggested t h a t consumers should have the r i g h t t o respond t o i s s u e s t h a t may a f f e c t them through the program o r e v a l u a t i o n .  T h i s may i n c l u d e not o n l y the  b e n e f i c i a r i e s o f s e r v i c e s but, p o t e n t i a l l y , p r o s p e c t i v e b e n e f i c i a r i e s and b e n e f i c i a r i e s excluded from s e r v i c e s .  It i s  important t o g i v e v o i c e to those most a f f e c t e d by the program being  evaluated,  p a r t i c u l a r l y i f they have some v u l n e r a b i l i t y  that c o u l d make them p o t e n t i a l " v i c t i m s "  (House, 1993, 1980; Guba  & L i n c o l n , 1989).  Mental H e a l t h C i t i z e n A d v i s o r y  Boards  Mental h e a l t h s e r v i c e s has a c t u a l l y been the l e a d e r i n advocating  f o r the r i g h t s of consumers i n e v a l u a t i o n .  T r a d i t i o n a l l y , c i t i z e n s and c l i e n t s had l i t t l e  i n p u t i n t o the  p r o f e s s i o n a l areas o f mental h e a l t h ; however, i n 1975, Community Mental H e a l t h Centre  the  (CMHC) Amendments i n the U n i t e d  S t a t e s mandated c i t i z e n involvement, s p e c i f i c a l l y i n the e v a l u a t i o n o f community h e a l t h c e n t r e programs. C i t i z e n Advisory Zinober,  As a r e s u l t ,  Boards were developed i n many communities.  D i n k e l , & Windle  (1984) argue t h a t  evaluation  a c t i v i t y i n mental h e a l t h s e r v i c e s should be viewed:  15 l e s s as a s p e c i a l i z e d domain of a t e c h n i c i a n e v a l u a t o r and more as a c o l l a b o r a t i v e endeavor....This argues f o r i n c r e a s e d p a r t i c i p a t i o n i n e v a l u a t i o n by program managers and s e r v i c e s t a f f , program c l i e n t s , and even the g e n e r a l p u b l i c , (p. 224) Advocates r e v e a l , however, that although attempts to c i t i z e n s have been e s t a b l i s h e d , e v a l u a t i o n s  include  are o f t e n  still  i n e f f e c t i v e because the a c t u a l consumers of s e r v i c e s are involved Due  (Morrison,  1978;  Prager, 1986).  to t h e i r " p a r t i c i p a n t - o b s e r v e r "  are regarded as b e i n g  Prager  are a p p r o p r i a t e receive.  r o l e , program c l i e n t s  a b l e to p r o v i d e more meaningful  than members of the community (1978) and  provide  Morrison  (1986) propose that even p s y c h i a t r i c p a t i e n t s  and  knowledgable e v a l u a t o r s  Whereas i t i s v e r y  of the s e r v i c e s they  time-consuming f o r a c i t i z e n  the case w i t h c l i e n t s .  c l i n i c and  evaluations  (Pinto & F i e s t e r , 1979).  become s u f f i c i e n t l y f a m i l i a r w i t h a c l i n i c and i s not  seldom  to  i t s services,  Because they are f a m i l i a r w i t h a  i t s treatment programs, a c l i e n t i s b e t t e r a b l e  s u b s t a n t i a l and meaningful i n f o r m a t i o n  and  the s e r v i c e s  to  suggestions to  enhance the e v a l u a t i o n . Thus, the c l i e n t - c o n s u m e r i s the person to e v a l u a t e  this  best  received.  Since community r e p r e s e n t a t i v e s are not always cognizant of the needs of p a t i e n t s or devoted to the p r o t e c t i o n of t h e i r r i g h t s , a governing board should i n c l u d e among i t s members p a s t and present p a t i e n t s and t h e i r r e l a t i v e s . (Chu & T r o t t e r , 1974, p. 85) Prager  (1986) b e l i e v e s that c l i e n t s are becoming  i n c r e a s i n g l y more s o p h i s t i c a t e d and more aware of t h e i r needs. They are a l s o becoming more w i l l i n g to c o n f r o n t advocate f o r change.  organizations  and  16 I t i s almost p a r a d o x i c a l t h a t , although mental h e a l t h c l i e n t s have spent a good p a r t of t h e i r l i v e s l e a r n i n g and implementing s k i l l s i n r e l a t i o n s h i p s and communication, h e l p i s s t i l l l a r g e l y d e f i n e d , planned, and e v a l u a t e d f o r them, not by them. (Prager, 1986, p. 5)  Paradigm Approaches t o Stakeholder  Participation  S e v e r a l paradigms, summarized b r i e f l y below, encourage the use o f b e n e f i c i a r i e s and consumers as c o l l a b o r a t o r s i n the research process. and  Two o f these paradigms, p a r t i c i p a t o r y  research  f e m i n i s t r e s e a r c h , r e v e a l a - p r o a c t i v e posture--one o f  empowerment--whereby s u b j e c t s and b e n e f i c i a r i e s a r e no longer v i c t i m s but have the knowledge and t o o l s t o shape t h e i r own destiny.  The t h i r d paradigm, f o u r t h g e n e r a t i o n  research,  encompasses a unique form o f q u a l i t a t i v e r e s e a r c h which has been d i f f e r e n t i a t e d as a d i s t i n c t approach t o e v a l u a t i o n .  P a r t i c i p a t o r y Research  P a r t i c i p a t o r y r e s e a r c h i s i n h e r e n t l y connected t o p o p u l a r e d u c a t i o n i n L a t i n America and i s sometimes known as p a r t i c i p a t o r y a c t i o n research or a c t i o n research.  Although  some  r e s e a r c h e r s i n North America have r e f e r r e d t o s t a k e h o l d e r involvement  i n r e s e a r c h as " p a r t i c i p a t o r y r e s e a r c h , "  the term  was o r i g i n a l l y c o i n e d i n South America and r e f e r s t o an i n h e r e n t l y d i s t i n c t r e s e a r c h p h i l o s o p h y and methodology.  P a r t i c i p a t o r y r e s e a r c h i s an e d u c a t i o n a l p r o c e s s intended to produce change and i s most o f t e n c a r r i e d out by the  oppressed,  w i t h the p o s s i b l e f a c i l i t a t i o n of an e x t e r n a l r e s e a r c h e r . Drawing on the teachings of Paulo F i e r r e r e s e a r c h advocates understand  (1972),  participatory  t h a t o r d i n a r y people have the a b i l i t y to  and analyze t h e i r own  a b i l i t y to shape t h e i r own  s o c i a l r e a l i t y , as w e l l as the  l i v e s and  destinies.  Only f u l l and complete c o n t r o l by the group under study would c o n s t i t u t e proper p a r t i c i p a t o r y r e s e a r c h by e d u c a t i o n standards  ( L a t a p l , 1988;  Tandon, 1988;  popular Maclure,  1990).  Thus, common people are c o n s i d e r e d to be the r e s e a r c h e r s , determining the q u e s t i o n s , g a t h e r i n g data, p a r t i c i p a t i n g i n the a n a l y s i s , and d e c i d i n g how  the i n f o r m a t i o n w i l l be  used.  Proponents of p a r t i c i p a t o r y r e s e a r c h c l a i m t h a t the advantage of t h i s extent of c o o p e r a t i o n and e s s e n t i a l l y , c o - i n v e s t i g a t i o n , i s the a c q u i s i t i o n of knowledge normally beyond the reach of other r e s e a r c h e r s . A key premise people w i l l  i s t h a t the involvement  of  local  f a c i l i t a t e an even c l e a r e r p i c t u r e of s o c i a l  reality  than p o s s i b l e otherwise. " P a r t i c i p a t o r y r e s e a r c h emphasizes the use of knowledge as one of the major bases f o r power and c o n t r o l i n our s o c i e t y .  It  has enormous p o t e n t i a l as a major c o n t r i b u t o r i n t r a n s f o r m i n g the s t r u g g l e s of poor and d e p r i v e d people"  (Tandon, 1988,  p. 7 ) .  It  i s the p r i n c i p l e s of shared ownership, l e a r n i n g , and a c t i o n which d i s t i n g u i s h p a r t i c i p a t o r y r e s e a r c h from o t h e r models of science research.  social  18 F e m i n i s t Research  F e m i n i s t r e s e a r c h v a l i d a t e s knowledge t h a t i s gained women's experiences  as important,  r e a l , and v a l u a b l e .  through  It  emphasizes the n e c e s s i t y o f i n t e r a c t i o n w i t h r e s e a r c h s u b j e c t s as p a r t i c i p a n t s - - o b s e r v i n g responses and l i s t e n i n g t o women's d e s c r i p t i o n s o f t h e i r own r e a l i t i e s . F e m i n i s t r e s e a r c h f i n d s i t s r o o t s i n the b e l i e f t h a t conceptions o f knowledge and t r u t h t h a t a r e accepted today have been shaped throughout h i s t o r y by the maledominated c u l t u r e . Drawing on t h e i r own perspectives...men have s e t v a l u e s t h a t have become the g u i d i n g p r i n c i p l e s f o r men and women a l i k e . (Belenky e t a l . , 1986, p. 5) Mies  (1983) r e v e a l s t h a t a r e s u l t of the t r a d i t i o n a l ,  h i e r a r c h i c a l r e s e a r c h format has been t h a t much o f the data c o l l e c t e d through s o p h i s t i c a t e d r e s e a r c h methodology i s i r r e l e v a n t and p o s s i b l y i n v a l i d because data o f t e n r e f l e c t e d e x p e c t a t i o n s r a t h e r than  truth.  Many f e m i n i s t r e s e a r c h e r s b e l i e v e t h a t r e s e a r c h s u b j e c t s should become f u l l c o l l a b o r a t o r s i n the r e s e a r c h process K l e i n , 1983; Mies, 1983; K i r b y , 1991).  (Duelli  They share a s t r o n g  c o n v i c t i o n t h a t i n f e m i n i s t r e s e a r c h , power must be shared among participants.  Belenky e t a l . (1986) d e s c r i b e s t h a t " i n a  community, u n l i k e a h i e r a r c h y , people g e t t o know each o t h e r . They do n o t a c t as r e p r e s e n t a t i v e s o f p o s i t i o n s o r as occupants of r o l e s b u t as i n d i v i d u a l s w i t h p a r t i c u l a r s t y l e s o f t h i n k i n g " (p. 221) . F e m i n i s t r e s e a r c h e r s advocate r e s e a r c h s t u d i e s t h a t f a c i l i t a t e and support  the movement a g a i n s t the o p p r e s s i o n and  19 i s o l a t i o n o f women.  "When women accept r e s p o n s i b i l i t y f o r  e v a l u a t i n g and c o n t i n u a l l y r e v e a l i n g t h e i r assumptions about knowledge, the a t t e n t i o n and r e s p e c t t h a t they might have once awarded t o the expert i s transformed"  (Belenky  e t a l . , 1986,  p. 40) . Although  f e m i n i s t r e s e a r c h holds s i m i l a r i d e o l o g i e s as  p a r t i c i p a t o r y r e s e a r c h , i t s r o o t s l a y i n the North American f e m i n i s t movement.  I t s focus i s on empowering women and  i n c l u d i n g women as s t a k e h o l d e r s i n the r e s e a r c h p r o c e s s . F e m i n i s t r e s e a r c h can a l s o be accomplished  by "expert" women  r e s e a r c h e r s and does not r e q u i r e complete c o n t r o l o f the group under study,  as does p a r t i c i p a t o r y r e s e a r c h .  Feminist  research  may, a t times, draw on some of the p h i l o s o p h i e s and techniques used i n p a r t i c i p a t o r y r e s e a r c h t o accomplish  Fourth G e n e r a t i o n  i t s goals.  Evaluation  In t h e i r p r e s e n t a t i o n of f o u r t h g e n e r a t i o n e v a l u a t i o n , Guba and L i n c o l n (1989) a s s e r t t h a t f o u r t h g e n e r a t i o n e v a l u a t i o n can o n l y be implemented w i t h i n the methodological c o n s t r u e t i v i s t paradigm.  They suggest  (or q u a l i t a t i v e ) techniques  canons o f the  t h a t although  naturalistic  can be used w i t h i n the c o n f i n e s o f  the c o n v e n t i o n a l e v a l u a t i o n paradigm, n a t u r a l i s t i c e v a l u a t i o n r e q u i r e s a complete paradigm s h i f t . not making t h i s d i s t i n c t i o n , fourth generation evaluation.  Because some e v a l u a t o r s were  they renamed n a t u r a l i s t i c i n q u i r y as  20 The  o n t o l o g i c a l assumption of c o n s t r u c t i v i s m i s t h a t  r e a l i t i e s a r e o n l y a s e r i e s o f mental and s o c i a l There i s no o b j e c t i v e r e a l i t y . denies s u b j e c t - o b j e c t that " c r e a t e s " .  reconstructions.  Epistemologically,  constructivism  dualism, s t a t i n g that i t i s i n t e r a c t i o n  Fourth g e n e r a t i o n  evaluation i s described  " c o l l a b o r a t i v e p r o c e s s i n that s e v e r a l s t a k e h o l d e r c o n t r o l o f the e v a l u a t o r s  as a  groups share  over the m e t h o d o l o g i c a l and  i n t e r p r e t a t i v e d e c i s i o n s that a r e made" (Guba, 1987, p. 39). The evaluator The  i s seen as a r e a l i t y shaper, mediator and change agent.  d i r e c t i o n o f the e v a l u a t i o n i s shaped through i n t e r a c t i o n and  involvement w i t h the p a r t i c i p a n t s o r i n f o r m e r s . P r o f e s s i o n a l researchers  solicit  "concerns, needs and  i s s u e s " o f informants i n a hermeneutic, d i a l e c t i c p r o c e s s . p r o c e s s feeds i t s e l f by a s k i n g  This  informants about other i n d i v i d u a l s  or groups t h a t may h o l d a d i f f e r e n t o p i n i o n than t h e i r s , may b e n e f i t o r l o s e because o f the program o f e v a l u a t i o n , strong  f e e l i n g s about a program.  S o c i a l r e a l i t y i s constructed  by o b t a i n i n g a consensus, as much as i s p o s s i b l e , c o n t i n u a l l y r e c y c l i n g the r e s e a r c h e r ' s multiple  The  The  o r may have  through  f i n d i n g s through these  informants.  Stakeholder Approach:  Two C o n t r o v e r s i a l Attempts  stakeholder-approach was developed i n the mid 1970's by  the N a t i o n a l  I n s t i t u t e of Education  (NIE) i n an attempt t o  i n c r e a s e u t i l i z a t i o n s o f e v a l u a t i o n s by g i v i n g a wide v a r i e t y o f  21 p a r t i c i p a n t s a s t r o n g v o i c e i n the e v a l u a t i o n  process.  E v a l u a t i o n s were b e i n g h e a v i l y c r i t i c i z e d by the p u b l i c and c r e a t i o n of an e v a l u a t i o n approach which was m u l t i p l e concerns was  responsive  the  to  a d e l i b e r a t e attempt to address some of  these concerns. The  stakeholder  (1983), was  concept, as i d e n t i f i e d by Bryk & Raudenbush  designed to address the f o l l o w i n g concerns:  the e v a l u a t i o n of programs and p o l i c i e s be  "How  s t r u c t u r e d so t h a t  knowledge produced w i l l be r e l e v a n t and a c c e s s i b l e " and evaluations  f a i r l y represent  concerns of those who Two Schools  and  have an i n t e r e s t i n a program?"  Push/Excel  "How  the can  the d i v e r s e , p o t e n t i a l l y c o n f l i c t i n g  programs were s e l e c t e d f o r e v a l u a t i o n : (CIS)  can  (Excel).  (p.  97).  Cities in  Both programs were  determined to be h i g h l y complex, c o n t r o v e r s i a l , and  politically  "loaded"  CIS was  (Weiss, 1983a, 1983b; Stake, 1983,  1986).  c o l l e c t i o n of i n n e r - c i t y school p r o j e c t s f o r estranged which i n c o r p o r a t e d  T h i s program was  youth  t r a d i t i o n a l academic t a s k s , c u l t u r a l  athletic activities,  a  and  and one-on-one c o u n s e l l i n g and t u t o r i n g .  s t r o n g l y endorsed by Rosalyn C a r t e r and  White House a d m i n i s t r a t i o n .  E x c e l was  the  a m o t i v a t i o n a l program  s t a r t e d by the Reverend Jesse Jackson i n an e f f o r t to persuade b l a c k youth to e x c e l i n s c h o o l .  The program, which i s d e s c r i b e d  more as a movement than a program ( F a r r a r & House, 1983), taught a message of s e l f - h e l p and  s e t h i g h standards of achievement  through i n c e n t i v e programs, pledges to e x c e l l e n c e , and code of conduct.  a written  Parents were urged to take an a c t i v e i n t e r e s t  22 i n t h e i r c h i l d r e n ' s e d u c a t i o n and s c h o o l s were c h a l l e n g e d t o r a i s e t h e i r standards f o r both behaviour and academic performance. Weiss (1983) agreed w i t h Stake  (1983) t h a t the emergent  nature o f both programs made them extremely d i f f i c u l t t o evaluate.  The programs were determined  l i t t l e continuity.  t o have h i g h turnover and  They were:  c h a r a c t e r i z e d by h i g h l e v e l s o f ambiguity. Their a c t i v i t i e s d i f f e r e d from day t o day and from s i t e t o s i t e . The programs were not a d m i n i s t e r e d by o r d e r l y s c h o o l b u r e a u c r a c i e s a c c o r d i n g to p r e d i c t a b l e s e t s o f r u l e s but by o u t s i d e , movement-like groups. They depended i n c o n s i d e r a b l e p a r t on the d e d i c a t i o n o f committed workers and the charisma o f l e a d e r s . (Weiss, 1983a, p. 155-156) In t h e i r reviews o f these two e v a l u a t i o n s , Stake F a r r a r & House  (1983) and  (1983) conclude t h a t the q u a n t i t a t i v e  evaluation  designs were i l l - s u i t e d t o e i t h e r the programs o r t h e i r context and d i d n o t c o n s i d e r the e v a l u a t i o n needs o f d e c i s i o n makers. Although the e v a l u a t i o n s were designed t o be " s t a k e h o l d e r f r i e n d l y , " the f o l l o w i n g c r i t i q u e s r e v e a l t h a t t h e i r r e s u l t s were l e s s than s a t i s f a c t o r y .  C i t i e s i n Schools  In h i s e v a l u a t i o n of the C i t i e s - i n - S c h o o l s e v a l u a t i o n .  Stake  (1983) found t h a t the use o f the term s t a k e h o l d e r was l i m i t e d and d i d n o t p r o v i d e d i r e c t involvement f o r many groups.  Although  d e c i s i o n makers and consumers were i n i t i a l l y i n c l u d e d as s t a k e h o l d e r s , program s t a f f were n e g l e c t e d i n t h i s p r o c e s s .  23 Stakeholders  were a l s o viewed as ah audience or as  information  r e c e i v e r s and not as c o l l a b o r a t o r s i n the e v a l u a t i o n The p r i n c i p a l r e s e a r c h e r , Murray, e v e n t u a l l y  process.  focused  e x c l u s i v e l y on d e c i s i o n makers and abandoned consumers because he p e r c e i v e d them to be u n h e l p f u l and u n i n v o l v e d .  As Murray  reported: "The i d e a of u s i n g c l i e n t s as s t a k e h o l d e r s never r e a l l y got o f f the ground. How do you get parents and k i d s together i n Indianapolis? You have to use the program to get them together. So what do you get? You get the fan c l u b . You get parents who r e a l l y l i k e the program...But t h i s i s n ' t having a stake i n i t . " (Stake, 1983, p. 26) S e v e r a l f a c t o r s were found to impede the approach.  stakeholder  Stake determined t h a t the r e s e a r c h p r o p o s a l d i d not  a n t i c i p a t e t h a t there "might be d i f f i c u l t y communicating w i t h stakeholders p. 18).  or i n understanding  t h e i r i n f o r m a t i o n needs"  In f a c t , they t r e a t e d s t a k e h o l d e r  involvement "as i f i t  c o u l d be handled w i t h i n o r d i n a r y i n f o r m a t i o n operations" expectations  (1986, p. 29).  The  of s t a k e h o l d e r s ,  and  processing  evaluation a l s o held high  assuming t h a t they would a c t i v e l y  pursue a p a r t i c i p a t o r y r o l e and e x p l i c i t of t h e i r own  CIS  (1983,  t h a t they were c o g n i z a n t  need. Stake r e v e a l s t h a t these  the r e s u l t i n g l a c k of p l a n n i n g may  have been due,  the u n e n t h u s i a s t i c treatment of s t a k e h o l d e r  and  assumptions i n p a r t , to  involvement by  members of the e v a l u a t i o n r e s e a r c h s o c i e t y , who  commissioned  the  study. Although Stake  (1983) assessed  t h a t the f i n d i n g s gave an  a c c u r a t e p o r t r a y a l of program d i f f i c u l t i e s and i n c l u d i n g which s t a k e h o l d e r s  accomplishments,  or c o n s i s t e n c i e s supported  or  24 opposed CIS, "the e v a l u a t i o n team d i d not take f u l l advantage of the s t a k e h o l d e r concept"  (p. 29).  Push/Excel In  t h e i r case study o f t h i s e v a l u a t i o n , F a r r a r & House  (1983) r e v e a l e d t h a t the s t a k e h o l d e r n o t i o n was n o t w e l l used and that s t a k e h o l d e r s c o n t r i b u t e d l i t t l e dissemination of f i n d i n g s .  t o the e v a l u a t i o n d e s i g n o r  The p r i n c i p a l i n v e s t i g a t o r s , Murray  and Gold, agreed that t h e i r method needed c o n s i d e r a b l e improvement. S e v e r a l b a r r i e r s were a t t r i b u t e d to the s t a k e h o l d e r s : s t a k e h o l d e r n a i v e t e about what an e v a l u a t i o n can accomplish, concern over p o l i t i c a l i s s u e s versus community  involvement,  s t a k e h o l d e r s ' l i m i t e d knowledge about l o c a l o p e r a t i o n s o r g o a l s , and a g e n e r a l l a c k o f s o p h i s t i c a t i o n i n e v a l u a t i o n r e s e a r c h . Researchers were blamed f o r b e i n g lukewarm about involvement  stakeholder  and f o r p a y i n g scant a t t e n t i o n t o s t a k e h o l d e r needs  ( F a r r a r & House, 1983, pp. 45-46).  Murray remarked that although  some p a r e n t s expressed a g e n e r a l i n t e r e s t i n t h e i r c h i l d r e n doing w e l l , many f e l t powerless his  t o do anything, and Gold acknowledged  f r u s t r a t i o n o f working w i t h p a r e n t s u n f a m i l i a r w i t h  about  these  issues.  talking  25 Collaborative Evaluation  E v a l u a t o r A t t i t u d e s Concerning Implementation  Issues  Many o f the same r e s e a r c h e r s who m a i n t a i n the importance of m u l t i p l e p e r s p e c t i v e s and agree that b e n e f i c i a r i e s and c i t i z e n s o f t e n have a stake i n program e v a l u a t i o n a l s o r e v e a l  their  scepticism of a nonresearcher s a b i l i t y  to e f f e c t i v e l y  p a r t i c i p a t e i n the e v a l u a t i o n p r o c e s s .  They express p e r s o n a l  1  apprehensions about the p r a c t i c a l l i m i t a t i o n s o f such an approach. S e v e r a l concerns a r e a t t r i b u t e d t o the nonresearcher and, particular,  the b e n e f i c i a r y o f s e r v i c e s .  in  These i n c l u d e both  p e r s o n a l and s i t u a t i o n a l f a c t o r s i n that n o n r e s e a r c h e r s : may ask uninformed,  trivial,  o r u n s o p h i s t i c a t e d q u e s t i o n s (Cook &  Shadish, 1986; Shadish e t a l , 1991; Guba & L i n c o l n , 1989; Weiss, 1983b); do not have a b e t t e r understanding o f the needs and problems than p r o f e s s i o n a l s o r do not know enough t o make u s e f u l suggestions f o r change (Pinto & F i e s t e r , 1979); may n o t understand the purposes of the e v a l u a t i o n , may f e a r r e p r i s a l i f they make n e g a t i v e remarks, due t o poor h e a l t h  o r may be unable t o cooperate  (Knott, 1988);  c o n c e r n i n g such matters  fully  a r e too p o o r l y educated  (Knott, 1988; R o s s i & Freeman, 1993); may  have mainly n e g a t i v e a t t i t u d e s towards the program o r the evaluation interest  (Tronya & F o s t e r , 1988); do not have the time o r  (Shadish e t a l . , 1991); may t r y t o p r o t e c t t h e i r own  26 i n t e r e s t or p o s s i b l y mistake t h e i r own Krause & Howard, 1976); may are i n c l u d e d  (Reinharz,  i n t e r e s t s (House,  1980;  r e f u s e to p a r t i c i p a t e i f other  1983); and p r e f e r to m a i n t a i n  v i s i b i l i t y or keep a low p r o f i l e  groups  low  ( L i n c o l n & Guba, 1981;  Rossi  &  Freeman, 1993). Other concerns r e l a t e to the r e s e a r c h e r they: may  not understand the concept of  e v a l u a t i o n or be  or e v a l u a t o r  stakeholder-based  "lukewarm" about stakeholder  involvement  & House, 1983); are faced w i t h too many p e r s p e c t i v e s , competing ones (Shadish  et a l , 1991;  i n that  (Farrar  possibly  R o s s i & Freeman, 1993);  may  be unable to j u g g l e the competing demands of p a r t i c i p a t i o n and technical integrity  (Weiss, 1983a); may  r e t a i n c o n t r o l , or may stakeholder it  roles  have a n a t u r a l d e s i r e to  l a c k experience w i t h or understanding of  (Dinkel, Zinober,  & F l a h e r t y , 1981); may  too time consuming to i n c l u d e p e r i p h e r a l audiences or  o v e r l o o k them (Stake,  1983;  L i n c o l n & Guba, 1981); may  p o l i t i c a l r o l e and p l a y advocate i n s t e a d of e v a l u a t o r 1974); and/or may  leave  find may  embrace a (Gibbs,  the d e c i s i o n maker w i t h a problem i n s t e a d  of a s o l u t i o n (Cochran, 1980). Evaluators  have a l s o been concerned about  technical i n t e g r i t y , e s p e c i a l l y i n evaluations utilization rationale.  Chelimsky  jeopardize Leithwood  that maintain  and  Patton  get too c l o s e to s t a k e h o l d e r s  scientific integrity"  an  (1987) argues t h a t t e c h n i c a l  q u a l i t y i s a p r e r e q u i s i t e f o r meaningful use, r e v e a l s t h a t e v a l u a t o r s who  maintaining  (Patton,  1987).  Cousins  (1987) "may and  (1986), i n t h e i r assessment of u s e - r e l a t e d f a c t o r s ,  27 found t h a t t e c h n i c a l q u a l i t y had the s t r o n g e s t r e l a t i o n s h i p t o utilization.  Weiss (1983b) concurs t h a t s t a k e h o l d e r  involvement  i s l i k e l y t o compromise the t e c h n i c a l q u a l i t y o f e v a l u a t i o n s . S t u d i e s o f s t a f f a t t i t u d e s toward s t a k e h o l d e r  participation  i n e v a l u a t i o n r e v e a l t h a t there i s a wide range o f ambiguity i n regards The  t o the r o l e of s t a k e h o l d e r s  infrequency  of experience  i n the e v a l u a t i o n  process.  o f p a r t i c i p a t i o n may be the r e s u l t o f "the l a c k  with, and understanding  of, the r o l e which  [stakeholders] might p l a y i n e v a l u a t i o n , o r o f the a c t i v i t i e s they might perform"  (Dinkel e t a l . , 1981, p. 60). Other b a r r i e r s  i n c l u d e s t a f f r e s i s t a n c e to change, p e r c e p t i o n s  of stakeholder  involvement as an a t t a c k o r t h r e a t t o t h e i r p e r s o n a l  credibility,  and a l a c k o f knowledge o f demonstrated procedures f o r stakeholder  involvement.  In t h e i r i n v e s t i g a t i o n of the a t t i t u d e s o f governing  board  and management s t a f f a t t i t u d e s toward c i t i z e n p a r t i c i p a t i o n i n mental h e a l t h agencies,  Pinto & Fiester  (1979) found t h a t  " d e s p i t e the g e n e r a l p e r c e p t i o n t h a t c i t i z e n i n p u t i s t o be valued, both groups f e l t  t h a t c i t i z e n s d i d not...know enough  about community mental h e a l t h c e n t r e s to make u s e f u l  suggestions  f o r change i n s e r v i c e " (p. 265). A review of p a t t e r n s i n program e v a l u a t i o n p r a c t i c e among e v a l u a t o r s by Shadish  and E p s t e i n  (1987) r e v e a l e d t h a t " e v a l u a t o r s i d e n t i f i e d themselves as experts who came i n t o judge and educate--apparently  we e v a l u a t o r s have  l i t t l e doubt t h a t we have something to o f f e r beyond merely r e f l e c t i n g stakeholder  opinions"  (p. 576). They found t h a t few  28 respondents  were f a m i l i a r w i t h d i v e r s e t h e o r i e s of e v a l u a t i o n  such as w r i t i n g s by S c r i v e n and S t a k e - - l e a d e r s i n s t a k e h o l d e r based approaches.  They conclude  t h a t " i t i s hard to argue t h a t  t h i s low l e v e l of f a m i l i a r i t y w i t h theory i s d e s i r a b l e , i f f o r no other reason than the f a c t t h a t e v a l u a t o r s are d e p r i v e d of knowledge of the o p t i o n s they have a v a i l a b l e to use i n t h e i r practice"  (p.  586).  Stakeholder P a r t i c i p a t i o n i n A c t i o n  Researchers  who  have been i n v o l v e d i n c o l l a b o r a t i v e r e s e a r c h  w i t h s t a k e h o l d e r s r e v e a l t h a t there are both advantages disadvantages process.  and  to the i n c l u s i o n of s t a k e h o l d e r s i n the e v a l u a t i o n  They r e p o r t s i g n i f i c a n t s h i f t s i n the r o l e s  and  r e l a t i o n s h i p s between r e s e a r c h e r and s t a k e h o l d e r working together i n a c o l l a b o r a t i v e process. There seems to be g e n e r a l agreement t h a t the i n c l u s i o n of m u l t i p l e s t a k e h o l d e r s r e q u i r e s an i n c r e a s e d investment and energy 1988;  (Stake, 1986;  Routledge,  1993;  Reinharz,  Tovar,  1983;  1989).  i n time  and Tronya & F o s t e r ,  T h i s was  a l s o true f o r  s t a k e h o l d e r s , as c o l l a b o r a t i v e r e s e a r c h demanded a g r e a t d e a l of time which c o u l d be c o n s i d e r e d u n r e a l i s t i c w i t h i n the c o n f i n e s of busy l i v e s or work schedules(Tronya D'Amico, 1985;  Donmoyer, 1990).  & F o s t e r , 1988;  Dawson &  Having s t a f f as s t a k e h o l d e r s  a l s o found to h i n d e r the c o l l a b o r a t i v e p r o c e s s when s t a f f r e s e n t e d or f e a r e d b e i n g e v a l u a t e d  (Tronya & F o s t e r , 1988) .  was  29 Status d i f f e r e n t i a l problems was  observed  between upper-middle  c l a s s s t a f f and b l u e - c o l l a r working s t a k e h o l d e r s and,  at  times,  p r o f e s s i o n a l s t a f f d i s c o u n t e d s t a k e h o l d e r s ' views because of t h e i r l a c k of e d u c a t i o n or p r o f e s s i o n a l t r a i n i n g Stakeholders,  (Donmoyer,  p l a c e d i n a d e c i s i o n making arena,  have  demonstrated d i f f e r e n t i n f o r m a t i o n needs which can be  very  demanding on e v a l u a t o r s Newman, 1992'  (Palumbo & H a l l e t t ,  Couto, 1987).  1993;  199).  Lobosco &  F u r t h e r , r e s e a r c h e r s found  clear  d i v i s i o n i n s t a k e h o l d e r s ' assessments of programs ( M c G a r r e l l & Sabath, 1994).  T e c h n i c a l d i f f i c u l t i e s developed  s t a k e h o l d e r s c o n s t r u c t e d a survey t h a t was too d i f f i c u l t  to analyze, and  when  too l o n g to code and  the use of v o l u n t e e r s r e s u l t e d i n  i n c o r r e c t coding of the survey and other e r r o r s r e q u i r i n g e x t e n s i v e time to r e c t i f y  (Couto,  1987).  An e v a l u a t i o n based i n Mexico r e v e a l e d t h a t s t a k e h o l d e r s were r e q u i r e d by t h e i r employer to c o l l e c t data and  who  perform  a n a l y s i s were not v e r y e n t h u s i a s t i c p a r t i c i p a n t s i n the evaluation process.  In a d d i t i o n , s t a k e h o l d e r s were unable to  s u s t a i n an e v a l u a t i o n f o r which they had s o l e r e s p o n s i b i l i t y , p a r t l y due  to the l a c k of f i n a n c e s and other r e s o u r c e s .  (Brunner  &. Guzam, 1989) . E v a l u a t o r s have a l s o found  the unintended  s i d e e f f e c t of  r e s e a r c h e r t a k i n g more r e s p o n s i b i l i t y and power than planned to s t a k e h o l d e r dynamics or to consumers' demands and to p l a y the "expert r o l e " Researchers  (Kirkup, 1986;  have a l s o r e p o r t e d a reduced  Reinharz,  the due  expectations  1983) .  sense of c o n t r o l and  30 f e e l i n g t h a t they may  have s a c r i f i c e d t h e i r autonomy and  objectivity  1991;  (Reineke,  Tronya & F o s t e r , 1988;  or  Reinharz,  1983) . On the o t h e r hand, c o l l a b o r a t i v e r e s e a r c h was  found  s o l i c i t h i g h p a r t i c i p a t i o n r a t e s and a c c e l e r a t e the of f i n d i n g s w h i l e m a i n t a i n i n g t e c h n i c a l i n t e g r i t y Couto, 1987;  Dawson & d'Amico, 1985).  to  utilization  (Patton,  Stakeholders,  1987;  including  b e n e f i c i a r i e s , were w i l l i n g p a r t n e r s i n the e v a l u a t i o n and d e c i s i o n making process and were a b l e to c o n t r i b u t e m e a n i n g f u l l y to the e v a l u a t i o n process Lobosco & Newman, 1992; 1993).  Stakeholder  (Malekoff, 1994;  Lawrence, 1989;  involvement  Routledge,  Couto, 1987;  1993; Barkdoll,  a l s o helped to r e i n f o r c e  limited  f i n a n c i a l r e s o u r c e s f o r e v a l u a t i o n s (Dawson & D'Amico, 1985).  A  study on s t a k e h o l d e r i n f o r m a t i o n p r e f e r e n c e s r e v e a l e d t h a t among a d i v e r s e group of s t a k e h o l d e r s there was  a h i g h degree of  c o n s i s t e n c y i n the importance a t t a c h e d to d i f f e r e n t types of information The  (Deutsch & Malmborg, 1986).  i n c l u s i o n of d e c i s i o n makers and program s t a f f has been  shown to i n c r e a s e the meaningfulness  and v a l i d i t y of f i n d i n g s and  to make the r e s e a r c h process l e s s i n t r u s i v e thus r e d u c i n g r e a c t i o n to the e v a l u a t i o n (Patton, 1987, Stakeholder d i a l o g u e was  "Early stakeholder p a r t i c i p a t i o n  e s t a b l i s h e d the l e g i t i m a c y of  [ s t a f f ] i n p u t i n t o the e v a l u a t i o n  and r e i n f o r c e d the p e r c e p t i o n t h a t i t was 1991,  Reineke, 1991).  a l s o found to c o n t r i b u t e to s t a f f  ownership and empowerment.  them" (Reineke,  staff  p. 41).  b e i n g done w i t h and f o r  Staff p a r t i c i p a t i o n a l s o provided  31 c r e d i b i l i t y and  served  to improve communication among s t a f f  members (Dawson & D'Amico, 1985). Subject,  or b e n e f i c i a r y , p a r t i c i p a t i o n was  to minimize the r e s e a r c h e r ' s  bias  (Reinharz,  also  1983),  discovered increase  e v a l u a t o r understanding of program background f a c t o r s , and v a l u a b l e i n f o r m a t i o n networks and review (Lawrence, 1989).  create  l i a i s o n s f o r the programs under  Researchers have a l s o found t h a t  b e n e f i c i a r y p a r t i c i p a t i o n i n e v a l u a t i o n i n c r e a s e d the v a l i d i t y of the data c o l l e c t i o n instruments and 1992;  Owston, 1986).  the r e s u l t s (Carey & Smith,  B e n e f i c i a r y involvement a l s o c r e a t e d a  sense of empowerment among p a r t i c i p a n t s and (Malekoff,  1994;  Routledge, 1993).  "The  l e d to s o c i a l a c t i o n  process  demonstrates  that consumers not o n l y know what they want, and  can  articulate  very c l e a r l y the r e a l i t y they f i n d themselves i n , but a l s o work a c t i v e l y and p.  c o l l e c t i v e l y to a f f e c t changes (Routledge,  1993,  106). A review of f i v e case s t u d i e s of s t a k e h o l d e r  mental h e a l t h c e n t r e s  ( D i n k e l , Zinober,  r e v e a l e d t h a t g i v e n the means and  involvement i n  & Flaherty,  the o p p o r t u n i t y ,  1981) consumers of  mental h e a l t h s e r v i c e s are capable of being a c t i v e l y meaningfully  i n v o l v e d i n the p r o f e s s i o n a l tasks  o r g a n i z a t i o n a l r e s p o n s i b i l i t i e s of e v a l u a t i o n . i n v o l v e d i n a v a r i e t y of tasks from reviewing to having  and Consumers were  evaluation findings  f u l l r e g u l a t o r y a u t h o r i t y f o r implementing  They p r o v i d e d u s e f u l i n f o r m a t i o n and  and  evaluations.  recommendations which were  determined to have d i r e c t impact on the programs s t u d i e d .  One  32 r e s e a r c h e r found  t h a t b e n e f i c i a r i e s of mental h e a l t h s e r v i c e s  a c t u a l l y tended to be more p o s i t i v e about a program's outlook than s t a f f  (Morrison,  1978).  Roles and R e l a t i o n s h i p s  Researchers  r e v e a l t h a t the i n c l u s i o n of s t a k e h o l d e r s i n  e v a l u a t i o n changes the r o l e of e v a l u a t o r s .  Patton  (1987) r e p o r t s  that: the involvement of program s t a f f or c l i e n t s as c o l l e a g u e s i n program e v a l u a t i o n changes the r e l a t i o n s h i p between e v a l u a t o r s and s t a f f . The r e l a t i o n s h i p becomes i n t e r a c t i v e and c o o p e r a t i v e r a t h e r than one-sided and a n t a g o n i s t i c , (p. 212) Role c o n f l i c t was  determined to be a major stumbling  to the e f f e c t i v e n e s s of CIS and E x c e l e v a l u a t i o n s Gold  (Cohen, 1983).  (1983) s t a t e s t h a t "the e f f e c t s of competing v a l u e s  p r i o r i t i e s p l a c e d the e v a l u a t o r i n c o n s i d e r a b l e r o l e (p. 67).  Gold concedes t h a t when r o l e e x p e c t a t i o n s  A l k i n & Associates  traditional  and  conflict" caused  u n c e r t a i n t y or c o n f l i c t i n CIS or E x c e l , the e v a l u a t o r s to a more f a m i l i a r and  block  regressed  role.  (1985) suggest  t h a t the c h o i c e of  role  can be r e l a t e d to the e v a l u a t o r ' s commitment to i n v o l v e users i n evaluation. predetermined  How  s t a k e h o l d e r s p a r t i c i p a t e can o f t e n be  by the r o l e s e l e c t e d by the r e s e a r c h e r .  I f the  e v a l u a t o r takes the r o l e of the detached, n e u t r a l judge or dedicated s c i e n t i s t ,  stakeholder p a r t i c i p a t i o n c o n s t i t u t e s  something v e r y d i f f e r e n t than i f the r e s e a r c h e r takes the r o l e of  33 a c o l l e a g u e o r advocate.  S i m i l a r l y , the r o l e a s s i g n e d t o a  s t a k e h o l d e r o f answering q u e s t i o n s determined by experts o r p a r t i c i p a t i n g i n d e c i s i o n making w i l l u l t i m a t e l y a f f e c t the continuum o f p a r t i c i p a t i o n .  The task o f i d e n t i f y i n g  stakeholder  groups can a l s o v a r y g r e a t l y , depending on the s k i l l o f the e v a l u a t o r s and t h e i r  experience.  E v a l u a t o r s h o l d v a r i o u s views, some competing, on the r o l e of the e v a l u a t o r i n c o l l a b o r a t i v e r e s e a r c h .  Cronbach (1982)  e n v i s i o n s the r o l e o f the e v a l u a t o r as one o f a c o n s u l t a n t o r a d v i s o r t o the p o l i t i c a l community.  He i d e n t i f i e s the r o l e o f an  e v a l u a t o r as one who should speak to the concerns o f the many c o n s t i t u e n t s w i t h i n t e r e s t s a t stake.  The " e v a l u a t o r " s f u n c t i o n  i s t o h e l p each person who holds a stake i n s o c i a l and i n s t i t u t i o n a l a c t i o n s t o understand  a proposed a c t i o n j u s t as  w e l l as s t a k e h o l d e r s w i t h competing i n t e r e s t s do" (Cronbach e t al,  1980, p. 66) .  Although  the r e s e a r c h e r i s compelled  to bring  i n competing views, Cronbach f e e l s t h i s can be accomplished the r e s e a r c h e r and does not r e q u i r e the s t a k e h o l d e r ' s  by  direct  involvement. Patton  (1988a, 1987, 1986) c o n s i d e r s the e v a l u a t o r t o be an  expert s c i e n t i s t , group f a c i l i t a t o r ,  and t e a c h e r . The e v a l u a t o r  i s seen as a t r a i n e r and f a c i l i t a t o r - - o n e who t r a i n s program managers and d e c i s i o n makers how t o use r e s e a r c h and f a c i l i t a t e s t h e i r involvement accomplished atmosphere.  f o r the purpose of s t i m u l a t i n g use.  This i s  through shared d e c i s i o n making i n a c o l l a b o r a t i v e  34 L i n c o l n & Guba (1981) suggest  t h a t the e v a l u a t o r "has  r i g h t to p r i o r i t i z e the audience i n terms of the l e v e l of each h o l d s , and  stake  to respond to them i n t h a t p r i o r i t y o r d e r to the  extent t h a t r e s o u r c e s permit" r o l e as one  the  (p. 304).  of human instrument,  They see the  evaluator s 1  c o l l a b o r a t o r , and i n v e s t i g a t o r ,  as w e l l as t h a t of r e a l i t y shaper and change agent. Some r e s e a r c h e r s argue t h a t the i n c l u s i o n of especially certain beneficiaries, spontaneously  stakeholders,  i s not l i k e l y to  and w i l l need advocate support.  develop  House (1986b,  1980), w e l l known f o r h i s r o l e as an advocate of disadvantaged  or  m i n o r i t y groups, argues t h a t the e v a l u a t o r has a r e s p o n s i b i l i t y to s e a r c h f o r and  l i s t e n to v u l n e r a b l e s t a k e h o l d e r s .  attempts to speak f o r s t a k e h o l d e r s as p a t e r n a l i s t i c . b e l i e v e s t h a t the r o l e of a democratic broker, pursued.  He  House  e v a l u a t o r i s one  e n s u r i n g t h a t a l l i n t e r e s t s are r e p r e s e n t e d and He d i s t i n g u i s h e s between an e v a l u a t o r who  versus one who Inherent  regards  of a actively  i s impartial  i s indifferent. i n a s t a k e h o l d e r approach to e v a l u a t i o n are  problems i n p e r s o n a l and r o l e r e l a t i o n s h i p s between r e s e a r c h e r and nonresearcher stakeholder.  p a r t i c i p a n t , r e g a r d l e s s of who  i s s e l e c t e d as a  C o n s i d e r a t i o n of r e l a t i o n a l dynamics would appear  to be an important  issue i n c o l l a b o r a t i v e research.  35 Ethical  Considerations  Mark and Shotland  (1985) r e v e a l that s t a k e h o l d e r - b a s e d  e v a l u a t i o n r a i s e s unique v a l u e q u e s t i o n s f o r e v a l u a t o r s . Inherent i n a s t a k e h o l d e r approach a r e v a l u e  judgements  c o n c e r n i n g the s e l e c t i o n o f s t a k e h o l d e r groups f o r p a r t i c i p a t i o n and which s t a k e h o l d e r groups' v a l u e s w i l l c o n t r o l the e v a l u a t i o n . These v a l u e judgements r a i s e complex i s s u e s o f power and c o n t r o l . The e v a l u a t o r o f t e n p l a y s a c r u c i a l r o l e i n d e c i d i n g who i s chosen as a s t a k e h o l d e r , determining what i s adequate group r e p r e s e n t a t i o n , and d e c i d i n g how much a u t h o r i t y i s g r a n t e d t o s t a k e h o l d i n g groups.  Out o f n e c e s s i t y , c h o i c e s a r e o f t e n made  among competing agendas t o r e p r e s e n t the i n t e r e s t s o f one group over another (Smith, 1985). Weiss  (1983a) i n q u i r e s "who d e f i n e s which groups a r e  s t a k e h o l d e r s and which groups a r e not?....[and] does the r i g h t to decide who i s i n and who i s out reduce the e f f i c a c y o f s t a k e h o l d e r e v a l u a t i o n as an instrument o f d e m o c r a t i z a t i o n ? " (p. 10). Kenny (1982) remarks that " i t  s t r i k e s me t h a t the  h i g h e s t form o f e l i t i s m occurs when persons unchosen by the disadvantaged say that they speak f o r the disadvantaged o r they say t h a t they take the disadvantaged's i n t e r e s t s i n t o account" (p. 21 & 2 2 ) . Smith (1985) argues t h a t "moral problems can a r i s e from unforseen consequences o r from the f a c t that the methods used t o implement an a c t i o n c o u l d i n f a c t thwart i t s achievement o r cause  36 o t h e r v a l u e s to s u f f e r "  (p. 8 ) .  For example, Mark & Shotland  (1985) r e v e a l the p o t e n t i a l problems of having an approach to s t a k e h o l d e r involvement:  "all-inclusive"  should a r a p i s t  be  c o n s i d e r e d a s t a k e h o l d e r i n an e v a l u a t i o n of rape laws?,  or  should a businessman whose primary  be  i n t e r e s t s are p r o f i t s  c o n s i d e r e d a l e g i t i m a t e s t a k e h o l d e r i n a community care home e v a l u a t i o n where substandard  care can r e s u l t i n i n c r e a s e d  profits? E v a l u a t o r s are c h a l l e n g e d to e x p l o r e the r a t i o n a l e f o r s t a k e h o l d e r involvement and honest Shotland;  d u r i n g the p l a n n i n g phase and  about t h e i r v a l u e c h o i c e s 1985).  Mark & Shotland  (Smith, 1985;  to be open  Mark &  (1985) recommend c l a s s i f i c a t i o n  of s t a k e h o l d e r s i n t o groups w i t h p e r c e i v e d power and  those w i t h  p e r c e i v e d l e g i t i m a c y of the s t a k e h o l d i n g group's i n t e r e s t s . suggest  They  t h a t h i g h u t i l i z a t i o n and d e c i s i o n making r a t i o n a l e s  should emphasize powerful  s t a k e h o l d e r s w h i l e empowerment  r a t i o n a l e s should i n c l u d e low-power s t a k e h o l d e r s . E v a l u a t o r s may p r e f e r not to d e a l w i t h such i s s u e s , but they must u n l e s s they choose to i n v o l v e a l l s t a k e h o l d e r s e q u a l l y , r e g a r d l e s s of the groups' l e g i t i m a c y - - a s t r a t e g y t h a t i t s e l f c l e a r l y i n v o l v e s a v a l u e judgement. (Mark & Shotland, 1985, p. 614) How  s t a k e h o l d e r s are i n v o l v e d i n e v a l u a t i o n can  be  r e p r e s e n t e d on a continuum, from token p a r t i c i p a t i o n to genuine power s h a r i n g .  I t has been i d e n t i f i e d by Maclure  "the terms ' p a r t i c i p a t i o n ' and  (1990) t h a t  ' p a r t i c i p a t o r y ' are ambiguous.  They express good i n t e n t i o n s , but they are e l u s i v e r e f e r e n c e s to the fundamentals of p r a c t i c e "  (p. 7 ) .  Cousins  & Earl  (1992)  37 reveal that p a r t i c i p a t i o n i s often l o o s e l y defined.  The  i n c l u s i o n of b e n e f i c i a r i e s i n p a r t i c u l a r , has been c h a l l e n g e d  as  to whether t h e i r i n c l u s i o n i s genuine c o r r o b o r a t i o n or j u s t a form of c o o p t a t i o n D'Amico, 1985;  (Prager,  Stake, 1983;  Mark & S h o t l a n d  1986;  Mark & Shotland, 1985,  Weiss, 1983b).  (1985) d i s c l o s e t h a t s t a k e h o l d e r  does not n e c e s s a r i l y l e a d to s t a k e h o l d e r stakeholder-based evaluations  that  stakeholder  They p r e s e n t  an a l t e r n a t i v e  of "pseudoempowerment" which exposes the p o t e n t i a l  r i s k of e v a l u a t o r s  having f u l l c o n t r o l of the p r o c e s s ,  n e g l e c t of s t a k e h o l d e r maintaining  involvement  i n f l u e n c e and  assume t h a t  p a r t i c i p a t i o n i s a p o s i t i v e value. perspective  Dawson &  concerns, or the r i s k of  c o n t r o l over s u b s t a n t i v e  stakeholders  issues,  power o n l y on i n s i g n i f i c a n t  transform  researchers  granting  does not  i n t o a democratic e v a l u a t i o n i n which  t h e i r i n t e r e s t s w i l l be represented  (Mathison, 1991) .  Smith  (1985) s t a t e s t h a t "moral problems can a r i s e because we much on our good i n t e n t i o n s that we e f f e c t s of our a c t i o n s  (p. 8).  ignore  focus  evaluations  House (1993, 1990)  reveals  evaluation.  that,  for  they do not have equal power to i n f l u e n c e or  nor do  they have equal p r o t e c t i o n from  Further,  properly represent  stakeholder  the  r e p r e s e n t a t i v e s may  the i n t e r e s t s of powerless groups.  (1983b) acknowledges t h a t :  so  the harmful s i d e  even when the powerless or poor have equal o p p o r t u n i t i e s participation,  the  issues.  E v a l u a t i o n which i n v o l v e s s t a k e h o l d e r s automatically  to  not Weiss  use  38 The s t a k e h o l d e r approach c o u l d be construed as a way o f d e f l e c t i n g s t a k e h o l d e r a t t e n t i o n from d e c i s i o n s t h a t more d i r e c t l y a f f e c t them....Whether a r e d u c t i o n o f i n e q u i t i e s i n the e v a l u a t i o n p r o c e s s r e s u l t s i n n e t g a i n s f o r a l l s t a k e h o l d e r s i s a matter t h a t deserves a t t e n t i o n , (p. 93) Dawson & D'Amico (1985) r e v e a l f u r t h e r r i s k s t o e v a l u a t i o n integrity: E v a l u a t o r s r u n the r i s k o f a v o i d i n g t h r e a t e n i n g i s s u e s , e q u i v o c a t i n g n e g a t i v e f i n d i n g s , o r g e n e r a l l y b e i n g l e s s than c a n d i d i n order t o p r o t e c t t h e i r r e l a t i o n s h i p s w i t h developer p a r t i c i p a n t s . Program s t a f f , on the o t h e r hand, run the r i s k o f r e l y i n g too h e a v i l y on i n f o r m a t i o n they f e e l they have helped generate, a c c e p t i n g and a c t i n g on e v a l u a t i o n recommendations a t face v a l u e without s u f f i c i e n t c r i t i c a l a n a l y s i s , (p. 194) When s t a k e h o l d e r s a r e i n v o l v e d i n the e v a l u a t i o n p r o c e s s i s a l s o regarded as a v a l u e judgement (Reineke,  1991).  C o l l a b o r a t i v e attempts w i t h s t a k e h o l d e r s r e v e a l v a l u e c h o i c e s concerning e v a l u a t i o n content. In t h e i r  important reflections  of an attempt t o a p p l y a c l i e n t - c e n t e r e d model o f i n s e r v i c e e d u c a t i o n i n p r a c t i c e , Tronya & F o s t e r (1988) were f a c e d w i t h d i f f e r e n c e s o f o p i n i o n between e v a l u a t o r s and s t a f f on what c o n s t i t u t e d e f f e c t i v e n e s s and c o n s t r a i n t s w i t h i n a t e a c h i n g context.  As r e s e a r c h e r s committed t o a n t i - r a c i s t e d u c a t i o n a l  p r i n c i p l e s , p u r s u i n g a s a t i s f a c t o r y r e s o l u t i o n o f the meanings of such " a b s t r a c t , complex and ( p o l i t i c a l l y ) d i s p u t e d terms" was found t o have enormous e t h i c a l i m p l i c a t i o n s .  They r e a l i z e d t h a t  any d e c i s i o n s they made as r e s e a r c h e r s and the r e s u l t i n g s t r a t e g i e s would be both judgemental and v a l u e - l a d e n . Does the r e s e a r c h e r abandon h i s / h e r p r i n c i p l e s t o the v a g a r i e s o f the marketplace? I f so, the apparent e g a l i t a r i a n r e l a t i o n s h i p between teacher ( c l i e n t ) and r e s e a r c h e r i s abandoned i n favour o f the former's priorities. I f the r e s e a r c h e r d e c i d e s n o t t o engage i n  39 c o l l a b o r a t i v e r e s e a r c h then the p r i n c i p l e of e g a l i t a r i a n i s m and c o o p e r a t i o n i s again s a c r i f i c e d : t h i s time on the a l t a r of the r e s e a r c h e r ' s p r i n c i p l e s and i n t e r e s t s , (p. 2 97) Although the r e s e a r c h e r s had hoped to remain t r u e to  the  p r i n c i p l e s of c o l l a b o r a t i o n , they found they were f a c e d w i t h e x a c t l y what the c l i e n t - c e n t e r e d model proposed to a v o i d . conclude t h a t " i t i s important a context  They  t h a t the r e s e a r c h i s conducted i n  i n which the e t h i c a l commitments of the r e s e a r c h e r have  been c l a r i f i e d "  (p.  298).  Relevance to S o c i a l Work  S o c i a l program e v a l u a t i o n i s an i n t e g r a l component of r o l e of s o c i a l work i n s o c i e t y . has c o n t i n u e d  The  r o l e of program e v a l u a t i o n  to grow and expand to i n c l u d e m u l t i p l e kinds  e v a l u a t i o n s and m u l t i p l e approaches to the e v a l u a t i o n The  the  i n f l u e n c e of s o c i a l work has  process.  served to enhance the  of program e v a l u a t i o n - - t o r e v e a l t h a t q u e s t i o n s r e s e a r c h v a r i e d from t h e i r s c i e n t i f i c  of  role  suited for social  counterpart.  Social  workers r e q u i r e d e v a l u a t i o n s t h a t would p r o v i d e u s e f u l and accurate  i n f o r m a t i o n about programs i n order to suggest  improvements and a s s i s t i n d e c i s i o n making p r o c e s s e s .  As  result,  that  they i n s i s t e d t h a t e v a l u a t i o n s be r e l e v a n t and  address the concerns of a v a r i e t y of  a they  stakeholders.  S o c i a l workers d e d i c a t e d to advancing the cause of oppressed and disadvantaged  groups must be s c e p t i c a l of e v a l u a t i o n s  c a t e r o n l y to the needs and wants of d e c i s i o n makers and  that policy  40 makers.  As r e s u l t s t h a t are s t a t i s t i c a l l y  substantively significant,  s o c i a l workers need to be  about the r e l e v a n c y of outcomes. a b l a c k box when one  s i g n i f i c a n t may  Essentially,  the examination of  i s standing o u t s i d e the box  i n s i d e the  be  concerned  looking i n  produce a d i f f e r e n t r e s u l t than an examination where one a c t u a l l y standing  not  may  is  box.  House (1986) i n d i c a t e s t h a t e f f o r t s to democratize evaluations  through the p a r t i c i p a t i o n of n o n - e v a l u a t o r s have not  been v e r y s u c c e s s f u l .  " I t i s . . . c l e a r that p a r t i c i p a t o r y  approaches have a v e r y l o n g way a g a i n s t e v a l u a t i o n conceived by e x p e r t s "  (p. 9).  Murray  stakeholder  process  provide  to go and  f a c e an u p h i l l  struggle  as a p u r e l y t e c h n i c a l a r t conducted (1983) concludes t h a t "aspects cause f o r prolonged debate  of  the  and  d i s c u s s i o n w i t h i n the e v a l u a t i o n community", i n d i c a t i n g t h a t i t i s the " i n t e n s e , c o n t i n u a l i n t e r a c t i o n s t h a t i t r e q u i r e s w i t h a l l the p a r t i e s to an e v a l u a t i o n " which i s both i t s s t r e n g t h and i t s danger (p.  60).  Although the concept of s t a k e h o l d e r e v a l u a t i o n o r i g i n a t e d more than 3 0 years "taken o f f " as might be expected.  The  participation in ago,  l i t e r a t u r e reveals  d i s c r e p a n c i e s t h a t e x i s t between n e g a t i v e stakeholder  process.  expectations  p a r t i c i p a t i o n by e v a l u a t o r s and  genuine a b i l i t y  the concept has  to p a r t i c i p a t e m e a n i n g f u l l y  the  of  stakeholders'  i n the  evaluation  In f a c t , c o l l a b o r a t i v e e v a l u a t i o n i n a c t i o n  has  r e v e a l e d more p r a c t i c a l and o p e r a t i o n a l l i m i t a t i o n s than difficulties  due  to p e r s o n a l  a t t r i b u t e s of  the  stakeholders.  not  41 Routledge  (1993) suggests t h a t the b e n e f i t s outweigh the  c o n s t r a i n t s : "at times f a c i l i t a t o r s have to g i v e a g r e a t d e a l of themselves, but t h a t i s more than r e t u r n e d by the support encouragement they r e c e i v e " (p. 107). ambiguity and u n c e r t a i n t y continues stakeholders  However, e v a l u a t o r  to h i n d e r  the f a c i l i t a t i o n of  i n evaluation research.  Whereas the n o t i o n of b r i n g i n g i n v i e w p o i n t s r e s e a r c h e r s , c i t i z e n s , and stakeholder-based conceptualized. unresolved  and  service recipients i s  e v a l u a t i o n has not been E v a l u a t o r s are s t i l l  i s s u e s concerning  disagreements about how  of  non-  admirable,  adequately  c o n f r o n t e d w i t h a s e r i e s of  stakeholder  collaboration, including  i t should be implemented  (House, 1993).  Greene (1988b) r e v e a l s t h a t : many of the t h e o r e t i c a l and o p e r a t i o n a l elements of s t a k e h o l d e r p a r t i c i p a t i o n are not y e t w e l l understood, f o r example, s t a k e h o l d e r d e f i n i t i o n and s e l e c t i o n , the nature and meaning of p a r t i c i p a t i o n , and r o l e e x p e c t a t i o n s of p a r t i c i p a t o r y r e s e a r c h e r s , (p. 341) Cousins and E a r l  (1990) c o n f i r m t h a t "we  need to know a l o t more,  however, about the c o n d i t i o n s w i t h i n which p a r t i c i p a t i o n i s s e n s i b l e and  feasible"  (p.  408).  Far from b e i n g a n a t u r a l phenomenon, p a r t i c i p a t i o n o f t e n i n v o l v e s the r e s t r u c t u r i n g of r e l a t i o n s h i p s and  the r e s t r u c t u r i n g  of power w i t h i n those r e l a t i o n s h i p s . Good i n t e n t i o n s f o r m u l t i p l e s t a k e h o l d e r p a r t i c i p a t i o n can be i n f l u e n c e d by the  role  taken on by the e v a l u a t o r , the r o l e g i v e n to the s t a k e h o l d e r , u l t i m a t e l y , by how  a stakeholder  i s defined.  It i s essential  that c o n s i d e r a t i o n i s g i v e n to these r e l a t i o n a l dynamics  and  and  42  t h e i r i m p l i c a t i o n s before evaluation.  embarking on a s t a k e h o l d e r  approach to  Weiss (1983) a n t i c i p a t e s t h a t :  As experience accumulates and i f we c o n s c i e n t i o u s l y l e a r n from t h a t experience, we should be a b l e to s p e c i f y the c o n d i t i o n s under which the stakeholder approach i s l i k e l y to prove u s e f u l and probe the r e a l i s t i c l i m i t s of i t s p o t e n t i a l , (p. 10) Further  study on the nature and  p a r t i c i p a t o r y process,  dynamics of  the  i n c l u d i n g o p e r a t i o n a l dimensions  r e l a t i o n a l dynamics, i s necessary.  An  the c e n t r a l concepts of stakeholder  c o l l a b o r a t i o n may  o p e r a t i o n a l and  contextual  and  i n c r e a s e d u n d e r s t a n d i n g of reveal  the  f a c t o r s necessary to c o n s i d e r p r i o r  embarking on such an e v a l u a t i o n  approach.  to  43 Chapter 2  METHODOLOGY  Research Design  Although  the r a t i o n a l e f o r the i n c l u s i o n o f s t a k e h o l d e r s i n  e v a l u a t i o n r e s e a r c h i s compelling, c o l l a b o r a t i v e r e s e a r c h , i n practice, i s s t i l l rule.  found t o be the e x c e p t i o n r a t h e r than the  A p e r v a s i v e l a c k of i n f o r m a t i o n and understanding  o f the  nature and consequence o f s t a k e h o l d e r c o l l a b o r a t i o n has hampered s t a k e h o l d e r i n c l u s i o n i n the e v a l u a t i o n p r o c e s s .  This i s  r e v e a l e d i n the d i s c r e p a n c y between the p e r c e i v e d l i m i t a t i o n s o f the approach and the t a n g i b l e and v e r i f i a b l e r e s u l t s as d i s c u s s e d in  the l i t e r a t u r e review.  I t i s apparent  that stakeholder  c o l l a b o r a t i o n s t i l l p r e s e n t s a tremendous c h a l l e n g e t o most evaluation researchers. T h i s e x p l o r a t o r y document study employs q u a l i t a t i v e a n a l y s i s to  e x p l o r e the nature o f s t a k e h o l d e r involvement  evaluation.  i n program  The study i s guided by the r e s e a r c h q u e s t i o n :  What  c o n t r i b u t e s t o the e f f e c t i v e p a r t i c i p a t i o n o f s t a k e h o l d e r s i n program e v a l u a t i o n ?  L i t e r a t u r e r e p o r t s and e v a l u a t i o n s t u d i e s  r i c h i n feedback on s t a k e h o l d e r involvement  i n e v a l u a t i o n were  a n a l y z e d i n an attempt p r o v i d e an understanding  o f key f a c t o r s  that f a c i l i t a t e e f f e c t i v e c o l l a b o r a t i o n w i t h s t a k e h o l d e r s .  The  phenomenological focus of the a n a l y s i s h i g h l i g h t s the o p e r a t i o n a l  44 dimensions of s t a k e h o l d e r p a r t i c i p a t i o n .  Special consideration  i s g i v e n to the nature of i n t e r a c t i o n s between r e s e a r c h e r s t a k e h o l d e r and  the c o n t e x t u a l f a c t o r s which may  affect  and  the  development o r success of p a r t i c i p a t i o n .  Sampling  Design  An e x t e n s i v e review of the l i t e r a t u r e was  undertaken by  the  r e s e a r c h e r to l o c a t e documents p e r t a i n i n g to program e v a l u a t i o n and  the use or nonuse of s t a k e h o l d e r s and  participants.  Documents were sought which were r i c h i n d e t a i l s and i n c o n t e x t u a l i n f o r m a t i o n on the o p e r a t i o n a l dimensions of s t a k e h o l d e r involvement reviews found  i n program e v a l u a t i o n .  Case s t u d i e s and  of s t a k e h o l d e r approaches to e v a l u a t i o n r e s e a r c h were  to c o n t a i n v a l u a b l e f i r s t - h a n d accounts  processes.  of p a r t i c i p a t o r y  These documents were o f t e n the c u l m i n a t i o n of  e x t e n s i v e f i e l d notes by the r e s e a r c h e r concerning  the  p a r t i c i p a t o r y process and i n c l u d e d both r e s e a r c h e r  and  s t a k e h o l d e r r e f l e c t i o n s on the p a r t i c i p a t o r y s t r a t e g i e s . authors  i n c l u d e d a c t u a l q u o t a t i o n s of r e a c t i o n s from  to the p a r t i c i p a t o r y p r o c e s s .  Many  stakeholders  These documents allowed  the  o p p o r t u n i t y to examine a d i v e r s e range of p a r t i c i p a t o r y approaches o p e r a t i n g w i t h i n v e r y d i f f e r e n t  contexts.  P u b l i c a t i o n s and documents w r i t t e n from p o p u l a r f o u r t h g e n e r a t i o n r e s e a r c h , and  research,  f e m i n i s t r e s e a r c h were p u r p o s e l y  45 excluded as data f o r t h i s study, s p e c i f i c a l l y because each i s p r e s e n t e d as a unique  and d i f f e r e n t r e s e a r c h paradigm.  Sample  A p u r p o s e f u l sample of twelve documents comprise f o r t h i s study.  the data  These twelve documents i n c l u d e b o t h case s t u d i e s  and r e t r o s p e c t i v e reviews and p r o v i d e a u t h e n t i c f i r s t - h a n d accounts of s t a k e h o l d e r involvement settings.  i n a v a r i e t y of e v a l u a t i o n  The a r t i c l e s i l l u m i n a t e v a r i o u s o p e r a t i o n a l dimensions  of s t a k e h o l d e r p a r t i c i p a t i o n ,  the r e l a t i o n a l dynamics between  r e s e a r c h e r and s t a k e h o l d e r , and the c o n t e x t u a l f a c t o r s participation.  Patton  affecting  (1991) q u a l i f i e s t h a t " i n f o r m a t i o n - r i c h  cases are those from which one can l e a r n a g r e a t d e a l about i s s u e s of c e n t r a l importance the term  to the purpose of the r e s e a r c h , thus  ' p u r p o s e f u l sampling'"  (p. 169).  D e s c r i p t i o n of Sample  A l k i n , M. C , & Patton, M. Q. the S t r e e t , pp. 19-32.  (1987).  Working Both Sides of  U s i n g both a n a r r a t i v e and c o n v e r s a t i o n a l format, A l k i n  and  Patton d i s c u s s the process of e v a l u a t i o n and the elements i n v o l v e d i n t h e i r attempt director  to work c o l l a b o r a t i v e l y both as program  (Patton) and e v a l u a t o r (Alkin) on a  A g r i c u l t u r a l Extension Project. p r o j e c t was  Caribbean  The purpose of t h i s $5 m i l l i o n  to i n c r e a s e the e f f e c t i v e n e s s of a g r i c u l t u r a l  46 extensions  s e r v i c e s i n e i g h t Caribbean c o u n t i e s .  i n v o l v e d such key s t a k e h o l d e r s officials,  as program s t a f f , government  farmers, and academic s t a f f .  covered a two-year p e r i o d . the "process"  This project  The e v a l u a t i o n ' s  scope  The focus o f t h e i r d i s c u s s i o n i s on  i n which t h e i r p a r t n e r s h i p evolved,  i n c l u d i n g the  r o l e s and r e l a t i o n s h i p s between team members, how the e v a l u a t i o n was focused,  and the b e n e f i t s o f m u l t i p l e  stakeholder  involvement. A l l e n , D. G. e t a l . (1994). One System, many p e r s p e c t i v e s : Stakeholders and Mental H e a l t h System E v a l u a t i o n , pp. 47-51. In what appears t o be a p r e l i m i n a r y r e p o r t , these authors d e s c r i b e the use o f a m o d i f i e d  a c t i o n r e s e a r c h model t o i n t e g r a t e  the p e r s p e c t i v e s o f m u l t i p l e s t a k e h o l d e r s s t a t e ' s mental h e a l t h reform.  i n the e v a l u a t i o n o f a  T h i s two-year e v a l u a t i o n was  c o n t r a c t e d by the L e g i s l a t i v e Budget Committee.  The e v a l u a t i o n  team attempted t o broaden the range o f p e r s p e c t i v e s on mental h e a l t h system e v a l u a t i o n by i n c o r p o r a t i n g as many s t a k e h o l d e r s as p o s s i b l e i n t o the study through the use o f an a d v i s o r y panel and through i n t e r v i e w s and p r e s e n t a t i o n s  t o i n f l u e n t i a l and p o l i t i c a l  groups.  group, e v a l u a t o r s r e c r u i t e d  To comprise the s t a k e h o l d e r  members from consumer and f a m i l y groups, s e r v i c e p r o v i d e r s , administrators,  r e g i o n a l o f f i c e s and s t a t e h o s p i t a l s , and v a r i o u s  l e g i s l a t i v e and governmental members. stakeholders  The authors recount  were i n v o l v e d i n v a r i o u s aspects  from the d e f i n i t i o n o f r e s e a r c h q u e s t i o n s , measurement, and a n a l y s i s .  how  o f the e v a l u a t i o n ,  t o sampling,  47 Ayers, T. D. (1987). Stakeholders as P a r t n e r s i n E v a l u a t i o n : A S t a k e h o l d e r - C o l l a b o r a t i v e Approach, pp. 263-271. T h i s paper p r e s e n t s a case study o f a s t a k e h o l d e r c o l l a b o r a t i v e approach i n an e d u c a t i o n a l forum.  The e v a l u a t i o n  i n v o l v e d a p u b l i c s c h o o l system l o c a t e d on the i s l a n d o f Guam--an unincorporated  t e r r i t o r y o f the U n i t e d S t a t e s .  The s c h o o l system  had r e c e n t l y been r e o r g a n i z e d t o i n c l u d e a c o n t r o v e r s i a l s h i f t from a j u n i o r h i g h structure.  (grades 7-9) t o a middle  (6-8) s c h o o l  An e v a l u a t i o n o f the middle s c h o o l system was  o r g a n i z e d and, i n response t o the h i g h l y p o l i t i c i z e d surrounding  situation  t h i s i s s u e , r e s p o n s i b i l i t y was a s s i g n e d t o a task  force of non-evaluation  personnel.  T e c h n i c a l a s s i s t a n c e was  p r o v i d e d by an e v a l u a t i o n c o n s u l t a n t who was a non-voting of the task f o r c e d u r i n g t h i s y e a r - l o n g study.  member  The task f o r c e  i n c l u d e d s c h o o l a d m i n i s t r a t o r s , teachers, union r e p r e s e n t a t i v e s , and community r e p r e s e n t a t i v e s .  Ayers d i s c u s s e s the procedures  and r o l e s o f p a r t i c i p a n t s i n the design, a d m i n i s t r a t i o n , and i n t e r p r e t a t i o n phases o f the e v a l u a t i o n .  B a r r i c k , C. B., & Cogliano, J . F. (1993). Stakeholder Involvement: Mythology o r Methodology?, pp. 33-37. T h i s paper d e s c r i b e s the e f f o r t s of a committee t o i n v o l v e n u r s i n g f a c u l t y as s t a k e h o l d e r s i n a program e v a l u a t i o n to meeting a c c r e d i t a t i o n requirements. experiences  pursuant  The authors d e s c r i b e the  o f the committee over a f i v e - y e a r p e r i o d , p l a c i n g an  emphasis on s t r a t e g i e s used t o f o s t e r success.  The  combination  of a number o f f a c t o r s had r e s u l t e d i n the committee p e r c e i v i n g  48 the implementation of any paper d e s c r i b e s  e v a l u a t i o n p l a n to be  "myth".  This  the committee's s u c c e s s f u l move toward a s t r a t e g y  of methodology which c l a r i f i e d t h e i r r o l e as the e v a l u a t o r the s t a f f ' s r o l e i n data c o l l e c t i o n and  and  i n t e r p r e t a t i o n and use  of  findings.  Brandon, P. R., Newton, B. J . , & Harman, J . W. (1993). Enhancing V a l i d i t y Through B e n e f i c i a r i e s ' E q u i t a b l e Involvement i n I d e n t i f y i n g and P r i o r i t i z i n g Homeless C h i l d r e n ' s E d u c a t i o n a l Problems, pp. 287-293. Brandon et a l . d i s c u s s the involvement of b e n e f i c i a r i e s ' i n e v a l u a t i o n and p r e s e n t  an example of an e v a l u a t i o n designed to  enhance v a l i d i t y through e q u i t a b l e p a r t i c i p a t i o n .  This state-  wide study of the e d u c a t i o n a l problems of homeless c h i l d r e n i n Hawaii i n v o l v e d teachers,  homeless s h e l t e r p r o v i d e r s , and  (the program b e n e f i c i a r i e s ) as s t a k e h o l d e r s .  parents  They d i s c u s s  methods on i d e n t i f y i n g and p r i o r i t i z i n g the e v a l u a t i o n problem and  r e c o n c i l i n g d i f f e r e n c e s between groups, f o c u s i n g on how  i n v o l v e d a l l three s t a k e h o l d e r  groups.  The  authors r e v e a l  they the  procedures used to ensure e q u i t a b l e p a r t i c i p a t i o n by b e n e f i c i a r i e s and how  b e n e f i c i a r y involvement enhanced  relevance.  Crow, G. M., Levine, L., & Nager, N. (1992). Are Three Heads B e t t e r Than One? R e f l e c t i o n s on Doing C o l l a b o r a t i v e I n t e r d i s c i p l i n a r y Research, pp. 737-753. T h i s paper p r e s e n t s  r e f l e c t i o n s on an i n t e r d i s c i p l i n a r y  c o l l a b o r a t e approach to conducting who  l e f t o t h e r occupations  an in-house study of  f o r teaching.  and  students  The p o r t i o n s of  this  paper which r e l a t e to the i n t e r d i s c i p l i n a r y attempt among the  49 r e s e a r c h e r were not reference  i n c l u d e d as data,  although they p r o v i d e d  f o r data on t h e i r c o l l a b o r a t i v e attempts.  d e s c r i b e and  analyze  authors  t h e i r attempts at e x t e r n a l c o l l a b o r a t i o n  w i t h both f a c u l t y a d v i s o r s and observations  The  a  students  and  include  on what f a c i l i t a t e d and c o n s t r a i n e d  their  this  process.  T h i s a r t i c l e i s somewhat unique i n t h a t the g o a l s and c o l l a b o r a t i o n evolved  d u r i n g the course of the  roles for  study.  Faase, T. P., & Pujdak, S. (1987). Shared Understanding of O r g a n i z a t i o n a l C u l t u r e , pp. 75-82. T h i s a r t i c l e d e s c r i b e s the shared understanding t h a t developed between the c l i e n t and  evaluator  Roman C a t h o l i c r e l i g i o u s o r g a n i z a t i o n .  i n an e v a l u a t i o n of a  For some time,  the  a d m i n i s t r a t i v e s t a f f w i t h i n the r e l i g i o u s order had been the need and urgency f o r retrenchment and members.  sensing  c o n s o l i d a t i o n of  The purpose of the e v a l u a t i o n was  to c o n s i d e r  the  i m p l i c a t i o n s f o r the p r o v i n c e  of the occurrence and  t h i s retrenchment.  chosen as e v a l u a t o r because of h i s  Faase was  t i m i n g of  knowledge about the s o c i o l o g y of r e l i g i o n and p r e v i o u s in religious settings.  Both Faase (evaluator)  and  research  Pujdak  ( c l i e n t ) i n c l u d e t h e i r r e f l e c t i o n s on t h i s shared e v a l u a t i o n attempt which r e q u i r e d mutual support  and c o l l a b o r a t i o n .  G i l l , S. J . , & Zimmerman, N. R. (1990). R a c i a l / E t h n i c Gender B i a s i n the Courts; A Stakeholder-Focused E v a l u a t i o n , pp. 103-108. Gill  & Zimmerman d e s c r i b e a s t a k e h o l d e r - f o c u s e d  used to e v a l u a t e  and  approach  r a c i a l / e t h n i c and gender b i a s i n the Michigan  50 c o u r t system.  These two e v a l u a t o r s were i n v i t e d t o a s s i s t i n  g a t h e r i n g and a n a l y z i n g data f o r two task f o r c e s appointed by the S t a t e Supreme Court to study these i s s u e s .  The task f o r c e s  i n c l u d e d judges, a t t o r n e y s , c o u r t a d m i n i s t r a t o r s , and o t h e r p r o f e s s i o n a l s e x p e r i e n c e d w i t h the c o u r t system.  Stakeholders  were i n v o l v e d i n the i n i t i a l study d e s i g n , development of data c o l l e c t i o n instruments, data c o l l e c t i o n , and i n t e r p r e t a t i o n of r e s u l t s of t h i s two-year p r o j e c t .  Stakeholders were i n v o l v e d f o r  t h e i r content and language e x p e r t i s e and to enhance  utilization.  G i l l and Zimmerman d i s c u s s the a p p l i c a t i o n of a s t a k e h o l d e r focused model of e v a l u a t i o n to a s e n s i t i v e and complex r e s e a r c h p r o j e c t and d e s c r i b e each l e v e l of s t a k e h o l d e r  involvement.  Greene. J . C. (1987). Stakeholder P a r t i c i p a t i o n i n E v a l u a t i o n Design: Is i t Worth the E f f o r t ? , pp. 379-394. T h i s r e p o r t reviews the d e s i g n and p l a n n i n g phase of two case study e v a l u a t i o n s i n which a p a r t i c i p a t o r y d e s i g n p r o c e s s i s operational within a u t i l i z a t i o n rationale.  Two programs, a  youth employment s e r v i c e and a c h i l d c a r e i n f o r m a t i o n and r e f e r r a l s e r v i c e , were chosen f o r these two-year reviews.  Stakeholders  were i d e n t i f i e d w i t h the h e l p of agency s t a f f and i n c l u d e d funders, program s t a f f , a d m i n i s t r a t i v e s t a f f , board members, program c l i e n t s and u s e r s , and community  representatives.  An  e v a l u a t i o n team comprised a s e l e c t group of these s t a k e h o l d e r s and e v o l v e d i n t o a s e l f - c o n t a i n e d d e c i s i o n making body.  In  keeping w i t h the u t i l i z a t i o n focus, s t a k e h o l d e r s were g i v e n r e s p o n s i b i l i t y f o r content w h i l e the e v a l u a t o r m a i n t a i n e d  51 r e s p o n s i b i l i t y f o r process d i s c u s s e s the nature participation, and  and  t e c h n i c a l content.  and meaningfulness of  the r o l e s and  Greene  stakeholder  r e s p o n s i b i l i t i e s of p a r t i c i p a n t s ,  t h e i r involvement i n d e c i s i o n making.  Greene, J . C. (1988). Stakeholder P a r t i c i p a t i o n U t i l i z a t i o n i n Program E v a l u a t i o n , pp. 91-106.  and  Greene e l a b o r a t e s on her e a r l i e r work (see above) to the l i n k a g e s between p a r t i c i p a t i o n and use employment and daycare e v a l u a t i o n s .  i n the youth  An e x c e l l e n t d i s c u s s i o n on  the meaning of s t a k e h o l d e r p a r t i c i p a t i o n supplements her a r t i c l e , as Greene r e v e a l s the experiences r e p o r t e d by three s t a k e h o l d e r s .  explore  previous  of p a r t i c i p a t i o n  In t h i s study,  as  Greene d e l v e s  i n t o an e l a b o r a t e d i s c u s s i o n on the l i n k s between p a r t i c i p a t i o n and u t i l i z a t i o n ,  and  that they p r o v i d e d  these s e c t i o n s were o n l y used to the  i n f o r m a t i o n r e l e v a n t to t h i s  extent  study.  Shapiro, J . Z., & B l a c k w e l l , D. L. (1987). Large S c a l e E v a l u a t i o n s on a L i m i t e d Budget; The P a r t n e r s h i p Experience, pp. 53-62). Using an i n f o r m a l , c o n v e r s a t i o n a l format, t h i s r e p o r t s on the d e s i g n and  implementation of a p a r t n e r s h i p  e v a l u a t i o n a t Southeastern were s l a t e d f o r review. (Blackwell) and  article  Louisiana University.  Six projects  Project director/internal  evaluator  external evaluator  (Shapiro) proposed a s t r a t e g y  f o r s h a r i n g e v a l u a t o r r e s p o n s i b i l i t i e s between u n i v e r s i t y s t a f f and  e v a l u a t o r i n an e f f o r t to enhance e v a l u a t o r c o n s u l t a n t  time  52 and e x p e r t i s e and minimize c o s t .  During  t h i s year-long  study,  each program d i r e c t o r took r e s p o n s i b i l i t y f o r t h e i r own e v a l u a t i o n a c t i v i t i e s w i t h t e c h n i c a l a s s i s t a n c e and guidance p r o v i d e d by the e x t e r n a l e v a l u a t o r . as c l i e n t and program s t a f f , and  L i m i t e d t o the s t a k e h o l d e r  t h i s paper d e s c r i b e s the e x t e r n a l  i n t e r n a l p e r s p e c t i v e s on the sequence o f events,  as w e l l as  the meaning and i m p l i c a t i o n s o f p a r t n e r s h i p s t h a t emerged d u r i n g the e v a l u a t i o n .  Whitmore, E. (1900). Empowerment i n Program E v a l u a t i o n , pp. 215- 229. Whitmore e x p l o r e s  the ways i n which b e n e f i c i a r i e s a r e  empowered through t h e i r p a r t i c i p a t i o n as e v a l u a t i o n s t a f f o f a p r e n a t a l program f o r s i n g l e expectant  mothers.  Program  p a r t i c i p a n t s were h i r e d and t r a i n e d t o a s s i s t i n conducting the e v a l u a t i o n i n o r d e r to i n c r e a s e the v a l i d i t y o f the d a t a . e v a l u a t o r worked c l o s e l y w i t h an e x i s t i n g  The  community-based  a d v i s o r y committee o f s t a k e h o l d e r s who oversaw the e v a l u a t i o n . Whitmore d i s c u s s e s the r o l e o f group cohesion  on the  p a r t i c i p a t o r y p r o c e s s , w i t h p a r t i c u l a r emphasis on the program p a r t i c i p a n t s ' p e r c e p t i o n s o f empowerment through t h e i r involvement i n t h i s y e a r - l o n g  evaluation.  rjd co M 0"  H-  0 "0 (D 0  H  I- R> 1  vo vo o  H VO VO O —  , . H VD 00 -J  S n  H ft  —O  VD M 00 (D 00 ID  p> III  I  O  H  HH M  Ri  — *fl H tf VO PJ oo to ~J ID •—  N  <XJ  C  5  0.  ID  <«  H 3  vo 00 •J  Pi  **  tf  58 o  PI (Q ID M  — W H K vo PI VO 0 OJ q .  M  0 3  »  — 0 0  — f H ID vo <i VO H-  ID rt  — ID  Pi •  R»  O  W  >  IQ H HPl  H M HO  ID M ca  0 PI  0 0  H M VO M VO ID  *. 0  —'  ID rt  ff  VO 00 -J '—  ,—. H VO VO LO "•^  P> •  >  O  vo 00 H-J 0  o  —  nt PI lift  ID ca H-  CO rt  •0  (L  <  C  ft H-  •<  a> fl  0 0  0*  tf M  PI  u  C M rt H"  (D  H  ca H-  H rr  n> •<:  W ID Mi ID H  pi  c  co w o  O CO pl ID rt  ^ H- Hca pi PI rt ca ID S H- P  H H V  ID  <  N H PI 0 ID TJ  0  M Mi co ID M < H-  tfl O rt  0 fr> O g  gF M H-  3  pi rt H- H  O 0  o  rt  ft  O  O  ID  H  H0 IQ  M  ID IQ ca pl rt H-  O  0  0 0  IQ rt ID fl Pi H ID ID H  S O  >0 M g M Oi ID O C  cr o  C  M  PI  ID ID  TJ ft) CL H O-IQ  o  O W ID  "SI SI a  HK ID  O  rt  O M  a  Ml H" 10  ID 0 «• ca Hca H- rt o Pi 0 rt pl O _ M M ca ca  ID ID HCO M ca ca ID ca rt rt 0 •> H pi PI Ml rt pi TJ rt Mi rt M H- •> H- O •< (Q ID Ml ID M 3 ca 0 ca rt 0. O PI ID M Mi H H- Mi ca ID ' -  <  rt cr ca 0 » Pi M  0  0 0*  Pi  a>  0* © p rr  H-  O  4  es  rt  ID IQ  B X  ID  H  0"  3  ID  irj M pi  R  0  H-  O 0  0 IQ ca rt Pi Ml Ml  o.  HM ID O rt  0  O  C H  rt •<! Pi  ca  in ca rt  d  rt HH  HN  H  rt ca ID ca  (Q  •0  Pi  H ID  C  M  Pi  0  a> rtca ID »  H  ca rt  PI rt H"  s? O  •< ID PI M ca  ca H-  0  O M  rt  to rt  PI  ca rt Pi  ID  Pi  0" 0  0"  &  ID  M  o  C  a  ID  ID H  ID  0  ilea  H  to  ca  ca ca  0 0  1  •d  o  H Hrt H-  H H PI  0  o w H  to H-  X  ID PI  8 0  CO  0"  H  0  cr 0  M PI ft  rt  o 0  ID •< ID PI H  M  0 •fl <5 0" H- PI PI  0 M 0 0- H- o H- (h ID 0 H-  IQ ft ca  f  0  (Q H" ca rt H-  O  PI H  •d  0  H Hrt H-  O  PI H  3  «! ID  ID  •<! ID  ID PI  tf M ta  O M HID  » (D ca ID  0 tf ft H  > O  a  0 0  H  ft H-  H  rt HH-  N  tf rt H"  Pl O ft 0*  O  0  0  O  ID  •< ID tf  ca  ca  tf H  Hi co  ID  ID H ID  pi rt R> 0 HIQ M O C H0 PI M 0 IQ PI ft ID ft ID H-  ID H ft Hca 0 ID rt  1 H  0  1  •0  o>  0  x tr ID ID H H ID  ta  0  H  0* 0  t" c!  W X  0  cfl  1  ca 3  Hca  O M  0 M  CO HH- O  0 c 0 I-  On  ID PI  Pi  <J- tf cr ID M ID O Pl rt M 0  1  r x  ID  M £ M O H H-  ID  m rt  Of  O  0  0  IQ K ID (Q pi rt H-  •fl Pi  O ID  0 0  3  c  >d rt M  n rt  ion  ^  ID  3  M  M  0  Hft  H- O (D  o  •0 (t> HID O ca O ft  ID M HID  0  < 0"d p> P> o M 0 $  (D  •fl P  •a  B  o o o  ft  ca O O  ID rt ca  ID  o a  a  pi rt  ca H- tP ft 0* 0 O rt 0 O V C 0 ID M C M rr H rt  0  2£  O  (Q  to  g-i O S O HM a CO M ID  M ID  UOT  9  0  CO p)  rt ID M Hca rt H-  0 tf  0  •0  n  n H"  PI ca  (_l  ID 0  O IQ  ID  flM  O C  va ua  (D ft ft H«Q ca  9 s H" <!  CO ft O C 0" P. PI ID  s  0  rt H-  tf rt  H-  0 0  i-3 H-  Ml  n i  ID  54 Sample  Composition  Nine of the s e l e c t e d documents were w r i t t e n o r co-authored by e v a l u a t i o n r e s e a r c h e r s w i t h the e x c e p t i o n o f one with a p r i e s t .  Of these, three co-authors were a l s o the  evaluation c l i e n t s . the same author  co-authored  Two of these nine s t u d i e s were w r i t t e n by  (Greene,  aspects of the same data.  1988a, 1987) who r e p o r t e d on d i f f e r e n t The a d d i t i o n a l t h r e e documents were  w r i t t e n by m u l t i p l e authors w i t h one authored by a s o c i o l o g i s t , a n t h r o p o l o g i s t , and developmental by three U n i v e r s i t y s t a f f  p s y c h o l o g i s t ; another w r i t t e n  ( p r o f e s s i o n s not i n d i c a t e d ) ; and  another by s i x authors, f i v e of whom are u n i v e r s i t y r e p r e s e n t i n g v a r i o u s departments  faculty  (two from p s y c h o s o c i a l n u r s i n g ,  two from s o c i a l work, one from p s y c h i a t r y ) , w i t h the remaining author from a l e g i s l a t i v e committee.  The s e l e c t e d documents were  p u b l i s h e d between 1987 and 1994, w i t h f i v e of the documents p u b l i s h e d i n 1987. A l l of the a r t i c l e s are case s t u d i e s of c o l l a b o r a t i v e r e s e a r c h attempts,  however, some p r o v i d e a v e r y formal  r e t r o s p e c t i v e review w h i l e o t h e r s a r e w r i t t e n more as an i n f o r m a l review and d i a l o g u e of s p e c i f i c case examples. The e v a l u a t i o n s were conducted s e t t i n g s and c o n t e x t s : an elementary  i n strikingly  different  s c h o o l o u t s i d e the U.S., a  j u d i c i a l system, a r e l i g i o u s congregation, an employment program and c h i l d c a r e c e n t r e , a p r e n a t a l program f o r s i n g l e mothers, mental h e a l t h s e r v i c e s , and an overseas  expectant  agricultural  55 project. needs.  One study e v a l u a t e d homeless c h i l d r e n ' s e d u c a t i o n a l The t h r e e s i t e s that were u n i v e r s i t y based i n c l u d e d  e v a l u a t i o n s w i t h i n a f a c u l t y of n u r s i n g , a f a c u l t y of education, and a d i v e r s i t y of departmental  s e t t i n g s w i t h i n one u n i v e r s i t y  i n c l u d i n g e l e c t r o n i c l e a r n i n g , development f o u n d a t i o n , c o u n s e l l i n g , b u s i n e s s education, i n d u s t r i a l technology and chemistry/physics.  A l l of the e v a l u a t i o n s were i n i t i a t e d by  e v a l u a t o r s o r agencies--none were c i t i z e n o r consumer i n i t i a t e d . The  s t u d i e s o f f e r a v a r i e t y of s t a k e h o l d e r p a r t i c i p a n t s from  program funders, s t a f f , and p r o f e s s i o n a l s , t o b e n e f i c i a r i e s and c i t i z e n s , w i t h each of these s t a k e h o l d e r s i n t e r a c t i n g i n a v a r i e t y of r o l e s .  In a l l but f o u r s t u d i e s , s t a k e h o l d e r s  p a r t i c i p a t e d s u b s t a n t i a l l y i n almost a l l f a c e t s of the evaluation.  S t a k e h o l d e r s shared r e s p o n s i b i l i t i e s w i t h  r e s e a r c h e r s f o r f o r m u l a t i n g q u e s t i o n s , d e s i g n i n g instruments, data c o l l e c t i o n and data a n a l y s i s . and areas f o r r e s p o n s i b i l i t y v a r i e d .  W i t h i n these s t u d i e s ,  roles  For example, i n one study,  the e v a l u a t o r had s o l e r e s p o n s i b i l i t y f o r data a n a l y s i s w h i l e the s t a k e h o l d e r wrote the f i n a l r e p o r t ; i n another, r e v e r s e d ; and i n y e t another,  these r o l e s were  these r o l e s were shared.  In the  remaining f o u r s t u d i e s , s t a k e h o l d e r p a r t i c i p a t i o n was l i m i t e d t o areas i n c l u d i n g problem f o r m u l a t i o n , data c o l l e c t i o n , and a n a l y s i s ; and i n two of these s t u d i e s , s t a k e h o l d e r p a r t i c i p a t i o n w i t h i n these stages was e i t h e r somewhat r e s t r i c t e d o r was v e r y structured.  56 M o t i v a t i o n s f o r s t a k e h o l d e r involvement evaluation.  v a r i e d f o r each  They i n c l u d e d r a t i o n a l e s of u t i l i z a t i o n ,  r e s t r i c t i o n s , need f o r p r o f e s s i o n a l e x p e r t i s e , shared  budget decision  making ( c o l l a b o r a t i o n ) , enhanced c r e d i b i l i t y and v a l i d i t y o f data, and empowerment.  Some s t u d i e s espoused more than one  motivation. Method o f Data A n a l y s i s  T h i s e x p l o r a t o r y d e s c r i p t i v e r e s e a r c h paper employs content a n a l y s i s t o examine the q u a l i t a t i v e d a t a . way  "Content  analysis i s a  of transforming q u a l i t a t i v e m a t e r i a l i n t o q u a n t i t a t i v e data.  I t c o n s i s t s p r i m a r i l y o f coding and t a b u l a t i n g the occurrences o f c e r t a i n forms o f content t h a t a r e b e i n g communicated" Babbie,  1989, p. 370).  (Rubin &  Content a n a l y s i s r e q u i r e s the r e s e a r c h e r  to l a b e l and code the data i n t o primary p a t t e r n s , c l a s s i f y i n g them a c c o r d i n g t o some c o n c e p t u a l framework (Patton, 1990). C a t e g o r i z i n g o r c l a s s i f y i n g the data a s s i s t s the r e s e a r c h e r t o d i s t i n g u i s h the s u b s t a n t i v e themes w i t h i n the d a t a . In t h i s study, s e l e c t passages r e l a t i n g s p e c i f i c a l l y t o the r e s e a r c h focus o f p a r t i c i p a t o r y o p e r a t i o n a l dimensions and r e l a t i o n a l dynamics were s t u d i e d i n depth and i n d e t a i l .  Some  passages were analyzed l i n e by l i n e , o t h e r s were a n a l y z e d sentence by sentence.  Each "thought"  o r " i d e a " as c a p t u r e d i n a  l i n e o r sentence was coded by the r e s e a r c h e r .  Patton  (1990)  d e s c r i b e s the q u a l i t a t i v e experience as one o f " i l l u m i n a t i o n , understanding,  and e x t r a p o l a t i o n " (p. 424).  Data were n o t  57 c o n s t r a i n e d by the r e s e a r c h e r through predetermined c a t e g o r i e s . Important  a n a l y t i c a l dimensions  were allowed to emerge f r e e l y  from the data without any advance p r e s u p p o s i t i o n s of what those dimensions  would be  Although  (Patton, 1990) .  the data espouse a wide a r r a y of s t a k e h o l d e r r o l e s  and r e s p o n s i b i l i t i e s w i t h i n a v a r i e t y of e v a l u a t i o n c o n t e x t s , the a n a l y s i s attempted  to l o c a t e commonalities  e v a l u a t o r " s p a r t i c i p a t o r y attempts. (Strauss, 1987) researcher.  and themes w i t h i n  "In v i v o " codes were used  to d e f i n e themes, as were codes p r e s c r i b e d by  the  Passages of data i n which authors summarized c u r r e n t  l i t e r a t u r e or thought  on s t a k e h o l d e r s ( u s u a l l y found i n the  b e g i n n i n g of a r t i c l e s or s e c t i o n s ) were not i n c l u d e d f o r analysis.  The primary focus was  on the authors" own  experiences  or t h e i r r e p o r t i n g of s t a k e h o l d e r s ' e x p e r i e n c e s . Memos, as i n t r o d u c e d to the r e s e a r c h e r through  Strauss  (1987), were used by the r e s e a r c h e r to o r g a n i z e the data by grouping and l a b e l l i n g the emerging p a t t e r n s and codes i n t o r e l a t e d c a t e g o r i e s and themes--the b a s i s f o r content (Patton, 1990) .  analysis  The r e s e a r c h e r used memos to r e c o r d any  r e a c t i o n s t o the data t h a t might prove u s e f u l .  Ideas about codes  and t h e i r r e l a t i o n s h i p s to each o t h e r were c o n v e n i e n t l y captured i n a memo through a comment, q u e s t i o n s , or s e r i e s of q u e s t i o n s w h i l e the data a n a l y s i s was  still  ongoing.  M i l e s & Huberman (1984) suggest  t h a t "memoing h e l p s the  a n a l y s t move e a s i l y from data to a c o n c e p t u a l l e v e l , expanding  refining  codes f u r t h e r , d e v e l o p i n g key c a t e g o r i e s and  showing  and  58 t h e i r r e l a t i o n s h i p s " (p. 71). conceptually organize  Memos a s s i s t e d the r e s e a r c h e r  the p a t t e r n s found i n the data and  to  focus  emerging themes, a l l o w i n g f o r f u r t h e r e x p l o r a t i o n of the data a t a more i n d e p t h and  intense  level.  S e v e r a l main themes emerged and p r o v i d e d a framework f o r conveying  the s u b s t a n t i v e i s s u e s y i e l d e d from the d a t a .  are d i s c u s s e d i n the subsequent  chapter.  These  59 Chapter 3  RESULTS AND  The  DISCUSSION  p a r t i c i p a t i o n of s t a k e h o l d e r s i n e v a l u a t i o n r e s e a r c h  increasingly  gained acceptance as a way  a c c u r a t e l y r e p r e s e n t the d e c i s i o n empowerment.  to i n c r e a s e u t i l i z a t i o n ,  making p r o c e s s , and  e x p l o r a t o r y study i s to i d e n t i f y the  The  participation.  Special  interactions  To  operational  this  operational  stakeholder  c o n s i d e r a t i o n i s g i v e n to the nature of  between r e s e a r c h e r and  contextual factors  focus of  dynamics and  dimensions which c o n t r i b u t e to e f f e c t i v e  the  facilitate  However, e v a l u a t o r understanding of the  dynamics of p a r t i c i p a t i o n i s l i m i t e d .  has  affecting  s t a k e h o l d e r and  the  participation.  f a c i l i t a t e r e a d a b i l i t y of the  r e p o r t , o n l y the  first  author's name i s used as a r e f e r e n c e i n s t u d i e s w i t h more than one  author.  distinguished The  References to Greene's two as Greene/a (1987) and  content a n a l y s i s  articles  Greene/b  are  (1988).  r e v e a l e d f o u r major themes r e l a t e d  to  stakeholder p a r t i c i p a t i o n : 1.  The  participatory  p r o c e s s should r e i n f o r c e  genuineness of s t a k e h o l d e r p a r t i c i p a t i o n . participation i s legitimized  through  communication, i t e r a t i v e l y s t r u c t u r e d evidence of p a r t i c i p a t i o n ,  and  the Stakeholder  interactive feedback,  shared d e c i s i o n  making.  60  2.  P a r t i c i p a t i o n i s enabled by a t t e n t i o n to group p r o c e s s . The  r o l e of r e s e a r c h e r as f a c i l i t a t o r of the e v a l u a t i o n  and of the group process 3.  i s essential.  A s u c c e s s f u l p a r t i c i p a t o r y process c u l t i v a t e s a sense of ownership.  Ownership i s f o s t e r e d by  "hands-on"  p a r t i c i p a t i o n and by s t a k e h o l d e r s ' p e r c e p t i o n of e v a l u a t i o n as an " i n s i d e j o b " and as 4.  the  "tailor-made."  The p a r t i c i p a t o r y process c r e a t e s a r e a d i n e s s f o r use. There i s a n a t u r a l p r o g r e s s i o n from p a r t i c i p a t o r y e v a l u a t i o n to u t i l i z a t i o n which i s enhanced by ongoing access  to e v a l u a t i o n r e s u l t s , an i n c r e a s e i n program  knowledge, and the r e l a t i o n s h i p between r e s e a r c h e r  and  stakeholder. Each of these themes and subthemes, reproduced addressed  i n Table 2, i s  w i t h s p e c i a l c o n s i d e r a t i o n p a i d to the nature of  r o l e s and r e l a t i o n s h i p s between r e s e a r c h e r and  the  stakeholder.  R e i n f o r c i n g the Genuineness of P a r t i c i p a t i o n  I n t e r a c t i v e Communication  A key element i n l e g i t i m i z i n g p a r t i c i p a t i o n f o r s t a k e h o l d e r s was  i d e n t i f i e d as a process which promotes frequent and  communication w i t h s t a k e h o l d e r s . communication v e h i c l e s to m a i n t a i n recommended.  ongoing  The use of m u l t i p l e t h i s c o n t i n u i t y of c o n t a c t  was  61  Table 2 Main Themes and Subthemes Main Themes  Subthemes  The p a r t i c i p a t o r y process should r e i n f o r c e the genuineness o f stakeholder participation.  Stakeholder p a r t i c i p a t i o n i s l e g i t i m i z e d through: a. i n t e r a c t i v e communication b. i t e r a t i v e l y s t r u c t u r e d feedback c. evidence o f p a r t i c i p a t i o n d. shared d e c i s i o n making.  Participation i s enabled by a t t e n t i o n t o group p r o c e s s .  The r o l e o f r e s e a r c h e r as f a c i l i t a t o r o f the e v a l u a t i o n and o f the group p r o c e s s i s essential.  3. A s u c c e s s f u l p a r t i c i p a t o r y process c u l t i v a t e s a sense o f ownership.  Ownership i s f o s t e r e d by: a. "hands-on" p a r t i c i p a t i o n b. p e r c e p t i o n s o f the e v a l u a t i o n as an " i n s i d e job" c. p e r c e p t i o n s o f the e v a l u a t i o n as " t a i l o r - m a d e . "  4. The p a r t i c i p a t o r y process c r e a t e s a r e a d i n e s s f o r use.  There i s a n a t u r a l p r o g r e s s i o n from p a r t i c i p a t o r y e v a l u a t i o n to u t i l i z a t i o n which i s enhanced by: a. ongoing access t o evaluation results b. an i n c r e a s e i n program knowledge c. the r e l a t i o n s h i p between r e s e a r c h e r and s t a k e h o l d e r .  Communication between the r e s e a r c h e r and s t a k e h o l d e r (and among s t a k e h o l d e r s ) was e s s e n t i a l t o s o l i c i t i n g participation.  and f a c i l i t a t i n g  Communication i n c l u d e d a) p e r s o n a l  contact  through i n t e r v i e w s , meetings and phone c a l l s , b) w r i t t e n m a i l c o n t a c t s through q u e s t i o n n a i r e s and w r i t t e n r e p o r t s , c) group meetings through both v e r b a l p r e s e n t a t i o n s and w r i t t e n r e p o r t s .  62  A c e n t r a l component was t h a t communication be i n t e r a c t i v e . W r i t t e n r e p o r t s and q u e s t i o n n a i r e s i n c l u d e d space f o r feedback o r comments by the s t a k e h o l d e r .  Communication  further assisted  e v a l u a t o r s t o e s t a b l i s h d e a d l i n e s by i n c o r p o r a t i n g a timeframe for  response. F u r t h e r , ongoing communication d u r i n g the e v a l u a t i o n process  enhanced p a r t i c i p a t i o n by r e c o g n i z i n g s t a k e h o l d e r s as p a r t n e r s and p r o v i d i n g them w i t h access t o a l l o f the i n f o r m a t i o n a v a i l a b l e t o the e v a l u a t o r , thereby v a l i d a t i n g participation.  their  Communication a l s o served t o promulgate the  e v a l u a t o r ' s i n t e r e s t , energy, and commitment t o the task. Ongoing and i n t e r a c t i v e communication helped t o s u s t a i n s t a k e h o l d e r i n t e r e s t and v a l i d a t e t h e i r v a l u e as p a r t i c i p a n t s . P a r t i c i p a t i o n i s enhanced by communication e f f o r t s which a r e responsiveness and needs.  t o i n d i v i d u a l s t a k e h o l d e r s communication s t y l e s  E v a l u a t o r s recount  stakeholder perceptions of  c o n t i n u i t y of contact: She [stakeholder] commented t h a t the updates were e s p e c i a l l y u s e f u l i n keeping h e r informed o f the e v a l u a t i o n ' s p r o g r e s s , i n making the i n f o r m a t i o n c o l l e c t e d open and a c c e s s i b l e t o a l l , and i n r e i n f o r c i n g the genuineness o f h e r p a r t i c i p a t i o n , thereby c o n t r i b u t i n g t o the t r u s t and c o o p e r a t i v e s p i r i t o f the v e n t u r e . (Greene/b: 104) In r e c o u n t i n g t h e i r p a r t i c i p a t o r y experiences, many s t a k e h o l d e r s . . . h i g h l i g h t e d the v a l u e o f the m u l t i p l e o p p o r t u n i t i e s a f f o r d e d f o r d i s c u s s i o n , r e f l e c t i o n , and c r e a t i v e a n a l y s i s o f s u b s t a n t i v e program i s s u e s . (Greene/b: 107) R e c e i v i n g these communications r e g u l a r l y " i n s p i r e d a d e a l o f c o n f i d e n c e t h a t the p r o j e c t wouldn't j u s t p e t e r out," s a i d Tim [ s t a k e h o l d e r ] . More i m p o r t a n t l y , Tim observed, these communications made him f e e l important and v a l u e d , r e i n f o r c e d h i s f e e l i n g t h a t "what I wrote r e a l l y counted" and helped him  63 to f e e l i n v o l v e d even when he d i d not p a r t i c i p a t e . . . . ( G r e e n e / b : 105) Evaluators  reflect:  I n s t e a d of a s k i n g q u e s t i o n s prepared i n advance, we i n i t i a t e d t o p i c s f o r d i s c u s s i o n and then, u s i n g s t u d e n t s ' own language, encouraged them to expand on t h e i r responses (Crow: 748) The content of t h i s [ongoing] communication was comprehensive, g i v i n g s t a k e h o l d e r s f u l l access to and p o t e n t i a l c o n t r o l over a l l i n f o r m a t i o n generated i n the e v a l u a t i o n s . V e h i c l e s f o r t h i s communication comprised both p e r s o n a l and w r i t t e n contacts. J o i n t l y , these v e h i c l e s a f f o r d e d m u l t i p l e o p p o r t u n i t i e s f o r 'stakeholder engagement w i t h key program i s s u e s , were r e s p o n s i v e to s t a k e h o l d e r d i f f e r e n c e s i n l e a r n i n g and communication s t y l e s , and emphasized the importance of d i a l o g u e as a key v e h i c l e f o r g e n e r a t i n g meaning and p l a n n i n g a c t i o n . (Greene/b: 102-103) In both the i n t e r v i e w s and survey, we d i d not s t a n d a r d i z e the procedures but t a i l o r e d them to s u i t the groups...For example, to keep the i n t e r v i e w s as n o n t h r e a t e n i n g as p o s s i b l e , some homeless p a r e n t s were i n t e r v i e w e d i n groups, and t o ensure that a l l p a r e n t s ' pair-comparison q u e s t i o n n a i r e s were p r o p e r l y completed, we a d m i n i s t e r e d them i n person, l e s s e n i n g problems due t o u n f a m i l i a r i t y w i t h the pair-comparison procedure. Teachers and s h e l t e r p r o v i d e r s , however, who were more f a m i l i a r w i t h survey instruments, completed the q u e s t i o n n a i r e s by m a i l . (Brandon: 289) A l l s t a k e h o l d e r s were g i v e n equal access to the e v a l u a t i o n f i n d i n g s and were p r o v i d e d w i t h equal chances to speak i n the meeting. (Brandon: 2 91) Stakeholders seemed to p r e f e r p e r s o n a l c o n t a c t .  Greene  found  that m a i l e d q u e s t i o n n a i r e s were not w e l l - r e c e i v e d .  Stakeholders  found them to be too wordy and hard to understand.  Personal  c o n t a c t and i n t e r a c t i o n , however, p r o v i d e d o p p o r t u n i t i e s f o r c l a r i f i c a t i o n and d i s c u s s i o n p r i o r to p r o v i d i n g feedback. importance  The  of p e r s o n a l i n t e r a c t i o n w i t h the r e s e a r c h e r i s  i d e n t i f i e d as an important  ingredient i n s o l i d i f y i n g  enhancing  E v a l u a t o r s d i s c u s s the importance  participation.  p e r s o n a l c o n t a c t s w i t h both i n d i v i d u a l s and  groups:  and of  64 The p e r s o n a l c o n t a c t s - - t h e d i v e r g e n t phase i n t e r v i e w s and the day care c o u n c i l a n a l y s i s phase m e e t i n g - - r e c e i v e d n e a r l y universal acclaim. Reasons c i t e d i n c l u d e the importance o f the p e r s o n a l c o n n e c t i o n and the v a l u e o f group i n t e r a c t i o n f o r c o n s i d e r a t i o n s o f important i s s u e s . (Greene/a: 388) The i n t e r v i e w s were c o n f i r m i n g , r e a f f i r m i n g , and helped f a c i l i t a t e [Tim's] own investment i n the p r o c e s s . "I d i d n ' t get l o s t i n a c r o w d . . . [ p a r t i c i p a t i o n ] was p e r s o n a l , one on one." (Greene/b: 105 The s t r e n g t h s o f the e v a l u a t i o n came from an a r r a y o f needs and cues about the c l i e n t t h a t emerged out o f many c o n v e r s a t i o n s . (Faase: 80) The h e a r t o f the p a r t n e r s h i p i n f r a s t r u c t u r e was developed d u r i n g the s e r i e s o f i n d i v i d u a l p l a n n i n g meetings. (Shapiro: 55) P a r t n e r s were encouraged t o c a l l the e x t e r n a l p a r t n e r and t o ask q u e s t i o n s and supply i n f o r m a t i o n about t h e i r p r o j e c t s . (Shapiro: 57) Tom's involvement i n the New Orleans meeting.... e s t a b l i s h e d h i s commitment t o the e v a l u a t i o n . He saw i t s scope and knew the people whom you were going t o be i n t e r v i e w i n g . I c o u l d see t h a t he was d e v e l o p i n g a sense o f commitment t o the e v a l u a t i o n as he looked over the i n t e r v i e w q u e s t i o n s and made suggestions f o r a few changes. The process o f r e v i e w i n g the i n s t r u m e n t a t i o n w i t h Tom helped i n c r e a s e the extent t o which those were Tom's q u e s t i o n s . ( A l k i n : 27). T h i s problem u l t i m a t e l y was r e s o l v e d as p r o g r e s s i v e c o n t a c t between c l i e n t and e v a l u a t o r brought assurances t h a t important s e n s i t i v i t i e s c o u l d and would be honoured i n c o n t a c t s w i t h these s t a k e h o l d e r s . The e v a l u a t i o n c o u l d n o t have begun without shared understanding and t r u s t . (Faase: 76) Group i n t e r a c t i o n and communication were found c o l l e g i a l and s u p p o r t i v e environment.  t o promote a  Stakeholders  enjoyed  i n t e r a c t i n g w i t h other s t a k e h o l d e r s and l e a r n e d from t h e i r views and p e r s p e c t i v e s . d e s i r e d access  Stakeholders  v a l u e d d i f f e r e n t p e r s p e c t i v e s and  t o t h e i r views on e v a l u a t i o n i s s u e s .  Evaluators  r e l a t e the importance o f these group i n t e r a c t i o n s : I n d i v i d u a l Task Force members brought a v a r i e t y o f d i f f e r e n t i s s u e s t o the group based on t h e i r own c u r i o s i t y and t h e i r  65 i n d i v i d u a l p e r c e p t i o n and experience gender b i a s . ( G i l l : 105)  of r a c i a l / e t h n i c  and  Repeated group d i s c u s s i o n s . . . n o t o n l y p r o v i d e the d i r e c t experience of c o n t r i b u t i n g to a d e c i s i o n , but a l s o the p e r s o n a l connections and the s h a r i n g of s u b s t a n t i v e a n a l y s i s and r e f l e c t i o n t h a t f u r t h e r enhance the p r o c e s s . (Greene/a: 3 90) Pam [stakeholder] p a r t i c u l a r l y v a l u e d the s t a k e h o l d e r group meetings, which she p e r c e i v e d as f r u i t f u l and c r e a t i v e d i a l o g u e s among people w i t h d i f f e r i n g views and p e r s p e c t i v e s about important program i s s u e s . "I l i k e d h e a r i n g what other people are t h i n k i n g . " Pam o n l y wished t h a t t h e r e had been more such meetings and t h a t some a d d i t i o n a l program c o n s t i t u e n c i e s had been b e t t e r r e p r e s e n t e d w i t h i n the group ( l i k e low-income p a r e n t s ) . (Greene/b: 104) Communication and  i n t e r a c t i o n with stakeholders r e f l e c t e d  c o l l a b o r a t i v e g o a l s of the p a r t i c i p a t o r y p r o c e s s . communication strengthened  Vehicles for  the genuineness of p a r t i c i p a t i o n  r e i n f o r c i n g s t a k e h o l d e r s as p a r t n e r s i n the e v a l u a t i o n . r e v e a l s d i f f i c u l t i e s when t h e i r communication w i t h  the  by  Crow  these  s t a k e h o l d e r s had not promoted a c o l l a b o r a t i v e environment: A review of our own notes and messages to a d v i s o r s r e v e a l s a c l e a r "us and them" o r i e n t a t i o n . We appear r e p e a t e d l y as the " r e s e a r c h e r s " ; they are b e i n g asked to p r o v i d e data to h e l p us " t e s t our f i n d i n g s . " (Crow: 745) Stakeholder  communication was  found  to not o n l y be marked by  frequent and c l o s e c o n t a c t , i t i n v o l v e d s t a k e h o l d e r s e a r l y i n the evaluation process.  Timing was  a l s o i n t r o d u c e d by two initial  a f a c t o r i n Crow, above, and  other e v a l u a t o r s who  c o n t a c t and s t a k e h o l d e r involvement  was  r e v e a l t h a t l a c k of r e s u l t e d i n problems.  They r e p o r t : Weakness of the e v a l u a t i o n p l a n were t h a t n e i t h e r the p l a n nor the instruments were p i l o t t e s t e d and both were developed without s i g n i f i c a n t involvement of the f a c u l t y . This issues would l a t e r plague the program e v a l u a t i o n committee's implementation of the e v a l u a t i o n p l a n f o r y e a r s . ( B a r r i c k : 34)  66  The s h o r t time frames r e q u i r e d submission o f a f a i r l y d e t a i l e d r e s e a r c h p l a n t h a t d i d n ' t evolve from i n p u t by m u l t i p l e s t a k e h o l d e r . . . . U n f o r t u n a t e l y [the panel] was o n l y c o n s t i t u t e d a f t e r the o r i g i n a l p r o p o s a l was funded, and thus wasn't i n v o l v e d i n some c r u c i a l , e a r l y d e s i g n d e c i s i o n s . ( A l l e n : 50)  I t e r a t i v e l y S t r u c t u r e d Feedback  Regardless  o f which tasks s t a k e h o l d e r s a r e i n v o l v e d i n , the  p a r t i c i p a t o r y process from s t a k e h o l d e r s .  r e q u i r e s c o n t i n u a l s o l i c i t a t i o n o f feedback  T h i s was a c e n t r a l theme t o r e i n f o r c i n g the  a u t h e n t i c i t y o f s t a k e h o l d e r p a r t i c i p a t i o n and i n v o l v e s s e v e r a l factors. Stakeholder  feedback p r o v i d e d d i r e c t i o n t o the e v a l u a t o r .  Feedback was e i t h e r w r i t t e n o r v e r b a l and i n v o l v e d a c t i v i t i e s such as r a n k i n g , p r i o r i t i z i n g ,  supporting, p r o v i d i n g  r e f l e c t i o n , and/or g i v i n g a d v i c e .  critical  C o n t i n u a l and r e p e t i t i v e  o p p o r t u n i t i e s f o r feedback seemed e s s e n t i a l . Within t h i s process,  the e v a l u a t o r seems t o make d e l i b e r a t e  attempts t o communicate t o s t a k e h o l d e r s welcome and necessary.  t h a t t h e i r i n p u t i s both  For example, e v a l u a t o r s l a b e l l e d w r i t t e n  communication and r e p o r t s as d r a f t s o r p r e l i m i n a r y r e p o r t s .  This  seemed t o t r a n s f e r r e s p e c t f o r s t a k e h o l d e r c o n t r i b u t i o n s and allowed o p p o r t u n i t i e s f o r s t a k e h o l d e r s to be i n v o l v e d i n tasks which may be p r i m a r i l y the r e s p o n s i b i l i t y o f the e v a l u a t o r . E v a l u a t o r s recount  t h e i r e f f o r t s to promote feedback:  Reports were shared w i t h a l l s t a k e h o l d e r s and then d i s c u s s e d i n meetings w i t h the e v a l u a t i o n team and/or the l a r g e r s t a k e h o l d e r group. Agendas f o r these meetings comprised s t a k e h o l d e r s h a r i n g of (a) r e f l e c t i o n s on and i n t e r p r e t a t i o n s  67 of the r e s u l t s , and (b) f u r t h e r questions generated by the r e s u l t s , which c o u l d be addressed i n a d d i t i o n a l analyses and/or f u t u r e data c o l l e c t i o n a c t i v i t i e s . These s u b s t a n t i v e d i s c u s s i o n s c o n t r i b u t e d to ongoing d a t a - c o l l e c t i o n d e c i s i o n s ; they a l s o r e p r e s e n t e d s t a k e h o l d e r ' s e v o l v i n g understanding of key program concerns and p o s s i b l e ways to address them. (Greene/b: 94) F i r s t d r a f t s of r e p o r t s were prepared and l a b e l l e d as such and d i s t r i b u t e d to a l l members of meeting....Then a meeting was h e l d to review major f i n d i n g s , to i d e n t i f y gaps i n the r e p o r t s or i n the p a r t i c i p a n t s ' understanding of the documents, and to p l a n f o r u t i l i z a t i o n of the i n f o r m a t i o n by the Task F o r c e s . (Gill: 107) A l l instruments developed by the e x t e r n a l e v a l u a t o r were t r e a t e d as d r a f t forms. The d i r e c t o r s had the f i n a l a u t h o r i t y to determine q u e s t i o n s , coding procedures, l e n g t h and other c h a r a c t e r i s t i c s . (Shapiro: 56) The f i r s t r e p o r t was p r e l i m i n a r y . . . . I t p e r m i t t e d the s t a f f to put q u e s t i o n s to the e v a l u a t o r f o r a d d i t i o n a l or subsequent a n a l y s i s . (Faase: 79) These r e p o r t s were n o n t e c h n i c a l n a r r a t i v e s t h a t p r e s e n t e d the f u l l s e t of d e s c r i p t i v e r e s u l t s and s e l e c t e d r e l a t i o n a l r e s u l t s , but no c o n c l u s i o n s or recommendations. Rather, the r e p o r t s were shared w i t h a l l s t a k e h o l d e r s and then d i s c u s s e d i n meetings w i t h the e v a l u a t i o n team and/or the l a r g e r s t a k e h o l d e r group (Greene/b: 94) S o l i c i t i n g feedback i n c o r p o r a t e d a process between r e s e a r c h e r and r e c e i v e r s of feedback. l i v e l y i n t e r a c t i o n and process  stakeholder.  of mutual exchange  E v a l u a t o r s are not merely  They are a c t i v e p a r t i c i p a n t s engaged i n the mutual exchange of i d e a s .  i n v o l v e s i n t e r d i r e c t i o n a l w r i t t e n or v e r b a l  concerning  i s s u e s a f f e c t i n g the study,  understanding  or i n process,  t h i s process.  dialogue  difficulties in  and o b s e r v a t i o n s  Mutual accommodation i s presented  This  and  impressions.  as an e s s e n t i a l component i n  E v a l u a t o r s d e s c r i b e t h i s s h a r i n g of  roles:  Stakeholders were viewed not as a d v i s o r s or c o n s u l t a n t s , as a c t i v e , engaged " c o l l a b o r a t o r s i n i n q u i r y " . . . . , w i t h  but  68 primary r e s p o n s i b i l i t y f o r d e t e r m i n i n g the substance or content of the e v a l u a t i o n s . (Greene/b: 101) In the d e s i g n process, the c l i e n t t o l d the e v a l u a t o r what he wanted, and the e v a l u a t o r o u t l i n e d procedures t h a t might b r i n g i t about. In q u e s t i o n n a i r e c o n s t r u c t i o n , the e v a l u a t o r t o l d the c l i e n t what he was t r y i n g to do, and the c l i e n t helped a r r i v e a t the e x p r e s s i o n of i t that would be b e s t understood by the r a n k - a n d - f i l e membership. (Faase:78) The approach to e v a l u a t i o n d e s i g n on my p a r t r e q u i r e d f l e x i b i l i t y and r e a c t i o n , r a t h e r than p r o a c t i o n , because the a c t i v i t y d i r e c t o r i n i t i a l l y was to determine the course and g o a l s of the e v a l u a t i o n r e s e a r c h p r o j e c t . (Shapiro: 55) It  i s e s s e n t i a l that t h i s p r o c e s s of s o l i c i t i n g feedback occur  w i t h i n an i t e r a t i v e s t r u c t u r e .  This i t e r a t i v e process  c h a r a c t e r i z e d by e v a l u a t o r s  "second go-around,"  "recirculated,"  as:  " s e v e r a l rounds o f , " " c o n t i n u a l i n t e r p l a y , "  "ongoing," " r e c i p r o c a l , " " s e r i e s , " "back and junctures,"  and  Constant and  updates of i n f o r m a t i o n  f o r p a r t i c i p a t i o n i s enhanced by r e c u r r i n g feedback p r o v i d e d  and  ensured t h a t e v a l u a t o r s  a c c u r a t e l y s y n t h e s i z i n g m a t e r i a l and stakeholder's  f o r t h , " "number of  "repeated."  Multiple opportunities repetition.  original intention.  of the e v a l u a t i o n .  various  continual  were  i n t e r p r e t i n g the  T h i s appeared to serve as a  safeguard--a k i n d of guarantee of the c r e d i b i l i t y and  expanding and  was  integrity  I t e r a t i v e feedback r e s u l t e d i n a c o n t i n u a l l y  e v o l v i n g process,  c o n s t i t u e n t groups.  whereby m a t e r i a l was  group were i n c o r p o r a t e d  before being  d i s s e m i n a t e d to a d i f f e r e n t s t a k e h o l d i n g  i n t o the e v a l u a t i o n  to enhance c l a r i t y .  the n a t u r e of t h i s i t e r a t i v e  by  At times, suggestions of a l a r g e r  stakeholding  I t e r a t i o n a l s o served  discussed  process:  Evaluators  material  group. illustrate  69 Pam a l s o l i k e d the m u l t i p l e , ongoing, and v a r i e d o p p o r t u n i t i e s p r o v i d e d f o r her p a r t i c i p a t i o n , commenting, " I had p l e n t y o f o p p o r t u n i t i e s t o have a say". (Greene/b: 104) In w r i t i n g the r e p o r t , we o u t l i n e d what we wanted t o say as a group, I d r a f t e d i t i n w r i t t e n form, and then we made r e v i s i o n s t o g e t h e r . Again, t h i s was an i t e r a t i v e p r o c e s s o f many r e d r a f t s u n t i l we f e l t the r e s u l t s were c l e a r and comprehensible. (Whitmore: 221) A f t e r each d i s c u s s i o n , i m p l i c a t i o n s and recommendations generated by t h a t group were i n c o r p o r a t e d i n t o the summary b e f o r e i t was shared w i t h the next audience. A p l u r a l i s t i c and c o n t i n u a l l y expanding s e t o f program i m p l i c a t i o n s and s p e c i f i c i d e a s f o r change e v o l v e d from t h i s p r o c e s s . (Greene/b: 94) T h i s t r a n s l a t i o n and d e l i m i t a t i o n was a thoroughgoing negotiation. F i v e t o s i x hours o f p l a n n i n g and subsequent review, f o r i n s t a n c e , went i n t o the c o n s t r u c t i o n o f a s i n g l e item on one s c a l e . (Faase: 78) [Evaluators] t r a n s l a t e d the Task Force's l i s t o f important content areas i n t o a l i s t o f q u e s t i o n n a i r e items t h a t would generate the data t o answer t h e i r q u e s t i o n s . . . . the d r a f t was c i r c u l a t e d t o the Design Subcommittee f o r comments. A phone c a l l f o l l o w - u p y i e l d e d h e l p f u l comments and s u g g e s t i o n s f o r almost a l l , o f the Subcommittee members.... F o l l o w i n g the i n c o r p o r a t i o n o f feedback from the Design Subcommittee, [evaluators] met w i t h the Subcommittee and reviewed the r e v i s e d d r a f t . ( G i l l : 106) Stakeholder feedback and i n p u t i s c r e d i t e d w i t h p r o v i d i n g i n s i d e r knowledge and v a l u a b l e i n f o r m a t i o n r e g a r d i n g l o c a l c o n t e x t s which was c e n t r a l t o the e v a l u a t i o n p r o c e s s . From the b e g i n n i n g , the c l i e n t sensed the k i n d s o f f a c t o r s , dilemmas, and so f o r t h that the membership was b e s e t by...the c l i e n t had g r e a t e r f a m i l i a r i t y w i t h t h i s p a r t i c u l a r group; he knew who would respond t o the q u e s t i o n and how they might r e a c t t o the c h o i c e o f a word o r a t w i s t o f a phrase. (Faase: 78) T h e i r [ r e s e a r c h a s s i s t a n t s ] experience and u n d e r s t a n d i n g o f l i f e i n t h e i r community was profound; they knew i n t u i t i v e l y what would work and what would n o t . . . . t h e i r i n s t i n c t s about what would work and what would not i n approaching the respondents were sound. (Whitmore: 220 & 223)  70 We wanted...help a n a l y z i n g the data to balance our own b l i n d spots and frame i n t e r p r e t a t i o n s i n a l a r g e r c o n t e x t of what was happening i n the s t a t e . ( A l l e n : 50) The p r o c e s s r e p r e s e n t s an e x c e l l e n t v e h i c l e f o r the e v a l u a t o r to l e a r n about the e v a l u a n d - - i t s people, a c t i v i t i e s , context, and p o l i t i c s . The i n d e p t h i n f o r m a t i o n gained from repeated i n t e r a c t i o n s w i t h s t a k e h o l d e r s f a r exceeds t h a t a v a i l a b l e from documents, r e c o r d s , or o n l y a few such i n t e r a c t i o n s . (Greene/a: 391) Stakeholder knowledge c o n t r i b u t e d to d e c i s i o n s r e g a r d i n g t e c h n i c a l f a c t o r s , enhanced data c o l l e c t i o n and a n a l y s i s , helped to i n c r e a s e the response impacted  rate.  and  Their contributions  the c l a r i t y of the language and content of the data  c o l l e c t i o n instruments, u n i q u e l y shaping the instruments to the audience b e i n g surveyed and e n s u r i n g r e l e v a n c e t o the i s s u e s a t hand.  T h i s i n s i d e r knowledge was  by e v a l u a t o r s .  regarded as a c r i t i c a l  element  They r e p o r t :  T h e i r f a m i l i a r i t y w i t h the women they had i n t e r v i e w e d helped enormously i n f i l l i n g i n the context and understanding the f u l l meaning of what people had s a i d . (Whitmore: 220) The c l i e n t and the audience f o r the e v a l u a t i o n spoke a t e c h n i c a l , p r o f e s s i o n a l language ( i . e . , l e g a l terminology) which was e s s e n t i a l f o r the o p e r a t i n g language of data c o l l e c t i o n instruments.... The Task Force members....were a b l e t o i d e n t i f y language and q u e s t i o n s t h a t d i d not " r i n g t r u e " based on t h e i r experience. ( G i l l : 104 & 106) I met r e g u l a r l y w i t h a subcommittee of the l o c a l a d v i s o r y committee, whose members c o n t r i b u t e d a knowledge of the community and of the program over time. Members not o n l y helped me to r e f l e c t on the process as I kept them up to date, but a l s o o f f e r e d i n v a l u a b l e a d v i c e on the d e s i g n of q u e s t i o n n a i r e s sent to a d v i s o r y committee members and community p r o f e s s i o n a l s . . . . W o r k i n g so c l o s e l y w i t h t h i s group a l s o enhanced the t r u s t w o r t h i n e s s of the data by c o n t r i b u t i n g ideas to the p r o c e s s and s p e c i f i c a l l y by c r i t i q u i n g q u e s t i o n n a i r e d e s i g n i n l i g h t of the l o c a l c o n t e x t . (Whitmore: 219/220) The A d v i s o r y Panel p r o v i d e d s i m i l a r v i t a l a d v i c e c o n c e r n i n g which R e g i o n a l S e r v i c e Network to i n c l u d e i n the sample as  71 w e l l as which p r o v i d e r s o r subproviders t o make sure we covered. As i n i t i a l r e s u l t s were r e p o r t e d , they both r a i s e d q u e s t i o n s about f u r t h e r analyses t h a t would c l a r i f y i n t e r p r e t a t i o n . . . a n d on s i t u a t i o n f a c t o r s t h a t i n f l u e n c e d r e s u l t s . ( A l l e n : 50) Consideration of stakeholder  feedback i n Crow's study on  c a r e e r changes l e d e v a l u a t o r s t o u n a n t i c i p a t e d data when t h e i r i n t e r p r e t a t i o n o f data was r e a c t e d t o n e g a t i v e l y by s t a k e h o l d e r s . Crow r e p o r t s : Some students r e a c t e d n e g a t i v e l y t o the use o f the phrase c a r e e r change, i n s i s t i n g t h a t e a r l i e r work experiences had not been c a r e e r s and t h a t t e a c h i n g r e p r e s e n t e d t h e i r o n l y r e a l c a r e e r . . . . the students' r e a c t i o n t o our p r e l i m i n a r y f i n d i n g s l e d us t o i n v e s t i g a t e t h e i r concepts o f c a r e e r and the ways i n which these concepts d i f f e r e d from c o n v e n t i o n a l n o t i o n s o f c a r e e r . (Crow: 746) Feedback was a l s o found to c r e a t e d i f f i c u l t i e s .  Crow was  c o n f r o n t e d w i t h f a c u l t y a d v i s o r s who d i d not agree w i t h the o r i g i n a l i n t e n t and purpose and were c r i t i c a l o f the e v a l u a t i o n design.  As t h i s disagreement evolved around i s s u e s c e n t r a l t o  t h e i r study and were not r e c o n c i l a b l e , Crow chose t o l i m i t c o l l a b o r a t i o n with advisors.  their  Crow admits t h a t the c o n f u s i o n was  the r e s u l t o f r o l e c o n f l i c t and may have been impacted by communication which d i d not t r e a t a d v i s o r s as p a r t n e r s .  Visible  Results  I t was important evidence  to stakeholders  t h a t there was t a n g i b l e  or v i s i b l e r e s u l t s of t h e i r p a r t i c i p a t i o n .  When t h e i r  own v o i c e was heard and r e f l e c t e d i n the e v a l u a t i o n , they  felt  72 t h a t t h e i r p a r t i c i p a t i o n was contributed.  Evaluators  v a l u e d and  t h a t they  had  report:  W i t h i n t h i s view, n e a r l y a l l s t a k e h o l d e r s f e l t t h e i r o p i n i o n had been heard and i t counted, t y p i c a l l y c i t i n g as evidence (a) r e c o g n i t i o n of t h e i r own concern on a q u e s t i o n n a i r e , (b) space f o r comments on a l l w r i t t e n communications, (c) " [ j u s t b e i n g asked] made me f e e l I had something worth l i s t e n i n g to ( c l i e n t s t a k e h o l d e r ) " . (Green/a: 388) She [stakeholder] f e l t t h a t her views were g e n u i n e l y sought and used throughout the process, her v o i c e was c l e a r l y heard, and her c o n t r i b u t i o n s d e f i n i t e l y counted. (Greene/b: 103) R e p r e s e n t a t i v e s of both Task Forces were i n c l u d e d i n the Design Subcommittee to i n s u r e t h a t concerns about gender and r a c i a l / e t h n i c b i a s would be i n c l u d e d i n the survey instruments.... The content p r i o r i t i e s of each Task Force member were g i v e n equal c o n s i d e r a t i o n f o r i n c l u s i o n i n the d r a f t instrument which r e s u l t e d from the meeting. ( G i l l : 105) They were assured t h a t t h e i r input was important and t h a t i t would be r e f l e c t e d i n the methodology. For example, b e f o r e any instruments f o r data c o l l e c t i o n were developed, much time was spent d i s c u s s i n g the d e s i r e d outcomes, the most c o s t - e f f e c t i v e way of o b t a i n i n g the data, and the need to e l i m i n a t e items t h a t appeared p o l i t i c a l l y i n a p p r o p r i a t e because of t i m i n g or of the o r g a n i z a t i o n a l i s s u e s . (Shapiro: 57) The e v a l u a t i o n summary and these program/policy areas were i t e r a t i v e l y d i s c u s s e d by v a r i o u s c o n s t i t u e n t groups, w i t h each group's i d e a s i n c o r p o r a t e d i n t o the summary b e f o r e i t was shared w i t h the next group. T h i s process again culminated i n a f i n a l r e p o r t , which i n c l u d e d a complete l i s t i n g of a l l program recommendations generated by s t a k e h o l d i n g groups, as w e l l as a h i g h l i g h t e d s e t of the most important recommendations f o r change as viewed by both the s t a k e h o l d e r s and the e v a l u a t o r . (Greene/b: 98) The c l i e n t p a r t i c i p a t e d a c t i v e l y i n a p e r v a s i v e r e v i s i o n of a f i r s t d r a f t of the f i n a l r e p o r t . . . . The c l i e n t brought to the w r i t i n g f i n e e d i t o r i a l and c l a r i f y i n g s k i l l s . More than a few f l o u r i s h e s were e l i m i n a t e d and some i n c o n s i s t e n c i e s were o b v i a t e d . (Faase: 79/80) Stakeholder  feedback a l s o served  to r e f i n e or " f i n e tune"  m a t e r i a l s , i n c l u d i n g data c o l l e c t i o n instruments.  R e f l e c t i o n s of  t h e i r c o n t r i b u t i o n s of t e c h n i c a l knowledge or an u n d e r s t a n d i n g of  73 the evaluand's l o c a l context i n the e v a l u a t i o n c o n t r i b u t e d t o the genuineness o f s t a k e h o l d e r  participation.  Shared D e c i s i o n Making  C o l l a b o r a t i o n w i t h s t a k e h o l d e r s r e q u i r e s shared d e c i s i o n making on s u b s t a n t i v e e v a l u a t i o n i s s u e s .  Shared d e c i s i o n making  l e g i t i m i z e s the r o l e o f s t a k e h o l d e r s i n the e v a l u a t i o n p r o c e s s , r e c o g n i z e s t h e i r i n p u t as a u t h e n t i c and v a l i d , and g e n u i n e l y  acts  on t h e i r i n p u t . E v a l u a t o r s r e v e a l e d t h a t s o l i c i t i n g feedback o f t e n r e q u i r e d a c t i o n on the p a r t o f s t a k e h o l d e r .  There appeared t o be an  e s s e n t i a l step o f s t a k e h o l d e r v e r i f i c a t i o n i n t h i s  process.  D r a f t s and f i n a l d r a f t s were presented n o t o n l y f o r i n f o r m a t i o n but f o r d i r e c t i o n , by the s t a k e h o l d e r .  the next  "go ahead"--implying  an endorsement  In one i n s t a n c e when the r e s e a r c h e r s d i d not  r e c e i v e a response a f t e r m a i l i n g a f i n a l e v a l u a t i o n d e s i g n  draft,  they " f i n a l i z e d the d e s i g n " w i t h a c o n f i r m a t o r y l e t t e r t o a l l s t a k e h o l d e r s b e f o r e moving on t o the next phase. they c o u l d n o t proceed without  I t was as i f  some k i n d o f c o n f i r m a t i o n and d i d  not want t o e r r i n assuming t h a t no response meant acceptance o f the d e s i g n .  The i t e r a t i v e s t r u c t u r e o f the p a r t i c i p a t o r y  process  impacted o p p o r t u n i t i e s f o r d e c i s i o n making and the l e v e l o f c o n t r o l f e l t by s t a k e h o l d e r s .  Evaluators report:  A f t e r the r e v i s e d q u e s t i o n n a i r e s were r e c i r c u l a t e d t o the Subcommittee members and they agreed t o the changes, the survey instruments were ready to be p i l o t e d . ( G i l l : 106)  74 The s t a g i n g i s what allowed [stakeholder group] to f e e l t h a t they were m a i n t a i n i n g some c o n t r o l over t h i s p r o c e s s , which they were. ( A l k i n : 26) E v a l u a t o r s r e p o r t s e v e r a l contexts i n which the p a r t n e r s h i p was  r e i n f o r c e d through shared d e c i s i o n making.  They share:  Moreover, g i v e n the s t a k e h o l d e r s ' genuine d e c i s i o n making r o l e , these [group] d i s c u s s i o n s were o r i e n t e d toward some c o n c r e t e d e c i s i o n or a c t i o n . That i s , s t a k e h o l d e r a c t i o n not j u s t r e a c t i o n was sought. (Greene/b: 109) D i f f e r e n c e s among the s t a k e h o l d e r groups' s c a l e d q u e s t i o n n a i r e r e s u l t s were r e c o n c i l e d i n a meeting of s t a k e h o l d e r s . The task of the meeting was to decide d e m o c r a t i c a l l y about the f i n a l p r i o r i t i z a t i o n of the e d u c a t i o n a l problems of homeless c h i l d r e n . . . . A f t e r the d i s c u s s i o n , s t a k e h o l d e r s completed s e c r e t b a l l o t s showing t h e i r f i n a l ranks f o r problems. (Brandon: 290/291) To e l i m i n a t e i n e q u i t y t h a t might be due to l e n g t h y d e c i s i o n making, we kept meetings b r i e f and the s t a k e h o l d e r s made d e c i s i o n s by p l u r a l i t y , not consensus....Thus, the power of the t e a c h e r s , whom we might have expected to w i e l d s t r o n g l e a d e r s h i p i n group decision-making, d i d not d i m i n i s h the p a r e n t s ' i n f l u e n c e . (Brandon: 2 91) For some, but not a l l d e c i s i o n s , these s t a k e h o l d e r s were asked to c o n t r i b u t e t h e i r views, which were then summarized by the e v a l u a t o r as m a j o r i t y o p i n i o n s . Sometimes these m a j o r i t y o p i n i o n s c o n s t i t u t e d d e c i s i o n s ; other times they were shared w i t h the e v a l u a t i o n team f o r f u r t h e r d i s c u s s i o n . (Green/a: 388) The p r o c e s s was viewed as a consensual democratic one, i n which the e v a l u a t o r s p r o v i d e d the d e s i r e d guidance and s t a k e h o l d e r s , p a r t i c u l a r l y the team, made the d e c i s i o n s about the c o n t e n t . (Greene/a: 3 89) In Brandon's study, d i f f e r e n c e s i n communication p r a c t i c e s among s t a k e h o l d e r s was makers.  shown to impact t h e i r r o l e as d e c i s i o n  Greene r e v e a l s t h a t because of i t s s m a l l s i z e  and  e x t e n s i v e e v a l u a t o r i n t e r a c t i o n , a s t a k e h o l d i n g team t h a t o r i g i n a l l y intended to serve as a communication l i n k or w i t h the l a r g e r s t a k e h o l d i n g group evolved i n t o a  was  liaison  decision-making  75 body.  She r e f l e c t s on the reasons  leadership  the team took on t h i s  role:  The l o g i s t i c a l ease o f working w i t h a s m a l l group o f three o r f o u r s t a k e h o l d e r s versus a l a r g e r group o f 15 t o 20; and...the f a c t t h a t team p a r t i c i p a t i o n was n e a r l y always i n the form o f group d i s c u s s i o n and i n t e r a c t i o n versus the s i n g u l a r , d i s t a n t , w r i t t e n c h a r a c t e r i z a t i o n o f most s t a k e h o l d e r p a r t i c i p a t i o n . (Greene/a: 3 88) In most o f the documents reviewed, e v a l u a t o r s  maintained  c o n t r o l f o r the t e c h n i c a l q u a l i t y w h i l e the s t a k e h o l d e r r e t a i n e d s u b s t a n t i v e c o n t r o l of the e v a l u a t i o n p r o c e s s , although how t h i s was observed  i n p r a c t i c e v a r i e d t o some degree.  Maintaining  t e c h n i c a l q u a l i t y o f the e v a l u a t i o n r e q u i r e d the e v a l u a t o r to take r e s p o n s i b i l i t y f o r g u i d i n g the r e s e a r c h process and e s t a b l i s h i n g c e r t a i n parameters.  However, even under  these  c o n d i t i o n s , Greene r e p o r t s t h a t the s t a k e h o l d e r s s t i l l had the f i n a l say:  "The e v a l u a t o r ' s attempts t o f a c i l i t a t e  meaningful  stakeholder consideration of t e c h n i c a l f a c t o r s i n t h e i r d e c i s i o n s were, on balance,  unsuccessful"  design  (Greene/a: 389).  Shapiro i n d i c a t e s t h a t s t a k e h o l d e r i n p u t made the development of  the e v a l u a t i o n e a s i e r t o undertake, a t times, because the  p a r t n e r s had t o a s s i s t i n making f i n a l c h o i c e s .  He comments:  Knowing t h a t the i n s t r u m e n t a t i o n would be s u b j e c t t o p a r t n e r s h i p review made the development e a s i e r t o undertake. Whenever I c o u l d n o t come t o s a t i s f a c t o r y d e c i s i o n s - - f o r example, on the b e s t coding scheme t o use f o r a p a r t i c u l a r instrument--I knew t h a t the p a r t n e r s would be o b l i g e d t o a s s i s t i n making f i n a l c h o i c e s . (Shapiro: 60) The p a r t i c i p a t o r y process r e q u i r e s an e v a l u a t o r who i s agreeable  t o b e i n g accountable  t o s t a k e h o l d e r s , open t o b e i n g  76 c h a l l e n g e d and c r i t i q u e d , and w i l l i n g to take E v a l u a t o r s r e f l e c t on t h e i r e v a l u a t i o n  direction.  experiences:  Even more than u s u a l , the e v a l u a t o r must develop s e n s i t i v i t y to the a d v i c e and c r i t i c i s m of p a r t n e r s - - a c r u c i a l element of the p a r t n e r s h i p e v a l u a t i o n i f the c l i e n t i s t r u l y going to a c t and f e e l l i k e a p a r t n e r . (Shapiro: 57) The c o n s c i o u s d e c i s i o n to engage i n p a r t n e r s h i p e v a l u a t i o n means t h a t the c o n v e n t i o n a l power r e l a t i o n s h i p s w i l l be a l t e r e d , and t h i s departure may have an u n s e t t l i n g e f f e c t on e v a l u a t o r and c l i e n t a l i k e . (Shapiro: 60) C o l l a b o r a t i o n can have the d i s a g r e e a b l e r e s u l t of making the e v a l u a t o r f e e l v u l n e r a b l e and uncomfortable. What was the most d i f f i c u l t of a l l (and, i n an i r o n i c way, a l s o most g r a t i f y i n g ) , the e v a l u a t o r assumed the stance of a peer, r a t h e r than t h a t of a s p e c i a l i z e d p r o f e s s i o n a l . I t would have been e a s i e r to r e s t on p r o t o c o l than to be exposed i n t e n t a t i v e n e s s . The e v a l u a t o r would have "looked b e t t e r " b r i n g i n g i n a p r i n t e d v e r s i o n of a survey instrument than showing the cut-and-paste v e r s i o n of a survey instrument . The e v a l u a t o r would have seemed so much more competent handing i n a word-processed and s p i r a l - b o u n d f i n a l r e p o r t i n s t e a d of coming i n w i t h packs of handwritten pages, w i t h r e v i s i o n s and c o r r e c t i o n s p e n c i l l e d i n and ready to undergo the awful s c r u t i n y of the c l i e n t ' s e d i t o r i a l e x p e r t i s e . The e v a l u a t o r was v u l n e r a b l e i n such s i t u a t i o n s . (Faase: 81) A c o o p e r a t i v e working r e l a t i o n s h i p i s r e q u i r e d between e v a l u a t o r s and s t a k e h o l d e r s r a t h e r than an independent... [and] perhaps a n t a g o n i s t i c r e l a t i o n s h i p . (Ayers: 266) Shapiro  found t h a t p a r t i c i p a t i o n of s t a k e h o l d e r s  resulted in  the e v a l u a t o r t a k i n g the r o l e of " e v a l u a t i o n educator" more s e l f - c o n s c i o u s l e v e l .  a t a much  He s t a t e s :  T h i s i s t r u e not o n l y w i t h r e s p e c t to formal i n s t r u c t i o n ( f o r example, the i n t r o d u c t o r y workshop) but a l s o w i t h r e s p e c t to the constant barrage of q u e s t i o n s , comments, i n s i g h t s , and r e f l e c t i o n s t h a t c l i e n t s may o f f e r w h i l e p l a n n i n g and c a r r y i n g out i n d i v i d u a l e v a l u a t i o n p r o j e c t s . Ignorance i s not b l i s s when the c l i e n t undertakes h i s or her p a r t n e r s h i p o b l i g a t i o n s e r i o u s l y ; thus, I found myself spending much more time e x p l a i n i n g , j u s t i f y i n g , even defending e v a l u a t i o n theory, p r a c t i c e , and p h i l o s o p h y t h a t I would have done i n a c o n v e n t i o n a l e v a l u a t i o n . (Shapiro: 60)  77 The  s t a k e h o l d e r process was  d e s c r i b e d by e v a l u a t o r s  as  "lengthy" because of the c o n t i n u a l i n t e r p l a y of feedback revisions,  "time consuming", and  E v a l u a t o r s recount  i n one  instance,  and  "tiresome".  that:  At times the constant c r i t i c i s m to which my instrument d r a f t s (approximately 15 l o c a l c o n s t r u c t i o n s ) were s u b j e c t e d became tiresome, p a r t i c u l a r l y because the more the p a r t n e r s c r i t i c i z e , the b e t t e r they got at i t . I had to keep i n mind, however, t h a t such c r i t i c i s m would not o n l y improve the e v a l u a t i o n but a l s o bond the p a r t n e r s to the e v a l u a t i o n p r o c e s s . (Shapiro: 57/58) The sometimes lengthy gaps between steps were troublesome because of (subsequently v e r i f i e d ) concerns t h a t s t a k e h o l d e r s would experience such gaps as d i s j o i n t e d n e s s i n the p r o c e s s . (Greene/a: 389) C o l l a b o r a t i o n was a l s o the most d i f f i c u l t way to do evaluation. I t demanded a g r e a t d e a l of t i m e . . . . I t was d i f f i c u l t to work w i t h a wide a r r a y of p e r s o n n e l and t r y to r e c o n c i l e the needs of a l l of them. Every step of the process was amended at l e a s t once. (Faase: p. 81) Evaluators e x p l a i n that stakeholders "long  (cumbersome, d i s j o i n t e d ) , "  d e s c r i b e d the process  "slow moving," " c h a l l e n g i n g  ( r e q u i r i n g r e f l e c t i o n and c o n c e n t r a t i o n ) , " and They r e p o r t some of these  as  "complicated".  perspectives:  L i k e most s t a k e h o l d e r s , Pam d i d t h i n k the o v e r a l l process was too l o n g . But, she a l s o r e c o g n i z e d t h a t w i t h a l o n g e r p r o c e s s , the p a r t i c i p a t i o n demands are spread out and thus e a s i e r to meet, e s p e c i a l l y f o r agency s t a f f . She observed that "as you broaden the p a r t i c i p a t i o n , you lengthen the time; a democratic process i s a slow one. (Greene/b: 104) He a l s o found the o v e r a l l l e n g t h and slow pace of the process made i t more d i f f i c u l t f o r him to s u s t a i n i n t e r e s t and harder to p a r t i c i p a t e , i n t h a t he had to r e f r e s h h i s memory a t each s t e p . (Greene/b: 105) "I d i d n ' t know t h a t doing an e v a l u a t i o n was so complicated. You r e a l l y have to t h i n k hard about making up q u e s t i o n s , " one person b l u r t e d out at the end of a s e s s i o n on q u e s t i o n n a i r e d e s i g n . (Whitmore: 223)  78  "I c o u l d do i t b e t t e r i f I were to do i t again, and I would do i t a g a i n . . . I ' d never done anything l i k e t h i s b e f o r e , t o take a b u l k of i n f o r m a t i o n , t r y to draw from i t , put i t i n t o c a t e g o r i e s , and develop workable s o l u t i o n s t h a t were realistic. As we went though the process, i t became more simple." (Ayers: 2 66)  Enabling  Facilitating  Participation  the E v a l u a t i o n  I t appears t h a t a major task of the r e s e a r c h e r i s t h a t of e n a b l i n g the p a r t i c i p a t o r y p r o c e s s e s .  T h i s r e q u i r e s a t t e n t i o n to  e v a l u a t i o n tasks and t h a t p a r t i c i p a t o r y process group process  needs.  and a t t e n t i o n t o  T h i s s e c t i o n d i s c u s s e s the f a c i l i t a t o r  as r e l a t e d t o e v a l u a t i o n tasks and p a r t i c i p a t i o n . s e c t i o n addresses group process E v a l u a t o r s regarded to group p r o c e s s .  The subsequent  needs.  t h e i r r o l e as f a c i l i t a t o r  t o be  important  The f a c i l i t a t o r ensured t h a t the groups  f u n c t i o n e d w e l l , t h a t i t was e f f i c i e n t , smoothly.  role  T h i s process  p r o d u c t i v e , and flowed  r e q u i r e d the e v a l u a t o r t o be i n v o l v e d i n  p r e s e n t i n g o p t i o n s , t r a n s l a t i n g , and s y n t h e s i z i n g i n f o r m a t i o n . These a c t i v i t i e s  were d e s c r i b e d by e v a l u a t o r s a s : " d i v i d e s  t a s k s , " "packages," "step by s t e p , " " c a r e f u l l y s t r u c t u r e d sequence of t a s k s , " and "break down the e v a l u a t i o n p r o c e s s . " example, i n Greene's study,  d e c i s i o n s about method s e l e c t i o n were  guided by c r i t e r i a p r o v i d e d by the r e s e a r c h e r stakeholders  For  to a s s i s t  i n j u d g i n g the most a p p r o p r i a t e method.  Whitmore  r e p o r t e d u s i n g a sequence of tasks t o break down the e v a l u a t i o n  79 process,  helping stakeholders  it.  evaluator  One  understand what to do and how  to  do  explains:  Throughout the process, the c l i e n t was s u b s t a n t i v e l y i n c o n t r o l but needed to be p r o c e d u r a l l y c o u n s e l l e d . For example, the c l i e n t knew the o r g a n i z a t i o n and what was wanted from i t s members, but the e v a l u a t o r suggested how to go about g e t t i n g d e s i r e d information....The working paper was p r e s e n t e d to a p l a n n i n g team and began w i t h i n f o r m a l l e s s o n s i n e v a l u a t i o n r e s e a r c h and step-by-step mapping of c o n s i d e r a t i o n s , o p t i o n s , and d e c i s i o n s . (Faase: 77) Further,  the e v a l u a t o r  i s r e q u i r e d to s y n t h e s i z e  p r e v i o u s l y r e c e i v e d from s t a k e h o l d e r s  and  c i r c u l a t e t h i s back to  them to ensure t h a t the t r a n s l a t i o n i s c o r r e c t . i n t e r p l a y of a c t i v i t i e s , and  the r e s e a r c h e r  ensures t h a t involvement was The  work.  e v a l u a t o r was  and planned due  to the r e s e a r c h e r ' s  m a i n t a i n e d and  evaluator's  f o r "hands-on"  and  often  responsibility for infusing  that the flow of the  evaluation  Regardless,  hands-on work r e s u l t e d i n making  e v a l u a t i o n p r o c e s s more manageable f o r some Evaluators  evaluation  then, were i n t e n t i o n a l  t i m e l i n e s s were e s t a b l i s h e d .  preparation  involvement  through the p r o v i s i o n of  Certain a c t i v i t i e s ,  t e c h n i c a l q u a l i t y or f o r e n s u r i n g was  facilitates  e v a l u a t i o n phase to the next was  the r e s p o n s i b i l i t y of the r e s e a r c h e r d r a f t s f o r review.  Through t h i s  meaningful.  also responsible  Movement from one  material  r e p o r t stakeholder  the  stakeholders.  reflections:  One key element of the p r o c e s s f o r Pam was the e v a l u a t o r ' s r o l e i n c o n d u c t i n g a l l of the " n i t t y - g r i t t y " work. "The e v a l u a t i o n wouldn't have been done otherwise," Pam s a i d , n o t i n g t h a t s t a f f do not have time f o r such work. The e v a l u a t o r ' s r e s p o n s i b i l i t y f o r process, i n c l u d i n g p l a n n i n g follow-up, a l s o made i t "easy" f o r Pam to p a r t i c i p a t e . (Greene/b: 104)  and  80 The e v a l u a t i o n d e s i g n showed a h i g h degree of o r g a n i z a t i o n and competence, which added to the team's c r e d i b i l i t y , but a l s o added to the design's v a l i d i t y and i n t e g r i t y . I t a l s o communicated to me [ c l i e n t ] and to the s t a f f t h a t we weren't going to have to worry about making the e v a l u a t i o n happen. ( A l k i n : 24) You [the e v a l u a t o r s ] s e t the framework, l i k e you s a i d i n the b e g i n n i n g you c o u l d o n l y focus on one e v a l u a t i o n q u e s t i o n . But w i t h i n t h a t framework, i t was democratic. You l i s t e n e d and combined q u e s t i o n s sometimes. But the process was s t i l l open. People c o u l d have i n p u t at any time....Too much involvement would have been oppressive....We wanted you to do t h i s f o r us [e.g., combine q u e s t i o n s ] . We t r u s t e d you to do these kinds of a d m i n i s t r a t i v e t h i n g s . I f we d i d n ' t t r u s t you, we would have wanted to do these o u r s e l v e s ( s t a f f ) . (Greene/a: 389) E v a l u a t o r a t t e n t i o n to e s t a b l i s h i n g the p a r t i c i p a t o r y f a c i l i t a t e s the i n t e r a c t i o n between r e s e a r c h e r and P a r t i c i p a t i o n t h a t was  t h e i r own  essential.  t h a t s t a k e h o l d e r s must be allowed  l e v e l of p a r t i c i p a t i o n without  expectations.  stakeholder.  v o l u n t a r y and open and not c o n d i t i o n a l or  c o n t i n g e n t on p r i o r response i s i d e n t i f i e d as Some e v a l u a t o r s f e l t  process  to f i n d  i m p o s i t i o n s or  E v a l u a t o r s assume t h a t , i f g i v e n the  opportunity,  s t a k e h o l d e r s w i l l p a r t i c i p a t e when they want to and are a b l e t o . They r e p o r t : The d e s i g n a t i o n of s t a k e h o l d e r p a r t i c i p a n t s c o n t i n u e d to change throughout the d e s i g n p r o c e s s . For example, one a d d i t i o n a l youth bureau p a r t i c i p a n t was i d e n t i f i e d v i a snowball sampling invoked i n a l l d i v e r g e n t phase i n t e r v i e w s , and two a d d i t i o n a l day care c o u n c i l s t a f f members i d e n t i f i e d themselves by v o l u n t a r i l y responding to the convergent phase q u e s t i o n n a i r e . (Greene/a: 386) The day c a r e c o u n c i l team v a l u e d the p r o c e s s ' s repeated emphasis on u t i l i z a t i o n and non-judgemental o p t i o n s f o r p a r t i c i p a t i o n , t h a t i s , i t allowed people to f i n d t h e i r l e v e l of p a r t i c i p a t i o n and be comfortable w i t h i t . (Greene/a: 388) Tim viewed h i s "somewhat i n v o l v e d person" p a r t i c i p a t o r y as c o n t r i b u t i n g h i s own views and o p i n i o n s to a l o n g  own  role  81 e v a l u a t i o n p r o c e s s , when he had the time o r when the task was p a r t i c u l a r l y i n t e r e s t i n g o r important. (Greene/b: 105) "Making i t OK t h a t I c o u l d do as much o r as l i t t l e [as I wanted] was r e a l l y important" (Board s t a k e h o l d e r ) . (Greene/a: 388) Attendance discussed.  [by stakeholders] ( A l l e n : 50)  v a r i e d w i t h the i s s u e s t o be  However, whether o r not the e v a l u a t o r takes an a c t i v e r o l e i n pursuing  s t a k e h o l d e r p a r t i c i p a t i o n was i n d i s p u t e .  evaluators f e l t at  times,  t h a t s t a k e h o l d e r s needed t o take r i s k s and t h a t ,  e v a l u a t o r s must h o l d s t a k e h o l d e r s  participation.  Some  accountable  for their  Others attempted t o s t r u c t u r e p a r t i c i p a t i o n t o  ensure an e q u i t a b l e and v a l i d p r o c e s s .  Evaluators  report:  A p a r t n e r s h i p e v a l u a t i o n produces b e n e f i c i a l data when everyone i s a t r u e p a r t n e r who wants t o be i n v o l v e d f o r the purpose o f o b t a i n i n g i n f o r m a t i o n . Each p a r t n e r must be w i l l i n g t o take a r i s k , p a t i e n t about what he o r she w i l l g e t f o r the time and e f f o r t i n v e s t e d , and w i l l i n g t o p r o v i d e i n f o r m a t i o n as i n p u t data. (Shapiro: 61) The assignment o f tasks and assurance t h a t each member " p u l l s t h e i r own weight" i s l i k e l y t o be the r e s p o n s i b i l i t y o f the e v a l u a t o r , i n c o n j u n c t i o n w i t h the group c h a i r p e r s o n . (Ayers: 270) F u l l "ownership" o f the study i m p l i e s c o n t r i b u t i o n s o f work by every member, n o t j u s t by a few committed i n d i v i d u a l s ; a s s i g n i n g r e s p o n s i b i l i t y f o r a d i s t i n c t task o r p a r t o f the study t o each member o r s m a l l subgroup i s l i k e l y t o enhance members' p e r c e p t i o n o f ownership. (Ayers: 270) In our study, we s t r o v e t o i n v o l v e program b e n e f i c i a r i e s e q u i t a b l y , thus h e l p i n g them i n c r e a s e t h e i r power and a v o i d i n g a b i a s i n f a v o r o f teachers and s h e l t e r p r o v i d e r s . (Brandon: 292) P a r t i c i p a t i o n was a l s o impacted by whether o r n o t c l e a r r o l e s and g u i d e l i n e s were e s t a b l i s h e d f o r the c o l l a b o r a t i v e p r o c e s s . Most e v a l u a t o r s e s t a b l i s h e d the parameters o f p a r t i c i p a t i o n from t h e i r r a t i o n a l e f o r stakeholder  involvement such as e v a l u a t i o n  82 u t i l i z a t i o n o r enhanced v a l i d i t y of d a t a .  To h e l p p r o v i d e  c l a r i t y , Whitmore and her p a i d b e n e f i c i a r y a s s i s t a n t s j o i n t l y drew up a c o n t r a c t o u t l i n i n g the terms o f r e f e r e n c e f o r participation.  R e p o r t i n g on the importance o f e s t a b l i s h i n g  clear  goals, evaluators said: The p r o c e s s e s used t o guide and f a c i l i t a t e s t a k e h o l d e r c o l l a b o r a t i o n were s e l f - c o n s c i o u s l y open and p l u r a l i s t i c . Views and o p i n i o n s from each member of the d e l i b e r a t e l y d i v e r s e groups o f s t a k e h o l d e r p a r t i c i p a n t s were a c t i v e l y sought and openly v a l u e d . (Greene/b: 101) That meeting l a i d out Henderson's [stakeholder] c l e a r c o n t r o l of access t o a l l the people w i t h whom the e v a l u a t o r s were going t o t a l k . I t a l s o c l e a r l y e s t a b l i s h e d a l l t h a t he would have t o do t o make sure t h a t the team's s i t e v i s i t s o c c u r r e d p r o p e r l y and e f f i c i e n t l y . ( A l k i n : 27) Crow p r e s e n t e d a vague i n v i t a t i o n to students t o p a r t i c i p a t e as s t a k e h o l d e r s and l a t e r r e g r e t t e d t h a t these r o l e s and s t a k e h o l d e r f u n c t i o n s had not been c l a r i f i e d .  They admit t h a t  t h e i r own d e f i n i t i o n o f c o l l a b o r a t i o n was not c l e a r l y u n t i l they began working together and u n t i l they  determined  experienced  r e l u c t a n c e by the f a c u l t y a d v i s o r s t o p a r t i c i p a t e i n p r o v i d i n g data.  They recount:  Our i n i t i a l l a c k o f c l a r i t y about the r e s e a r c h r o l e o f students became an o b s t a c l e i n e s t a b l i s h i n g and m a i n t a i n i n g p a r i t y and r e c i p r o c i t y w i t h them. We d i s c o v e r e d the d i f f i c u l t y o f i n v i t i n g student p a r t i c i p a t i o n i n a game w i t h e v o l v i n g r u l e s . (Crow: 748) Although the students remained i n the study, t h e i r r o l e was not as e x t e n s i v e as we had i n i t i a l l y envisioned....we a r e a l s o aware o f missed o p p o r t u n i t i e s . (Crow: 753) S p e c i f i c r o l e - r e l a t e d concerns c r e a t e d d i f f e r e n c e s i n p e r s p e c t i v e and emphasis. F o r example, our c o l l e a g u e s , i n t h e i r advisement r o l e s , were concerned t h a t the r e s e a r c h p r o c e s s might preempt t o p i c s o r developmental i s s u e s w i t h students. T h i s concern l e d them t o oppose s p e c i f i c r e s e a r c h q u e s t i o n s and methodologies. A n t i c i p a t i n g the importance o r  83 these r o l e p e r s p e c t i v e s and the ways they can c o n f l i c t w i t h r e s e a r c h r o l e s i s e s s e n t i a l to the success of c o l l a b o r a t i v e r e s e a r c h . (Crow: 752) C l a r i f i c a t i o n of both the g o a l s f o r c o l l a b o r a t i o n and and r e s p o n s i b i l i t i e s of e v a l u a t o r and  stakeholder  the r o l e s  facilitate  the  development of p a r t i c i p a t i o n . Some s t a k e h o l d e r s  i n Greene shared d i f f i c u l t i e s i n  p a r t i c i p a t i o n a r i s i n g from a l a c k of program i n f o r m a t i o n . outcome was  p e r c e i v e d to have important  s e l e c t i o n and these  recruitment  i m p l i c a t i o n s f o r the  of s t a k e h o l d e r s ,  as p r i o r a t t e n t i o n to  f a c t o r s by the r e s e a r c h e r might have avoided  problem.  She  This  such a  reports:  Ann's p e r c e i v e d l a c k of program knowledge was a major o b s t a c l e to her p a r t i c i p a t i o n . Because of i t , she f e l t "shut out" from participation. She commented t h a t perhaps the s t a k e h o l d e r group had been spread too f a r to i n c l u d e people, l i k e h e r s e l f , w i t h o n l y marginal involvement i n the program. (Greene/b, 106)  F a c i l i t a t i n g the Group  The  Process  e v a l u a t o r must be r e s p o n s i v e  to group dynamics and affirmation.  to group p r o c e s s  and  s t a k e h o l d e r needs f o r v a l i d a t i o n  Although t h i s i s regarded  attend  and  as p r i m a r i l y the  e v a l u a t o r ' s r e s p o n s i b i l i t y , i t can extend to s t a k e h o l d i n g group members as w e l l . enables  A t t e n t i o n to i n d i v i d u a l and group process  the p a r t i c i p a t o r y process by b u i l d i n g t r u s t and  credibility.  E v a l u a t o r s d i s c u s s t h i s r o l e and  needs  group  i t s effect  on  stakeholders: Throughout the p r o c e s s , but e s p e c i a l l y at the b e g i n n i n g , we spent c o n s i d e r a b l e time b u i l d i n g group t r u s t , f o r the key to  84 t h e i r c o n t i n u e d p a r t i c i p a t i o n was m o t i v a t i o n . . . . T h i s process c o n s i s t e d of some s t r u c t u r e d e x e r c i s e s , c h e c k i n g i n a t the b e g i n n i n g of each s e s s i o n and r e f l e c t i n g on our i n t e r a c t i o n a t the end, and b e i n g i n touch by telephone between meetings when necessary. (Whitmore: 221) The importance of p e r s o n a l f e e l i n g s of worth and v a l u e w i t h i n the e v a l u a t i o n process were underscored, e s p e c i a l l y by somewhat-involved-person s t a k e h o l d e r s i n the... e v a l u a t i o n s . (Greene/b: 109) At times, the e x t e r n a l e v a l u a t o r takes on a s o r t of mentorship r o l e i n working w i t h c l i e n t s on t h e i r own e v a l u a t i o n s . (Shapiro: 60) The i n t e r n a l advocate must be i n a p o s i t i o n to understand the networking among those i n v o l v e d i n the e v a l u a t i o n . He or she must m a i n t a i n frequent c o n t a c t and open communication w i t h a l l s t a f f and must hone n e g o t i a t i n g s k i l l s so as to c u l t i v a t e the development of a t r u e p a r t n e r s h i p . Throughout the e v a l u a t i o n , the i n t e r n a l advocate must serve as l i a i s o n , both to the s t a f f and to the e x t e r n a l p a r t n e r , never f o r g e t t i n g to assure s t a f f t h a t t h e i r i n p u t i s needed and t h a t no one i s g o i n g to "do i t to them." (Shapiro: 61/62) A t t e n t i o n to group process by the e v a l u a t o r f a c i l i t a t e d mutual support and encouragement among s t a k e h o l d e r s i n the s t u d i e s reviewed.  Stakeholders' i n t e r a c t i o n with other stakeholders  a l s o important  to f a c i l i t a t i n g the p a r t i c i p a t i o n , as the process  should promote r e s p e c t and a t t e n t i o n to o t h e r s ' experiences views.  The  and  c r e a t i o n of a s a f e environment i n which s t a k e h o l d e r s  c o u l d share and  f e e l they had a sense of b e l o n g i n g  stakeholder p a r t i c i p a t i o n . was  was  a l s o seen as important  establishing trust.  assisted  Informal s h a r i n g amongst each other to b u i l d i n g group r e l a t i o n s h i p s  and  Evaluators r e f l e c t :  The c o f f e e break midway through each meeting was an important time f o r p e r s o n a l s h a r i n g and exchange of i n f o r m a t i o n , e s p e c i a l l y around t h e i r own experience as s i n g l e mothers. (Whitmore: 221) An i n i t i a l i n t r o d u c t o r y meeting of a l l s t a k e h o l d e r s was planned, and, w i t h h i n d s i g h t , was needed. T h i s meeting would  85 serve to i n t r o d u c e the e v a l u a t i o n process and r a t i o n a l e to the s t a k e h o l d e r s and to i n t r o d u c e the s t a k e h o l d e r s to each other...many s t a k e h o l d e r s had no i n f o r m a t i o n on what to expect or on the number and r o l e s of the other p a r t i c i p a n t s . (Greene/a: 389) The e v a l u a t o r s p l a y e d the r o l e of n e u t r a l f a c i l i t a t o r s making sure t h a t the ideas of Task Force members...were heard by a l l members of the Design Subcommittee. ( G i l l : 105) So the team emerged as a c r e d i b l e u n i t , but the i n d i v i d u a l i t y of team members--and what each brought to the team--was never f o r g o t t e n . ( A l k i n : 21) As the p a r t n e r s h i p developed, t r u s t was b u i l t , so t h a t a l l p a r t n e r s began to f e e l t h a t they c o u l d open up and share, without f e a r of b e i n g h u r t . No longer was t h e r e an i d e a t h a t the e x t e r n a l e v a l u a t o r would "do i t a l l . " (Shapiro: 57) Many e v a l u a t o r s i d e n t i f i e d a n o t i c e a b l e change i n the r e l a t i o n a l dynamics d u r i n g the course of the e v a l u a t i o n . was  There  a sense of c r e d i b i l i t y and i n t e g r i t y , of working together  a team.  Whitmore n o t i c e d t h a t her s t a f f were more c o n f i d e n t  as and  i n t e r a c t e d more w i t h o t h e r s i n the program, and Faase r e f e r s to a "chemistry"  t h a t developed  over the course of the  study.  A t t e n t i o n to group process a l s o r e q u i r e d the r e s e a r c h e r to v a l i d a t e and  support  stakeholders i n t h e i r p a r t i c i p a t i o n .  s t r a t e g y seemed e s p e c i a l l y important s t a k e h o l d e r s who or experience  This  when working w i t h  l a c k e d c o n f i d e n c e and d i d not have the  of other s t a k e h o l d e r groups.  education  Whitmore d i s c u s s e s  t h i s aspect a t l e n g t h i n her work w i t h program b e n e f i c i a r i e s paid research s t a f f .  Her d e s c r i p t i o n of her a c t i v i t i e s  t h a t she took on the r o l e of n u r t u r e r w i t h s t a k e h o l d e r s .  as  revealed She  reports: They a l s o had l i m i t e d confidence i n t h e i r a b i l i t y to do complex i n t e l l e c t u a l tasks and responded p o s i t i v e l y to p r a i s e and my h i g h e x p e c t a t i o n s of them. (Whitmore: 224)  86 Once they r e a l i z e d that t h e i r own experiences and i d e a s were r e s p e c t e d and l i s t e n e d t o [by the a d v i s o r y committee], the p a r t i c i p a n t s got q u i t e involved i n t a l k i n g e n t h u s i a s t i c a l l y about what they were doing. I n the p r o c e s s , they gained c o n s i d e r a b l e c o n f i d e n c e i n themselves and t h e i r a b i l i t y to speak t o o t h e r s about the p r o j e c t . (Whitmore: 225) The group p r o c e s s r e i n f o r c e d t h e i r sense o f d i g n i t y and l e g i t i m a c y as they encouraged one another t o stand up f o r t h e i r r i g h t s . (Whitmore: 225) In a study on homelessness, small remuneration  s t a k e h o l d e r s were p r o v i d e d w i t h a  for their participation.  T h i s s e r v e d as a  v e h i c l e f o r m o t i v a t i n g f u l l p a r t i c i p a t i o n and, appreciation for their  i n a sense, showed  contributions.  C u l t i v a t i n g group process allowed the r e s e a r c h e r access t o " i n s i d e r i n f o r m a t i o n " which was important to understanding s t a k e h o l d e r p a r t i c i p a t i o n i n the e v a l u a t i o n p r o c e s s .  Evaluator  awareness o f these f a c t o r s was important because i t c o u l d ( i n ) d i r e c t l y i n f l u e n c e the success o f the e v a l u a t i o n .  Evaluators  d i s c u s s these e x t e r n a l f a c t o r s : I t was important f o r the e x t e r n a l p a r t n e r (no l o n g e r "the e v a l u a t o r " ) t o know that the atmosphere a t the u n i v e r s i t y was changing. New v i c e - p r e s i d e n t had been h i r e d , m e r i t pay was a t o p i c o f major concern, and promotion and tenure p o l i c i e s were b e i n g r e w r i t t e n . A l l these i s s u e s a f f e c t e d the s t a f f ' s m o t i v a t i o n f o r p a r t i c i p a t i o n i n an e v a l u a t i o n . These i s s u e s c o u l d be d i s c u s s e d w i t h i n the p a r t n e r s h i p , but an e x t e r n a l e v a l u a t o r under c o n t r a c t might never have l e a r n e d o r been i n t e r e s t e d i n l e a r n i n g about them. (Shapiro: 57) Under the time p r e s s u r e of completing a complex task w i t h s i g n i f i c a n t p o l i t i c a l and funding consequences, c o n f l i c t s and misunderstandings arose w i t h some frequency and r e q u i r e d time to r e s o l v e . ( A l l e n : 48) Two o r the three advisors...were new to the u n i v e r s i t y . In our view, t h e i r l a c k o f experience i n a d v i s i n g students and the u n i v e r s i t y ' s new c h a l l e n g e o f p r e p a r i n g c a r e e r changers to teach may have made f a c u l t y p a r t i c i p a t i o n i n an u n f a m i l i a r mode o f i n q u i r y h i g h l y p r o b l e m a t i c (Crow: 745)  87 P r e v i o u s experience had made the membership wary and uncooperative. There was a r e a l r e s i s t a n c e to the i s s u e s p r e s e n t e d by the e v a l u a t i o n . . . . T h e p r e l i m i n a r y r e p o r t . . . calmed some f e a r s on the p a r t of s t a f f people who were i n v o l v e d i n p l a n n i n g the c h a p t e r . (Faase: 78) G i l l r e v e a l s t h a t the e x t e r n a l p r e s s u r e s of working i n a complex, politically  s e n s i t i v e system l e d e v a l u a t o r s to i n v o l v e  s t a k e h o l d e r s i n the e v a l u a t i o n process i n the f i r s t p l a c e . A t t e n t i o n to group process r e q u i r e d f l e x i b i l i t y . e x t e r n a l demands can i n f l u e n c e s t a k e h o l d e r ' s a b i l i t y  Because to  p a r t i c i p a t e , e v a l u a t o r s had to be s e n s i t i v e to o u t s i d e p r e s s u r e s and a d j u s t the r e s e a r c h process a c c o r d i n g l y . t h i s meant c r e a t i n g new  opportunities for  For some e v a l u a t o r s  involvement.  S t a k e h o l d e r s ' s needs were addressed by making changes and providing choices for p a r t i c i p a t i o n .  Evaluators describe:  I t was apparent t h a t the amount of data requested of the f a c u l t y was e x c e s s i v e , c o n s i d e r i n g the s m a l l number of f a c u l t y , t h e i r time c o n s t r a i n t s , and t h e i r l i m i t e d s e c r e t a r i a l support.... One of the f i r s t s t r a t e g i e s adopted by the committee was to change the time frames f o r the r e p o r t i n g of e v a l u a t i o n d a t a . Reports, whose due dates had p r e v i o u s l y been staggered throughout the academic year, were now due d u r i n g one d e s i g n a t e d " E v a l u a t i o n Week," which was a t the end of the academic y e a r when c l a s s e s and f i n a l grades had been completed. A l l o t h e r a c t i v i t i e s , such as committee meetings were suspended....The i n t e n t of t h i s s t r a t e g y was t o s t r u c t u r e a v a i l a b l e time f o r busy f a c u l t y . E v a l u a t i o n forms were r e v i s e d t o c h e c k l i s t type e n a b l i n g a much f a s t e r completion time. ( B a r r i c k : 34/35) As the academic year progressed, i t became i n c r e a s i n g l y d i f f i c u l t to i n v o l v e a l l of the students as c o l l a b o r a t o r s i n inquiry. Student i n t e r e s t remained h i g h , as evidenced by t h e i r c o n t i n u e d attendance and p a r t i c i p a t i o n i n meetings. However, time p r e s s u r e prevented them from engaging as f u l l y as we had hoped....We sought to extend o p t i o n on p a r t i c i p a t i o n by i n d i c a t i n g t h a t d i f f e r e n t r o l e s and l e v e l s of p a r t i c i p a t i o n were p o s s i b l e . (Crow: 747)  88 To h e l p m a i n t a i n the s t a k e h o l d e r s ' m o t i v a t i o n t o p a r t i c i p a t e f u l l y and a c t i v e l y , the study was designed t o l i m i t demands on t h e i r time. (Brandon: 290) Furthermore, the way i n which the e x e c u t i v e summary was w r i t t e n was h e l p f u l i n f a c i l i t a t i n g [stakeholder review] w i t h i n a t i g h t t i m e l i n e . The e v a l u a t o r ' s commitment t o summarizing f i n d i n g s i n two pages and t o r a i s i n g a s e t o f p a r t i c u l a r q u e s t i o n s v e r y much helped. ( A l k i n : 2 9) In two o f the e v a l u a t i o n s , an i n t e r n a l e v a l u a t o r o r advocate p l a y e d an important  r o l e i n a t t e n d i n g t o the group  Because the i n t e r n a l advocate u s u a l l y entered process w i t h b u i l t - i n  the e v a l u a t i o n  t r u s t and c r e d i b i l i t y w i t h the l a r g e r  s t a k e h o l d i n g body, they served as a l i a i s o n u n t i l credibility  c o u l d develop between the s t a k e h o l d e r s  primary e v a l u a t o r .  process.  Evaluators  t r u s t and and the  report:  T h i s was a p r o j e c t t h a t c o u l d not have been c a r r i e d out u n l e s s an e x t e r n a l e v a l u a t o r had p r o v i d e d t e c h n i c a l a s s i s t a n c e and an i n t e r n a l advocate had been not o n l y s e n s i t i v e t o u n i v e r s i t y p o l i t i c s b u t a l s o a b l e t o h o l d the p r o j e c t together u n t i l the external partner's r e l a t i o n s h i p s with i n d i v i d u a l a c t i v i t y d i r e c t o r s had been s o l i d i f i e d The p a r t n e r s h i p had t o be brokered by the i n t e r n a l advocate, and so f i r s t there was a c e n t r a l p a r t n e r s h i p between myself and him, c o n c u r r e n t w i t h his i n i t i a l partnership with a c t i v i t y d i r e c t o r s . Eventually I developed p a r t n e r s h i p s w i t h a c t i v i t y d i r e c t o r s . (Shapiro: 61) The p r o j e c t d i r e c t o r c o n t i n u o u s l y s t r e s s e d the competence of the team and the r e p r e s e n t a t i v e n e s s o f i t s s e l e c t i o n "We s u r e l y d i d have some i n i t i a l c r e d i b i l i t y but i t was extended by your [ i n t e r n a l e v a l u a t o r ] a c t i o n s " . ( A l k i n : 20) In the overseas a g r i c u l t u r a l study by A l k i n ,  the e v a l u a t o r s  a b i l i t y t o develop r e l a t i o n s h i p s was hampered by d i s t a n c e and the t r a v e l required f o r personal contacts. communication was fundamental.  As such, w r i t t e n  The e v a l u a t o r r e p o r t s :  Given these c o n s t r a i n t s , one o f the ways t h a t the e v a l u a t o r s t r i e d t o b u i l d confidence i n the e v a l u a t i o n was t o make sure t h a t a l l o f the steps l e a d i n g to the conduct o f the e v a l u a t i o n  89 were w e l l d e f i n e d and t h a t a c l e a r r e l a t i o n s h i p between steps e x i s t e d . ( A l k i n : 24) The program d i r e c t o r ' s p e r s o n a l involvement,  supplemented by a  c l e a r e v a l u a t i o n document, f a c i l i t a t e d the e v a l u a t o r / s t a k e h o l d e r relationship. In one study a t r a i n e d group f a c i l i t a t o r conducted group meetings u s i n g a technique designed  to help lessen  d i f f e r e n c e s i n group p a r t i c i p a t i o n due t o s t a t u s , behaviour, making.  stakeholder  domineering  and/or s t a k e h o l d e r u n f a m i l i a r i t y w i t h group d e c i s i o n -  Group s i z e was a l s o l i m i t e d i n an e f f o r t t o decrease the  l i k e l i h o o d t h a t the meeting would be i n t i m i d a t i n g f o r b e n e f i c i a r i e s , and, i n some stages o f the decision-making p r o c e s s , p r i v a t e r a n k i n g and s e c r e t b a l l o t s were used.  They  report: The p a r t i c i p a t i n g homeless parents i n our study were u n f a m i l i a r w i t h program e v a l u a t i o n r a t i o n a l e s o r methods and might have been i n t i m i d a t e d by formal decision-making s e t t i n g s and procedures. T h e r e f o r e , e q u i t y o f p a r t i c i p a t i o n was p a r t i c u l a r l y important i n the s t a k e h o l d e r meeting and steps were taken t o use t e c h n i c a l l y adequate procedures f o r b e n e f i c i a r y involvement. (Brandon: 291) A t t e n d i n g t o group p r o c e s s a l s o meant c o n f r o n t i n g i s s u e s as they came up.  Negative  group dynamics hampered p a r t i c i p a t i o n i n  some s t u d i e s .  Evaluators reveal:  Perhaps the b i g g e s t o b s t a c l e was the u n d e r l y i n g l a c k o f t r u s t . The women simply d i d not t r u s t each o t h e r s and were q u i c k t o assume the worst whenever something happened.... Though t h e i r s t r o n g commitment t o the task allowed them t o work p r o d u c t i v e l y together, there was always an u n d e r l y i n g t e n s i o n which o c c a s i o n a l l y rose to the s u r f a c e ; we then had t o spend time d e a l i n g w i t h the r e s u l t i n g h o s t i l i t y and h u r t . The group dynamics were sometimes d i f f i c u l t t o handle, and one o f the o r i g i n a l f o u r , f e e l i n g caught i n the middle, d i d drop o u t . (Whitmore: 22 6)  90 While the meetings proved a c h a l l e n g e to manage due to the s t r o n g l y h e l d p o s i t i o n s of f o r c e f u l i n d i v i d u a l s , they l a r g e l y served the purposes f o r which they were i n t e n d e d . ( A l l e n : 50) D i f f i c u l t i e s i n group dynamics t h a t were p r e s e n t i n one advisement group may have i n t e n s i f i e d a d v i s o r ' s d i s c o m f o r t about a d d i t i o n a l d i s c l o s u r e . (Crow: 745) In t h i s l a t t e r context, a f a c u l t y member a t t a c k e d the r e s e a r c h e r s i n t e g r i t y and c h a l l e n g e d the grounds on which the r e s e a r c h e r s were p r o c e e d i n g w i t h the study.  Crow's s i t u a t i o n was  unique because the r e s e a r c h team was i n t e r d i s c i p l i n a r y r e s e a r c h approach.  attempting They  a  somewhat  first-time  conclude:  We b e l i e v e i t i s extremely d i f f i c u l t to s i m u l t a n e o u s l y e s t a b l i s h i n t e r n a l i n t e r d i s c i p l i n a r y c o l l a b o r a t i o n and two a d d i t i o n a l k i n d s of e x t e r n a l c o l l a b o r a t i o n . (Crow: 753) The complexity of our e f f o r t to c r e a t e and develop p a r i t y and r e c i p r o c i t y w i t h i n the i n t e r d i s c i p l i n a r y team assumed p r i o r i t y (Crow: 748) Awareness of these f a c t o r s by A l l e n , who  was  also establishing  a f i r s t - t i m e c o l l a b o r a t i v e r e s e a r c h e f f o r t between two r e s u l t e d i n h i r i n g an extremely f a c i l i t a t e the e v a l u a t i o n team.  agencies,  s t r o n g p r o j e c t d i r e c t o r to Evaluators report:  These normal d i f f e r e n c e s [of opinion] were a m p l i f i e d by the problem of a complex study which both depended upon t i m e l y completion of tasks and y e t l a c k e d a l i n e - r e l a t i o n s h i p among p a r t i c i p a n t s . ( A l l e n : 50)  S u c c e s s f u l P a r t i c i p a t i o n C u l t i v a t e s Ownership  Ownership was was  an important  theme emerging from the d a t a .  a c e n t r a l concept which seemed to be interwoven  with  themes of r e i n f o r c i n g the genuineness of p a r t i c i p a t i o n f a c i l i t a t i n g i n t e r a c t i o n between r e s e a r c h e r and  the  and  stakeholder.  It  91 I n i t i a l l y i t was a d i f f i c u l t  theme t o c o n c e p t u a l i z e , however, a  number o f f a c t o r s c o n t r i b u t e to ownership of the e v a l u a t i o n by stakeholders.  These f a c t o r s a r e d i s c u s s e d as f o l l o w s .  "Hands On" P a r t i c i p a t i o n  P a r t i c i p a t i o n which i s "hands on" c o n t r i b u t e d t o a sense of ownership by s t a k e h o l d e r s .  Evaluators reveal:  The p r o c e s s of r e v i e w i n g the i n s t r u m e n t a t i o n w i t h Tom helped i n c r e a s e the extent t o which those were Tom's q u e s t i o n s . . . . The more people who can review i t , the more ownership t h e r e i s - - i t makes i t t h e i r q u e s t i o n s . Even though we sent i t out t o a l l of these o t h e r s t a k e h o l d e r s t o get t h e i r review, i t was not as e f f e c t i v e as i f someone was there t o hand-hold and take them through the d e t a i l s . ( A l k i n : 27) The time devoted t o the c o l l a b o r a t i o n process heightened the investment of a l l concerned and s t r o n g l y f o s t e r e d the p r o v i n c e ' s a p p r o p r i a t i o n of the e v a l u a t i o n as i t s own. (Faase: 77) The e v a l u a t o r s maintained involvement of the c l i e n t group throughout the p r o j e c t . T h i s i n s u r e d t h a t the most important q u e s t i o n s were addressed, i n c r e a s e d a sense of ownership i n the p r o c e s s and products, maintained a h i g h l e v e l o f i n t e r e s t of the course of a l o n g p r o j e c t (one y e a r ) , and c r e a t e d an investment i n the p r o j e c t as a whole. ( G i l l : 108) F u l l "ownership" of the study i m p l i e s c o n t r i b u t i o n s of work by every member, not j u s t by a few committed i n d i v i d u a l s ; a s s i g n i n g r e s p o n s i b i l i t y f o r a d i s t i n c t task o r p a r t of the study t o each member o r small subgroup i s l i k e l y t o enhance members' p e r c e p t i o n of ownership. (Ayers: 270) Stakeholder  ownership was enhanced by a sense of c o n t r o l o f  the p r o c e s s , due t o t h e i r ongoing involvement  and i n p u t .  The s t a g i n g [of the e v a l u a t i o n ] i s what allowed [stakeholder group] t o f e e l t h a t they were m a i n t a i n i n g some c o n t r o l a l s o co-opted them i n t o having t o take the t h i n g more s e r i o u s l y , because they were approving i t a l l along the way. ( A l k i n : 26)  92 An  e s s e n t i a l component of ownership appeared to be  stakeholders  that  needed to take advantage of o p p o r t u n i t i e s  participation.  Stakeholders have a r e s p o n s i b i l i t y to ensure that  the e v a l u a t i o n a c c u r a t e l y r e f l e c t s the d i r e c t i o n and they have p r o v i d e d .  feedback  Stakeholders must take an a c t i v e r o l e i n  t h i s p r o c e s s - - t h e y cannot p a s s i v e l y s i t by and views w i l l be  for  incorporated.  assume that  A c l i e n t stakeholder  their  reports:  The e v a l u a t o r should help the c l i e n t formulate the focus, but the c l i e n t has to be sure that the questions are h i s own and that they address h i s own i s s u e s . The more the c l i e n t understands and can take ownership f o r what went i n t o the " f r o n t end" of the p r o j e c t , the more l i k e l y i t i s t h a t he w i l l comprehend and accept the r e s u l t s . (Faase: 82)  Stakeholder P e r c e p t i o n s  of an  "Inside  Job"  Ownership i s f o s t e r e d when the e v a l u a t i o n i s seen as " i n s i d e job" and not as something done to them by Evaluators  outsiders.  r e v e a l t h e i r s e n s i t i v i t y of t h i s f a c t o r when they used  such phrases as:  "so they wouldn't f e e l i t was  was  t h e i r p r o j e c t , " "never had  one  i s g o i n g to  stakeholder  an  'do  the sense of b e i n g  i t to them.'"  perceptions  my  Evaluators  s e t up," " i t 'snowed,'"  illustrate  "no  how  i n f l u e n c e d ownership:  The p r o d u c t s of the s t u d i e s were not viewed as simply the r e s e a r c h of an independent, o u t s i d e i n v e s t i g a t o r . Members worked hard to understand the f i n d i n g s w e l l enough so t h a t they c o u l d r e p o r t the f i n d i n g s as t h e i r own. ( G i l l : 107) S t a k e h o l d e r s c h a r a c t e r i z e d the d e s i g n p r o c e s s as...open and v a l i d ( c r e d i b l e , broad-based, coming from w i t h i n r a t h e r than from the top or o u t s i d e ) . (Greene/a: 388) F a c u l t y need to be i n v o l v e d i n p r o j e c t s to o b t a i n t h e i r and to f o s t e r t h e i r sense of ownership. ( B a r r i c k : 36)  input  93 As a r e s u l t o f the i n t e r a c t i v e feedback and m u l t i p l e o p p o r t u n i t i e s f o r p a r t i c i p a t i o n , even those tasks completed by the r e s e a r c h e r were p e r c e i v e d evaluator  to be owned by s t a k e h o l d e r s .  One  reveals:  I n d i c a t i v e o f t h e i r sense o f ownership i n the p r o j e c t , Task Force members t a l k e d as i f they had produced the f i n d i n g s and the r e p o r t . ( G i l l : 107) Another e v a l u a t o r r e v e a l s how the c l o s e n e s s  o f the  r e l a t i o n s h i p which developed between one s t a k e h o l d e r and researcher  enhanced s t a k e h o l d e r  p e r c e p t i o n o f the e v a l u a t i o n  b e i n g an i n s i d e j o b : The r e c e p t i o n o f the o r a l p r e s e n t a t i o n o f the e v a l u a t i o n r e s u l t s and recommendations evidenced an e x t e n s i o n o f the "chemistry" t h a t had gone i n t o the c o l l a b o r a t i o n o f c l i e n t and researcher. The r e p o r t had a compelling presence i n the [ r e l i g i o u s ] chapter due to shared understanding. (Faase: 80) Further,  the presence o f r e p r e s e n t a t i v e s t a k e h o l d e r s as  p a r t i c i p a n t s c o n t r i b u t e d t o t h i s sense o f ownership and i n s i d e c o n t r o l , even f o r s t a k e h o l d e r s evaluation p a r t i c i p a n t s .  not d i r e c t l y i n v o l v e d as  Evaluators  report:  They had recommended o n l y one member o f the e v a l u a t i o n team [Jerry]. He was t h e i r o n l y p r o t e c t i o n , o r guarantee o r independent judgement....His agreement t o the p l a n added a p i e c e o f c r e d i b i l i t y t o i t , t h a t the teaming was working o u t . C e r t a i n l y i f i t hadn't worked J e r r y would have l e t them [ s t a k e h o l d i n g group] know. I f he had any concerns about you or the team o r the process a t the d e s i g n stage, I t h i n k they expected he would l e t them know. ( A l k i n : 26) Marlene, a UWI f a c u l t y member, represented the Caribbean p e r s p e c t i v e and, i n i d e o l o g i c a l terms, h e r p e r s p e c t i v e was n e c e s s a r y t o make the whole process c r e d i b l e t o UWI s t a f f . Her presence was an assurance t h a t the e v a l u a t i o n wasn't something t h a t was b e i n g c o n t r o l l e d by the o u t s i d e , because she was an " i n s i d e " Caribbean person. That was important t o the team makeup and i t s wider a c c e p t a b i l i t y . ( A l k i n : 21)  94  Without t h i s " i n s i d e r " or " p a r t n e r s h i p " d e s i g n a t i o n , e v a l u a t i o n e f f o r t s c o u l d be thwarted by uncooperative For example, one  insiders.  e v a l u a t o r r e p o r t s the i r o n i c occurrence  p r o t e c t i n g the i n t e r n a l d i r e c t o r from the e v a l u a t i o n .  of  staff  I t was  not  u n t i l the i n t e r n a l d i r e c t o r v i s i b l y came on board as a p a r t n e r i n the e v a l u a t i o n t h a t " i n s i d e r s " were w i l l i n g to respond.  They  e x p l a i n t h i s development: I r o n i c a l l y , the s t a f f t r i e d to p r o t e c t me [ i n t e r n a l advocate] from the e v a l u a t i o n p r o c e s s . When the survey of my performance as grant d i r e c t o r went out to a sample of f a c u l t y and s t a f f , w i t h a cover l e t t e r b e a r i n g o n l y the e x t e r n a l e v a l u a t o r ' s s i g n a t u r e , s e v e r a l people came to assure me not to worry; they were not going to send the survey i n and would not p r o v i d e i n f o r m a t i o n r e g a r d i n g a c o l l e a g u e to an o u t s i d e r . I had to i n f o r m them t h a t the survey was my i d e a and t h a t I wanted them to respond. I r e a l i z e d t h a t the problem c o u l d have been avoided i f the cover l e t t e r had c a r r i e d my name on i t as w e l l as Dr. S h a p i r o ' s . T h i s would have demonstrated that our p a r t n e r s h i p was important and p r o v i d e d c o m p e l l i n g evidence t h a t we r e a l l y wanted sound i n f o r m a t i o n . (Shapiro: 61) The  i n s i d e r / o u t s i d e r issue a l s o hindered  efforts.  However, i n t h e i r s i t u a t i o n ,  i n s i d e r s presented  Crow's c o l l a b o r a t i o n  the f a c t t h a t they were  a t h r e a t to other s t a k e h o l d e r s .  They r e p o r t :  The f a c t t h a t we were f a c u l t y members i n the same c o l l e g e as the a d v i s o r s and students presented an i n s i d e r / o u t s i d e r i s s u e s t h a t a l s o threatened the c o l l a b o r a t i o n . As the r e s e a r c h moved i n t o teacher education areas, the v u l n e r a b i l i t y f o r a d v i s o r s was heightened, and we were p e r c e i v e d as o u t s i d e r s i n r e g a r d to access to data. Although there are b e n e f i t s to in-house r e s e a r c h . . . a p e r c e i v e d t h r e a t f o r o t h e r s i n v o l v e d i n the r e s e a r c h may weaken the c o l l a b o r a t i v e nature of the d e s i g n . (Crow: 752) Ownership i s a l s o a c o n s i d e r a t i o n f o r those o u t s i d e  the  p a r t i c i p a t o r y process--whether they p e r c e i v e the e v a l u a t i o n as b e i n g an i n s i d e job has  i m p l i c a t i o n s f o r the e v a l u a t i o n .  v i s i b l e p a r t i c i p a t i o n of s t a k e h o l d e r s who  The  are members from the  95 l o c a l community can p r o v i d e an i n s i d e c o n n e c t i o n .  Whitmore  r e v e a l s t h a t she h i r e d f o u r program p a r t i c i p a n t s as h e r a s s i s t a n t s i n order t o p r o v i d e a l i n k w i t h women who took the program.  She r e p o r t s t h a t , i n the eyes o f community members,  they had c r e d i b i l i t y : They i n t e r v i e w e d program p a r t i c i p a n t s , a h a r d - t o - r e a c h p o p u l a t i o n who do not tend t o cooperate w i t h o u t s i d e r s , l e t alone o u t s i d e e v a l u a t o r s . The e v a l u a t i o n a s s i s t a n t s were p a r t of the community, knew the c u l t u r e , and understood i n t u i t i v e l y how t o approach t h e i r peers i n a way no o u t s i d e r c o u l d match. (Whitmore: 219) Because the e v a l u a t i o n was viewed as an i n s i d e job, t h e i r peers t r u s t e d t h a t t h e i r "stake" would be s a f e w i t h them. In another i n s t a n c e , because o f the weighty and s e n s i t i v e issues explored,  the e v a l u a t i o n r e q u i r e d a r e s e a r c h e r who was an  o u t s i d e r and c o u l d be viewed as an o b j e c t i v e p a r t i c i p a n t . approach was necessary because o f a p r e v i o u s  t o ensure the success r e s e a r c h experience  o f the j o i n t  This venture  t h a t had proved  inadequate.  As a r e s u l t , s t a k e h o l d e r s were wary o f the  evaluation.  Evaluators  report:  The e v a l u a t o r was a b l e t o enter i n t o and understand the uniqueness o f the r e l i g i o u s community without thereby shedding the " o u t s i d e r " p e r s p e c t i v e and " o b j e c t i v i t y " he was expected to b r i n g t o the p r o j e c t . (Faase: 77) E v a l u a t o r s a r e t e c h n i c a l a d v i s o r s and s e r v i c e p r o v i d e r s , and t h e i r p e r c e i v e d o b j e c t i v i t y and p o l i t i c a l n e u t r a l i t y may be c r i t i c a l t o the study's acceptance and the group's p e r c e p t i o n of ownership. (Ayers: 270)  96 Tailor-Made  Evaluation  C r e a t i n g an e v a l u a t i o n t h a t i s s p e c i f i c t o the needs and d e s i r e s o f p a r t i c i p a t i n g s t a k e h o l d e r s heightens the e v a l u a t i o n i s seen as " t a i l o r made." another  ownership because  T h i s p e r c e p t i o n adds  dimension t o the theme o f ownership.  In the s t u d i e s reviewed, s t a k e h o l d e r s p e r c e i v e d t h a t the e v a l u a t i o n was designed  s p e c i f i c a l l y f o r them and f o r t h e i r  program, thus i t was s p e c i f i c to t h e i r needs and r e l e v a n t t o t h e i r l o c a l context. was made by them.  In f a c t , not o n l y was i t made f o r them, i t  Evaluators reveal:  By i n v o l v i n g the c l i e n t group as c o n s u l t a n t s throughout the e v a l u a t i o n p r o c e s s , the r e s u l t i n g i n f o r m a t i o n was h i g h l y s p e c i f i c t o the c l i e n t ' s needs. ( G i l l : 108) Word spread t h a t the e v a l u a t i o n under way was grounded i n and w e l l attuned t o the concerns o f the membership....Tailor making the e v a l u a t i o n r e s e a r c h d e s i g n b u i l t c o n f i d e n c e . Relevance seemed ensured. (Faase: 79/80) E v a l u a t o r s r e v e a l the d i s c r e p a n c y between p a r t i c i p a t o r y e v a l u a t i o n and the i m p o s i t i o n o f p r e c o n c e i v e d  n o t i o n s f o r what  the e v a l u a t i o n should be: One cannot impose a r e s e a r c h design, a s t a t i s t i c a l technique a l o c a l l y c o n s t r u c t e d instrument, o r even a "good i d e a " when the p r e s c r i b e d ground r u l e s g i v e s u b s t a n t i a l r e s p o n s i b i l i t y and a u t h o r i t y t o the c l i e n t . (Shapiro: 57) I f you had come i n w i t h p r e d i s p o s i t i o n s about what t o e v a l u a t e , based upon conceptions and understandings from "the l i t e r a t u r e " on what a g r i c u l t u r a l e x t e n s i o n i s supposed t o do, there would have been problems. I f you had spent your time f e e l i n g t h a t you had t o t e s t out some k i n d o f a d u l t - l e a r n i n g model o r i n s t i t u t i o n a l change model out o f the l i t e r a t u r e , t h a t would have been d i f f i c u l t . And you might have d e r i v e d from such a model a bunch o f v a r i a b l e s i d e n t i f i e d as important to look a t , and maybe some i n s t r u m e n t a t i o n . That would not have gone w e l l . ( A l k i n : 22)  97  T h i s i s confirmed by a c l i e n t  stakeholder:  Any e v a l u a t o r who i n t e n d s t o meet the needs o f a c l i e n t has to be c a r e f u l l y attuned to the uniqueness o f the c l i e n t ' s p a r t i c u l a r group wholesale t r a n s p o s i t i o n o f one environment t o another, without necessary and a p p r o p r i a t e adjustments, i l l serves any c l i e n t . (Faase: 81)  C r e a t i n g a Readiness f o r Use  There seems t o be a n a t u r a l p r o g r e s s i o n f o r s t a k e h o l d e r s o f moving from a c t i v e e v a l u a t i o n p a r t i c i p a t i o n t o f o l l o w through o f the e v a l u a t i o n recommendations. p a r t i c i p a t o r y process prepare stakeholders.  Stakeholder  e v a l u a t i o n , access  V a r i e d aspects o f the  the groundwork f o r u t i l i z a t i o n by  awareness and f a m i l i a r i t y o f the  t o ongoing i n f o r m a t i o n and p r o g r e s s r e p o r t s ,  and a n t i c i p a t i o n o f i t s r e s u l t s , a l l c o n t r i b u t e t o e v a l u a t i o n use.  Evaluators report:  User commitment i s r e f l e c t e d i n s t a k e h o l d e r w i l l i n g n e s s t o a l l o c a t e s u b s t a n t i a l time t o the e v a l u a t i o n and i t s f o l l o w through. Commenting t h a t i t was time t o " s h i f t from an e v a l u a t i o n mode t o a p l a n n i n g mode" the youth employment c o o r d i n a t o r w i l l i n g l y a s s i s t e d the e v a l u a t o r i n a d v o c a t i n g f o r use....the agency board and e s p e c i a l l y program committee s t a k e h o l d e r s a s s i g n e d themselves the task o f m o n i t o r i n g the f o l l o w i n g through on e v a l u a t i o n f i n d i n g s , most c l e a r l y r e p r e s e n t e d i n the i n t e n s i v e agency p l a n n i n g e f f o r t they helped t o i n i t i a t e (Greene/b: 113) The important s t a k e h o l d e r s "bought i n t o " the f i n a l r e s u l t s because the sequence was so c l e a r and they had reviewed and approved i t a t a number of j u n c t u r e s along the way. ( A l k i n : 24) As one [stakeholder] s t a t e d , "By doing an e v a l u a t i o n themselves, people a r e more l i k e l y t o see why they should change. (The u s u a l r e p o r t ) would g e t stuck on a s h e l f ; people would laugh about i t . " (Ayers: 266)  98 "I [stakeholder] don't t h i n k e v e r y t h i n g i n the r e p o r t w i l l be implemented, e s p e c i a l l y r i g h t away--they a r e recommendations to shoot f o r - - b u t s i n c e the ideas came from t e a c h e r s , the union w i l l read i t , and o t h e r s w i l l read i t . " (Ayers: 269) The key people who were going t o have t o a c t on immediate u t i l i z a t i o n were i n v o l v e d and were prepared t o hear answers, because they knew what the questions were. ( A l k i n : 28) Because the r e s u l t i n g e v a l u a t i o n i n f o r m a t i o n was h i g h l y r e l e v a n t t o s t a k e h o l d e r needs and r e f l e c t e d t h e i r concerns and t h e i r q u e s t i o n s , u t i l i z a t i o n was enhanced. involvement o f s t a k e h o l d e r process  In a d d i t i o n , the  r e p r e s e n t a t i v e s i n the p a r t i c i p a t o r y  ensured t h a t the e v a l u a t i o n was r e s p o n s i v e  s i t u a t i o n and p o l i t i c a l c o n t e x t s .  Evaluators  to t h e i r  local  reflect:  By i n v o l v i n g the c l i e n t group as c o n s u l t a n t s throughout the e v a l u a t i o n p r o c e s s , the r e s u l t i n g i n f o r m a t i o n was h i g h l y s p e c i f i c t o the c l i e n t ' s needs and, t h e r e f o r e , i n c r e a s e d the l i k e l i h o o d t h a t the i n f o r m a t i o n would be used. ( G i l l : 108) A d e f i n i t e s t r e n g t h o f the r e p o r t was t h a t i t was prepared f o r the e n t i r e membership o f the p r o v i n c e [ r e l i g i o u s group], a f t e r 90 p e r c e n t o f the membership had responded t o the s e n s i t i v e inquiry. Candid and r e l e v a n t i n f o r m a t i o n was conveyed w i t h the sense t h a t i t would have impact on everyone concerned. (Faase: 80) R e p r e s e n t a t i v e membership, w i t h a broad spectrum o f d i f f e r e n t v i e w p o i n t s c h a r a c t e r i s t i c o f the v a r i o u s s t a k e h o l d e r groups, i s l i k e l y t o p r o v i d e b e t t e r i n f o r m a t i o n and p o l i t i c a l acceptance. (Ayers: 270) [Stakeholder] recommendations s u b s t a n t i a l l y improved the q u a l i t y o f the i n f o r m a t i o n and s u c c e s s f u l l y a n t i c i p a t e d q u e s t i o n s t h a t might have undermined the r e c e p t i o n o f the e v a l u a t i o n . ( A l l e n : 50) We c o u l d have done some e x t e r n a l s t u d i e s o f our own choosing, which the s t a f f would have had l e s s of a v e s t e d i n t e r e s t i n , and the e v a l u a t i o n r e s u l t s might not have been so r e l e v a n t o r r e a d i l y accepted a t the end. ( A l k i n : 22)  99 Evaluators  a l s o r e v e a l the importance of the  r e l a t i o n s h i p s which developed between r e s e a r c h e r to u t i l i z a t i o n of  close and  stakeholders  results:  Tom's involvement i n the New Orleans meeting was of s i g n i f i c a n t importance w i t h r e s p e c t to f u t u r e impact because g e t t i n g to know us i n an i n f o r m a l way added to the c r e d i b i l i t y t h a t each of the members of the e v a l u a t i o n team had as i n d i v i d u a l s - - t h e y were r e a l people....He was g e t t i n g e x c i t e d about h e a r i n g what the m i n i s t e r s of a g r i c u l t u r e were going to say about the p r o j e c t . . . . A s he looked a t the q u e s t i o n s , I c o u l d see t h a t he shared my i n t e r e s t i n knowing what was going to happen. ( A l k i n : 27) The c l o s e c o l l a b o r a t i o n between c l i e n t and e v a l u a t o r had the e f f e c t of e l i c i t i n g s t r o n g a d m i n i s t r a t i v e support from the l e a d e r s of the p r o v i n c e f o r every phase of the research....Furthermore, something e n l i v e n i n g and g r a t i f y i n g i n the "chemistry" that developed between c l i e n t and e v a l u a t o r f u e l l e d a p a s s i o n f o r the success of the p r o j e c t , which l e f t i t s mark on the f i n a l w r i t t e n r e p o r t and on the r e p o r t to the chapter. (Faase: 78/81) Evaluators  r e v e a l that the b e n e f i t of g r e a t e r understanding of  the program, of the e v a l u a t i o n , stakeholders  was  and  of i t s r e s u l t s f o r  found to impact use.  Greene r e v e a l s :  These s u b s t a n t i v e d i s c u s s i o n s c o n t r i b u t e d to ongoing datac o l l e c t i o n d e c i s i o n s ; they a l s o represented stakeholder's e v o l v i n g understanding of key program concerns and p o s s i b l e ways to address them. (Greene/b: 94) Members of t h i s committee, along w i t h many other stakeholders both i n and o u t s i d e the agency, observed t h a t the e v a l u a t i o n r e s u l t s h e l p e d to i n c r e a s e program understanding, to c o n f i r m and document i n t u i t i o n s about program e f f e c t s , and to v a l i d a t e important d i r e c t i o n s f o r program development. (Greene/b: 99) S t a k e h o l d e r s r e p o r t e d two major b e n e f i t s of t h e i r p a r t i c i p a t i o n i n the e v a l u a t i o n p r o c e s s : (a) l e a r n i n g more about the program and the agency, and (b) l e a r n i n g more about and d e v e l o p i n g more f a v o r a b l e a t t i t u d e s toward e v a l u a t i o n . (Greene/b: 110/111) Immediate access to the data produced by the e v a l u a t i o n p r o v i d e momentum f o r u t i l i z a t i o n .  Evaluators  report:  helped  100 One s t a f f s t a k e h o l d e r observed, "We r e a l l y l e a r n e d a l o t about p u t t i n g t o g e t h e r an e v a l u a t i o n , and a t every s t e p of the p r o c e s s we got new i n s i g h t s i n t o d i r e c t i o n s the program might take." (Greene/b: 111) Tom knowing what was going to be asked and g e t t i n g e x c i t e d about i t . . . s e t up the u t i l i z a t i o n t h a t would occur out of immediate feedback to our f i e l d s t a f f . We were anxious to get e v a l u a t i o n answers i n t h e i r i n f o r m a l , c a s u a l content, as soon as you gathered the data, w h i l e i t was f r e s h and Tom was s e t up and prepared, as I was, to hear the answers to those q u e s t i o n s and to b e g i n a c t i n g on them r i g h t away. ( A l k i n : 27/28) The s t a f f made d e c i s i o n s about m o d i f y i n g and changing programs based upon the i n i t i a l data t h a t were p r e s e n t e d . These d e c i s i o n s were made a l l along, so t h a t the a c t u a l e v a l u a t i o n p r o c e s s had c o n t i n u a l impact. ( A l k i n : 28) The presence a l s o p r o p e l use. involvement  of i n f l u e n t i a l s t a k e h o l d e r s i n the e v a l u a t i o n can Not o n l y i s t h i s impacted  e a r l y i n the p r o c e s s ,  by s o l i c i t i n g  t h e i r a c t u a l presence  endorsement of the e v a l u a t i o n can h e l p s t i m u l a t e use by  their  i n and/or other  stakeholders. I n v e s t i g a t o r s met w i t h s e v e r a l s t a t e s e n a t o r s , r e p r e s e n t a t i v e s , and t h e i r s t a f f to get t h e i r p e r s p e c t i v e s on what c o n s t i t u t e d a p o l i c y - r e l e v a n t r e s e a r c h r e p o r t and hear any concerns they had about the p r o p o s a l . As a r e s u l t of these c o n t a c t s , we had p o s i t i v e working r e l a t i o n s h i p s throughout the study...and when LBC r e c e i v e d formal responses to the r e p o r t , these were v e r y c o n s t r u c t i v e . ( A l l e n : 50) [Stakeholder group's] p a r t i c i p a t i o n as an a d v i s o r y group to the p r o j e c t was not as a primary user, but r a t h e r as a s t r o n g and i n t e r e s t e d c o n s t i t u e n c y (a p o l i t i c a l s t a k e h o l d e r , i f you will). Thus, [ t h e i r ] p a r t i c i p a t i o n helped to c o n v i n c e . . . ( t h e funder) t h a t the process should be taken s e r i o u s l y and the e v a l u a t i o n r e s u l t s used. ( A l k i n : 28) By i n v o l v i n g the more powerful board members and funders, p a r t i c i p a t o r y element a l s o " s t i r r e d up i n t e r e s t i n " and generated a t t e n t i o n to the program and the i s s u e s b e i n g i n v e s t i g a t e d . (Greene/b: 110)  this  101 Summary  This  study was  guided by  the g e n e r a l purpose of e x p l o r i n g  the  nature of the p a r t i c i p a t o r y dynamics which enable e f f e c t i v e stakeholder p a r t i c i p a t i o n . participatory operational of r e s e a r c h e r and  This exploration  considered  mechanisms, the r o l e s and  s t a k e h o l d e r , and  relationships  the c o n t e x t u a l f a c t o r s which  b e n e f i t or l i m i t p a r t i c i p a t i o n . The  analysis  reveals  that  the p a r t i c i p a t o r y p r o c e s s should  r e i n f o r c e the genuineness of s t a k e h o l d e r p a r t i c i p a t i o n ,  that  p a r t i c i p a t i o n i s enabled by a t t e n t i o n  process  and  group p r o c e s s needs, and  Data r e v e a l ,  evaluation  that s u c c e s s f u l  c u l t i v a t e s a sense of ownership, and readiness for  to the  participation  that p a r t i c i p a t i o n c r e a t e s a  use. first,  t h a t the genuineness of  stakeholder  p a r t i c i p a t i o n i s l e g i t i m i z e d through i n t e r a c t i v e communication, i t e r a t i v e feedback, v i s i b l e r e s u l t s , and Communication between r e s e a r c h e r and  shared d e c i s i o n making.  stakeholder i s e s s e n t i a l i n  establishing a collaborative relationship. and  frequent, p r o v i d e equal and  I t must be  open access to i n f o r m a t i o n ,  r e s p o n s i v e to i n d i v i d u a l communication s t y l e s , and s t a k e h o l d e r as a p a r t n e r i n c o l l a b o r a t i o n . reveals  the  The  Malmburg, 1986;  i n evaluation  Donmoyer, 1990;  research  Tovar, 1989).  treat  be  the  literature  importance of ongoing communication and  for stakeholder p a r t i c i p a n t s  continual  equal access (Deutsch &  102 P e r s o n a l communication i s p r e f e r r e d by s t a k e h o l d e r s . Communication v e h i c l e s f a c i l i t a t e m u l t i p l e and o p p o r t u n i t i e s f o r s t a k e h o l d e r feedback.  repetitive  An i t e r a t i v e process of  feedback and i n t e r a c t i o n conveys r e s p e c t f o r s t a k e h o l d e r c o n t r i b u t i o n s and a l l o w s them to be i n v o l v e d i n m u l t i p l e f a c e t s of the e v a l u a t i o n p r o c e s s .  Stakeholders p r o v i d e d i r e c t i o n  c o n t r i b u t e v a l u a b l e i n f o r m a t i o n r e l e v a n t to l o c a l through  and  contexts  i t e r a t i v e feedback mechanisms, thereby enhancing  the  c r e d i b i l i t y of the e v a l u a t i o n and s t r e n g t h e n i n g e v a l u a t i o n t o o l s . The v a l u e of s t a k e h o l d e r knowledge of the l o c a l c o n d i t i o n s and o r g a n i z a t i o n a l contexts f o r p a r t i c i p a t o r y evaluations i s underscored  i n the l i t e r a t u r e  (Tovar, 1989;  Wagner, 1991;  Owston,  1986) . The genuineness of p a r t i c i p a t i o n i s r e i n f o r c e d through t a n g i b l e and v i s i b l e r e s u l t s of s t a k e h o l d e r p a r t i c i p a t i o n i n the e v a l u a t i o n and through evaluation issues.  shared d e c i s i o n making i n s u b s t a n t i v e  Although  a v a r i e t y of d e c i s i o n making models  were r e p r e s e n t e d and s t a k e h o l d e r s ' d e c i s i o n making r e s p o n s i b i l i t i e s v a r i e d among s t u d i e s , the data r e v e a l t h a t e v a l u a t o r s must be w i l l i n g to g e n u i n e l y a c t on the feedback and d i r e c t i o n p r o v i d e d by s t a k e h o l d e r s .  As Greene (1988a) i n d i c a t e s ,  " s t a k e h o l d e r a c t i o n not j u s t r e a c t i o n was  sought"  (p. 109).  This  f a c t o r i s r e i n f o r c e d i n the l i t e r a t u r e , which recommends openended, two-way d i a l o g u e w i t h s t a k e h o l d e r s i n which e v a l u a t i o n f i n d i n g s are e x p l o r e d and d i s c u s s e d , not j u s t p r e s e n t e d 1986;  Greene, 1988b).  Maclure  (Patton,  (1990) and Carey & Smith (1992)  103 c o n f i r m t h a t systems of g e n u i n e l y shared c o n t r o l should be i n place f o r e f f e c t i v e stakeholder p a r t i c i p a t i o n . Because of the suggested  l i n k between decision-making  a c t i o n , p a r t i c i p a t i o n t h a t does not l e a d to a c t i o n may p a r t i c i p a t i o n t h a t i s not a u t h e n t i c .  and  result in  T h e r e f o r e , d e c i s i o n s and  a c t i o n s are a d i r e c t r e s u l t or consequence of the genuineness of participation. Although  i t i s not necessary  to have each of these f o u r  components p r e s e n t i n a study i n order f o r p a r t i c i p a t i o n to occur, the p a r t i c i p a t i o n of s t a k e h o l d e r s i s l e g i t i m i z e d by  the  i n c l u s i o n of each of these f a c e t s i n t o the e v a l u a t i o n p r o c e s s . Second, the data demonstrate t h a t the p a r t i c i p a t o r y process i s enabled by the r o l e of r e s e a r c h e r as f a c i l i t a t o r of e v a l u a t i o n and of the group p r o c e s s .  the  F a c i l i t a t i n g the e v a l u a t i o n  r e q u i r e s a t t e n t i o n to group tasks and the e v a l u a t i o n needs of s t a k e h o l d e r s or, as Faase put i t , Patton  "procedural c o u n s e l l i n g " .  (1987) v e r i f i e s the c r i t i c a l r o l e of e v a l u a t o r as  facilitator.  T h i s r o l e r e q u i r e s such tasks as s y n t h e s i z i n g  i n f o r m a t i o n and p r e s e n t i n g o p t i o n s . enables  T h i s p r e p a r a t o r y work  s t a k e h o l d e r p a r t i c i p a t i o n by making the e v a l u a t i o n more  manageable and more meaningful.  C a r e f u l and thorough p l a n n i n g  and p r e p a r a t i o n i s promoted by Malekoff  (1994) as e s s e n t i a l to a  successful p a r t i c i p a t o r y process. F a c i l i t a t i n g stakeholder p a r t i c i p a t i o n requires c l e a r goals and g u i d e l i n e s f o r p a r t i c i p a t i o n and a v o l u n t a r y and open process.  T h i s f i n d i n g i s supported by Rudd & A s s o c i a t e s (1993)  104 who  found  t h a t p a r t i c i p a t i o n was  l i m i t e d when the mechanisms and  o b j e c t i v e s f o r c o l l a b o r a t i o n were not d e f i n e d or a r t i c u l a t e d e a r l y i n the p r o c e s s .  Researchers  have r e c o g n i z e d the need f o r  c l e a r l y d e f i n e d purposes and methods of s t a k e h o l d e r p r i o r to embarking on p a r t i c i p a t o r y approaches 1985;  involvement  (Mark &  Shotland,  Weiss, 1983b).  A t t e n t i o n to group process b u i l d s t r u s t and group c r e d i b i l i t y . The  importance of a t t e n t i o n to group process  Patton  i s recognized  by  (1987) as a p r e r e q u i s i t e to e x p e r i e n c i n g mutual r e s p e c t  and i n t e g r i t y w i t h i n e v a l u a t i o n . understanding  B u i l d i n g r a p p o r t and mutual  enables p a r t i c i p a n t s to experience  a l e v e l of  mutual r e s p e c t t h a t i s o f t e n l o s t i n more formal or r i g i d r o l e s (Malekoff, 1994) .  Rudd & A s s o c i a t e s  a t t e n t i o n to group process  (1993) found  that "  i s e s s e n t i a l f o r the establishment  a t r u e c o l l e g i a l atmosphere"  (p. 250).  The use of a t r a i n e d  f a c i l i t a t o r to f a c i l i t a t e s t a k e h o l d e r group decision-making supported The  by Greene  of  is  (1988b).  r o l e of n u r t u r e r i n the p e r s o n a l development of  s t a k e h o l d e r s was  seen as e s p e c i a l l y important  i n an e v a l u a t i o n  conducted w i t h b e n e f i c i a r i e s and o p e r a t i n g w i t h i n an empowerment rationale.  G u t i e r r e z and Ortega  (1991) a f f i r m the v i t a l r o l e of  the e v a l u a t o r i n f a c i l i t a t i n g i n t r a - g r o u p i n t e r a c t i o n and cohesion w i t h b e n e f i c i a r i e s .  Donmoyer (1990) r e p o r t s the  importance of p r o v i d i n g support i n p u t and r e v e a l s how beneficiaries.  t h i s was  group  and l e g i t i m a t i o n f o r b e n e f i c i a r y found  to be r e a s s u r i n g f o r  105 A t t e n t i o n t o group process  allows the e v a l u a t o r access t o  v a l u a b l e i n s i d e r i n f o r m a t i o n concerning contextual  external pressures or  f a c t o r s which c o u l d a f f e c t the success  evaluation.  A t times,  o f an  the e v a l u a t o r may need t o be f l e x i b l e and  w i l l i n g t o a d j u s t the e v a l u a t i o n a c c o r d i n g l y .  An i n t e r n a l  e v a l u a t o r o r advocate can enhance the r e l a t i o n s h i p between the e v a l u a t o r and s t a k e h o l d e r s . T h i r d , the data r e v e a l t h a t s u c c e s s f u l p a r t i c i p a t i o n c u l t i v a t e s ownership.  When p a r t i c i p a t i o n i s "hands on" and  the e v a l u a t i o n i s p e r c e i v e d by s t a k e h o l d e r s  as an " i n s i d e j o b "  and as " t a i l o r made", ownership i s f o s t e r e d . enhanced by s t a k e h o l d e r s activities.  when  Ownership i s  active p a r t i c i p a t i o n i n evaluation  These f i n d i n g s a r e c o n s i s t e n t w i t h those o f Malekoff  (1994) and Patton  (1986) who found t h a t a c t i v e  stakeholder  involvement i n p r o j e c t s s t i m u l a t e s a sense o f ownership. (1991) a f f i r m s the importance o f r e a s s u r i n g s t a k e h o l d e r s  Reineke that  e v a l u a t i o n s a r e b e i n g done w i t h them and f o r them, and Tovar (1989) r e v e a l s how s t a k e h o l d e r p e r c e p t i o n s o f e v a l u a t i o n  evolved  "from something 'others do t o them' to something they can h e l p create"  (p. 5 5 ) .  E v a l u a t i o n which i s p e r c e i v e d as an " i n s i d e j o b " can both b e n e f i t o r h i n d e r c o l l a b o r a t i v e e f f o r t s , depending on whether o r not a p a r t n e r s h i p i s p e r c e i v e d between the r e s e a r c h e r and stakeholders.  T a i l o r made e v a l u a t i o n s a r e r e s p o n s i v e  needs and do n o t impose p r e c o n c e i v e d from o t h e r s e t t i n g s without  designs  to l o c a l  o r methodologies  c o n s i d e r a b l e adjustments.  The  106 l i t e r a t u r e concurs  t h a t r e f l e c t i n g l o c a l c o n t e x t s i n c r e a s e s the  r e l e v a n c y of the e v a l u a t i o n and  the support  i t receives  (Tovar,  1989) . Finally,  the data r e v e a l t h a t the p a r t i c i p a t o r y  creates ready-for-use  process  c o n d i t i o n s among s t a k e h o l d e r s .  There seems  to be a n a t u r a l p r o g r e s s i o n from a c t i v e e v a l u a t i o n p a r t i c i p a t i o n to f o l l o w through f o r s t a k e h o l d e r s .  T h i s i s a m p l i f i e d by  their  ongoing awareness of the e v a l u a t i o n , access to i n f o r m a t i o n , a n t i c i p a t i o n of e v a l u a t i o n r e s u l t s , and r e l a t i o n s h i p s which may stakeholder.  through the c l o s e  develop between r e s e a r c h e r  Researchers  and  c o n f i r m t h a t an a c c e l e r a t i o n of  e v a l u a t i o n use i s l i n k e d w i t h : a c t i v e p a r t i c i p a t i o n  (Dawson &  D'Amico, 1985), access to and r e l e v a n c y of i n f o r m a t i o n  (Kennedy,  1984;  (Tovar,  L e v i t o n & Hughes, 1981), a n t i c i p a t i o n of r e s u l t s  1989), d i r e c t and ongoing communication between r e s e a r c h e r s users  (Rudd & A s s o c i a t e s , 1993;  L e v i t o n & Hughes, 1981), and  i n t e n s i v e i n t e r a c t i o n between r e s e a r c h e r and (Huberman, 1990;  Trochim & L i n t o n , 1986).  Stakeholder the  (Greene, 1988b).  E v a l u a t i o n u t i l i z a t i o n was  a l s o p r o p e l l e d by an i n c r e a s e i n  program knowledge by p a r t i c i p a n t s , by s t a k e h o l d e r ' s access to the data, and by the involvement powerful  the  stakeholders  p a r t i c i p a t i o n i n e v a l u a t i o n i s l i n k e d to documented uses of evaluation  and  stakeholders.  The  immediate  of i n f l u e n t i a l  or  l i t e r a t u r e reveals that evaluations  can y i e l d a g r e a t e r understanding and c r e a t e a focus on process,  of an o r g a n i z a t i o n or program  thereby e n a b l i n g s t a k e h o l d e r s  to  107 t h i n k more about the meanings of f i n d i n g s and e v a l u a t i o n utilization  (Huberman, 1993; Trochim & L i n t o n ,  B a r k d o l l , 1983) .  Some authors conclude that the ongoing  and a s s i m i l a t i o n of i n f o r m a t i o n (Cronbach & A s s o c i a t e s , 1986) .  1986; Tovar, 1989; learning  i n and of i t s e l f c o n s t i t u t e s use  1980; Greene, 1988b; Cousins & Leithwood,  108 CHAPTER 4  IMPLICATIONS FOR PRACTICE AND CONCLUSIONS  Implications f o r Evaluation Practice  The  r e s u l t s from t h i s study have important  s t a k e h o l d e r involvement  i n evaluation practice.  s e l e c t e d i m p l i c a t i o n s a r e addressed.  implications for A number o f  As w i t h chapter  three,  r e f e r e n c e s t o the data o f t h i s study a r e i n d i c a t e d by the f i r s t name o n l y o f the authors.  A Strategy f o r Partnership  The data c l e a r l y i l l u s t r a t e t h a t i f s t a k e h o l d e r p a r t i c i p a t i o n i s taken s e r i o u s l y by e v a l u a t o r s , s t a k e h o l d e r s must be i n v o l v e d as t r u e p a r t n e r s i n c o l l a b o r a t i o n . the f o u r elements p r e s e n t e d :  Implementation o f  ongoing communication,  iteratively  s t r u c t u r e d feedback, v i s i b l e r e s u l t s , and shared d e c i s i o n making, r e i n f o r c e the commitment o f e v a l u a t o r s t o t r e a t s t a k e h o l d e r s as genuine p a r t i c i p a n t s i n the e v a l u a t i o n p r o c e s s . are motivated  E v a l u a t o r s who  t o p a r t i c i p a t e and who i n c l u d e s t a k e h o l d e r s as  p a r t n e r s a r e more l i k e l y t o experience (Cousins & E a r l ,  1992).  successful collaboration  Successful p a r t i c i p a t i o n i s l i n k e d to  s t a k e h o l d e r ownership i n the data.  109 Two o f the s t u d i e s r e v e a l d i f f i c u l t i e s t h a t can a r i s e when one o r more o f these elements i s m i s s i n g .  In Crow and i n  Brandon, s t a k e h o l d e r s were i n v o l v e d i n v e r y s t r u c t u r e d activities.  R e v i s i t i n g t h i s f i r s t - t i m e attempt a t c o l l a b o r a t i o n ,  Crow r e g r e t s missed  o p p o r t u n i t i e s w i t h s t a k e h o l d e r s due t o  u n c l e a r g o a l s f o r c o l l a b o r a t i o n , n o n c o l l a b o r a t i v e communication, and,  p o s s i b l y , lack of i n i t i a l  Although  involvement  w i t h the study.  t h i s l a t t e r element a l s o a f f e c t e d A l l e n somewhat,  s t a k e h o l d e r p a r t i c i p a t i o n w i t h i n A l l e n ' s study was much more s u b s t a n t i v e and appeared t o compensate f o r the l a c k o f i n i t i a l input by s t a k e h o l d e r s . In Brandon's study, although s t a k e h o l d e r p a r t i c i p a t i o n was shown t o i n c r e a s e v a l i d i t y , of  s t a k e h o l d e r ownership.  e v a l u a t o r s d i d not a l l u d e t o a sense U n l i k e the o t h e r s t u d i e s , s t a k e h o l d e r  p a r t i c i p a t i o n was not ongoing.  Stakeholders were i n v i t e d t o  p a r t i c i p a t e on a one-time b a s i s a t c e r t a i n j u n c t u r e s i n the e v a l u a t i o n p r o c e s s and, w i t h i n these a c t i v i t i e s , p a r t i c i p a t i o n was v e r y s t r u c t u r e d .  their  To Brandon's c r e d i t , however,  some o f t h i s s t r u c t u r e was a d e l i b e r a t e attempt t o ensure e q u i t a b l e p a r t i c i p a t i o n through facilitator,  the use o f an expert group  s p e c i a l a t t e n t i o n to group p r o c e s s , and the format  and o r g a n i z a t i o n o f i n f o r m a t i o n presented. i n c o r p o r a t e d each element necessary  Although  the study  t o r e i n f o r c e genuine  p a r t i c i p a t i o n w i t h i n each j u n c t u r e , the l a c k o f ongoing p a r t i c i p a t i o n among s t a k e h o l d e r s d i d not seem t o c r e a t e a sense of  ownership by s t a k e h o l d e r s as i n o t h e r s t u d i e s .  110 An important may  i n f e r e n c e t h a t can be drawn i s t h a t continuity-  be a s i g n i f i c a n t f a c t o r i n l e g i t i m i z i n g s t a k e h o l d e r  participation.  E f f o r t s to s t r u c t u r e a c t i v i t i e s e q u i t a b l y d i d not  seem to have the same e f f e c t f o r p a r t i c i p a n t s as p a r t i c i p a t i o n had i n o t h e r s t u d i e s .  ongoing  Bunker (1978) suggests t h a t  the " c o n t i n u i t y of r e l a t i o n s h i p s permit the e s t a b l i s h m e n t of f a m i l i a r i t y and mutual t r u s t necessary to o v e r r i d e the adversary r e l a t i o n s h i p and d e f e n s i v e n e s s which so o f t e n b l o c k mutual i n f l u e n c e and l e a r n i n g " involvement  (p. 132).  be continuous,  Not o n l y must s t a k e h o l d e r  the elements of r e i n f o r c i n g a u t h e n t i c  p a r t i c i p a t i o n must operate c o n t i n u o u s l y and throughout  iteratively  the e v a l u a t i o n p r o c e s s .  Organizational  Support  A commitment to the involvement  of m u l t i p l e s t a k e h o l d e r s  a l s o r e q u i r e s u n e q u i v o c a l support on the p a r t of agencies h o s t i n g evaluations.  With g r e a t e r acceptance  and occurrence of  stakeholder p a r t i c i p a t i o n i n evaluation research, organizations p l a n n i n g or h o s t i n g e v a l u a t i o n s need to be c o g n i z a n t of the a d d i t i o n a l needs of p a r t i c i p a n t s t a k e h o l d e r s .  Agencies must not  o n l y p r o v i d e the time and r e s o u r c e s r e q u i r e d , but they must a t t e n d t o the a d d i t i o n a l demands on s t a f f time, i n p a r t i c u l a r . It i s v i t a l  t h a t they s u f f i c i e n t l y f r e e s t a f f from r o u t i n e tasks  i n o r d e r f o r them to p a r t i c i p a t e m e a n i n g f u l l y i n the e v a l u a t i o n process  (Cousins & E a r l y , 1992).  The data r e v e a l t h a t  Ill s t a k e h o l d e r s were " a p p r e c i a t i v e of any  tangible assistance  r e c e i v e d from the a d m i n i s t r a t i o n i n the form of e x t r a h e l p w i t h t h e i r r e g u l a r job d u t i e s which allowed e v a l u a t i o n study"  (Ayers: 270).  them adequate time f o r the  C o n s i d e r a t i o n of these f a c t o r s  w i l l impact meaningful p a r t i c i p a t i o n f o r s t a k e h o l d e r s . O r g a n i z a t i o n s may  a l s o be r e q u i r e d to advocate on b e h a l f of  s t a k e h o l d e r s from other o r g a n i z a t i o n s or agencies the importance of the study and stakeholder p a r t i c i p a t i o n .  the fundamental need f o r m u l t i p l e  T a n g i b l e a s s i s t a n c e p r o v i d e d by  h o s t i n g o r g a n i z a t i o n to t h e i r own hallmark  to promulgate  the  s t a k e h o l d e r s would serve as a  to o t h e r o r g a n i z a t i o n s and r e v e a l t h e i r commitment to  the e v a l u a t i o n .  Tovar  (1989) r e v e a l s t h a t i n s t i t u t i o n a l  f o r e v a l u a t i o n s which i s o n l y p a r t i a l can l i m i t the  support  involvement  of key s t a k e h o l d e r s and reduce the e f f e c t i v e n e s s of e v a l u a t i o n s .  The Role of the Expert  Evaluator  An i n t r i g u i n g by-product of the a u t h e n t i c p a r t i c i p a t o r y process  i s t h a t even a c t i v i t i e s completed by the r e s e a r c h e r  p e r c e i v e d to be  "owned" by the s t a k e h o l d e r s .  are  Ongoing  communication and e a r l y p a r t i c i p a t i o n l a y s the groundwork f o r t h i s outcome by i n f o r m i n g s t a k e h o l d e r s of the e v a l u a t i o n as i t proceeds and e n s u r i n g t h a t the e v a l u a t i o n i s r e l e v a n t and c r e d i b l e . Once the p a r t i c i p a t o r y elements are entrenched,  it  appears t h a t r e s e a r c h e r s can choose to be i n v o l v e d more e x c l u s i v e l y i n the t e c h n i c a l r e s e a r c h tasks such as the c r e a t i o n  112 of data c o l l e c t i o n instruments, data c o l l e c t i o n , o r data a n a l y s i s without compromising genuine  collaboration.  C o l l a b o r a t i o n and shared d e c i s i o n making do n o t n e c e s s a r i l y imply t h a t s t a k e h o l d e r s have t o be i n v o l v e d t a n g i b l y i n each and every phase o f the e v a l u a t i o n .  Genuine p a r t n e r s h i p can occur  w i t h shared t a s k s and shared e x p e r t i s e and s t i l l d e f i n i t e s t a k e h o l d e r ownership.  result i n  Although the data i n d i c a t e  that  s t a k e h o l d e r s need t o be i n v o l v e d s u b s t a n t i v e l y , the e x p e r t i s e and l e a d e r s h i p r o l e p r o v i d e d by e v a l u a t o r s was shown t o be important to the e v a l u a t i o n process and was even a p p r e c i a t e d by stakeholders.  As s t a k e h o l d e r s i n d i c a t e d , they d i d n o t always  have the time, t r a i n i n g , o r d e s i r e t o be i n v o l v e d i n some o f the " n i t t y g r i t t y " work.  The data r e v e a l t h a t these v a r i o u s tasks do  not have t o be r e s t r i c t e d t o r e s e a r c h e r s , however, i t may be r e a s s u r i n g f o r some r e s e a r c h e r s t o know that they can be r e s p o n s i b l e f o r such tasks without h i n d e r i n g the p a r t i c i p a t i o n process.  T h i s may be e s p e c i a l l y important f o r s t u d i e s i n which  technical i n t e g r i t y i s e s s e n t i a l or desired.  Group Process  Routledge  Skills  (1993) i n d i c a t e s that one o f the most o v e r l o o k e d  aspects i n p a r t i c i p a t o r y approaches  i s the "importance o f  f a c i l i t a t o r s h a v i n g a t h e o r e t i c a l understanding o f groupwork p r o c e s s and well-honed p r a c t i c a l s k i l l s i n working w i t h  groups"  (p. 107). The data r e v e a l s the v i t a l r o l e o f e v a l u a t o r s as  113 f a c i l i t a t o r s o f the group p r o c e s s i n working w i t h  stakeholders.  Group p r o c e s s s k i l l s a r e e s s e n t i a l t o i n s u r i n g adequate communication among p a r t i c i p a n t s .  Donmoyer (1990) r e p o r t s  that  the q u a l i t y o f group p r o c e s s e s and who p a r t i c i p a t e s a r e key f a c t o r s i n s u c c e s s f u l c o l l a b o r a t i v e approaches, s t a t i n g t h a t "an evaluator's  group p r o c e s s s k i l l s a r e c l e a r l y as important as h i s  or her t e c h n i c a l e x p e r t i s e "  (p. 277).  I t i s l i k e l y that many r e s e a r c h e r s  have o n l y minimal  t r a i n i n g i n the area o f group p r o c e s s and f a c i l i t a t i o n .  This  would appear t o be an important c o n s i d e r a t i o n f o r e v a l u a t o r s wishing to incorporate  a p a r t i c i p a t o r y approach t o e v a l u a t i o n .  Although some e v a l u a t o r s may be s k i l l e d communicators and l i s t e n e r s , group f a c i l i t a t i o n may r e q u i r e a d d i t i o n a l s k i l l s and s t r a t e g i e s e s p e c i a l l y when c o n s i d e r i n g d i f f e r e n t l e a r n i n g s t y l e s or when working w i t h stakeholders  w i t h v a r i e d o r unique needs,  such as b e n e f i c i a r i e s . Church and R e v i l l e (1990) r e v e a l  that  beneficiaries: who a r e b e g i n n i n g t o speak out o f t e n have no exposure t o the r u l e s f o r speech and b e h a v i o r which c h a r a c t e r i z e p r o f e s s i o n a l / a g e n c y m e e t i n g s . . . . F a c i l i t a t i n g the p a r t i c i p a t i o n o f people who do not u s u a l l y speak out r e q u i r e s someone no more d e f e r e n t i a l t o p r o f e s s i o n a l s and f a m i l y members than t o s e r v i c e r e c i p i e n t s , (p. 80) I n s u f f i c i e n t researcher  s k i l l s may h i n d e r  the c o l l a b o r a t i o n  p r o c e s s by l i m i t i n g the e v a l u a t i o n p r o c e s s and the meaningfulness of p a r t i c i p a t i o n f o r s t a k e h o l d e r s , and  thereby r i s k i n g the v a l i d i t y  r e l i a b i l i t y o f the e v a l u a t i o n r e s u l t s .  Evaluators  may wish  to b r i n g i n t r a i n e d group f a c i l i t a t o r s to generate e f f e c t i v e and e f f i c i e n t d i s c u s s i o n and feedback, e s p e c i a l l y i n t h e i r  first  114 attempts w i t h s t a k e h o l d e r c o l l a b o r a t i o n . Greene (1988b) r e g r e t s not h i r i n g an experienced d e c i s i o n making.  p r o f e s s i o n a l t o f a c i l i t a t e group  In the f u t u r e , i t may n o t be uncommon t o f i n d  e v a l u a t o r s teaming up w i t h group f a c i l i t a t o r s t o conduct p a r t i c i p a t o r y approach e v a l u a t i o n s . i n c o r p o r a t e group process  I d e a l l y , educators  should  s k i l l s training into educational  programs f o r r e s e a r c h e r s and e v a l u a t o r s who wish t o pursue p a r t i c i p a t o r y approaches.  Responsiveness o f Communication  Making the e v a l u a t i o n i n f o r m a t i o n r e s p o n s i v e t o s t a k e h o l d e r s ' needs may r e s u l t i n e v a l u a t o r s t a k i n g on new r o l e s and l e a r n i n g i n n o v a t i v e communication s k i l l s .  Although  many  r e s e a r c h e r s c o n s i d e r these f a c t o r s i n the d i s s e m i n a t i o n o f r e s u l t s phase t o i n c r e a s e u t i l i z a t i o n  (Patton, 1987; Greene,  1988b; Weiss, 1988b; Cronbach e t a l . , 1980), the data r e v e a l s that e v a l u a t o r s need t o promote responsiveness  i n t h e i r ongoing  communication w i t h s t a k e h o l d e r s throughout the e v a l u a t i o n process. The data d i s c u s s the use o f l a y terms and n o n t e c h n i c a l language i n d r a f t s and progress  r e p o r t s t o s t a k e h o l d e r s and the  need t o be s e n s i t i v e t o v a r i o u s l e a r n i n g and communication s t y l e s of s t a k e h o l d e r s .  Greene (1988b) addresses  this topic further i n  a l a t e r d i s c u s s i o n o f h e r two s t a k e h o l d e r s t u d i e s .  She r e v e a l s  that g e n e r a t i n g an engaging n a r r a t i v e o r s t o r y t o p r e s e n t  115 i n f o r m a t i o n i s much more e a s i l y understood t a b l e s or graphs.  In a s i t u a t i o n where d e t a i l e d and  t a b l e s and graphs were presented  Reineke  than  colour-coded  to s t a k e h o l d e r s , she r e p o r t s  that the r e s u l t i n g d i s c u s s i o n was illuminating.  by s t a k e h o l d e r s  d i s j o i n t e d and not v e r y  (1991) d i s c l o s e s t h a t communication based  i n content f a m i l i a r to s t a k e h o l d e r s can h e l p to keep d i a l o g u e s focused.  Case s t u d i e s and o r a l r e p o r t s are more r e a d i l y  comprehended by s t a k e h o l d e r s  ( L i n c o l n , 1990) .  The  i d e a of u s i n g  n a r r a t i v e s i s f u r t h e r s a n c t i o n e d by Cronbach & A s s o c i a t e s (1980) who  encourage the use of a n e c d o t a l s t o r i e s to p r e s e n t data  and  results. I t would seem t h a t t h i s process r e q u i r e s c r e a t i v e e v a l u a t o r s who  are s e n s i t i v e to s t a k e h o l d e r l e a r n i n g needs.  of p a r t i c i p a t o r y approaches, Cousins & E a r l  In t h e i r  (1992) conclude  review that  e v a l u a t o r s "must be s e n s i t i v e to p r i n c i p l e s of a d u l t l e a r n i n g and ought to have the a p p r o p r i a t e i n t e r p e r s o n a l and skills"  (p. 413).  Such responsiveness may  communication  r e q u i r e the  i n c o r p o r a t i o n of an a r r a y of communication mediums such as p o s t e r - b o a r d p r e s e n t a t i o n s , s l i d e s , c h a r t s , and  props,  figures.  I n c o r p o r a t i o n of v a r i e d communication s t r a t e g i e s would serve to i n c r e a s e s t a k e h o l d e r understanding  and p o s s i b l y h e i g h t e n  their  i n t e r e s t i n the e v a l u a t i o n (Greene, 1988b). Researchers  who  work i n academic s e t t i n g s may  need to pay  s p e c i a l a t t e n t i o n to i n d i v i d u a l l e a r n i n g s t y l e s i n t h e i r communication and d i a l o g u e w i t h nonacademic s t a k e h o l d e r groups. T h i s may  i n v o l v e a d d i t i o n a l t r a i n i n g f o r some e v a l u a t o r s and/or  116 c o n s u l t a t i o n p r i o r to or d u r i n g e v a l u a t i o n s . s u b j e c t c o u l d be addressed  Ideally,  this  i n academic s e t t i n g s so t h a t  e v a l u a t i o n students would be a b l e to i d e n t i f y d i f f e r e n t l e a r n i n g s t y l e s and be  t r a i n e d to respond to these needs p r i o r to engaging  in evaluation practice. Lincoln responsiveness  (1990) o f f e r s a somewhat elementary s o l u t i o n to the of r e s u l t s d i s s e m i n a t i o n which c o u l d be used by  e v a l u a t o r s i n a l l f a c e t s of the p a r t i c i p a t o r y p r o c e s s : i t would be u s e f u l i f we would ask s t a k e h o l d e r s how they would l i k e to use f i n d i t most u s a b l e "  "Perhaps  they t h i n k  i n f o r m a t i o n and i n what form they would  (p. 3 ) .  L i m i t a t i o n s of Stakeholder  E x t e n s i v e P r e p a r a t i o n and  Participation i n Evaluation  Planning  The data r e v e a l t h a t c o l l a b o r a t i o n w i t h s t a k e h o l d e r s can an arduous task and p l a c e s heavy demands on e v a l u a t o r s . s i g n i f i c a n t of these tasks may r e q u i r e d of r e s e a r c h e r s who suggest  be the p r e p a r a t i o n and  work w i t h s t a k e h o l d e r s .  Malekoff and  (1994) has underscored  The most  planning The  t h a t the r e s e a r c h e r ' s preparedness impacts how  and manageable the p a r t i c i p a t o r y experience  be  data  meaningful  i s f o r stakeholders.  the importance of c a r e f u l  thorough p r e p a r a t i o n and p l a n n i n g when working w i t h m u l t i p l e  stakeholders i n evaluation.  However, not o n l y does t h i s seem  time consuming, but i t would a l s o r e q u i r e e x t e n s i v e  117 o r g a n i z a t i o n a l s k i l l s on the p a r t o f the r e s e a r c h e r .  Activities  such as p r e s e n t i n g o p t i o n s and s y n t h e s i z i n g i n f o r m a t i o n not be taken l i g h t l y .  should  They r e q u i r e a s o l i d knowledge base f o r  e v a l u a t o r s i n e v a l u a t i o n r e s e a r c h , group p r o c e s s , and v a r i e d l e a r n i n g s t y l e s , as w e l l as e x c e p t i o n a l communication and w r i t i n g skills. P l a n n i n g a l s o has i m p l i c a t i o n s f o r f a c i l i t a t i n g group p r o c e s s e s .  Kurland  effective  (1978) i d e n t i f i e s p l a n n i n g as the  n e g l e c t e d component o f group development.  She r e v e a l s t h a t when  working w i t h groups, "the p r i c e f o r l a c k o f thorough and thoughtful planning i s high"  (p. 173). Lack o f p l a n n i n g can  r e s u l t i n s p o r a d i c o r i r r e g u l a r attendance and d i s s a t i s f i e d group members who f e e l t h a t t h e i r needs have not been met.  Kurland  argues t h a t pregroup p l a n n i n g enhances o p p o r t u n i t i e s f o r s e l f determination and informed  and i n c r e a s e s the c l i e n t ' s a b i l i t y t o make c l e a r decisions regarding t h e i r p a r t i c i p a t i o n .  Planning  not o n l y e n t a i l s the f u n c t i o n a l and a d m i n i s t r a t i v e tasks surrounding  group meetings but a l s o i n c l u d e s d e c i s i o n s on group  purpose, group composition, for  and the s u i t a b i l i t y o f meeting space  s t a k e h o l d e r needs.  Collaboration with B e n e f i c i a r i e s  The r o l e o f the r e s e a r c h e r when working w i t h b e n e f i c i a r y s t a k e h o l d e r s i s ambiguous.  Both Whitmore and Brandon p a i d  c o n s i d e r a b l e a t t e n t i o n to group dynamics, each f o r d i f f e r e n t  118 reasons.  Whitmore, t o f a c i l i t a t e empowerment among  b e n e f i c i a r i e s , and Brandon t o ensure e q u i t a b l e involvement f o r beneficiaries.  I t i s unclear,  however, whether t h e i r  shared  focus i s due t o the uniqueness of t h e i r s t u d i e s o r t o a l a c k o f r e p o r t i n g by other authors o f t h i s dynamic. Both s t u d i e s , however, r e v e a l d i s t i n c t i v e c h a r a c t e r i s t i c s o f working w i t h b e n e f i c i a r y groups who a r e both poor and powerless: uneducated s i n g l e mothers on s o c i a l a s s i s t a n c e , and homeless parents.  They i d e n t i f y t h a t these s t a k e h o l d e r s  came t o the  evaluation process with a d i f f i c u l t y t r u s t i n g others,  a sense o f  i s o l a t i o n and p o s s i b l e l o n e l i n e s s , and a p r o b a b l e i n t i m i d a t i o n by formal  decision-making processes.  need f o r the e v a l u a t o r  Donmoyer (1990) confirms the  t o v a l i d a t e and l e g i t i m a t e b e n e f i c i a r y  i n p u t , e s p e c i a l l y i n e v a l u a t i o n s where a d i s p a r i t y i n s t a t u s o r power i s e v i d e n t . Although both s t u d i e s i n v o l v e d s t a k e h o l d e r s  t o enhance  v a l i d i t y and both p a i d s p e c i a l o r e x t r a a t t e n t i o n t o group dynamics, the r e s u l t s were d r a s t i c a l l y d i f f e r e n t .  Brandon took a  much more s t r u c t u r e d approach to ensure e q u i t a b l e group involvement, f o c u s i n g on s i t u a t i o n a l adjustments t o ensure equity:  h i r i n g a t r a i n e d group f a c i l i t a t o r ,  taking  secret  b a l l o t s , p r o v i d i n g equal chances t o speak, l i m i t i n g group d i s c u s s i o n , and d e c i s i o n making by p l u r a l i t y v e r s u s consensus. Although e q u i t y o f p a r t i c i p a t i o n was achieved, Brandon does not r e v e a l any a d d i t i o n a l b e n e f i t s t o any o f the s t a k e h o l d i n g with this  process.  groups  119 Whitmore's r o l e took a d i f f e r e n t t w i s t because o f her empowerment focus and because o f the c o n t i n u i t y o f p a r t i c i p a t i o n which was not p r e s e n t i n Brandon's study. if,  the age o f p a r t i c i p a n t s  I t i s u n c l e a r how, and  ( a l l young mothers) may have  her r o l e w i t h t h i s group o f b e n e f i c i a r i e s .  impacted  Whitmore focused on  b u i l d i n g p e r s o n a l r e l a t i o n s h i p s , t a k i n g on the r o l e o f n u r t u r e r and working hard t o develop group cohesiveness and t r u s t among the s t a k e h o l d e r s .  She r e p o r t s t h a t both the group p r o c e s s and  the e v a l u a t i o n p r o c e s s empowered s t a k e h o l d e r s . Whitmore's study r e v e a l s t h a t , w i t h the h e l p o f a t r a i n e d and s e n s i t i v e e v a l u a t o r , o r d i n a r y community people a r e capable o f p r o d u c i n g knowledge t h a t i s important and v a l i d .  Empowerment and  v a l i d i t y outcomes w i t h b e n e f i c i a r y p a r t i c i p a t i o n a r e supported i n the l i t e r a t u r e  (Malekoff, 1994; Wagner, 1991; Routledge, 1993;  G u t i e r r e z & Ortega, 1991).  The importance  i n d i c a t e d i n these  s t u d i e s o f i n t r a - g r o u p i n t e r a c t i o n and c r i t i c a l d i a l o g u e t o empowerment a r e c o n s i s t e n t w i t h the d a t a . One wonders whether empowerment would have r e s u l t e d i f Whitmore's s t a k e h o l d i n g group had c o n s i s t e d o f a v a r i e t y o f s t a k e h o l d e r s and i n c l u d e d program s t a f f and a d m i n i s t r a t o r s . C e r t a i n l y , Brandon's more s t e r i l e and hands-off approach t o e q u i t a b l e p a r t i c i p a t i o n among m u l t i p l e s t a k e h o l d e r s d i d not f a c i l i t a t e such r e s u l t s .  T h i s f a c t o r would seem t o have  important i m p l i c a t i o n s f o r s t u d i e s that employ an empowerment rationale.  120 There are f u r t h e r i m p l i c a t i o n s f o r e v a l u a t o r s working s o l e l y with b e n e f i c i a r i e s . perspectives  The data r e v e a l s the v a l u e  i n multiple  and i n t e r a c t i v e s h a r i n g which l e d to a g r e a t e r  awareness o f the d i v e r s i t y and complexity of a program's a c t i v i t i e s and r e s p o n s i b i l i t i e s . v a l u e o f t h i s mutual exchange, constituents with a strong, i n stakeholders  Associates  (1983) r e v e a l s the  i n d i c a t i n g that e s p e c i a l l y f o r  s i n g l e focus, p a r t i c i p a t i o n r e s u l t e d  " r e c o g n i z i n g and w i l l i n g to share the burden of  a l l o c a t i n g s c a r c e resources (p. 37).  Barkdoll  to important but competing demands"  In t h e i r d i s c u s s i o n of e v a l u a t i o n use, Cronbach & (1980) i n d i c a t e that " s t i m u l a t i n g a d i s c u s s i o n  leads to g r a d u a l  change i n p r e v a i l i n g views i s v e r y  most important e f f e c t o f e v a l u a t i o n r e s e a r c h " Decisions  that  l i k e l y the  (p. 193).  about m u l t i p l e group p a r t i c i p a t i o n may  be  e s p e c i a l l y important when working w i t h h i g h l y s e n s i t i v e o r p o l i t i c a l issues.  Because the e x c l u s i o n o f c e r t a i n  views o r p e r c e p t i o n s  stakeholder  does not f a c i l i t a t e such mutual exchange, i t  would be important to c o n s i d e r  the e f f e c t s t h i s might have w i t h i n  an empowerment focus and whether the b e n e f i t s of working w i t h s i n g l e b e n e f i c i a r y groups outweigh the o p p o r t u n i t i e s  f o r mutual  exchange o f p e r s p e c t i v e s .  Inadequate R e p r e s e n t a t i o n  of B e n e f i c i a r i e s  Another s i g n i f i c a n t element r e v e a l e d  i n the d a t a was  b e n e f i c i a r i e s were o f t e n under-represented i n m u l t i p l e  that  121 stakeholder  evaluations.  stakeholders valued  Greene r e v e a l s t h a t  the involvement of program p a r t i c i p a n t s and  r e g r e t t e d t h a t they were not more i n v o l v e d . Lincoln's at  nonbeneficiary  Brandon agrees w i t h  (1990) a s s e r t i o n t h a t e v a l u a t o r s are not v e r y  effective  l o c a t i n g stakeholders. Church & R e v i l l e  (1990) a t t e s t t h a t l a c k of i n i t i a t i o n  s t a f f to l o c a t e or p e r s o n a l l y i n v i t e s t a k e h o l d e r s consumer involvement. through ads  inhibit  For example, i n v i t a t i o n s sent by m a i l or  i n newspapers may  b e n e f i c i a r i e s w i t h no  can  not be v e r y s u c c e s s f u l to l o c a t e  f i x e d address or who  might be  illiterate.  B a r r i e r s of c l a s s , c u l t u r e , and u n f a m i l i a r s o c i a l c o n t e x t s s e t t i n g s are a l s o i d e n t i f i e d as i s s u e s a f f e c t i n g  (Aronson, 1993;  & Beresford,  1990).  Church & R e v i l l e , 1989,  and  participation  and r e p r e s e n t a t i o n of c e r t a i n s t a k e h o l d e r s 1989;  by  Croft  L i n c o l n (1990)  i n d i c a t e s t h a t "the v e r y argument used not to f i n d [ s t a k e h o l d e r s ] - - t h a t i t c o s t s time and r e s o u r c e s - - i s the reason we  need to f i n d them: When s c a r c e resources  every i n t e r e s t e d p a r t y should be c o n s u l t e d "  very  are a t  stake,  (p. 3 ) .  I t i s s i g n i f i c a n t t h a t , i n almost a l l of the s t u d i e s , b e n e f i c i a r y s t a k e h o l d e r s were not s e l e c t e d as r e p r e s e n t a t i v e s by b e n e f i c i a r y groups.  As Brandon r e p o r t s :  A p o s s i b l e weakness i n the t e c h n i c a l q u a l i t y , and thus the v a l i d i t y , of our study was the process f o r s e l e c t i n g s t a k e h o l d e r r e p r e s e n t a t i v e s f o r the meeting....we asked s h e l t e r p r o v i d e r s and elementary s c h o o l teachers to recommend r e p r e s e n t a t i v e s f o r a l l three s t a k e h o l d e r groups....We d i d not s y s t e m a t i c a l l y canvass the s t a k e h o l d e r groups f o r suggestions about r e p r e s e n t a t i v e n e s s . (Brandon: 291)  122 The  data r e v e a l s the v a l u e i n having  own  r e p r e s e n t a t i v e s , c l a i m i n g the sense of ownership  c r e d i b i l i t y t h i s provided--a  stakeholders  recommend t h e i r and  "guarantee of independent  judgement." T h i s s t r a t e g y , however, may groups who may  be e s p e c i a l l y d i f f i c u l t  are unorganized such as the homeless.  have been a l o c a l l y o r g a n i z e d  whom Brandon's e v a l u a t o r s could present  Although  there  group of homeless parents  c o u l d have c o n s u l t e d ,  with  their inclusion  a r i s k of skewing r e s u l t s by having more m i l i t a n t  or one-sided views represented. stakeholder  with  House (1993) r e v e a l s t h a t  r e p r e s e n t a t i v e s do not always p r o p e r l y r e p r e s e n t  the  i n t e r e s t s of disadvantaged groups. These f a c t o r s would seem to operate much e a s i e r i n a l o c a l program i n which the s t a k e h o l d e r organized  or a t l e a s t v i s i b l e .  s i t u a t i o n i n the data,  were u s u a l l y chosen by  agency.  the the  Having the a b i l i t y to s e l e c t  r e p r e s e n t a t i v e s c o u l d have important i m p l i c a t i o n s f o r  stakeholder The  However, even when t h i s was  stakeholders  e v a l u a t o r or the sponsoring t h e i r own  groups are p o s s i b l y b e t t e r  groups and  i s not l i m i t e d j u s t to b e n e f i c i a r y groups.  under-representedness of b e n e f i c i a r i e s was  w i t h i n the e v a l u a t i o n p r o c e s s .  In Greene's study,  also  reported  the  r e s p o n s i b i l i t y f o r s e l e c t i n g an " e v a l u a t i o n team" to serve as a l i a i s o n between the r e s e a r c h e r was  and  the l a r g e r s t a k e h o l d i n g  g i v e n to the agencies sponsoring  reports  the e v a l u a t i o n s .  body  She  that:  w h i l e we viewed such a team as r e p r e s e n t a t i v e of t h i s l a r g e r group [of s t a k e h o l d e r s ] , the s e l f - i d e n t i f i e d teams i n both  123 agencies comprised o n l y the d i r e c t o r , the s t a f f person i n charge o f the evaluand, and one t o three o t h e r agency s t a f f . (Greene/a: 381) T h i s came t o be v e r y s i g n i f i c a n t when the team  unexpectedly  emerged as the major decision-making body f o r s t a k e h o l d e r s . Although Greene r e p o r t s t h a t the r e s u l t s were n o t b i a s e d , she r e v e a l s an u n c e r t a i n t y among the l a r g e r s t a k e h o l d i n g body on how e v a l u a t i o n d e c i s i o n s were made. These f a c t o r s would appear t o be e s p e c i a l l y important i n s t u d i e s where the b e n e f i c i a r i e s have the most stake i n the evaluation.  I t may be t h a t e v a l u a t o r s need t o determine, i f  p o s s i b l e , the l e v e l o f stake f o r d i f f e r e n t groups and mould group composition on t h a t b a s i s . disadvantaged  Some e v a l u a t o r s might argue t h a t  groups should be o v e r - r e p r e s e n t e d , t o compensate  f o r t h e i r l a c k o f power.  Regardless, e v a l u a t o r s need t o  c o n s c i e n t i o u s l y make d e c i s i o n s r e g a r d i n g group composition to r e c r u i t m e n t and s e l e c t i o n o f s t a k e h o l d e r s .  prior  Consideration of  these f a c t o r s r e q u i r e s e v a l u a t o r s to be c l e a r and open about t h e i r v a l u e judgements and the r a t i o n a l e s used of s t a k e h o l d i n g groups.  f o r the i n c l u s i o n  Lack o f a t t e n t i o n to these matters can  r e s u l t i n u n d e r - r e p r e s e n t a t i o n o f key s t a k e h o l d e r s .  Access  to Information  The  importance  o f s t a k e h o l d e r s having access t o a l l  i n f o r m a t i o n a v a i l a b l e t o the e v a l u a t o r i s addressed However, what was not c l e a r l y s t a t e d was how t h i s i s  i n the d a t a .  124 accomplished.  The data r e v e a l t h a t r e s e a r c h e r s have a  s i g n i f i c a n t r o l e i n p r e s e n t i n g o p t i o n s and i n t r a n s l a t i n g and synthesizing information.  These tasks can p l a c e a profound  r e s p o n s i b i l i t y on the e v a l u a t o r t o ensure f u l l a c c e s s . c l e a r t h a t access  I t seems  t o i n f o r m a t i o n does not merely imply access to  hard c o p i e s o f agency r e c o r d s , data, e v a l u a t i o n r e p o r t s , e t c .  It  r e q u i r e s an openness and w i l l i n g n e s s to communicate w i t h s t a k e h o l d e r s and  provide assistance i n i n t e r p r e t i n g evaluation  i n f o r m a t i o n throughout the e v a l u a t i o n . Stakeholders  who l a c k the a p p r o p r i a t e knowledge base may be  unable t o understand f u l l access.  the i n f o r m a t i o n p r o v i d e d , even i f there i s  There may a l s o be i n c o n s i s t e n c i e s i n the knowledge  base among s t a k e h o l d e r s o r s t a k e h o l d i n g groups.  Potter  (1988)  r e p o r t s t h a t "the imbalance i n the amount o f i n f o r m a t i o n possessed It for  by p r o v i d e r s and consumers i s o f t e n so wide"  (p. 153).  i s u n c l e a r t o what extent the r e s e a r c h e r i s r e s p o n s i b l e  e n s u r i n g adequate understanding  o f these matters o r f o r  e n s u r i n g e q u i t y o f access among s t a k e h o l d e r s . f a c t o r s be addressed  without  c r e a t i n g such a s t r u c t u r e d  environment such as Brandon d i d i n t h e i r study? are n o t addressed  adequately  Can some o f these  i n this  These q u e s t i o n s  study.  Equity of P a r t i c i p a t i o n  Issues o f access t o i n f o r m a t i o n a l s o have i m p l i c a t i o n s f o r equitable p a r t i c i p a t i o n .  Some e v a l u a t o r s assume t h a t d i a l o g u e  125 e s t a b l i s h e s a c e r t a i n e q u a l i t y among p a r t i c i p a n t s (Reineke, 1991).  Others may assume that equal access t o i n f o r m a t i o n and  equal o p p o r t u n i t i e s  f o r involvement r e s u l t i n equal  participation. However, e v a l u a t o r s  r e p o r t that "those persons who c o n t r o l  the sources o f i n f o r m a t i o n  and who can manipulate i t s meaning  t y p i c a l l y wield substantial influence" that " t o an a d m i n i s t r a t o r (Palumbo, 1987, p. 24) .  information  ( K e l l y , 1987, p. 293) and i s a source o f power"  Although s u b s t a n t i a l i n f o r m a t i o n  about  programs can c o n f e r r e a l power ( P o t t e r , 1988; K e l l y , 1987; L i n c o l n & Guba, 1986), d i f f u s i o n o f i n f o r m a t i o n  t o groups who a r e  unequal i n s t a t u s o r power should not presume equal d i f f u s i o n o f power. Brandon ensured that these important i s s u e s were addressed i n t h e i r study w i t h the thorough and c a r e f u l p r e c a u t i o n s facilitate  taken t o  equitable p a r t i c i p a t i o n , including h i r i n g a trained  group f a c i l i t a t o r ,  l i m i t i n g d i s c u s s i o n , and making d e c i s i o n s by  p l u r a l i t y and s e c r e t b a l l o t s . p a r t i c i p a t i o n recognized  T h e i r attempts a t e q u a l i t y o f  a discrepancy  i n s t a t u s and power among  p a r t i c i p a n t s and an u n f a m i l i a r i t y w i t h group d e c i s i o n making, a l l f a c t o r s which can a f f e c t the balance o f p a r t i c i p a t i o n .  Aronson  (1993) remarks that consumer p a r t i c i p a t i o n p r o c e s s e s a r e o f t e n i n s u b s t a n t i a l because "they do not,  t y p i c a l l y , a l t e r the  d i s t r i b u t i o n o f power and i n f l u e n c e among consumers, p r o v i d e r s , p l a n n e r s and p o l i c y makers"  (p. 375).  126 Donmoyer (1990) r e v e a l s t h a t attempts to minimize s t a t u s at a p e r s o n a l l e v e l by v a l i d a t i n g or l e g i t i m i z i n g  stakeholder's  i n p u t were l a r g e l y u n s u c c e s s f u l because of a f i x e d agenda and because of the p r e v a l e n t a t t i t u d e s of p r o f e s s i o n a l s t a k e h o l d e r s towards uneducated s t a k e h o l d e r s .  He r e p o r t s :  The tragedy here was t h a t t h i s p a r t i c u l a r p a r e n t ' s ideas were o f t e n q u i t e s o p h i s t i c a t e d , even i f she d i d not always express them i n a s o p h i s t i c a t e d way....The s c h o o l s t a f f would have b e n e f i t t e d from at l e a s t c o n s i d e r i n g t h i s p a r e n t ' s p o i n t of view. (pp. 282-283) House (1993), who of the disadvantaged,  urges e v a l u a t o r s to a t t e n d to the p l i g h t i n d i c a t e s t h a t "making c e r t a i n  i n t e r e s t s of the disadvantaged process  i n the e v a l u a t i o n  i s not b e i n g b i a s e d , but r a t h e r i s c o r r e c t i n g the b i a s e s  that a l r e a d y e x i s t " extend  are r e p r e s e n t e d  the  (p. 123).  Some of these i s s u e s might a l s o  to groups of s t a k e h o l d e r s who  are not n o r m a l l y  considered  poor and powerless but have a s i g n i f i c a n t stake i n e v a l u a t i o n s , such as program s t a f f or c i t i z e n s . i t w i l l be important  to c o n s i d e r how  Aronson (1993) suggests l a c k of power or  among c e r t a i n s t a k e h o l d e r s w i l l l i m i t t h e i r a b i l i t y  that  resources  to  participate. Assuming t h a t equal access and o p p o r t u n i t i e s f o r  involvement  r e s u l t i n e q u a l i t y of p a r t i c i p a t i o n p r o v i d e s a d i s s e r v i c e to stakeholders.  C a r e f u l and t h o u g h t f u l c o n s i d e r a t i o n of  these  i s s u e s , w i t h i n the context of the program b e i n g e v a l u a t e d , must be f a c t o r e d i n t o the p a r t i c i p a t o r y process by e v a l u a t o r s .  127 Use  of Stakeholder  Feedback  Another l i m i t a t i o n of the study r e l a t e s to how use  stakeholder  feedback.  Crow r e p o r t s t h a t : "how  researchers this  should be used i s a concern f o r both problem f o r m u l a t i o n i n t e r p r e t a t i o n i n q u a l i t a t i v e research"  (Crow: 747).  feedback and  They  d i s c u s s t h i s as an i m p l i c a t i o n of the c o l l a b o r a t i v e model.  It  appears t h a t these e v a l u a t o r s were unsure about whether or not use s t a k e h o l d e r  feedback and how  to  such feedback would be used.  Although one might assume t h a t d e c i s i o n s about feedback are b u i l t i n t o the p a r t i c i p a t o r y p r o c e s s , consideration. arrangement  i t i s a f a c t o r worthy of  I f an e v a l u a t o r e n t e r s a c o l l a b o r a t i v e  (a g o a l i n Crow's s t u d y ) , does she or he have the  o p t i o n of choosing whether or not to use  stakeholder  Or does c o l l a b o r a t i o n assume t h a t s t a k e h o l d e r used and  feedback?  feedback w i l l  i n c o r p o r a t e d i n t o the e v a l u a t i o n process?  The  be  data  suggest t h a t shared d e c i s i o n making i s an e s s e n t i a l component to r e i n f o r c i n g the genuineness of p a r t i c i p a t i o n , however, t h i s  can  c r e a t e e t h i c a l problems f o r e v a l u a t o r s . Smith researchers  (1985) r e v e a l s p o t e n t i a l moral problems f a c e d by i f c l i e n t s wish to censor  have produced. ownership may  information  researchers  The p a r t i c i p a t o r y process which c u l t i v a t e s c a r r y the i m p l i c a t i o n t h a t s t a k e h o l d e r s  i n f o r m a t i o n concerning  them.  "own"  Smith r e v e a l s the r i s k t h a t a  s t r o n g r e a c t i o n to a d r a f t r e p o r t may  cause an e v a l u a t o r  to  the  128 r e v i s e a r e p o r t and a v o i d the " s t r i n g of moral problems" t h a t might ensue.  He concludes  that:  e v a l u a t o r s have been t r a i n e d p r i m a r i l y as r e s e a r c h e r s , have l i t t l e formal p r e p a r a t i o n f o r or experience i n d e a l i n g w i t h problems of competing v a l u e s or i n d e c i d i n g which a c t i o n s are m o r a l l y j u s t i f i e d and which are not. (p. 5) House (1994) r e v e a l s t h a t a f u r t h e r dilemma of  stakeholders  owning the e v a l u a t i o n i s t h a t d i s c l o s u r e of r e s u l t s can e n t i r e l y i n t h e i r hands.  A d m i n i s t r a t o r s who  be  are p r e s s u r e d  to  l e g i t i m a t e t h e i r a c t i v i t i e s w i l l t h r e a t e n the autonomy of evaluators  (House, 1993).  Palumbo (1987) r e v e a l s a s i t u a t i o n i n  which an agency d i r e c t o r asked him  to c o r r e c t what the d i r e c t o r  p e r c e i v e d to be an i n c o r r e c t e v a l u a t i o n f i n d i n g .  When Palumbo  r e f u s e d , the a d m i n i s t r a t o r f o l l o w e d through on h i s t h r e a t to limit  the d i s s e m i n a t i o n of the f i n d i n g s .  Tronya & F o s t e r  q u e s t i o n whether c o l l a b o r a t i o n i m p l i e s t h a t the r e s e a r c h e r  (1988) should  "abandon h e r / h i s p r i n c i p l e s to the v a g a r i e s of the marketplace?" (p.  297). Such c o n s i d e r a t i o n s can c r e a t e d i f f i c u l t i e s  working c l o s e l y w i t h s t a k e h o l d e r s . c o l l a b o r a t i o n g o a l s may  for evaluators  In some i n s t a n c e s ,  need to be s a c r i f i c e d i n favour of  e v a l u a t o r ' s e t h i c a l p r i n c i p l e s or i n t e r e s t s .  In  s i t u a t i o n s , working w i t h m u l t i p l e s t a k e h o l d e r s who stakes might h e l p circumvent these problems.  the  other have m u l t i p l e  129 Creating Structure  Another l i m i t a t i o n o f the p a r t i c i p a t o r y approach i s how much s t r u c t u r e should be imposed i n the e v a l u a t i o n process by the researcher.  S e v e r a l authors  and s t a k e h o l d e r s r e v e a l a p o t e n t i a l  r i s k o f the e v a l u a t o r t a k i n g too much power.  The data r e v e a l the  v i t a l r o l e o f the r e s e a r c h e r t o demonstrate e v a l u a t i o n e x p e r t i s e and i n t r o d u c e c e r t a i n a c t i v i t i e s t h a t a r e i n t e n t i o n a l and planned.  Researchers  who a r e e x p l i c i t about the v a l u e judgements  made r e g a r d i n g what i n f o r m a t i o n gets i n t r o d u c e d and when i t gets i n t r o d u c e d a r e l e s s l i k e l y t o f i n d themselves f a c e d w i t h risk.  As w e l l , i t would appear t h a t the elements i n d i c a t e d to  r e i n f o r c e p a r t i c i p a t i o n would of  this  (and did) p r o v i d e a constant  source  a c c o u n t a b i l i t y f o r the e v a l u a t o r and h e l p t o l e s s e n these  risks.  V u l n e r a b i l i t y o f Researcher  Both Faase and Shapiro t a l k about the v u l n e r a b i l i t y o f the r e s e a r c h e r i n the s t a k e h o l d e r s e t t i n g .  Faase i m p l i e s t h a t the  c o l l a b o r a t i o n process can make a r e s e a r c h e r f e e l uneasy:  "It  would have been e a s i e r t o r e s t on p r o t o c o l than t o be exposed i n tentativeness"  (Faase: 81). He a l s o d i s c u s s e s the v u l n e r a b i l i t y  he f e l t coming i n w i t h s t a c k s o f handwritten cut-and-paste  pages which were  v e r s i o n s and how t h i s made him f e e l uncomfortable  and p o s s i b l y , u n p r o f e s s i o n a l and incompetent.  Shapiro  reveals  130 that he l e a r n e d  a g r e a t d e a l about "the hidden agenda, the  unquestioned assumptions, and the accepted r o l e s and r e l a t i o n s h i p s adopted by e v a l u a t o r s and c l i e n t s i n more t r a d i t i o n a l evaluation  arrangements"  (Shapiro: 60). He concludes  that t h e r e can be an u n s e t t l i n g e f f e c t f o r e v a l u a t o r s because o f the a l t e r e d power r e l a t i o n s h i p s . the l i t e r a t u r e .  These e f f e c t s a r e supported i n  In t h e i r e a r l y i n s i g h t s i n t o  Krause & Howard (1976) r e v e a l  collaboration,  that e v a l u a t o r s e n t e r a realm o f  uncertainty  w i t h p a r t i c i p a t o r y approaches because the r e s e a r c h i s  continually  subject  to renegotiation  and change.  The realignment  of r e l a t i o n s h i p s w i t h s t a k e h o l d e r s i n c o l l a b o r a t i v e can  approaches  cause e v a l u a t o r s t o experience a reduced sense o f c o n t r o l  (Reineke, 1991) and a " r o l e s t r a i n " (Church & R e v i l l e , 1989) . These r e a c t i o n s be  reveal  the emphasis p l a c e d on e v a l u a t o r s t o  the " s p e c i a l i z e d p r o f e s s i o n a l "  evaluation  and "expert."  Collaborative  may r e s u l t i n the l o s s o f r e s e a r c h e r s t a t u s .  cases, t r a d i t i o n a l r o l e s and power s t r u c t u r e s altered within  the p a r t i c i p a t o r y p r o c e s s .  I n many  are d r a s t i c a l l y  Breton  (1991) a d v i s e s  that r e s e a r c h e r s cannot be d e f e n s i v e about p r o f e s s i o n a l o r organizational  t u r f s when i n v o l v e d  stakeholders.  Researchers may need time t o a d j u s t  r o l e s and r e l a t i o n s h i p s w i t h i n  i n collaboration  with t o these new  the p a r t i c i p a t o r y p r o c e s s .  E v a l u a t o r s who a r e used t o e x p e r i e n c i n g a c e r t a i n l e v e l o f c o n t r o l and s t a t u s need t o aware o f t h e i r p e r s o n a l r e a c t i o n s to t h i s l o s s and how t h a t might a f f e c t the p a r t i c i p a t o r y p r o c e s s .  131 Implications  f o r Policy-  There are a number of s i g n i f i c a n t p o l i c y i m p l i c a t i o n s that a r i s e from t h i s document study. because e v a l u a t i o n s e v a l u a t i o n has issues.  are o f t e n i n v o l v e d w i t h government programs,  an e s t a b l i s h e d l i n k w i t h i d e o l o g i c a l and  Many e v a l u a t o r s  stakeholders  House (1993) i n d i c a t e s t h a t  have begun to advocate f o r the use  and b e n e f i c i a r i e s i n the p l a n n i n g  that determine p o l i c i e s , programs, and 1992;  Morrison,  Beresford,  1988,  1989).  Potter  perspective  Church & R e v i l l e ,  negotiations (Aronson,  Croft  and  must be  offered  f o r them to have r e a l  the i n t e r e s t s of consumers and  (p.  the wider  s e r v i c e s are planned, d e l i v e r e d  162).  These advocates r e v e a l that s t r u c t u r a l r e o r g a n i z a t i o n s h i f t s i n managerial p r a c t i c e s do not ensure s u b s t a n t i v e i n d e c i s i o n making and power ( C r o f t & B e r e s f o r d , 1992;  of  (1988) recommends a "fundamental s h i f t i n  that places  evaluated"  1989;  They f e e l that stakeholders  p u b l i c a t the h e a r t of the way and  and  evaluations  r e a l i n p u t i n t o p o l i c y s e t t i n g s i n order impact.  political  Hambleton, 1988;  meaningful s t a k e h o l d e r  House, 1993).  1989;  or  changes  Aronson,  They argue t h a t i s s u e s of  p a r t i c i p a t i o n and  impact cannot  adequately addressed through program e v a l u a t i o n .  be  Stakeholder  p a r t i c i p a t i o n i n s o c i a l program e v a l u a t i o n w i l l r a r e l y t u r n c i t i z e n s i n t o p a r t n e r s who 1988).  a c t i v e l y shape programs  These advocates d i s t i n g u i s h between  p a r t i c i p a t i o n and  stakeholder  i n f l u e n c e and  (Potter,  stakeholder are u n e q u i v o c a l i n  132 t h e i r a s s e r t i o n s t h a t those a f f e c t e d by s o c i a l and  economic  p o l i c i e s s h o u l d have the r i g h t to c o n t r i b u t e to t h e i r development and implementation.  As such,  s t a k e h o l d e r s must e n t e r  the  p o l i t i c a l arena and be engaged i n a l l stages of d e c i s i o n making i n p l a n n i n g and s e r v i c e s t r u c t u r e s . people a r e a l sense of involvement  " I t i s a matter of o f f e r i n g  so t h a t they can b e g i n to  experience a c l o s e r r e l a t i o n s h i p between t h e i r needs and services"  ( B e r e s f o r d , 1987,  p.  48).  A p r e r e q u i s i t e to consumer impact  i n p o l i c y i s regarded  as  the a d o p t i o n of p r i n c i p l e s which r e f l e c t the r i g h t s of s t a k e h o l d e r s to guide and reform the development of p o l i c i e s programs.  The  and  i d e o l o g i c a l foundations of consumer-led s t r a t e g i e s  must be e x p l i c i t  (Jones, 1987).  A p h i l o s o p h y of s e r v i c e and  o v e r a l l p o l i c y must be based on a p h i l o s o p h y which supports  and  r e s p e c t s the r i g h t s of s t a k e h o l d e r s to p a r t i c i p a t e . The use of s t a k e h o l d e r s i n e v a l u a t i o n r e s e a r c h p r o v i d e s a s t a r t i n g point f o r enabling stakeholder i n f l u e n c e i n p o l i c y settings.  As s t a k e h o l d e r s are shown to p r o v i d e v a l i d  and  r e l e v a n t i n f o r m a t i o n , and as t h e i r p a r t i c i p a t i o n i s v a l u e d  and  l e g i t i m i z e d , the p a r t i c i p a t i o n of s t a k e h o l d e r s i n p o l i c y making and d e c i s i o n s w i l l be impacted.  E v a l u a t o r s can a s s i s t  makers to develop g u i d e l i n e s and statements  policy  of p r i n c i p l e s which  r e f l e c t the r i g h t s of consumers to p a r t i c i p a t e  meaningfully.  The p r a c t i c e of e v a l u a t i o n i s p a r t of the a u t h o r i t y s t r u c t u r e of s o c i e t y , and e v a l u a t i o n as an a i d to p u b l i c d e c i s i o n making i n v o l v e s concepts of democracy and s o c i a l j u s t i c e , although o f t e n these ideas are i m p l i c i t . Public e v a l u a t i o n should be an i n s t i t u t i o n f o r d e m o c r a t i z i n g p u b l i c d e c i s i o n making, f o r making d e c i s i o n s , programs, and  133 p o l i c i e s more open t o p u b l i c s c r u t i n y and d e l i b e r a t i o n . (House, 1993, p. 127) The  development o f p r i n c i p l e s which r e f l e c t the r i g h t s to  c i t i z e n p a r t i c i p a t i o n may h e l p t o s h i f t the p r e v a i l i n g a t t i t u d e s of many p r o f e s s i o n a l s and p o l i t i c a l r e p r e s e n t a t i v e s  regarding  consumer i n p u t . McGrath (1989) i d e n t i f i e s t h a t there i s a c o n f l i c t between b e l i e f i n consumerism and p r o f e s s i o n a l v a l u e s .  Many p l a n n e r s do  not  f e e l t h a t c o n s t i t u e n t groups can c o n t r i b u t e m e a n i n g f u l l y  the  "reasoned d e l i b e r a t i o n s " that such a p r o c e s s r e q u i r e s .  However, B a r k d o l l  to  (1983) r e v e a l s t h a t i t i s i n d i v i d u a l consumers,  consumer o r g a n i z a t i o n s ,  and other  stakeholding  groups who a r e  o f t e n the source o f the "hard data" used by p l a n n e r s and p o l i c y makers t o make t h e i r i n t e n s e d e l i b e r a t i o n s . P r o f e s s i o n a l and p o l i t i c a l p u b l i c s e r v i c e managers need t o c o n s i d e r what v a l u e  they p l a c e on users,  the p u b l i c .  (1988) argues t h a t p r o f e s s i o n a l s have the  Potter  c i t i z e n s , and members o f  most t o Ipse from consumer-oriented p o l i c y and p r a c t i c e . recommends a s e a r c h i n g and  He  review o f r e l a t i o n s h i p s between p r o v i d e r s  consumers, due to the f a c t that many p r o f e s s i o n a l s a r e  u n w i l l i n g to consider opinions sources.  from uninformed and uneducated  F u l l commitment by p o l i c y makers i s e s s e n t i a l to i n s u r e  that i n f l u e n c e i s n o t j u s t c o n f i n e d  t o minor i s s u e s .  Explicit  i d e o l o g i c a l p r i n c i p l e s f o r p a r t i c i p a t i o n would enhance t h i s process. How can we ensure that consumers, o r consumer groups, i n f l u e n c e s e r v i c e p l a n n i n g f o r the f u t u r e ? We must q u e s t i o n how w e l l our p o l i c y making p r o c e s s e s hear the consumer a t  134 t h a t stage. How o f t e n i s our response e i t h e r or token? (White, 1987, p. 9) In order  bureaucratic,  f o r p a r t i c i p a t i o n t o be s u c c e s s f u l , i s s u e s o f power  must be addressed.  Consumers o f s e r v i c e s a r e almost always l e s s  powerful than p r o v i d e r s  (House 1990a).  Church & R e v i l l e (1989)  b e l i e v e t h a t "the growth o f user involvement i n Canada i s i n e x t r i c a b l y bound up w i t h how q u i c k l y r e l a t i o n s h i p s . . . . change"  (or how slowly)  (p. 24). Aronson  the power  (1992) concurs that  enhancing consumer power: cannot be r e a l i z e d without a d d r e s s i n g the d i s p a r i t y i n power between p r o v i d e r s and u s e r . I f the b o l d e r , more d e m o c r a t i z i n g aims o f p a r t i c i p a t i o n a r e t o be r e a l i z e d , i t i s e v i d e n t that p r o f e s s i o n a l s and a d m i n i s t r a t o r s i n s e r v i c e and p l a n n i n g o r g a n i z a t i o n s w i l l be r e q u i r e d t o share power and c o n t r o l that has, t y p i c a l l y , been c o n c e n t r a t e d w i t h i n f a i r l y h i e r a r c h i c a l s t r u c t u r e s , (p. 375) Jones  (1987) i n d i c a t e s that we must c r i t i c a l l y examine whether  there i s s u f f i c i e n t b a c k i n g w i t h i n the p o l i t i c a l and p r o f e s s i o n a l forums t o b e g i n t o r e n e g o t i a t e  the balance o f power and command  w i t h consumers.  that s t r u c t u r a l changes might be  She i n t i m a t e s  more c h a l l e n g i n g f o r p r o f e s s i o n a l s than they would be f o r clients.  Legitimizing Participation  When consumers a r e granted the r i g h t t o be i n v o l v e d i n p o l i c y d e c i s i o n s which a f f e c t them, the genuineness o f t h e i r p a r t i c i p a t i o n must be r e i n f o r c e d .  Aronson (1992) r e v e a l s  i n i t i a t i v e s t o seek consumer p e r s p e c t i v e s  that  need t o move beyond  r e t r o s p e c t i v e responses t o the r e c e i p t o f s e r v i c e s o r resources  135 to a more a c t i v e r o l e .  She  r e v e a l s t h a t , a l l too  often,  c o n t r i b u t i o n s are sought o n l y a f t e r i r r e v e r s i b l e p o l i c y d i r e c t i o n s have a l r e a d y been e s t a b l i s h e d , thereby p a r t i c i p a t i o n to i n v i t a t i o n s r e g a r d i n g  questions  limiting of  implementation. In order  f o r consumer i n p u t to be  systems must accommodate stakeholder  legitimized,  formal  involvement and  create a  l o c a l p o l i c y environment i n which p a r t i c i p a t i o n i s enhanced. McGrath (1989) and McGrath and  Grant  (1992) i d e n t i f y  consumers must be o f f e r e d m u l t i p l e o p p o r t u n i t i e s participation.  that  for  T h i s p r o c e s s e n t a i l s s i m i l a r elements as  i d e n t i f i e d i n the data i n c l u d i n g :  a t t e n t i o n to group  process;  c o n s i d e r a t i o n of venues and group s i z e ; c l e a r g u i d e l i n e s parameters f o r p l a n n i n g f u n c t i o n and  groups i n c l u d i n g a t t e n t i o n to the group  the range of group a u t h o r i t y ; p r a c t i c a l  such as t r a v e l l i n g expenses and  comprehensive access to i n f o r m a t i o n ;  of  group meetings; e f f e c t i v e and  open communication systems, both i n t e r n a l l y and and  Responsive communication and  s e t t i n g s which are f a m i l i a r and  assistance  childcare; consideration  p s y c h o l o g i c a l b a r r i e r s to a t t e n d i n g  contact.  and  externally;  d i r e c t , personal  the importance of c r e a t i n g  comfortable to consumers are  emphasized. Church & R e v i l l e (1990) and G u t i e r r e z & Ortega  (1991) a l s o  address the importance of s e t t i n g s which are consumer f r i e n d l y e s p e c i a l l y to encourage the involvement of m a r g i n a l i z e d p o l i c y c o n s u l t a t i o n s and p l a n n i n g .  groups i n  C o n s u l t a t i o n w i t h some groups  136 may  be more e f f e c t i v e i n s m a l l f a m i l i a r groups t a k i n g p l a c e where  people l i v e and i n c l u d i n g i s s u e s which concern them (Church & R e v i l l e , 1990;  C r o f t & B e r e s f o r d , 1990).  Clashes i n vocabulary,  u n f a m i l i a r s o c i a l c o n t e x t s and s e t t i n g s , and b a r r i e r s of c l a s s and c u l t u r e can h i n d e r the c o n s u l t a t i o n p r o c e s s and need to be addressed Reville,  (Aronson, 1989,  1992;  C r o f t & B e r e s f o r d , 1989;  Church &  1990).  S e v e r a l e v a l u a t o r s have r e c o g n i z e d the v a l u a b l e l i n k t h a t f r o n t - l i n e p r o f e s s i o n a l s w i l l p r o v i d e between consumers and p l a n n i n g systems 1989;  (McGrath & Grant,  Aronson, 1983;  1992;  B e r e s f o r d , 1988;  Tower, 1994;  McGrath,  M o r r i s o n , 1988) .  The  use  of development workers and community a c t i v i s t s i s encouraged to engage consumers, assure them of the importance involvement, want.  of  their  and prepare them to work toward the changes they  In f a c t , e v a l u a t o r s r e g a r d p r o f e s s i o n a l h e l p and  as a s i g n i f i c a n t f a c t o r i n d e v e l o p i n g s t r u c t u r e s which maximum p a r t i c i p a t i o n and r e a l i n f l u e n c e by consumers 1989,  support ensure  (McGrath,  C r o f t & B e r e s f o r d , 1989). T r a i n i n g i s regarded as e s s e n t i a l f o r s t a f f to s e n s i t i z e  them to consumer needs and to teach them techniques which f o s t e r r a t h e r than d i s c o u r a g e p a r t i c i p a t i o n .  The  importance  i n t e r m e d i a r y r o l e i s c r u c i a l because the involvement  of t h i s of  local  people i s not an experience w i t h which b e n e f i c i a r i e s or consumers are f a m i l i a r or c o m f o r t a b l e . in c i v i l  Often the development of c i t i z e n s  r i g h t s and r e s p o n s i b i l i t i e s i s not encouraged  B e r e s f o r d , 1989).  (Croft &  137 The i m p l i c a t i o n s . . . a r e s i g n i f i c a n t s i n c e they imply a v e r y d i f f e r e n t s t y l e from that most t r a d i t i o n a l l y found i n l o c a l government. I t i m p l i e s a c o n f i d e n t r e l a t i o n s h i p s t y l e and one which i s u n a f r a i d of b e i n g c r i t i c i s e d or c h a l l e n g e d . . . . I t i m p l i e s an approach which goes to the v e r y h e a r t of t r a i n i n g our s t a f f s i n c e i t i m p l i e s t h a t we w i l l need to teach people how to innovate, how to communicate i n a d i f f e r e n t way, and how to be u n a f r a i d of c l i e n t i n f l u e n c e . (White, 1987. p. 10) Proponents r e c o g n i z e and wonder how  the c h a l l e n g e  t h i s p r e s e n t s i n some s e t t i n g s  f a r the p r o c e s s c o u l d extend i n s i t u a t i o n s where  c o n t r o l v e r s u s care i s a major element, such as c h i l d abuse or delinquency Jones  (McGrath, 1989). (1987) suggests that i n order to ensure t h a t consumer  response permeates the p l a n n i n g , s e r v i c e s a t every l e v e l ,  management and  d e l i v e r y of  f i v e elements need to be pursued:  development of l o c a l consumer c o n s u l t a n t  groups; b u i l d i n g  the on  d e c e n t r a l i z a t i o n ; b u i l d i n g p o l i t i c a l support; working i n partnership  w i t h trade unions; and  t r a i n i n g (p. 60-61) .  Consumerism i n p e r s o n a l s o c i a l s e r v i c e s i s s e l f - e v i d e n t l y a 'good t h i n g ' . I t commands a broad p o l i t i c a l consensus and o f f e r s the p o s s i b i l i t y of a new v i s i o n f o r the weary and embattled p e r s o n a l s o c i a l s e r v i c e s . (Jones, 1987, p. 53)  Social Service  Practice  C o l l a b o r a t i o n w i t h consumers of s e r v i c e s i n e v a l u a t i o n p o l i c y s e t t i n g s has practice. be  Not  s i g n i f i c a n t implications for social  only w i l l  service  the foundations of s o c i a l work p r a c t i c e  impacted by consumer-led s e r v i c e s , v o l u n t a r y  organizations  and  which tend to be  smaller  social  service  and more f l e x i b l e  could  s u b s t a n t i a l l y impact the r o l e of consumers i n p o l i c y s e t t i n g s  by  138 p r o v i d i n g o p p o r t u n i t i e s t o d i r e c t p o l i c y making and p l a n n i n g (Jones,  1987) .  S o c i a l work p r a c t i c e i s i n f l u e n c e d when c l i e n t s a r e viewed as r e s o u r c e s and when i n d i v i d u a l s r e d e f i n e t h e i r r o l e from t h a t of c l i e n t t o t h a t o f consumer.  "The h i s t o r i c a l n o t i o n o f  consumer as a p a s s i v e r e c i p i e n t o f a s e r v i c e no l o n g e r true"  (McGrath & Grant,  holds  1992, p. 75). Tower (1994) endorses  adopting a consumer-centered approach t o p r a c t i c e .  She r e v e a l s  that s o c i a l workers who serve even the most v u l n e r a b l e  clients  "are f i n d i n g t h a t the consumers themselves can be t h e i r own greatest resources"  (p. 191). When b e n e f i c i a r i e s  (or consumers)  are viewed as r e s o u r c e s ,  they a r e a b l e to take a c t i v e  i n t h e i r own treatment.  Church and R e v i l l e  involvement  (1990) d e s c r i b e  mental h e a l t h s e l f - h e l p groups as an example o f consumercontrolled  activity.  S o c i a l workers must a s s i s t c l i e n t s t o develop confidence necessary  the s k i l l s and  t o move from a p a s s i v e r o l e o f s e r v i c e  r e c i p i e n t t o a more a c t i v e r o l e which allows them g r e a t e r say i n the s e r v i c e s they use.  This s h i f t e n t a i l s a greater  r e s p o n s i b i l i t y f o r s o c i a l workers i n the areas o f advocacy and includes teaching c l i e n t s s t r a t e g i e s f o r e f f e c t i v e and  "coaching"  1994).  them through the maze o f bureaucracy  (Tower,  In some areas o f s o c i a l work p r a c t i c e t h i s may i n v o l v e  advocating  the use o f brokers  needed s e r v i c e s u s i n g vouchers 1988).  communication  I t i s suggested  t o a l l o w consumers t o purchase (Church & R e v i l l e ,  1989; P o t t e r ,  t h a t "by d e c i d i n g how and where t o use  139 vouchers, consumers can have g r e a t e r impact on are p r o v i d e d "  (Morrison, 1988).  w i l l determine demand and w i l l be  redefined  P r o v i d e r s who development and Transferring  that  supply, and  the s o c i a l s e r v i c e  system  by g r e a t e r consumer c o n t r o l . seek to engage and  power r e l a t i o n s h i p s and advocate f o r the  groups i n s o c i e t y w i l l  enable consumers i n p o l i c y conflict.  authority  i s unsettling.  i n t e r e s t s of m a r g i n a l i z e d  f i n d themselves i n c o n f l i c t w i t h o t h e r  s t a k e h o l d e r groups, e s p e c i a l l y those i n p o l i t i c a l themselves as r e p r e s e n t i n g (Breton, 1991)  services  In essence, consumer "spending"  s e r v i c e d e l i v e r y must expect  S o c i a l workers who  the  the  forums who  see  i n t e r e s t s of a l l groups i n s o c i e t y  C o n f l i c t i s i d e n t i f i e d as a by-product of  consumer-oriented p r a c t i c e  (Tower, 1994).  The p r o c e s s w i l l not be t i d y , i t w i l l not be o r d e r l y , but i f we succeed i n s u r r e n d e r i n g some of our p r o f e s s i o n a l monopoly f o r d e t e r m i n i n g the outcome of our consumers' l i v e s , we w i l l have h e l p e d c r e a t e some space f o r them to come a l o n g s i d e us i n the mainstream of s o c i e t y . (Jones, 1987, p. 61)  Limitations  I d e n t i f y i n g the o p e r a t i o n a l  of the  mechanisms of  p a r t i c i p a t i o n through r e t r o s p e c t i v e The  Study  studies  stakeholder  i s tenuous a t b e s t .  v a l i d i t y of t h i s study i s impacted by a number of Evaluator reporting  events may  be  inaccurate  and  r e c o l l e c t i o n s of the  or incomplete.  factors.  participatory  Evaluators,  faced  academic p r e s s u r e s to p u b l i s h might have m i s r e p r e s e n t e d  with  the  a c t u a l p r o c e e d i n g s of the p a r t i c i p a t o r y p r o c e s s , p r e s e n t i n g t h e i r  140 e v a l u a t i o n s i n the b e s t l i g h t . the s t u d i e s were co-authored  Although  the f a c t t h a t s e v e r a l of  w i t h s t a k e h o l d e r s might enhance the  v a l i d i t y of e v a l u a t o r f i n d i n g s , s t a k e h o l d e r s may  have f e l t  should be p o s i t i v e i n t h e i r comments and support  the e v a l u a t o r ' s  conclusions.  on a p a r t i c u l a r  F u r t h e r , the focus of some authors  e v a l u a t i o n r a t i o n a l e may  they  have impacted the types of i n f o r m a t i o n  i n c l u d e d i n the documents.  Relevant m a t e r i a l or p a r t i c i p a t o r y  dynamics which d i d not n e c e s s a r i l y r e l a t e to the e v a l u a t i o n ' s r a t i o n a l e f o r s t a k e h o l d e r i n c l u s i o n may The  have been  excluded.  s t u d i e s were p u b l i s h e d over a l i m i t e d p e r i o d of time  the r e s e a r c h e r may  have not comprehensively  searched  and  the  l i t e r a t u r e f o r a l l of the a v a i l a b l e s t u d i e s .  As w e l l , the  r e s e a r c h e r i s aware of some s t u d i e s which may  have proved u s e f u l  but were u n a v a i l a b l e because they were  unpublished.  Because documents are f i r s t - h a n d accounts,  they l e n d  themselves to more r i g o r o u s checks on face v a l i d i t y ( B a i l e y , 1994), however, i t i s not known whether the coding and memoing was  comprehensive and r e l i a b l e .  Some areas may  not have been  e x p l o r e d i n s u f f i c i e n t breadth or depth by the r e s e a r c h e r , other important The  i s s u e s may  and  have been i n a d v e r t e n t l y overlooked.  r e s e a r c h method chosen i l l u m i n a t e s the i s s u e s but i t  n e c e s s a r i l y r e s t r i c t s the a b i l i t y to g e n e r a l i z e beyond the F u r t h e r , the q u a l i t a t i v e d e s i g n o p e r a t i n g w i t h i n the s i n g l e source of p u b l i s h e d documents l i m i t s the r i g o r of the data the depth and breadth of the i n f o r m a t i o n produced.  data. data and  141 The  sample s i z e i s too s m a l l f o r c o n f i d e n t or r i g o r o u s  g e n e r a l i z a t i o n s and l i m i t s the a b i l i t y to p o s t u l a t e beyond the documents s t u d i e d .  Although  the sample i n c l u d e d an assortment  of  m i l i e u s and p a r t i c i p a t o r y mechanisms and n e g a t i v e cases were not excluded,  the p e r s p e c t i v e s r e p r e s e n t e d i n the data cannot be  g e n e r a l i z e d to a l l e v a l u a t i o n s .  F u r t h e r , the l i m i t e d number of  some r a t i o n a l e s , i . e . , empowerment, cannot be regarded r e p r e s e n t a t i v e of a l l empowerment e v a l u a t i o n s .  as  Although  there i s  v a l u e i n s e l e c t i n g i n f o r m a t i o n - r i c h cases, a l a r g e random sample of p a r t i c i p a t o r y approaches would i n c r e a s e the r e l i a b i l i t y of the r e s u l t s and might permit reasonable e x t r a p o l a t i o n . At b e s t , the study can serve to shed some l i g h t on phenomenon of s t a k e h o l d e r p a r t i c i p a t i o n and how  the r o l e s  r e l a t i o n s h i p s between r e s e a r c h e r and s t a k e h o l d e r impact p a r t i c i p a t o r y process.  The  the  the  e x p l o r a t o r y d e s i g n i s an a p p r o p r i a t e  approach to attempt to b e g i n the process of understanding c e n t r a l concepts  and  the  and r e l a t i o n a l dynamics of s t a k e h o l d e r  p a r t i c i p a t i o n i n e v a l u a t i o n r e s e a r c h and p r o v i d e s i n s i g h t s  into  areas f o r f u r t h e r study.  Future Research P o s s i b i l i t i e s  T h i s r e s e a r c h p r o j e c t examines the dynamics of  stakeholder  p a r t i c i p a t i o n i n evaluation research with s p e c i a l consideration of r o l e s and r e l a t i o n s h i p s and c o n t e x t u a l f a c t o r s .  Because of  i t s e x p l o r a t o r y focus, t h i s study r a i s e d many more q u e s t i o n s  than  142 i t answered.  R e p l i c a t i o n s t u d i e s would i n c r e a s e the v a l i d i t y and  r e l i a b i l i t y o f the study and r e v e a l f u r t h e r i n s i g h t s i n t o the dynamics o f s t a k e h o l d e r p a r t i c i p a t i o n i n e v a l u a t i o n r e s e a r c h . T r i a n g u l a t i o n o f q u a l i t a t i v e data sources  (Patton, 1990) by  c r o s s - c h e c k i n g the c o n s i s t e n c y o f i n f o r m a t i o n a c r o s s s e v e r a l data sources would enhance the i n t e g r i t y of the data and the analysis.  Conducting  a c r o s s - s e c t i o n o f case s t u d i e s o f  p a r t i c i p a t o r y approaches, both w i t h i n the same r a t i o n a l e s and compared a c r o s s r a t i o n a l e s , would add weight t o the study and p r o v i d e an i l l u m i n a t i o n o f the s u b s t a n t i v e i s s u e s o f p a r t i c i p a t i o n which were not addressed investigation.  through t h i s p r e l i m i n a r y  Comparing e v a l u a t o r p e r c e p t i o n s a g a i n s t those o f  p a r t i c i p a n t s would f u r t h e r enhance v a l i d i t y R e t r o s p e c t i v e accounts  (House, 1980) .  by s t a k e h o l d e r s , i r r e s p e c t i v e o f e v a l u a t o r  i n p u t o r d i r e c t i o n , c o u l d be s t u d i e d i n and o f themselves o r cross-checked  w i t h r e t r o s p e c t i v e reviews w r i t t e n by e v a l u a t o r s o r  other s t a k e h o l d e r s .  Comparing s t a k e h o l d e r p e r c e p t i o n s  across  s t a t u s o r power d i f f e r e n c e s would f u r t h e r impact the v a l i d i t y and r e l i a b i l i t y o f the data. Other q u e s t i o n s emanate from the l i m i t a t i o n s o f the study. Does c o n t i n u i t y o f p a r t i c i p a t i o n e f f e c t the genuineness o f participation? of  access  What i s the r e s e a r c h e r ' s r o l e i n e n s u r i n g e q u i t y  for participants?  What f a c t o r s c o n t r i b u t e t o over-  r e p r e s e n t a t i o n o f some s t a k e h o l d e r s ?  Are r e s e a r c h e r s more l i k e l y  to c o l l a b o r a t e w i t h r e g a r d to some s t a k e h o l d e r s but n o t others? What a r e the e f f e c t s o f ownership f o r p a r t i c i p a n t s ,  f o r programs,  143 and  for utilization?  ownership?  How much p a r t i c i p a t i o n c o n s t i t u t e s  Does a l a c k o f power o r r e s o u r c e s among s t a k e h o l d e r s  l i m i t t h e i r sense o f ownership?  How does ownership e f f e c t the  ongoing o p e r a t i o n s o f the programs studied?  Although  there has  been c o n s i d e r a b l e study on l i n k a g e s between use and p a r t i c i p a t i o n (Greene, 1990, 1988b; Cousins  & Leithwood, 1986; L e v i t o n &  Hughes, 1981), t h e r e i s l i m i t e d understanding  o f the concept o f  ownership and i t s l i n k s t o p a r t i c i p a t i o n . Finally,  the i n c l u s i o n o f b e n e f i c i a r y s t a k e h o l d e r s  appears  to p r e s e n t unique c h a l l e n g e s to e v a l u a t i o n r e s e a r c h and f u r t h e r i n v e s t i g a t i o n should ensue both w i t h b e n e f i c i a r i e s working together as a u n i t and working among m u l t i p l e s t a k e h o l d e r s .  The  data p r o v i d e c l u e s about some o f these i s s u e s but a r e g e n e r a l l y weak i n i l l u m i n a t i n g these i s s u e s .  Conclusions  The  i n c l u s i o n of stakeholders i n evaluation research i s a  complex and arduous task. research i s s i g n i f i c a n t .  However, the p o t e n t i a l f o r e v a l u a t i o n The data r e v e a l t h a t a p a r t i c i p a t o r y  process which r e i n f o r c e s the genuineness o f p a r t i c i p a t i o n through i n t e r a c t i v e communication, i t e r a t i v e feedback, evidence o f p a r t i c i p a t i o n , and shared d e c i s i o n making, l e g i t i m i z e s stakeholder p a r t i c i p a t i o n . s t a k e h o l d e r s generated  The involvement  of m u l t i p l e  e v a l u a t i o n t o o l s which were s e n s i t i v e t o  144 l o c a l c o n t e x t s and i n c r e a s e d the v a l i d i t y and u t i l i z a t i o n o f the evaluation. The  elements o f a l e g i t i m a t e p a r t i c i p a t o r y p r o c e s s which  operated  c o n t i n u o u s l y and i t e r a t i v e l y throughout the e v a l u a t i o n  process c u l t i v a t e d a sense o f ownership and r e a d y - f o r use c o n d i t i o n s among p a r t i c i p a n t s .  When these same p a r t i c i p a t o r y  elements were p r e s e n t w i t h i n mutually  exclusive evaluation  phases, t h i s dynamic was not observed.  Effective  participation  may a l s o be impacted by the c o n t i n u i t y o f s t a k e h o l d e r  involvement  throughout the e v a l u a t i o n p r o c e s s . C o l l a b o r a t i o n with stakeholders requires extensive p r e p a r a t i o n and p l a n n i n g .  The preparedness o f the r e s e a r c h e r  impacts meaningful and manageable p a r t i c i p a t i o n f o r s t a k e h o l d e r s . P r o f i c i e n t p r e p a r a t i o n and p l a n n i n g a s s i s t the e v a l u a t o r t o f a c i l i t a t e e f f e c t i v e group p r o c e s s e s , stakeholder p a r t i c i p a t i o n . a shift  thereby  enabling  These dynamics, however, may r e q u i r e  f o r some e v a l u a t o r s o p e r a t i n g i n u n f a m i l i a r t e r r i t o r i e s  of group p r o c e s s e s  and d i v e r s e l e a r n i n g needs.  Advanced  training  f o r e v a l u a t o r s and/or c o l l a b o r a t i o n w i t h t r a i n e d group f a c i l i t a t o r s may become p r e v a l e n t p r a c t i c e i n e v a l u a t i o n research.  I f , as Donmoyer (1990) suggests,  are as important  group p r o c e s s  as the t e c h n i c a l e x p e r t i s e o f e v a l u a t o r s ,  skills this  would s i g n i f i c a n t l y impact e v a l u a t i o n t r a i n i n g . When p a r t i c i p a t i o n i s l e g i t i m i z e d ,  the data r e v e a l t h a t  r e s e a r c h e r s can be i n v o l v e d more e x c l u s i v e l y i n the t e c h n i c a l e v a l u a t i o n t a s k s without  compromising c o l l a b o r a t i o n .  This  145 ensures an important r o l e f o r expert e v a l u a t o r s i n program evaluation.  In f a c t , the r o l e of the e v a l u a t o r i s even more  s i g n i f i c a n t i n p a r t i c i p a t o r y approaches due for  facilitators  who  to the c r i t i c a l need  are both knowledgeable i n r e s e a r c h  methodology and e v a l u a t i o n tasks and who  can ensure  meaningful  p a r t i c i p a t i o n of p a r t i c i p a n t s . Understanding  of the unique  c h a l l e n g e s of b e n e f i c i a r y  i n c l u s i o n i n e v a l u a t i o n i s l i m i t e d due evaluation.  to the scope of the  C o n s i d e r a b l e a t t e n t i o n to group p r o c e s s appears  be e s s e n t i a l , e s p e c i a l l y when working w i t h groups who and powerless.  to  are poor  Responding to group process and l e a r n i n g needs,  however, seems rudimentary  i n comparison to d e t e r m i n i n g  how  e v a l u a t o r s s h o u l d a t t e n d to equal access and e q u i t y of participation.  I t i s l i k e l y t h a t s i g n i f i c a n t problems w i l l  arise  when there are g l a r i n g d i s c r e p a n c i e s i n knowledge bases, power, and s t a t u s among s t a k e h o l d e r s . are determined be apparent, difficult  i s ambiguous.  How  and when these d i f f e r e n c e s  Although d i f f e r e n c e s i n s t a t u s  d i f f e r e n c e s i n power or knowledge bases may  to d e t e c t .  I t i s not c l e a r how  may  be more  the p a r t i c i p a t o r y  process i s a f f e c t e d when these d i f f e r e n c e s s u r f a c e or what the e v a l u a t o r ' s r o l e i s i n determining how d i f f e r e n c e s are addressed.  and/or i f these  F u r t h e r , the r e s e a r c h e r ' s  i n t e r f e r e n c e i n such f a c t o r s may  have s i g n i f i c a n t i m p l i c a t i o n s  the group dynamics and the p a r t i c i p a t o r y p r o c e s s . s i g n i f i c a n t concerns  These are  t h a t pose c o m p l i c a t i o n s f o r e v a l u a t i o n  researchers interested i n s o c i a l j u s t i c e  issues.  on  146 The b e n e f i t s of s t a k e h o l d e r involvement s o c i a l s e r v i c e p r a c t i c e are c o n s i d e r a b l e .  for policy  and  Providing  o p p o r t u n i t i e s f o r s t a k e h o l d e r i n p u t opens up p o s s i b i l i t i e s f o r t h e i r involvement  i n program p l a n n i n g and development and f o r  s u b s t a n t i a l i n p u t and say i n t h e i r own  treatment.  S o c i a l workers  are e n t e r i n g an e r a where they are r e d e f i n i n g t h e i r r o l e s  and  t h e i r s k i l l s by d e v e l o p i n g c o n s t r u c t i v e r e l a t i o n s h i p s w i t h consumers or s e r v i c e r e c i p i e n t s to enable  them to have a say.  T h i s r e s t r u c t u r i n g p r o v i d e s an o p p o r t u n i t y to f u r t h e r s o c i a l work g o a l s f o r s o c i a l change (Wagner, 1991). I t i s apparent  t h a t the i n c l u s i o n of s t a k e h o l d e r s i n  e v a l u a t i o n i s a formidable task which r e q u i r e s c o n s i d e r a b l e a t t e n t i o n to v a r i o u s aspects of p a r t i c i p a t o r y Although  processes.  s t a k e h o l d e r p a r t i c i p a t i o n shows c o n s i d e r a b l e promise,  much remains to be understood  before r e l i a b l e p a r t i c i p a t o r y  elements can be recommended f o r c o n s i s t e n t implementation current evaluation practices. expedient  research i s evident.  The need f o r c o n t i n u e d  and  in  147 BIBLIOGRAPHY A l k i n , M. C. (1990). Debates on e v a l u a t i o n . Sage P u b l i c a t i o n s .  Newbury Park,  CA:  A l k i n , M. C , & Patton, M. Q. (1987). Working both s i d e s of the s t r e e t . In J . Nowakowski (Ed.), The c l i e n t p e r s p e c t i v e on e v a l u a t i o n (pp. 19-32). San F r a n c i s c o : Jossey-Bass. A l k i n , M. C , Jacobson, P., Burry, J . , Ruskus, J . , White, P., Kent, L. (1895) . A guide f o r e v a l u a t i o n d e c i s i o n makers. B e v e r l y H i l l s : Sage P u b l i c a t i o n s .  &  A l l e n , D. G., G i l c h r i s t , L. D., Brown, L., Cox, G. B., Semke, J . , Thomas, M. D., Perry, R. D. (1994). One system, many p e r s p e c t i v e s : Stakeholders and mental h e a l t h system e v a l u a t i o n . E v a l u a t i o n and Program Planning, 17, 47-51. A r n o l d , R., Barndt, D., & Burke, B. (1987). The new weave: Popular e d u c a t i o n i n Canada and C e n t r a l America. CUSO: OISE. Aronson, J . (1993). G i v i n g consumers a say i n p o l i c y development: I n f l u e n c i n g p o l i c y or j u s t being heard? Canadian P u b l i c P o l i c y , 19 (4), 367-378. Ayers, T. D. (1987). Stakeholders as p a r t n e r s i n e v a l u a t i o n : A s t a k e h o l d e r - c o l l a b o r a t i v e approach. E v a l u a t i o n and Program Planning, 10. 263-271. B a i l e y , K. D. York: The  (1994). Methods of s o c i a l r e s e a r c h Free P r e s s .  (4th Ed).  New  B a r k d o l l , G. L. (1983) . I n v o l v i n g c o n s t i t u e n t s i n agency p r i o r i t y s e t t i n g . E v a l u a t i o n and Program Planning, 6, 31-37. B a r r i c k , C. B., & Cogliano, J . F. (1993). Stakeholder involvement: Mythology or methodology? E v a l u a t i o n P r a c t i c e , 14# 33-37. Belenky, M. F., C l i n c h y , B. M., Goldverty, N. R., & Taruk, J . (1986) . Womens' ways of knowing. New York: B a s i c Books.  M.  B e r e s f o r d , P. (1988). Consumer views: Data c o l l e c t i o n or democracy? In I. A l l e n (Ed.), Hearing the v o i c e of the consumer, (pp. 37-51). London: P o l i c y S t u d i e s I n s t i t u t e . Berk, R. A., & R o s s i , P. H. (1976). Doing good or worse: E v a l u a t i o n r e s e a r c h p o l i t i c a l l y reexamined. S o c i a l Problems, 23, 327-349.  148 Brandon, P. R., Newton, B. J . , & Harman, J . W. (1993). Enhancing v a l i d i t y through b e n e f i c i a r i e s ' e q u i t a b l e involvement i n i d e n t i f y i n g and p r i o r i t i z i n g homeless c h i l d r e n ' s e d u c a t i o n a l problems. E v a l u a t i o n and Program Planning, 16, 287-293. Braskamp, L. A., Brandenburg, D. C , & Ory, J . C. (1987). Lessons about c l i e n t s ' e x p e c t a t i o n s . In J . Nowakowski (Ed.), The c l i e n t p e r s p e c t i v e on e v a l u a t i o n (pp. 19-32). San F r a n c i s c o : Jossey-Bass. Breton, M. (1991). R e f l e c t i o n s on s o c i a l a c t i o n p r a c t i c e i n France. S o c i a l Work w i t h Groups, 14, 91-107. Brunner, I . , & Guzman, A. (1989). P a r t i c i p a t o r y e v a l u a t i o n : A t o o l to assess p r o j e c t s and empower people. In R. F. Conner, & M. Hendricks (Eds.), I n t e r n a t i o n a l i n n o v a t i o n s i n e v a l u a t i o n methodology (pp. 9-18). San F r a n c i s c o : Jossey-Bass. Bryk, A. S., & Raudenbush, S. W. (1983). The p o t e n t i a l c o n t r i b u t i o n of program e v a l u a t i o n to s o c i a l problem s o l v i n g : A view based on the CIS & Push/Excel experience. In A. S. Bryk (Ed.), Stakeholder-based e v a l u a t i o n (pp. 97-108). San Francisco: Jossey-Bass. Bunker, D. R. (1978) . O r g a n i z i n g e v a l u a t i o n to serve the needs of program p l a n n e r s and managers. E v a l u a t i o n and Program P l a n n i n g . 1, 129-134. Campbell, D. T. (1984). Can we be s c i e n t i f i c i n a p p l i e d s o c i a l s c i e n c e ? E v a l u a t i o n S t u d i e s Review Annual. 9_, 26-48. Carey, M. A., & Smith, M. W. (1992). Enhancement of v a l i d i t y through q u a l i t a t i v e approaches. E v a l u a t i o n and the H e a l t h P r o f e s s i o n s , 15, 107-114. Chambers, D. E., Wedel, K. R., & Rodwell, M. K. (1992). E v a l u a t i n g s o c i a l programs. Boston: A l l y n & Bacon. Chelimsky, E. (1987). What have we l e a r n e d about the p o l i t i c s program e v a l u a t i o n ? E v a l u a t i o n P r a c t i c e , 8_, 5-21.  of  Chen, H. T. (1994) . Current trends and f u t u r e d i r e c t i o n s i n program e v a l u a t i o n . E v a l u a t i o n P r a c t i c e , 15, 229-238. Chu, F. D., & T r o t t e r , S. (1974). The madness e s t a b l i s h m e n t : Ralph Nader's study group r e p o r t on the N a t i o n a l I n s t i t u t e of Mental H e a l t h . New York: Grossman P u b l i s h e r s . Chubin, D. E. (1987). Designing r e s e a r c h program e v a l u a t i o n s : A s c i e n c e s t u d i e s approach. Science and P u b l i c P o l i c y , 14, 8290.  149 Church, K., & R e v i l l e , D. (1990). Do the r i g h t t h i n g Canadian Review of S o c i a l P o l i c y , 26, 77-81.  right.  Church, K., & R e v i l l e , D. (1989). User involvement i n the mental h e a l t h f i e l d i n Canada. Canada's Mental H e a l t h , 37 (2), 22-25. Cochran, N. (1980). S o c i e t y as emergent and more than r a t i o n a l : An essay on the i n n a p p r o p r i a t e n e s s of program e v a l u a t i o n . P o l i c y S c i e n c e s , 12, 113-129. Cohen, D. K. (1983). E v a l u a t i o n and reform. In A. S. Bryk (Ed.), Stakeholder-based e v a l u a t i o n (pp. 73-81). San F r a n c i s c o : Jossey-Bass. Cook, T. D. (1985). P o s t - p o s i t i v i s t c r i t i c a l m u l t i p l i s m . In R. L. Shotland, & M. M. Mark (Eds), S o c i a l s c i e n c e and s o c i a l p o l i c y (pp. 21-62) . B e v e r l y H i l l s : Sage P u b l i c a t i o n s . Cook, T. D., & Shadish, W. R. (1986). Program e v a l u a t i o n : The wordly s c i e n c e . Annual Review of Psychology, 37, 193-232. Cook, T. D., and Campbell, D. T. (1976). The d e s i g n and conduct of quasi-experiements and t r u e experiments i n f i e l d s e t t i n g s . In M.D. Dunnette (Ed.), Handbook of I n d u s t r i a l and O r g a n i z a t i o n a l Psychology. Chicago: Rand McNally. Cousins, J . B., & E a r l , L. M. (1992). The case f o r p a r t i c i p a t o r y e v a l u a t i o n . E d u c a t i o n a l E v a l u a t i o n and P o l i c y A n a l y s i s , 14, 397-418. Cousins, J . B., & Leithwood, K. A. (1986). C u r r e n t e m p i r i c a l r e s e a r c h on e v a l u a t i o n u t i l i z a t i o n . Review o f E d u c a t i o n a l Research, 56, 331-364. Cousins, J . B., Ross, J . A., & Maynes, F. L. (1994). The r e p o r t e d nature and consequences of t e a c h e r s ' j o i n g work i n t h r e e exemplary s c h o o l s . The Elementary School J o u r n a l , 94, 441-465. Couto, R. A. (1987). P a r t i c i p a t o r y r e s e a r c h : Methodology c r i t i q u e . C l i n i c a l S o c i o l o g y Review, 5_, 83-90. Crane, J . A. (1982). The e v a l u a t i o n of s o c i a l p o l i c i e s . Kluwer-Nijhoff Publishing.  and Boston:  C r o f t , S., & B e r e s f o r d , P. (1989). User-involvement, c i t i z e n s h i p and s o c i a l p o l i c y . C r i t i c a l S o c i a l P o l i c y , 9. (2), 5-18. Cronbach, L. J . (1982) . D e s i g n i n g e v a l u a t i o n s of e d u c a t i o n a l and s o c i a l programs. San F r a n c i s c o : Jossey-Bass.  150 Cronbach, L. J . (1982). In p r a i s e of u n c e r t a i n t y . I n R o s s i , P. H. (Ed.), Standards F o r E v a l u a t i o n P r a c t i c e , (pp. 49-58). San F r a n c i s c o : Jossey-Bass. Cronbach, L. J . , Ambron, S. R., Dornbusch, S. M., Hess, R. D., Hornik, R. C , P h i l l i p s , D. C , Walker, D. F., & Weiner, S. S. (1980) . Toward reform o f s o c i a l programs. San F r a n c i s c o : Jossey-Bass. Crow, G. M., Levine, L., & Nager, N. (1992). A r e three heads b e t t e r than one? R e f l e c t i o n s on doing c o l l a b o r a t i v e i n t e r d i s c i p l i n a r y r e s e a r c h . American E d u c a t i o n a l Research J o u r n a l , 29, 737-753. Dawson, F. A., & D'Amico, J . J . (1985). I n v o l v i n g program s t a f f i n evaluation studies: A strategy for increasing evaluation use and e n r i c h i n g the database. E v a l u a t i o n Review, 9_, 173-188. Deutsch, S. J . , & Malmborg, C. J . (1986). A study on the c o n s i s t e n c y o f s t a k e h o l d e r p r e f e r e n c e s f o r d i f f e r e n t types o f i n f o r m a t i o n i n e v a l u a t i o n p o l i c e s e r v i c e s . E v a l u a t i o n and Program Planning, 9, 13-24. D i n k e l , N. R., Zinober, J . W., & F l a h e r t y , E. W. (1981). C i t i z e n p a r t i c i p a t i o n i n community mental h e a l t h c e n t r e e v a l u a t i o n : A n e g l e c t e d p o t e n t i a l . Community Mental H e a l t h J o u r n a l , 17, 5465. Donmoyer, R. (1990). C u r r i c u l u m e v a l u a t i o n and the n e g o t i a t i o n o f meaning. Language A r t s , 67, 274-286. D r i s c o l l , K., & McFarland, J . (1987). The impact o f a f e m i n i s t p e r s p e c t i v e on r e s e a r c h methodologies: S o c i a l s c i e n c e s . I n W. Tomm (Ed.), The e f f e c t s o f f e m i n i s t approaches on r e s e a r c h methodologies (pp. 185-203). Waterloo, O n t a r i o : W i l f r e d L a u r i e r Press. DuBois, B. (1983). Passionate s c h o l a r s h i p : Notes on v a l u e s , knowing and method o f f e m i n i s t s o c i a l s c i e n c e . I n G. Bowles, & R. D u e l l e K l e i n (Eds.), T h e o r i e s o f women's s t u d i e s (pp. 105116). New York: Routeldge. D u e l l i K l e i n , R. (1983). How t o do what we want t o do: Thoughts about f e m i n i s t methodology. I n G. Bowles, & R. D u e l l e K l e i n (Eds.), T h e o r i e s o f women's s t u d i e s (pp. 88-104). New York: Routeldge. Dunn, W. N. (1982). Reforms as arguments. Knowledge: C r e a t i o n , d i f f u s i o n , u t i l i z a t i o n , 3_, 293-326.  151 Faase, T. P., & Pujdak, S. (1987). Shared u n d e r s t a n d i n g of o r g a n i z a t i o n a l c u l t u r e . In J . Nowakowski (Ed.), The c l i e n t p e r s p e c t i v e on e v a l u a t i o n (pp. 75-82). San F r a n c i s c o : Jossey Bass. F a r r a r , E., & House, E..R. (1983). The e v a l u a t i o n of Push/Excel: A case study. In A. S. Bryk (Ed.), Stakeholder-based e v a l u a t i o n (pp. 31-57). San F r a n c i s c o : Jossey-Bass. F r e i r e , P. Herder.  (1970) . Pedagogy of the oppressed. New  York: Herder &  Gaventa, J . (1988). P a r t i c i p a t o r y r e s e a r c h i n North America. Convergence, 21, 19-27. Gibbs, L. (1983). E v a l u a t i o n r e s e a r c h : S c i e n t i s t o r advocate? J o u r n a l of S o c i a l S e r v i c e Review, 7, 81-92. G i l s i n a n , J . F., & Volpe, L. C. (1986). Do not c r y wolf u n t i l are sure: The manufactured c r i s e s i n e v a l u a t i o n r e s e a r c h . E v a l u a t i o n S t u d i e s Review Annual, 11, 175-191.  you  G i l l , S. J . , & Zimmerman, N. R. (1990). R a c i a l and e t h n i c and gender b i a s i n the c o u r t s : A s t a k e h o l d e r - f o c u s e d e v a l u a t i o n . E v a l u a t i o n P r a c t i c e , 11, 103-108. Gold, N. (1983). Stakeholders and program e v a l u a t i o n : C h a r a c t e r i s t i c s and r e f l e c t i o n s . In A. S. Bryk (Ed.), Stakeholder-based e v a l u a t i o n (pp. 63-72). San F r a n c i s c o : Jossey-Bass. Greene, J . C. (1990). T e c h n i c a l q u a l i t y versus user responsiveness i n e v a l u a t i o n p r a c t i c e . E v a l u a t i o n and Planning, 13, 267-274.  Program  Greene, J . C. (1988a). Stakeholder p a r t i c i p a t i o n and u t i l i z a t i o n i n program e v a l u a t i o n . E v a l u a t i o n Review, 12, 91-116. Greene, J . C. (1988b). Communication of r e s u l t s and u t i l i z a t i o n i n p a r t i c i p a t o r y program e v a l u a t i o n . E v a l u a t i o n and Program Planning, 11, 341-351 Greene, J . C. (1987). Stakeholder p a r t i c i p a t i o n i n e v a l u a t i o n d e s i g n : Is i t worth the e f f o r t ? E v a l u a t i o n and Program Planning, 10, 379-394. Greene, J . C , & McClintock, C. (1991). The e v o l u t i o n of e v a l u a t i o n methodology. Theory Into P r a c t i c e , 30, 13-21. Guba, E. G., and L i n c o l n , Y. S. (1989). Fourth g e n e r a t i o n e v a l u a t i o n . Newbury Park, CA: Sage P u b l i c a t i o n s .  152 Guba, E. G. (1987). N a t u r a l i s t i c e v a l u a t i o n . In D. S. Cordray, H. S. Bloom, & R. J . L i g h t (Eds.), E v a l u a t i o n p r a c t i c e i n review, (pp. 23-43). San F r a n c i s c o : Jossey-Bass. Guba, E. G., & L i n c o l n , Y. S. (1983). E f f e c t i v e e v a l u a t i o n . San F r a n c i s c o : Jossey-Bass. G u t i e r r e z , L. M., & Ortega, R. (1991). Developing methods t o empower L a t i n o s : The importance o f groups. S o c i a l Work w i t h Groups, 14, 23-41. Guttentag, M. (1977) . E v a l u a t i o n and s o c i e t y . In M. Guttentag & S. Saar (Eds.), E v a l u a t i o n S t u d i e s Review Annual ( V o l . 2, pp. 52-62). B e v e r l e y H i l l s : Sage P u b l i c a t i o n s . Gunn, W. J . (1987) . C l i e n t concerns and s t r a t e g i e s i n e v a l u a t i o n s t u d i e s . In J . Nowakowski (Ed.), The c l i e n t p e r s p e c t i v e on e v a l u a t i o n (pp. 19-32). San F r a n c i s c o : Jossey-Bass. Hambleton, R. (1988). Consumerism, d e c e n t r a l i z a t i o n and l o c a l democracy. P u b l i c A d m i n i s t r a t i o n , 66, 125-147. Haug, M. R., & Sussman, M. B. (1969). P r o f e s s i o n a l autonomy and the r e v o l t o f the c l i e n t . S o c i a l Problems, 17, 153-161. Hastings, J . T. (1966). C u r r i c u l u m e v a l u a t i o n : The why o f outcomes. J o u r n a l o f E d u c a t i o n a l Measurement, 3_ (1), 27-32). Heilman, J . G. (1983). U t i l i z a t i o n T h e o r i e s . E v a l u a t i o n Review, 7, 707-728. Holman, B. (1987) . Research from the u n d e r s i d e . The B r i t i s h J o u r n a l o f S o c i a l Work. 17, 669-683. Hope, A., & Timmel, S. (1984). T r a i n i n g f o r t r a n s f o r m a t i o n . Zimbabwe: Mambo P r e s s . House, E. R. (1994). The f u t u r e p e r f e c t o f e v a l u a t i o n . P r a c t i c e , 15, 239-247. House, E. R. (1993). P r o f e s s i o n a l E v a l u a t i o n . Sage P u b l i c a t i o n s .  Evaluation  Newbury Park, CA:  House, E. R. (1990a). Methodology and j u s t i c e . In K. A. S i r o t n i k (Ed.), E v a l u a t i o n and s o c i a l j u s t i c e : Issues i n p u b l i c education, (pp. 23-36). San F r a n c i s c o : Jossey-Bass. House, E. R. (1990b). Trends i n e v a l u a t i o n . Researcher, 19 (3), 24-28.  Educational  House, E. R. (Ed.). (1986a). New d i r e c t i o n s i n e d u c a t i o n a l e v a l u a t i o n . P h i l a d e l p h i a , PA: The Falmer P r e s s .  153 House, E. R. (1986b). J u s t i c e i n e v a l u a t i o n . In G. V. G l a s s (Ed), E v a l u a t i o n S t u d i e s Review Annual (Vol 1, pp. 75-100). B e v e r l y H i l l s : Sage P u b l i c a t i o n s . House, E. R. (1980). E v a l u a t i n g w i t h v a l i d i t y . Sage P u b l i c a t i o n s .  Beverly  Hills:  Huberman, M. (1990). Linkage between r e s e a r c h e r s and p r a c t i c i o n e r s : A q u a l i t a t i v e study. American E d u c a t i o n a l Research J o u r n a l , 27, 363-391. Jamieson, K. (1981). S i s t e r s under the s k i n : An e x p l o r a t i o n of the i m p l i c a t i o n s of f e m i n i s t - m a t e r i a l i s t p e r s p e c t i v e r e s e a r c h . Canadian E t h n i c S t u d i e s , 8. (1), 130-143 . Jones, A. (1988). C o l l a b o r a t i o n w i t h consumers: L e a r n i n g how to l i s t e n . In I . A l l e n (Ed.), Hearing the v o i c e of the consumer, (pp. 53-61). London: P o l i c y S t u d i e s I n s t i t u t e . K e l l y , R. M. (1987). The p o l i t i c s of meaning and p o l i c y i n q u i r y . In D. J . Palumbo (Ed.), The p o l i t i c s of program e v a l u a t i o n , (pp. 270-295). Newbury Park, CA: Sage P u b l i c a t i o n s . Kennedy, M. M. (1984). How evidence a l t e r s u n d e r s t a n d i n g and d e c i s i o n s . E d u c a t i o n a l E v a l u a t i o n and P o l i c y A n a l y s i s . 6., 226.  207-  Kenny, D. A. (1982) . [Review of E v a l u a t i n g w i t h v a l i d i t y ] . E d u c a t i o n a l E v a l u a t i o n and P o l i c y A n a l y s i s , 4, 121-122. K i r b y , S. L. (1991). What do f e m i n i s t methods have to do w i t h e t h i c s ? In S. K i r b y , D. D a n i e l s , K. McKenna, M. P u j o l , & V a l i q u e t t e , M. (Eds), Women changing academe, (pp. 167-179). Winnipeg: S o r o r a l P u b l i s h i n . Kirkup, G. (1986). The f e m i n i s t e v a l u a t o r . In House, E. R. New d i r e c t i o n s i n e d u c a t i o n a l e v a l u a t i o n (pp.68-84). P h i l a d e l p h i a , PA: The Falmer P r e s s .  (Ed.),  Knott, T. D. (1988). The impact of major consumer groups on h e a l t h c a r e e v a l u a t i o n . In J . A. McLaughlin, L. J . Weber, R. W. Covert, & R. B. I n g l e (Eds.), E v a l u a t i o n u t i l i z a t i o n (pp. 33-46). San F r a n c i s c o : Jossey-Bass. Krause, M. S., & Howard, K. J . (1976). Program e v a l u a t i o n i n the p u b l i c i n t e r e s t : A new r e s e a r c h methodology. Community Mental H e a l t h J o u r n a l , 12, pp. 291-300. Kurland, R. (1978). P l a n n i n g : The n e g l e c t e d component of group development. S o c i a l Work w i t h Groups, 1 (2), 173-178.  154 L a t a p i , , P. (1988). P a r t i c i p a t o r y r e s e a r c h : A new r e s e a r c h paradigm? A l b e r t a J o u r n a l of E d u c a t i o n a l Research, 34, 310319. Lawrence, J . E. S. (1989). Engaging r e c i p i e n t s i n development e v a l u a t i o n : The s t a k e h o l d e r approach. E v a l u a t i o n Review, 13., 243-256. Lawrence, J . E. S., & Cook, T. J . (1982). D e s i g n i n g u s e f u l e v a l u a t i o n s : The s t a k e h o l d e r survey. E v a l u a t i o n and Program Planning, 5, 327-336. Levine, A., & Levine, M. (1977). The s o c i a l context of e v a l u a t i o n r e s e a r c h : A case study. E v a l u a t i o n Q u a r t e r l y , 1, 515-542. L e v i t o n , L. C , & Hughes, E. F. X. (1981). Research on the u t i l i z a t i o n of e v a l u a t i o n s : A review and s y n t h e s i s . E v a l u a t i o n Review. 5, 525-548. L i n c o l n , Y. S., (1990). The a r t s and s c i e n c e s of program e v a l u a t i o n . E v a l u a t i o n P r a c t i c e , 12., 1-7. L i n c o l n , Y. S., & Guba, E. G. (1985). N a t u r a l i s t i c i n q u i r y . B e v e r l y H i l l s : Sage P u b l i c a t i o n s . Lobosco, A. F., & Newman, D. L. (1992). Stakeholder i n f o r m a t i o n needs: I m p l i c a t i o n s f o r e v a l u a t i o n p r a c t i c e and p o l i c y development i n e a r l y c h i l d h o o d s p e c i a l e d u c a t i o n . E v a l u a t i o n Review, 16, 443-463. Maclure, R. (1990). The c h a l l e n g e of p a r t i c i p a t o r y r e s e a r c h and i t s i m p l i c a t i o n s f o r funding agencies. I n t e r n a t i o n a l J o u r n a l of S o c i o l o g y and S o c i a l P o l i c y , 10, 1-21. Madaus, G. F., Stuffelbeam, D. L., & S c r i v e n , M. S. (1983). Program e v a l u a t i o n : A h i s t o r i c a l overview. In G. F. Madaus, M. S. S c r i v e n , & D. L. Stuffelbeam (Eds.), E v a l u a t i o n Models (pp. 3-20). Boston, MA: K l u w e r - N i j h o f f P u b l i s h i n g . Malekoff, A. (1994). A c t i o n r e s e a r c h : An approach to p r e v e n t i n g substance abuse and promoting s o c i a l competency. H e a l t h and S o c i a l Work, 19, 46-53. Mark, M. M., & Shotland, R. L. (1985). Stakeholder-based e v a l u a t i o n s and v a l u e judgements. E v a l u a t i o n Review, 9., 626. Mathison, S. (1991). What do we know about i n t e r n a l E v a l u a t i o n and Program Planning, 14, 159-165.  605-  evaluation?  155 M c G a r r e l l , E. F., & Sabath, M. J . (1994). S t a k e h o l d e r c o n f l i c t i n an a l t e r n a t i v e s e n t e n c i n g program. E v a l u a t i o n and Program P l a n n i n g , 17, 179-186. McGrath, M., & Grant, G. (1992). Supporting 'needs-led s e r v i c e s : I m p l i c a t i o n s f o r p l a n n i n g and management systems (A case study of mental handicap s e r v i c e s ) . J o u r n a l of S o c i a l P o l i c y , 21 (1), 71-97. 1  Mies, M. (1983) . Towards a methodology f o r f e m i n i s t r e s e a r c h . In G. Bowles, & R. D u e l l i K l e i n (Eds.), T h e o r i e s o f women's s t u d i e s , (pp. 117-13 9). New York: Routeledge. M i l e s , M. B., & Huberman, A. M. (1984). Q u a l i t a t i v e d a t a a n a l y s i s . Newbury Park, CA: Sage P u b l i c a t i o n s . M o r r i s o n , C. (1988) . Consumerism and the p u b l i c s e c t o r : How does the c o a t f i t ? P u b l i c A d m i n i s t r a t i o n , 66., 125-147.  well  M o r r i s o n , J . K. (1978). The c l i e n t as consumer and e v a l u a t o r o f community mental h e a l t h s e r v i c e s . American J o u r n a l of Community Psychology, 6, 147-155. Mowbray, C. T. (1988). G e t t i n g the system to respond to e v a l u a t i o n f i n d i n g s . In J . A. McLaughlin, L. J . Weber, R. W. Covert, & R. B. I n g l e (Eds.), E v a l u a t i o n u t i l i z a t i o n (pp. 4758). San F r a n c i s c o : Jossey-Bass. Murphy, J . W., & P i l o t t a , J . J . (1983). Community based evaluation for criminal j u s t i c e planning. Social Service Review, 57, 465-476. Murray, C. A. (1983). S t a k e h o l d e r s as deck c h a i r s . In A. S. Bryk (Ed.), Stakeholder-based e v a l u a t i o n (pp. 59-61). San F r a n c i s c o : Jossey-Bass. O r e f i c e , P. (1988). P a r t i c i p a t o r y r e s e a r c h i n Southern Convergence, 11, 39-45.  Europe.  Owston, R. D. (1986). E s t a b l i s h i n g v a l i d i t y i n e v a l u a t i o n : The Canadian I n d i a n s c h o o l e v a l u a t i o n s . E v a l u a t i o n and Program P l a n n i n g , 9, 319-323. Palumbo, D. J . (1987). P o l i t i c s and e v a l u a t i o n . In D. J . Palumbo, (Ed.), The p o l i t i c s of program e v a l u a t i o n , (pp. 12-46). Newbury Park, CA: Sage P u b l i c a t i o n s . Palumbo, D. J . , & H a l l e t t , M. A. (1993). C o n f l i c t vesus consensus models i n p o l i c y e v a l u a t i o n and implementation. E v a l u a t i o n and Program P l a n n i n g , 16, 11-23.  156 Patton, M. Q. (1990). Q u a l i t a t i v e e v a l u a t i o n methods B e v e r l y H i l l s : Sage P u b l i c a t i o n s .  (2nd E d . ) .  Patton, M. Q. (1988a). The e v a l u a t o r ' s r e s p o n s i b i l i t y f o r u t i l i z a t i o n . E v a l u a t i o n P r a c t i c e , 9, 5-24. Patton, M. A. evaluator?  (1988b). How primary i s your i d e n t i t y as an E v a l u a t i o n P r a c t i c e , 9., 87-92.  Patton, M. Q. (1987). C r e a t i v e e v a l u a t i o n H i l l s : Sage P u b l i c a t i o n s .  (Rev. E d . ) . B e v e r l y  Patton, M. Q. (1986). U t i l i z a t i o n - f o c u s e d e v a l u a t i o n B e v e r l y H i l l s : Sage P u b l i c a t i o n s .  (2nd E d . ) .  Patton, M. Q. (1982). P r a c t i c a l e v a l u a t i o n . B e v e r l y H i l l s : Publications.  Sage  P i n t o , R., & F i e s t e r , A. (1979). Governing board and management s t a f f a t t i t u d e s toward community mental h e a l t h c e n t e r c i t i z e n p a r t i c i p a t i o n . Community Mental H e a l t h J o u r n a l , 15, 259-266. P o t t e r , J . (1988). Consumerism and the p u b l i c s e c t o r : How w e l l does the coat f i t ? P u b l i c A d m i n i s t r a t i o n , 66, 149-164. Prager, E. (1986). E v a l u a t i o n i n mental h e a l t h : E n t e r the consumer. S o c i a l Work Research and A b s t r a c t s , 16 (2), 5-10. Rawls, J . (1971). A theory of j u s t i c e . Cambridge, MA:  Belknap.  Reineke, R. A. (1991). Stakeholder involvement i n e v a l u a t i o n : Suggestions f o r p r a c t i c e . E v a l u a t i o n P r a c t i c e , 12., 39-44. Reineke, R. A., & Welch, W. W. (1986). C l i e n t c e n t e r e d e v a l u a t i o n . E v a l u a t i o n P r a c t i c e . 7, 16-24.  meta  Reinharz, S. (1983). E x p e r i e n t i a l a n a l y s i s : A c o n t r i b u t i o n to f e m i n i s t r e s e a r c h . In G. Bowles, & R. D u e l l e K l e i n (Eds.), T h e o r i e s o f women's s t u d i e s (pp. 162-192). New York: Routeldge. Reinharz, S., Bombyk, M., & Wright, J . (1983). M e t h o d o l o g i c a l issues i n feminist research: A bibliography of l i t e r a t u r e i n women's s t u d i e s , s o c i o l o g y and psychology. Women's S t u d i e s I n t e r n a t i o n a l Forum, 6_, 437-454. R o s s i , P. H. and Freeman, H. (1993). E v a l u a t i o n : A s y s t e m a t i c approach (5 t h E d ) . B e v e r l y H i l l s , CA: Sage P u b l i c a t i o n s . Routledge, R. (1993). Grass t i p consumer p o l i c y i n p u t . Development J o u r n a l , 28, 101-107.  Community  157 Rubin, A., & Babbie, E. (1989). Research methods f o r s o c i a l work. Belmont, CA: Wadsworth P u b l i s h i n g . Rudd, R. E. & A s s o c i a t e s . (1993). E v a l u a t i o n c o l l a b o r a t i o n a c r o s s d i v e r s e a i d s - r e l a t e d e d u c a t i o n a l programs. E v a l u a t i o n P r a c t i c e , 14, 243-251. S c r i v e n , M. (1994). The f i n e l i n e between e v a l u a t i o n e x p l a n a t i o n . E v a l u a t i o n P r a c t i c e , 15, 75-77.  and  S c r i v e n , M. (1983). E v a l u a t i o n I d e o l o g i e s . In G. F. Madaus, M. S. S c r i v e n , & D. L. Stuffelbeam (Eds.), E v a l u a t i o n Models (pp. 229-260) . Boston, MA: K l u w e r - N i j h o f f P u b l i s h i n g . S c r i v e n , M. Press.  (1980). The l o g i c of e v a l u a t i o n . Iverness,  CA:  Edge  S i e g e l , K., & T u c k e l , P. (1985). The u t i l i z a t i o n o f e v a l u a t i o n r e s e a r c h : A case a n a l y s i s . E v a l u a t i o n Review, _, 307-328. Shadish, W. R., Cook, T. D., & L e v i t o n , L. C. (1991). Foundations of program e v a l u a t i o n . Newbury Park, CA: Sage P u b l i c a t i o n s . Shadish, W. R., & E p t e i n , R. (1987). P a t t e r n s o f program e v a l u a t i o n p r a c t i c e among members of the e v a l u a t i o n r e s e a r c h s o c i e t y and e v a l u a t i o n network. E v a l u a t i o n Review, 11, 555590. Shapiro, J . Z., & B l a c k w e l l , D. L. (1987). Large s c a l e e v a l u a t i o n s on a l i m i t e d budget: The p a r t n e r s h i p e x p e r i e n c e . In J . Nowakowski (Ed.), The c l i e n t p e r s p e c t i v e on e v a l u a t i o n (pp. 53-62). San F r a n c i s c o : Jossey-Bass. Smith, J . K. (1988). The e v a l u a t o r / r e s e a r c h e r as person v s . the person as e v a l u a t o r / r e s e a r c h e r . E d u c a t i o n a l Researcher 17.(2), 18-23. Smith, N. L. (1985). Some c h a r a c t e r i s t i c s of moral problems i n e v a l u a t i o n p r a c t i c e . E v a l u a t i o n and Program Planning, 8, 5-11. Smith, N. L., & Chircop, S. (1987). Weiss-Patton debate: I l l u m i n a t i o n o f the fundamental concerns. E v a l u a t i o n P r a c t i c e , 10/ 5-13. Smith, M. F. (1994). E v a l u a t i o n : Review of the p a s t , preview of the f u t u r e . E v a l u a t i o n P r a c t i c e , 15, 215-227. Smith, M. K. (1988). E v a l u a t i o n u t i l i z a t i o n r e v i s i t e d . In J . A. McLaughlin, L. J . Weber, R. W. Covert, & R. B. I n g l e (Eds.), E v a l u a t i o n u t i l i z a t i o n (pp. 7-19). San F r a n c i s c o : Jossey-Bass.  158 Stake, R. (1986). Q u i e t i n g reform. Chicago: U n i v e r s i t y of I l l i n o i s Press. Stake, R. (1983a). Stakeholder i n f l u e n c e i n the e v a l u a t i o n of C i t i e s - i n Schools. In A. S. Bryk (Ed.), Stakeholder-based e v a l u a t i o n (pp. 15-30). San F r a n c i s c o : Jossey-Bass. Stake, R. (1983b). Program e v a l u a t i o n , p a r t i c u l a r l y r e s p o n s i v e evaluation. In G. F. Madaus, M. S. S c r i v e n , & D. L. S t u f f e l b e a m (Eds.), E v a l u a t i o n Models (pp. 287-310). Boston, MA: K l u w e r - N i j h o f f P u b l i s h i n g . Stake, R. (1981). Case study methodology: An e p i s t e m o l o g i c a l advocacy. In W. Welch (Ed)., Case study methodology i n e d u c a t i o n a l e v a l u a t i o n . M i n n e a p o l i s : Minnesota Research and E v a l u a t i o n Centre. S t r a u s s , A. L. (1987). Q u a l i t a t i v e a n a l y s i s f o r s o c i a l s c i e n t i s t s . New York: Cambridge U n i v e r s i t y P r e s s . S t r a u s s , A. L., & Corbin, J . (1990). B a s i c s of g u a l i t a t i v e data a n a l y s i s . Newbury Park, CA: Sage P u b l i c a t i o n s . Stuffelbeam, D. L., & Webster, J . (1980). An a n a l y s i s of a l t e r n a t i v e approaches to e v a l u a t i o n . E d u c a t i o n a l E v a l u a t i o n and P o l i c y A n a l y s i s , 3_, 5-19. Tandon, R. (1988). S o c i a l t r a n s f o r m a t i o n r e s e a r c h . Convergence, 11, 5-15.  and p a r t i c i p a t o r y  Tomm, W. (Ed.). (1987). The e f f e c t s of f e m i n i s t approaches on r e s e a r c h methodologies. Waterloo, ON: W i l f r e d L a u r i e r P r e s s . Tovar, M. (1989). Representing m u l t i p l e p e r s p e c t i v e s : Collaborative-democratic evaluation i n distance education. American J o u r n a l of D i s t a n c e Education, 3_ (2), 44-56.  The  Tower, K. D. (1994) . Consumer-centered s o c i a l work p r a c t i c e : R e s t o r i n g c l i e n t s e l f - d e t e r m i n a t i o n . S o c i a l Work, 39, 191-196. Trochim, W. M. K., & L i n t o n , R. (1986). C o n c e p t u a l i z a t i o n f o r p l a n n i n g and f o r e v a l u a t i o n . E v a l u a t i o n and Program Planning, 9, 289-308. Tronya, B., & F o s t e r , P. (1988). Conceptual and e t h i c a l dilemmas of c o l l a b o r a t i v e r e s e a r c h : R e f l e c t i o n s on a case study. E d u c a t i o n a l Review, 40, 289-300. Wagner, D. (1991). R e v i v i n g the a c t i o n - r e s e a r c h model: Combining case and cause w i t h d i s l o c a t e d workers. S o c i a l Work, 36., 477481.  159 W a l l e r s t e i n , N., & Martinez, L. (1994). Empowerment e v a l u a t i o n : A case study of an adolescent substance abuse p r e v e n t i o n program i n New Mexico. E v a l u a t i o n P r a c t i c e , 15, 131-138. W a l l s t o n , B. S. (1981). What are the q u e s t i o n s i n the psychology of women? A f e m i n i s t approach to r e s e a r c h . Psychology of Women Q u a r t e r l y , 5, 597-617. Weiss, C. H. (1988a). E v a l u a t i n g f o r d e c i s i o n s . Is anybody Does anybody care? E v a l u a t i o n P r a c t i c e , 9_, 5-19.  there?  Weiss, C. H. (1988b). I f program d e c i s i o n s hinged o n l y on i n f o r m a t i o n . A response to Patton. E v a l u a t i o n P r a c t i c e , 9., 28.  15-  Weiss, C. H. (1983a). The s t a k e h o l d e r approach to e v a l u a t i o n : O r i g i n s and promise. In A. S. Bryk (Ed.), Stakeholder-based e v a l u a t i o n (pp. 3-14). San F r a n c i s c o : Jossey-Bass. Weiss, C. H. (1983b). Toward the f u t u r e of s t a k e h o l d e r approaches i n e v a l u a t i o n . In A. S. Bryk (Ed.), Stakeholder-based e v a l u a t i o n (pp. 83-96). San F r a n c i s c o : Jossey-Bass. Weiss, C. H. (1981). Doing s c i e n c e or doing p o l i c y ? [Review of Toward reform of program e v a l u a t i o n ] . E v a l u a t i o n and Program Planning, 4, 397-405. Weiss, C. H. (1972). E v a l u a t i o n r e s e a r c h : Methods f o r a s s e s s i n g program e f f e c t i v e n e s s . Newbury Park, CA: Sage P u b l i c a t i o n s . Whitaker, G. P. (1974). Who puts the v a l u e i n e v a l u a t i o n ? Science Q u a r t e r l y , 54, 759-761.  Social  White, I. (1988). Consumer i n f l u e n c e s : Challenges f o r the f u t u r e . In I . A l l e n (Ed.), Hearing the v o i c e of the consumer, (pp. 112). London: P o l i c y S t u d i e s I n s t i t u t e . Whitmore, E. (1990). Empowerment i n program e v a l u a t i o n : A case example. Canadian S o c i a l Work Review, 7, 215-229 Wholey, J . S. (1986). Managing f o r h i g h performance: The r o l e of e v a l u a t i o n . E v a l u a t i o n S t u d i e s Review Annual, 11, 194-204. W i l l i a m s , J . E. (1989) . A n u m e r i c a l l y developed taxomony of e v a l u a t i o n theory and p r a c t i c e . E v a l u a t i o n Review, 13, 18-31. Wortman, P. M. (1983). E v a l u a t i o n r e s e a r c h : A m e t h o d o l o g i c a l p e r s p e c t i v e . Annual Review of Psychology. 34, 223-260. Zinober, J . W., D i n k e l , N. R., & Windle, C. (1984). N o n s p e c i a l i s t e v a l u a t i o n of mental h e a l t h agencies. A d m i n i s t r a t i o n i n Mental H e a l t h . 11, 223-232.  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0086688/manifest

Comment

Related Items