Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

An assessment of illuminative evaluation as an approach to evaluating residential adult education programs Hasman, Ruth Margaret Reiner 1982

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-UBC_1983_A8 H38.pdf [ 5.11MB ]
Metadata
JSON: 831-1.0055871.json
JSON-LD: 831-1.0055871-ld.json
RDF/XML (Pretty): 831-1.0055871-rdf.xml
RDF/JSON: 831-1.0055871-rdf.json
Turtle: 831-1.0055871-turtle.txt
N-Triples: 831-1.0055871-rdf-ntriples.txt
Original Record: 831-1.0055871-source.json
Full Text
831-1.0055871-fulltext.txt
Citation
831-1.0055871.ris

Full Text

C.J  An Assessment of I l l u m i n a t i v e E v a l u a t i o n as an Approach to E v a l u a t i n g R e s i d e n t i a l Adult Education  Programs  by Ruth Margaret Reiner Hasman B.A., San Jose State C o l l e g e , 1967  A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE  REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS in  THE  FACULTY OF GRADUATE STUDIES  (Department of A d m i n i s t r a t i v e , Adult and Higher Education)  We accept  t h i s t h e s i s as conforming  to the r e q u i r e d  THE  standard  UNIVERSITY OF BRITISH COLUMBIA December 1982  © Ruth Margaret Reiner  Hasman, 1982  In p r e s e n t i n g  t h i s t h e s i s i n p a r t i a l f u l f i l m e n t o f the  requirements f o r an advanced degree a t the U n i v e r s i t y o f B r i t i s h Columbia, I agree t h a t the L i b r a r y s h a l l make it  f r e e l y a v a i l a b l e f o r reference  and study.  I further  agree t h a t p e r m i s s i o n f o r e x t e n s i v e copying o f t h i s t h e s i s f o r s c h o l a r l y purposes may be granted by the head o f my department o r by h i s o r her r e p r e s e n t a t i v e s .  It is  understood t h a t copying o r p u b l i c a t i o n o f t h i s t h e s i s f o r f i n a n c i a l gain  s h a l l not be allowed without my  permission.  Department o f  /AcLoJL£  Izdx^al^j  The U n i v e r s i t y o f B r i t i s h Columbia 1956 Main Mall Vancouver, Canada V6T 1Y3  D  a  t  e  o2<» I9?g. ;  written  i i  ABSTRACT The purpose of t h i s study was to t e s t illuminative  evaluation  value of r e s i d e n t i a l  as  a  adult  the  suitability  methodology f o r determining the  education  programs.  Illuminative  e v a l u a t i o n methodology was s e l e c t e d f o r s e v e r a l reasons. the  methodology  functioned  Second, i t p e r m i t t e d  the  developing  Third,  program.  spontaneous events.  independently  flexibility  been  those  reasons,  suitability A  (Miles,  i t seemed  to  program.  evaluate  a  a means of studying  for  r e p r e s e n t a t i o n of  few s t u d i e s of t h i s methodology  1981; P a r l e t t & King, 1971). important  residential  program  was  to  determined  investigate  For the  chief  advantage  traditional  types  temporarily  from  of  was  the  that  i n other program  of  h i s ongoing  which  is  formats.  r e s i d e n t i a l format over the more removing  the  responsibilities.  p o s s i b l e f o r the i n v e s t i g a t o r t o have participants  to be p a r t i c u l a r l y  i l l u m i n a t i v e e v a l u a t i o n because i t had some  unique advantages that d i d not e x i s t  the  the  First,  of i l l u m i n a t i v e e v a l u a t i o n .  suitable for testing  The  needed  Fourth, i t allowed  undertaken  of  i t provided  m u l t i p l e v i e w p o i n t s , and l a s t l y , had  of  continuous  important  participant T h i s made i t contact  f o r a methodology  with that  r e l i e s on f i e l d w o r k techniques. In t h i s applied  to  study, the  illuminative  evaluation  of  evaluation a  residential  J u s t i c e I n s t i t u t e of B r i t i s h Columbia. suitability frequently  of  the  methodology,  i n the l i t e r a t u r e were  methodology  In three  judged  order  was  program at the to  criteria appropriate  test  the  appearing to  this  study—technical literature  suggested  technically and  adequacy,  sound  that  utility an  information  and  efficiency.  evaluation that  should  The  produce  i s u s e f u l to some audience  i s worth more to the audience than i t c o s t s  (Grotelueschen,  1980). Evidence of the degree to which i l l u m i n a t i v e e v a l u a t i o n these  criteria  such as to  was  during  interviews, questionnaires,  collect  the  evidence.  q u a n t i t a t i v e and methodology met The  the  The  and  criteria  Although i t was  Techniques  observations  were  showed  was  used  analyzed  using  to determine whether  the standards set by the collected  the program.  evidence  q u a l i t a t i v e techniques  evidence  satisfied utility.  collected  met  the  criteria.  that  this  methodology  requirements of t e c h n i c a l adequacy  and  weak on the e f f i c i e n c y c r i t e r i o n ,  the  methodology compensated with p a r t i c u l a r s t r e n g t h s  in u t i l i t y  and  t e c h n i c a l adequacy. For  further  research,  there are a whole host  areas that i l l u m i n a t i v e e v a l u a t i o n opens up. to  be  done  to  develop  specific  tasks,  of p o s s i b l e  Further  work needs  questions,  and/or  procedures which c o u l d guide implementation of each stage of illuminative  evaluation  methodology.  Further  done to c o n t r i b u t e to the understanding of the studies  could  methodology formats.  for  be  done  to  evaluating  determine other  the  adult  studies could methodology  s u i t a b i l i t y of education  the be and the  program  iv  TABLE OF CONTENTS  ABSTRACT  i i  LIST OF TABLES  v  ACKNOWLEDGEMENT  vii  CHAPTER I: INTRODUCTION  1  The Problem  4  Research Approach  5  Summary  8  CHAPTER I I : LITERATURE REVIEW  9  H i s t o r i c a l Emergence Of E v a l u a t i o n Social  Sciences  9 11  Education  12  C l a s s i c a l Versus N a t u r a l i s t i c  Paradigm  13  C l a s s i c a l Paradigm  18  Naturalistic  24  Paradigm  Illuminative Evaluation Summary  26 .  CHAPTER I I I : METHODOLOGY Criteria  30 32 33  T e c h n i c a l Adequacy  33  Utility  37  Efficiency  40  Study S i t e  43  Summary  47  CHAPTER IV: ILLUMINATIVE EVALUATION STRATEGY  49  Pilot  Phase  50  Issues C l a r i f i e d  51  E v a l u a t i o n Process  51  Q u e s t i o n n a i r e Design  53  Data C o l l e c t i o n  56  O p e r a t i o n a l Program  61  Summary  61  CHAPTER V: RESULTS  63  T e c h n i c a l Adequacy  63  Utility  69  Efficiency  78  Summary  82  CHAPTER  VI:  SUMMARY,  CONCLUSIONS,  IMPLICATIONS  AND  RECOMMENDATIONS  85  Summary  85  Conclusions  87  T e c h n i c a l Adequacy  88  Utility  90  Efficiency  92  I m p l i c a t i o n s And Recommendations REFERENCES REFERENCE NOTES  95 99 ...106  APPENDIX A: E x p e c t a t i o n s Q u e s t i o n n a i r e  107  APPENDIX B: M i n i - S e s s i o n Q u e s t i o n n a i r e  110  APPENDIX C: F i n a l Q u e s t i o n n a i r e  112  APPENDIX D: Follow-Up Q u e s t i o n n a i r e  116  APPENDIX E: Revised Q u e s t i o n n a i r e s  119  vi  LIST OF TABLES  1: Some  Basic  Naturalistic  Differences  Between  Classical  And  Paradigms  18  2: I l l u m i n a t i v e E v a l u a t i o n Stages  29  3: L i s t Of E v a l u a t i o n Costs  42  4: Data C o l l e c t i o n Schedule  57  5: E f f o r t  79  And Time Spent On I l l u m i n a t i v e Stages  6: C r i t e r i a  And Standards Used For Determining S u i t a b i l i t y  Of I l l u m i n a t i v e E v a l u a t i o n Methodology  84  vii  ACKNOWLEDGEMENT The thesis  support and d i r e c t i o n p r o v i d e d by committee  research. of  Dr.  served  as  the members  the m o t i v a t i o n  of  t o complete  my this  I g r a t e f u l l y acknowledge the wisdom and encouragement  T.  thank Dr.  J.  Sork and Dr.  J . G.  J.  E.  Thornton.  Dickinson for h i s i n i t i a l  In  addition  guidance  I  i n getting  t h i s study underway. I  would l i k e a l s o to thank Mr.  Land T i t l e s programs  f o r a l l o w i n g me t o e v a l u a t e h i s  and Mr.  Land  Title  School  Paul Dampier, Program D i r e c t o r at the J u s t i c e  I n s t i t u t e f o r h i s h e l p and guidance this  Henry Kennedy, D i r e c t o r of  i n the  initial  stages  of  project. To Annette Buckmaster, a very s p e c i a l  thanks  f o r her u n f a i l i n g moral  my l a s t year. without  f r i e n d , I owe s i n c e r e  support and encouragement during  S p e c i a l thanks a l s o go t o  MarDell  Parrish, for  h i s h e l p i n word p r o c e s s i n g , t h i s t h e s i s might s t i l l be  on the computer. Throughout the l a s t in  f i v e years many of my  fellow  students  a d u l t education provided encouragement, advice and h e l p .  To  each of them I extend thanks and a p p r e c i a t i o n . Finally, who empathized who  kept  cajolement, have been  I would l i k e to express my s i n c e r e thanks with my " t h e s i s phases"  saying  and t o  Janet  "Are you f i n i s h e d y e t , Mommy?"  support and encouragement, t h i s completed.  t o David  and  Brent  Without  their  project  would  not  1  CHAPTER I  INTRODUCTION  The purpose of t h i s study was to t e s t illuminative value  of  evaluation  residential  specifically  As the numbers grown,  new  methods  a  adult  focused  strengths/weaknesses  as  on  the  educational  education the  programs.  in  study and  of t h i s methodology. of  people  and  seeking  adult  education  techniques have developed  developments  has  to meet the  One of the  been  have  fastest  use of short There  are  term few  education and b u s i n e s s today who have not attended a  r e s i d e n t i a l course, conference, These  The  advantages/disadvantages,  r e s i d e n t i a l group l e a r n i n g programs f o r a d u l t s . people  of  methodology f o r determining the  i n c r e a s e d demand (Apps, 1979; Houle, 1971). growing  suitability  programs  provide  seminar,  participants  c o l l o q u i u m or workshop. with  a  concentrated  experience, a change i n environment and an o p p o r t u n i t y f o r c l o s e interaction 1969;  and  mutual  Houle, 1971; M i l l e r ,  problem-solving 1964;  Schacht,  with  peers ( G a r s i d e ,  1960).  Because  of  t h e i r c o n c e n t r a t e d nature, the programs are capable of p r o v i d i n g  2  an experience with a powerful To  enhance  planners  or  impact.  the r e s i d e n t i a l program's c a p a c i t y f o r impact, evaluators  systematically,  should  collecting  information  can  be  used  ineffective  procedures,  evaluate  the  program  i n f o r m a t i o n from many sources. to  improve  and a s s i s t  effectiveness,  i n d e s i g n i n g both  a c t i v i t i e s and f u t u r e programs (Beckhard,  This modify  follow-up  1956).  Since program e v a l u a t i o n i n education i s i n the e a r l y of  stage  theory development, i t has become an area of intense academic  interest. and  As might be expected  terms  have  been  a plethora  created  by  of  those  divergent  trying  views  to describe,  analyze, e x p l a i n , t h e o r i z e , or otherwise capture the essence evaluation by  (Rusnell,  academics,  enthusiastic.  Note 1). D e s p i t e i n t e r e s t  practitioners "Among  in  theorists the  field  i n the process  have  been  less  e v a l u a t i o n i s one of the most  h o t l y debated  activities  practitioners  i t i s one of the most ignored" (Davis & McCallon,  1974,  in  the  of  educational  process;  among  p. 271). The  foremost  evaluation  reason  f o r reduced  i s the lack of guidance  the l i t e r a t u r e .  In any new f i e l d ,  experts through  their  program  evaluation  literature,  has  enthusiasm  regarding  provided to p r a c t i t i o n e r s by guidance but  i s expected  the  from the  literature  served more to confuse  about  than to guide.  Worthen (1974) noted that e v a l u a t i o n l i t e r a t u r e i s badly fragmented into unrelated pieces and is as d i f f i c u l t to s y n t h e s i z e as i t i s to make a meaningful picture from a random h a n d f u l of p i e c e s to a jigsaw puzzle. Looking at the i n d i v i d u a l pieces i s  3  little more h e l p f u l , for the l e v e l of d i s c o u r s e in i n d i v i d u a l w r i t i n g s i s o f t e n aimed at fellow evaluation t h e o r i s t s more than at schoolmen, thereby communicating a great deal of detail about a t o p i c which l a c k s a l a r g e r context w i t h i n which i t could be u s e f u l . Working under t h i s handicap, busy p r a c t i t i o n e r s can hardly be faulted f o r not expending the necessary time to try to develop a c l e a r p i c t u r e from the current e v a l u a t i o n l i t e r a t u r e (p. 2). A which  second  reason  definition  evaluation  has  to  is  that  use.  e v a l u a t o r s have the problem of  During  its  development,  program  come to have many d i f f e r e n t d e f i n i t i o n s .  d e f i n i t i o n s are d e r i v e d l a r g e l y  from  the  emphasis  These  placed  on  q u a n t i t a t i v e versus q u a l i t a t i v e s t u d i e s . Program  evaluation,  according  (1977), should g i v e i n d i v i d u a l s and their  mutual  growth  optimally.  It  information  from  and  should many  e f f e c t i v e n e s s , to modify serve  (1966)  Blackwell  systems  be  a  "systematic"  sources  part  of  the  (Steele,  to improve  procedures  where  necessary,  of  planning and  to  f u t u r e programs (Beckhard, 1956).  suggested  that  evaluations  total  program and  should  be  should c o n s t i t u t e an  from beginning  to  not done j u s t f o r i t s own  tries  is  likely  suits to f a l l  every  situation,  evaluation  (Stake,  1979).  mentioned i n the l i t e r a t u r e , two  a  end. sake  definition  short in numerous ways.  such odds e v a l u a t o r s u s u a l l y withdraw to t h e i r of  function  1970).  Since no d e f i n i t i o n that  over  collection  order  E v a l u a t i o n must be p u r p o s e f u l  Bolman  control  in  planned at the same time as the program and integral  some  and  development so that they can  as a guide in planning  Bass & Vaughan  to  From  own  Against  definitions  the numerous d e f i n i t i o n s  seem a p p r o p r i a t e  to t h i s  study:  4  By the term e v a l u a t i o n , we mean systematic examination of events o c c u r r i n g i n and consequent on a contemporary program--an examination conducted to a s s i s t i n improving t h i s program and other programs having the same general purpose (Cronbach, 1980, p. 14). E v a l u a t i o n i s a c o l l e c t i o n of methods, s k i l l s and sensitivities necessary to determine whether a human s e r v i c e i s needed and l i k e l y to be used, whether i t i s conducted as planned, and whether the human s e r v i c e a c t u a l l y does h e l p people i n need. While doing these tasks e v a l u a t o r s a l s o seek ways to improve programs (Posavac & Carey, 1980, p. 6 ) .  The Problem  Although  more  than  developed s i n c e T y l e r ' s are are  still  a hundred e v a l u a t i o n models have been objectives-centered  l o o k i n g f o r a l t e r n a t i v e models  seeking  new  ways  "Producing data i s one t h i n g ! 412).  Thus,  methodology i s q u i t e  evaluators  (Cronbach, 1980).  They  to evaluate programs as w e l l as ways t o  improve u t i l i z a t i o n of r e s u l t s .  (p.  model,  House (1972) put i t t h i s Getting  i t appears that  way:  i t used i s q u i t e another" i d e n t i f y i n g an a p p r o p r i a t e  significant.  Since programs evolve and change over time, a l t e r a t i o n s f o r improvement may need to be made during the program. a  criterion  of an e v a l u a t i o n methodology i s that i t should not  r e q u i r e the program to stand s t i l l be  evaluated  Therefore,  (Katz  &  Morgan,  words, the methodology should  be  or stay the same i n order 1974;  Stake, 1978).  independent  of  the  to  In other program  5  being  evaluated.  requirement 1977),  The  methodology i d e n t i f i e d that met  is "illuminative evaluation"  for  it  involves  interfering,  manipulating  Illuminative  evaluation  reasons.  First,  program. a  studying  it  the  restricting  methodology  program.  spontaneous  was  the  1971).  For  the  been undertaken  activities.  independently  it  it  of  the  needed  to  a means of  allowed  last,  (Miles,  reasons,  without  selected for several  Fourth, and  Hamilton,  program  T h i r d , i t provided  events.  those  &  flexibility  of m u l t i p l e viewpoints,  t h i s methodology had King,  or  permitted  developing  representation  examining  the methodology f u n c t i o n e d  Second,  evaluate  (Parlett  the above  few  1981;  seemed  for  s t u d i e s of Parlett  important  & to  i n v e s t i g a t e the s u i t a b i l i t y of i l l u m i n a t i v e e v a l u a t i o n .  Research Approach  Within be  found:  strategies, Most classical  educational the  classical  f o c i and formal  and  naturalistic.  educational which  psychology.  evaluation derives  These s t u d i e s assess  standards on p r e - s p e c i f i e d c r i t e r i a . to  statistically  paradigms Each has  yield analyzed.  studies  its  of a program by examining whether or not  designed  distinct  can  i t s own  assumptions.  paradigm  experimental  e v a l u a t i o n , two  objective  i t has  use  the  methodology  from  the e f f e c t i v e n e s s reached  Studies of t h i s  numerical  data  that  required kind can  are be  6  R e c e n t l y , however, there has been i n c r e a s i n g r e s i s t a n c e e v a l u a t i o n s of t h i s type ( P a r l e t t & Hamilton, Stake,  1978).  related  to  There  social  fundamentally  is a  different  1976; Smith, 1976;  movement to use a second  anthropology.  This  evaluation  with the c l a s s i c a l paradigm.  to  paradigm  paradigm  requires  methodology from that  a  used  These two paradigms are d i s c u s s e d  thoroughly i n Chapter I I . Frequently  evaluations  i n v o l v e a case study methodology  of  a  based on the n a t u r a l i s t i c program  a c c o r d i n g to Stake  or  project.  (1978) has f a l l e n  paradigm  Case  into disrepute  among s o c i a l s c i e n t i s t s ; however, he suggests that case are  still  needed i n c e r t a i n types of e v a l u a t i o n s .  when the e v a l u a t i o n program, not j u s t than  i s aimed  at  improvement  when the i n f o r m a t i o n c o l l e c t e d s c i e n t i s t s , when the concern  broad  generalizations,  i d e n t i f i e s unique invaluable The  then  characteristics  (Patton,  of  a and  idiosyncracies  tested  in  this  study,  It  i s not  the  ( P a r l e t t & Hamilton,  1978) .  that  can  be  process  operations This  illuminative naturalistic  which  i s not  1976). tied  treatment, predetermined g o a l s or outcomes, but  (Patton,  rather  a standard methodological package but a  general r e s e a r c h s t r a t e g y  actual  specific  case study approach  paradigm.  the  a  i s for individuals  i s r e l a t i v e l y new and i s based on  on  For example,  1978).  methodology  evaluation  studies  i s f o r p a r t i c i p a n t s and  evaluation,  dynamic  study  of  process  It  is a  to  a single  rather  focuses  a program over a p e r i o d of time requires  sensitivity  to  both  q u a l i t a t i v e and q u a n t i t a t i v e changes i n a program throughout i t s  7  development,  not  just  illuminative evaluation the  strategies  extremely  used  at  is built are  important  some  in  end-point  not,  and  as  adaptable program  planners  time.  on d i v e r s i t y and and  Since  adaptability,  eclectic.  evaluation,  programs are o f t e n changed as planners does  in  This  for  innovative  l e a r n what works and  experiment  and  is  change  what their  priorities. In t h i s  study,  applied  to  Justice  Institute  described  in  suitability, certain  the  illuminative  evaluation of  British  Chapter  III.  illuminative  criteria.  of  The  evaluation  a  residential  Columbia. In  evaluation  (2)  useful  audience than i t c o s t s use  descriptions  is  determine  its  methodology  should  that i s : (1)  some audience and (Grotelueschen,  the above as c r i t e r i a  Complete  to  of  meet  evaluation technically  (3) worth more to  1980).  I t was  for t e s t i n g i l l u m i n a t i v e these  the  program  l i t e r a t u r e suggests t h a t an  to  was  program at  This  order  methodology should produce i n f o r m a t i o n sound,  methodology  c r i t e r i a are  decided  the to  evaluation.  found in Chapter  III. Evidence o f the degree to which i l l u m i n a t i v e e v a l u a t i o n these c r i t e r i a such  as  to c o l l e c t  was  collected  during  interviews, questionnaires, the  evidence.  The  the and  observations  evidence  q u a l i t a t i v e and q u a n t i t a t i v e techniques methodology met  program.  the standards set by the  was  Techniques were used  analyzed  using  to determine whether criteria.  met  the  8  Summary  Chapter statement description thesis review  I  of  provided  purpose,  of  the  selected  a  background  statement  research  i s organized i n t o of  general  five  of  approach. chapters  the  as  well  problem  and  appendices.  l i t e r a t u r e appears i n Chapter I I .  operationalization  of  the  illuminative  i n Chapter V.  Chapter VI i n c l u d e s a  the  chapters  conclusions  f indings.  and  a  The  Chapter contains  strategy.  r e s u l t s appear previous  and  a  The remainder of t h i s  III p r o v i d e s the r e s e a r c h methodology while Chapter IV the  as  summary  The of  based on the r e s e a r c h  9  CHAPTER II  LITERATURE REVIEW  The review of the brief  description  literature  of  the  presented  historical  Then  naturalistic  the  review  paradigms  used  contains  in s o c i a l  i s directed  to  s c i e n c e s and  classical  in evaluation studies.  emerged  Emergence of E v a l u a t i o n  in  natural science established overturning has  traditional  developed  Cronbach's  model,  study.  Historical  Evaluation  and  The f i n a l  s e c t i o n c o n t a i n s a review of the i l l u m i n a t i v e e v a l u a t i o n the focus of t h i s  a  emergence of e v a l u a t i o n  followed by the development of e v a l u a t i o n education.  here  in  (1980)  a  the  1600's  1980) when  i t s e l f as a powerful instrument  beliefs.  variety review  (Cronbach,  Since i t s e a r l y beginnings i t  of of  for  ways  the  in  various  historical  fields.  emergence  of  10  evaluation human  "reminds us that a p p l i e d s o c i a l r e s e a r c h ,  endeavors,  s p u r t s and  slumps and  As one number  develops  reviews  of  not  in  a  substantive  current  (p.  evaluation  areas--education,  of the  field,  evaluation  design,  value  of  formative  influenced Evaluation academic outlooks  evaluation  the  methodologies  s t u d i e s have and  and  (Freeman  Solomon,  assumptions  critical  s c r u t i n y and  from these new Because  a  been  from  (Guttentag  of  of  each  i s reviewed below.  the development of e v a l u a t i o n  sciences. diverse political researchers  evaluation's  discipline,  have been subjected revisions  & Saar,  evaluation  and to  resulting  1977).  i n f l u e n c e on e d u c a t i o n a l of  strongly  the  evaluation  them,  have b e n e f i t e d  development  the  data.  have  Because  underlie  observation,  i d e o l o g i c a l and  the methods of  of i t s strong  of  reflection  1981).  perspectives  the h i s t o r y of the sciences  which  techniques,  of a l l the s o c i a l  past career commitments of  i n t e r d i s c i p l i n a r y nature, the  become  studies  professional identities,  &  occurs.  the use of o b j e c t i v e s , the  utilization  years,  by  a  evaluation,  collection  the value of  evaluation,  of long-term s t u d i e s , and Over the  in  community  pattern  " r o l e s " of  measurement and  the n e u t r a l i t y of the e v a l u a t o r , function  literature  the same i s s u e s or concerns reappear.  For example, common concerns i n c l u d e the the  in  23).  training,  a c t i o n , h e a l t h , psychotherapy--an i n t e r e s t i n g Regardless  other  steady expansion but  changes of d i r e c t i o n " the  like  in  evaluation, the  social  T h i s i s followed by a d i s c u s s i o n of in the f i e l d of  education.  11  Social  Sciences  In  the  emphasis. an  1930's,  social  Psychologists  experimental  science  research  changed  were beginning to undertake s t u d i e s  character  in  and  out  of the l a b o r a t o r y .  example: Newcomb's (1943) study of a t t i t u d e change at  Bennington  1930's  C o l l e g e , L i p p i t t and White's study of the  impact  democratic l e a d e r s h i p s t y l e s on (Lippitt,  and  Kurt  Lewin and  influence  undertaken  1940's.  Then,  there  the monumental a p p l i e d -  soldiers  out  during  Stouffer  World  Electric  Studies  of  Effect"  to  vocabulary  the  by  the  was  War  1930's  II and  that  of  and  during  his  social  program c a r r i e d  American  1940),  children's  s t u d i e s on  and  research  For girls  relationships  associates'  of  among  of a u t h o r i t a r i a n and group  its  associates  the  science  many  social  on  famous Western  contributed  social  the  "Hawthorne  (Bernstein  &  Freeman,1975). In  the  intervention  1950's efforts  and  1960's  were  s c r u t i n i z e d and  scientists--deliquency-prevention e f f o r t s , psychotherapeutic  and  p u b l i c housing p r o j e c t s and However, i t was during that  the  not u n t i l  sincerity  or  psychopharmacological  community o r g a n i z a t i o n  the massive U.S.  began  political  evaluated  to  head  by  and social  programs, p e n a l - r e h a b i l i t a t i o n  "Great S o c i e t y " programs d u r i n g  accountability  action  mean  treatments, activities.  f e d e r a l expenditures the  1960's and  1970's  more than a s s e s s i n g  counting  of  staff  opponents  and  proponents. In  the  1970's,  evaluation  During that time, e v a l u a t i o n s  were  emerged as a p o l i t i c a l regularly  required  tool. of  all  12  health,  education,  and  welfare programs.  e v a l u a t i o n was a p o l i t i c a l  The requirement f o r  response to the p e r c e i v e d demand  for  i n c r e a s e d governmental a c c o u n t a b i l i t y .  Educat ion The  f i e l d of education  evaluation  of  curricula,  and m a t e r i a l s . unique other  i n s t r u c t i o n , programs, p a r t i c i p a n t s ,  T h i s f i e l d has tended to c o n s i d e r  and i t s methods s p e c i a l and d i f f e r e n t kinds of programs.  followed the  t r a d i t i o n a l l y has had an i n t e r e s t i n  from e v a l u a t i o n of  However, as e d u c a t i o n a l e v a l u a t i o n has  i n n o v a t i v e programming beyond the classroom  social  issues  indistinguishable interventions.  of  from  the  day,  evaluation  of  i t has other  to  involve  become  almost  planned  social  Weiss (1972) s a i d : " E d u c a t i o n a l e v a l u a t o r s have  much to l e a r n from--and to teach--those they have much to loose by developing special  i t s problems  vocabulary  of experience"  (p.  in  other  that i n h i b i t s communication and 13).  Pooling  a  relatively  and  s p e c i a l p e r s p e c t i v e s and a  information  inactive  period  interchange  benefits  f a c i n g s i m i l a r problems a c r o s s the range of program Following  fields,  in  those  areas. the 1950's,  development of e d u c a t i o n a l e v a l u a t i o n theory was r e v i t a l i z e d the  mid 1960's.  (1963), S c r i v e n The  in  T h i s r e v i t a l i z a t i o n was i n f l u e n c e d by Cronbach (1967), Stake  (1967)  and  Stufflebeam  (1967).  f i e l d ' s development was f u r t h e r s t i m u l a t e d by the e v a l u a t i o n  requirements  of  U.S.  federal  1965,  and by the U.S.  early  1970's (Stufflebeam  education  programs launched i n  a c c o u n t a b i l i t y movement that began i n the & Webster, 1981).  13  This brief two  sketch  distinct  fields.  e v a l u a t i v e methods i n direction.  i l l u m i n a t e s the growth of  these  development  in  the  fields  Demands from funding  towards a c c o u n t a b i l i t y .  occurred.  Although  of  the  independently,  have  in  moved  agencies have helped  and  trend the  national  has  projects  inadequate to d e a l  issues  posed  by  with  these  the  projects  1963).  C l a s s i c a l vs.  What, studies?  the  found that e v a l u a t i o n approaches based on  questions  (Cronbach,  same  f i e l d , a q u a l i t a t i v e change  the c l a s s i c a l paradigm were simply evaluation  the  been c o n t i n u i t y in  With the emergence of l a r g e s c a l e  1960's, i t was  in  developing  While there has  evaluation  evaluation  then, The  are  Naturalistic  the  options  available  l i t e r a t u r e r e v e a l s two  guide e v a l u a t i o n s ;  they  are  the  Paradigm  for  evaluative  paradigms that are used to  classical  and  naturalistic  paradigms. The  classical  experimentation experimental  paradigm  comes  from  techniques  most  m u l t i v a r i a t e , parametric By way  in  the  widely  experimental  of  of  This design  s t a t i s t i c a l analysis.  anthropology  interviewing  of  basic  used in e v a l u a t i o n .  of c o n t r a s t , the n a t u r a l i s t i c paradigm has  fields  techniques  tradition  i n a g r i c u l t u r e , which gave us many of the  paradigm assumes q u a n t i t a t i v e measurement, and  the  and  and  ethnography.  personal  i t s roots Using  observation,  the this  14  paradigm  relies  on  d e r i v e d from c l o s e classical  q u a l i t a t i v e data, and d e t a i l e d  contact  paradigm  aims  with  the  target  of  description study.  The  at p r e d i c t i o n of phenomena, while the  n a t u r a l i s t i c paradigm aims at understanding  phenomena  (Patton,  1978) . Which  of  these  p r o v i d e s best guidance no  definitive  paradigms—classical f o r an e v a l u a t i o n ?  answer  to  that  or  naturalistic--  "There i s of  question....The  course  choice between  paradigms i n any- i n q u i r y or e v a l u a t i o n ought to be made basis  of  the  best  f i t between  the  phenomenon being s t u d i e d or e v a l u a t e d "  on the  assumptions.... and  (Guba & L i n c o l n ,  the  1981, p.  56) . Although classical  the  literature  has  that the two s i d e s " . . . w i l l debating  the  paradigms....[E]ach less of  the c r i t e r i a  neither  neither  merits  of  paradigm w i l l be shown to  each  their satisfy  more  that i t d i c t a t e s f o r i t s e l f and to f a l l  paradigm  by solves  i t s opponent"  (p.  a l l problems,  e v a l u a t o r should s e l e c t the paradigm and  or  short  109-110).  they should be  the methodology  The that  the type of program being e v a l u a t e d and the nature of the  e v a l u a t i o n q u e s t i o n s , f o r paradigms only t e l l to  other  respective  viewed as a l t e r n a t i v e s from which the e v a l u a t o r can choose.  suits  the  Kuhn (1970) has p o i n t e d out  i n e v i t a b l y t a l k through  relative  a few of those d i c t a t e d  Since  that  nor the n a t u r a l i s t i c paradigm i s i n t r i n s i c a l l y b e t t e r  than the other, the debate goes on.  when  shown  emphasize,  researchers  what  what t o look f o r , what q u e s t i o n s to be concerned  with, and what standards  to  apply.  In  order  to  make  those  15  choices,  i t i s necessary to be aware of the assumptions of each.  Although  these two paradigms d i f f e r on a number of assumptions,  the d i s c u s s i o n  below w i l l be l i m i t e d to nine  (Guba & L i n c o l n ,  the  assumptions  1981).  Philosophical between  major  base.  two  Bogdan and Taylor  relevant  (1975) d i f f e r e n t i a t e  philosophical  perspectives.  "One,  p o s i t i v i s m . . . s e e k s the f a c t s or causes of s o c i a l phenomena little  regard  for  the s u b j e c t i v e  states  second, phenomenology, " i s concerned behavior  from  the  actor's  own  of i n d i v i d u a l s . "  with  understanding  frame of r e f e r e n c e .  p o s i t i v i s t s and the phenomenologists approach d i f f e r e n t and seek d i f f e r e n t answers, different  methodologies"  investigator,  a  2).  phenomenologist,  and understanding investigator,  t h e i r research w i l l (p.  of  social  a positivist,  with  Thus,  The human  Since the problems  t y p i c a l l y demand  the  naturalistic  i s concerned with d e s c r i p t i o n  phenomena,  while  the  classical  i s concerned with " s c i e n t i f i c "  facts  and t h e i r r e l a t i o n s h i p to one another. Inquiry  paradigm.  A  second  difference  approaches can be found i n the guiding investigator, world as  with  composed  his of  positivist  variables.  paradigm. leanings,  Certain  between the two The  classical  tends to see the  variables  can  manipulated  to determine t h e i r e f f e c t s on other v a r i a b l e s .  naturalistic  investigator,  description  be The  on the other hand, i s concerned  with  and understanding, and i s guided by a paradigm  based  on ethnography. Purpose. purpose.  A third difference  between the two approaches i s  The c l a s s i c a l approach t e s t s some p r o p o s i t i o n  about  a  16  relationship  c a l l e d a hypothesis.  The purpose i s to v e r i f y the  h y p o t h e s i s by t e s t i n g ideas e m p i r i c a l l y . naturalistic  approach,  on  The  Framework/design.  of the  the other hand, i s the d i s c o v e r y of  r e l a t i o n s h i p s that can be observed rather to happen under c o n t r o l l e d  purpose  than a r r a n g i n g f o r i t  conditions.  Pre-ordinate,  f i x e d designs are one of  the hallmarks of a c l a s s i c a l approach, while emergent,  variable  designs are among the hallmarks of a n a t u r a l i s t i c approach. Setting. classical  It  i s c l e a r from the above statements that the  investigator  investigations,  leans toward the l a b o r a t o r y  i n a n a t u r a l , n o n - c o n t r i v e d , environment.  Conditions. conditions;  The c l a s s i c a l  the  Treatment.  investigator implies  the  to  investigator  conditions  control  opens  the  as much as p o s s i b l e .  The concept of treatment experimental  seeks  science.  i s extremely To  the  important  naturalistic  concept of treatment i s very f o r e i g n  since i t  some kind of manipulation or i n t e r v e n t i o n .  Scope. range  investigator  naturalistic  investigation to uncontrolled  classical  for  while the n a t u r a l i s t i c i n v e s t i g a t o r c a r r i e s out  investigations  in  setting  C l a s s i c a l i n v e s t i g a t o r s must  of v a r i a b l e s  i n order to be able  c o n t r o l l e d , systematic way that Conversely, consider  naturalistic  focus  to deal  characterizes  investigators  any v a r i a b l e that appears r e l e v a n t .  problem from a h o l i s t i c Methods.  Lastly,  on  limited  with them i n the this  are  a  more  approach. ready  They approach  to the  view. both  r e s e a r c h e r s wish to be o b j e c t i v e  classical  and  naturalistic  i n t h e i r methodology,  but the  17  meaning  which they a s c r i b e to that term i s q u i t e d i f f e r e n t .  classical  investigator  inter-subjective  s t r i v e s f o r o b j e c t i v i t y i n the sense of  agreement. in  The  that  naturalistic  places  little  instead  f o r c o n f i r m a b i l i t y , i . e . agreement  information  store  The  investigator,  form of o b j e c t i v i t y and s t r i v e s among a  variety  of  sources.  The nine p o i n t s of d i f f e r e n c e noted above are summarized i n Table the  1 (Guba, Note 2 ) .  The dimensions of the t a b l e  fundamental d i f f e r e n c e s  naturalistic  approaches.  b e l i e v e that every c l a s s i c a l to  the  points  illustrate  i n viewpoints between c l a s s i c a l Nevertheless,  it  i n v e s t i g a t o r would  and  would be naive to always  conform  of view mentioned, j u s t as i t would be absurd to  suppose that a n a t u r a l i s t i c  i n v e s t i g a t o r would never  deviate.  18  Table 1 Some Basic D i f f e r e n c e s Between C l a s s i c a l and N a t u r a l i s t i c  Paradigms  COMPARISON ITEM  CLASSICAL  NATURALISTIC  P h i l o s o p h i c a l base  Logical  Phenomenology  I n q u i r y paradigm  Exper imental physics  Anthropology  Purpose  Verification  Discovery  Framework/design  Fixed  Var i a b l e  Setting  Laboratory  Nature  Condit ions  Controlled  Invited interference  Treatment  Stable  Variable  Scope  Limited v a r i a b l e s  Holistic  Methods  Object i v e - - i n sense of inter-subject agreement  Objective--in sense of factual/ confirmable  positivism  Classical  The classical  l i t e r a t u r e reviewed paradigm  with  confirmed  (1963)  available  cumulative  belief part  to  called  this  progress"  i n and commitment t o the n a t u r a l of  most  prominent  the dominance  of the  i t s q u a n t i t a t i v e , experimental  Campbell and S t a n l e y route  Paradigm  academic  paradigm  bias.  "the only  (p. 3 ) . I t was t h i s  science  researchers  model that  on the made the  19  c l a s s i c a l paradigm dominant explained,  "a  paradigm  (Patton,  governs,  1978).  i n the f i r s t  s u b j e c t matter but r a t h e r a group" (p. committed  to  the  dominant  As  80).  Kuhn  (1970)  i n s t a n c e , not a  Those groups  most  paradigm are found i n u n i v e r s i t i e s  where they not only employ  the s c i e n t i f i c method  evaluation  where they a l s o nurture students i n a  research  but  commitment to that same methodology L i k e the m a j o r i t y of  evaluative  short-term r e s i d e n t i a l programs the c l a s s i c a l paradigm. evaluations  (Kuhn,  in  their  own  1970).  studies,  evaluations  belong to the group dominated by  A survey of the l i t e r a t u r e y i e l d e d  relying  of  heavily  on  the  only  assumptions  and  c h a r a c t e r i s t i c s of the c l a s s i c a l paradigm d e s c r i b e d  previously.  No s t u d i e s were i d e n t i f i e d that conformed  naturalistic  to  the  paradigm. Based studies  on  classical  reviewed  experimental discussion this  the  utilized  either  pre-experimental,  quasi-experimental designs.  The  concerned  testing  i n t h i s format.  an  evaluation  The d i s c u s s i o n  true  following  i s l i m i t e d to short-term r e s i d e n t i a l programs,  study  programs  or  paradigm, the r e s e a r c h e r s i n the  since  methodology  separates and  on  critiques  the s t u d i e s on the b a s i s of d e s i g n .  One-shot  Case Study  Much e v a l u a t i o n r e s e a r c h i n education conforms to a in  which  study) presumed  a  single  subsequent  group to  some  to cause change.  i s s t u d i e d only once treatment  design  (one-shot case  (conference,  workshop)  Three s t u d i e s reviewed used the one-  20  shot approach  (Havelock,  1971;  M i l o z a r e k , 1976;  Scruggs,  1976).  B a s i c a l l y , the planners i n the above s t u d i e s wanted to know program  participants  T h i s simple  felt  at  how  the c o n c l u s i o n of the program.  form of e v a l u a t i o n r e q u i r e s that a set of systematic  o b s e r v a t i o n s be made of one  group at some s p e c i f i e d time.  These  s t u d i e s i m p l i c i t l y compare the r e s i d e n t i a l experience with other observed  and/or remembered events.  general  expectations  of  The  i n f e r e n c e s are based  on  what the data would have been had  the  experience not o c c u r r e d .  In a d d i t i o n ,  "the  sources  between  one  of  differences  f u t u r e ones are so  numerous  any  that  many  uncontrolled  study and  justification  potential  in  terms  p r o v i d i n g a bench mark f o r f u t u r e s t u d i e s i s hopeless" & S t a n l e y , 1963, Where results  p.  only  group  i s measured, i n t e r p r e t a t i o n of the  and o f t e n unconvincing.  r a t e d s u c c e s s f u l by the p a r t i c i p a n t s . ended  responses  praise,  and  (Havelock,  were  requests 1971).  (Campbell  7).  one  is difficult  divided for  more  Without  a  into time  In two  "This workshop general,  the  for  program,  open-  and  depth  of  topic"  comparison group, i t i s hard to  or whether the program was  some  actually responsible  producing the r e s u l t s at a l l .  Pretest-Post If  Design  the q u e s t i o n an e v a l u a t o r i s seeking to answer cannot  addressed of  was  categories: outright  know whether the r e s u l t s would have been e q u a l l y good with other  of  through one  the program, then  set of o b s e r v a t i o n s made at the the  next  more  complex  be  completion  research  design  21  should  be  evaluator  used.  The p r e t e s t - p o s t t e s t design  i s used when the  wants to know i f p a r t i c i p a n t s improved or at l e a s t d i d  not d e t e r i o r a t e while being  served  by a program.  were  used  this  design  Dickinson  &  identified  that  Deantonio,1973; Densmore, 1965; Halverson 1971;  &  Thiesse,  1979;  Pattison,  V a l l a , 1975; Wohllenben, 1965).  study,  one  cannot  improvement. however,  Like  (Cox,  1974;  Lamoureux,  1975;  the  that  the  The program might  have  caused  design  articles  1968; Roberts & Holmes,  conclude  this  Nine  one-shot  program the  case  caused  the  improvement;  i s not r i g o r o u s enough to permit such a  conclusion. A l l of the above s t u d i e s study  or  pretest-posttest  localized.  T h e i r value  therefore localized  not  used design.  the  one-shot  s t u d i e s should  Sutton  be a p p r a i s e d  case  These s t u d i e s were h i g h l y  was l i m i t e d t o the program  generalizable.  o p e r a t i o n a l values  either  (1966)  only  in  studied  and  suggested terms  that  of  their  to the i n s t i t u t i o n making them.  True Experimental When  evaluators  want  to d i s c o v e r  program p a r t i c i p a n t s , e v a l u a t i o n s designed. else,  In  order  i t i s necessary  precedes  the  supposed  to to  show  with the e f f e c t ; and (3) no other the 1980,  effect p.  exist  except  of greater complexity must  be  that something caused something  demonstrate  effect  the cause of changes i n  that:  "(1) the  i n time; (2) the cause alternative  cause  covaries  explanations  of  the assumed cause" (Posavac & Carey,  196). Campbell and S t a n l e y  (1963) suggest  that  only  22  true  experimental  causal  and  q u a s i - e x p e r i m e n t a l designs  w i l l prove  relations.  S e v e r a l s t u d i e s of r e s i d e n t i a l programs have used the experimental d e s i g n . with  participants  They used a p r e t e s t - p o s t t e s t c o n t r o l  1978; Bunch,  1976).  Occasionally  1976; Conrad, 1976; D e v l i n , researchers  s e l e c t e d groups (Blaney & McKie,  it's  practice,  impossible  conditions.  group  randomly s e l e c t e d f o r the two groups (Bale &  Molitor,  In  true  evaluators to  In  randomly those  have  used  1966;  Jenkins,  three  randomly  1969; Peterson, 1971). are o f t e n select  cases,  in a p o s i t i o n  groups  evaluators  and  i n which  manipulate  choose  a  quasi-  experimental design which c o n t r o l s some but not a l l " t h r e a t s validity"  (Campbell  &  Stanley,  1963).  to  In q u a s i - e x p e r i m e n t a l  designs, an i n t a c t group i s s e l e c t e d as a c o n t r o l because of i t s similarity  to the experimental group.  group  called  is  true c o n t r o l  Sometimes  this  type  a "comparison group" to d i s t i n g u i s h  i t from a  group.  I n c l u d i n g the comparison group permits a d i s t i n c t i o n made  between  the  effects  alternative plausible comparison  of  the  program  i n t e r p r e t a t i o n s of  both  the  change.  be  several  Because  of  time  to  the  mature.  f o r c e s presumably a f f e c t the groups e q u a l l y .  groups  equivalent.  and  to  group i s t e s t e d at the same time as the experimental  group, both groups have the same, amount Historical  of  are  tested  Finally,  the  twice, rates  testing of  p r e t e s t and p o s t t e s t can be examined (Posavic & Carey, 1980).  effects  participant  Because  should  loss  be  between  to be sure they are s i m i l a r  23  Quasi-experimental Design The  quasi-experimental  s t u d i e s reviewed was design  necessitates  selection  bias.  equivalence the  the  be  through  groups.  i n v e s t i g a t e d one  (Bringle,  1967;  a  pretest  Touzel,  1975).  1961;  Smallegan,  group.  can  This  c o n t r o l for  intact  the e v a l u a t o r  F i v e of the s t u d i e s  group  to compare using  this  or more aspects of a p a r t i c u l a r program  George & Green, 1976;  1966;  control  of a p r e t e s t to provide  demonstrated, e n a b l i n g  r e s u l t s of the two  design  most f r e q u e n t l y used in the  nonequivalent  the use  Only  design  Four s t u d i e s  1971;  Wientge  Stewart,  1965;  Torrence,  (Edelbach,  1973;  Lacognata,  1966)  compared  &  Lahr,  r e s i d e n t i a l to n o n - r e s i d e n t i a l programs. The focused  above c l a s s i c a l / e x p e r i m e n t a l / q u a s i - e x p e r i m e n t a l on  anxiety,  isolated psychological variables, i . e . self-esteem.  Such  studies  i n t o the complex impact that Moreover,  they  information  needed to  much  evaluation  of negative anger, and The the  have  evaluators  always  have  which  the e v a l u a t i o n  for  evaluation  paradigm  with  appears  to  have  from  serious  1978).  program  decisions. I t has  been composed  typically  quantitative,  off  the of  great any  planners Finally,  discourage,  audience (Patton,  consideration  e v a l u a t i o n paradigm (Patton,  participants.  i s that the very  its cut  on  given  programmatic  statements  satisfaction,  have not allowed i n s i g h t  been h i g h l y c r i t i c a l .  disappoint problem  not make  punitive  classical  emphasis  has  programs  studies  1978).  dominance of experimental majority  of  alternative  24  Naturalistic  Recently, utilizing &  however, there has been i n c r e a s i n g r e s i s t a n c e to  the c l a s s i c a l paradigm  Hamilton,  to  the  The  classical  naturalistic  r o o t s i n ethnography is  in evaluation studies  1976; Patton, 1978; Smith,  alternative paradigm.  Paradigm  1976; Stake,  paradigm paradigm  and anthropology.  is is  A  the  (Parlett  1978).  One  naturalistic  not new but has i t s  naturalistic  inquiry  a dynamic process which i s not t i e d to a s i n g l e treatment or  predetermined g o a l s or  outcomes,  actual  a  operations  Lincoln, process  of  1981; P a r l e t t requires  & Hamilton,  sensitivity  not j u s t at some end-point  of  focuses  1976; to  Patton,  both  throughout  on  the  (Guba &  1978).  qualitative  This and  i t s development,  i n time.  (1977) has c h a r a c t e r i z e d those a l t e r n a t i v e models  as " p l u r a l i s t " e v a l u a t i o n models. account  rather  program over a p e r i o d of time  q u a n t i t a t i v e changes i n a program  Hamilton  but  the  value  positions  pluralist  That of  practical  terms,  Hamilton,  1976; Patton, 1975; Stake,  i s , models multiple  evaluation  models  that  take  audiences.  In  (Parlett  &  1967) can be c h a r a c t e r i z e d  in the f o l l o w i n g manner: Compared with the c l a s s i c models, they tend to be more e x t e n s i v e (not n e c e s s a r i l y centered on numerical d a t a ) , more n a t u r a l i s t i c (based on program a c t i v i t y r a t h e r than program i n t e n t ) , and more adaptable (not c o n s t r a i n e d by experimental or p r e o r d i n a t e d e s i g n s ) . In turn they are l i k e l y to be s e n s i t i v e t o the d i f f e r e n t values of program p a r t i c i p a n t s , to endorse empirical methods which are couched i n the n a t u r a l language of the r e c i p i e n t s , and to s h i f t the l o c a l e of  25  formal judgment p a r t i c i p a n t s (p. There  evaluator  are many methodological questions  about n a t u r a l i s t i c issues  from the 339).  to  i n q u i r y , ranging  operational  from  or procedural  The  major  problem,  a u t h e n t i c i t y — t h e establishment outcomes  of  the e v a l u a t i o n .  s e t t i n g l i m i t s to the which  data  can  Despite  the d i f f i c u l t i e s  just  begun  to  gain  have been s t r o n g l y  matters.  Guba (1978) has  congenial  are:  the  a  naturalistic  it,  on  assimilated  mentioned,  is  that  of  in  the  trust  the and  categories understood.  naturalistic  Leading e v a l u a t i o n  Guba & L i n c o l n , 1981;  inquiry theorists  1977;  Patton,  naturalistic Fienberg,  1978)  techniques to  1977;  Lutz,  1974;  emerged  which  seem  R i s t , 1975).  evaluation  F i v e emergent models inquiry  saw  focusing  be  (Erickson,  P a r l e t t & Hamilton, 1976;  especially  face  i n t e r e s t e d in moving away from more c l a s s i c a l  studies  A number of  raised  epistomological  he  p r a c t i t i o n e r s have begun to apply  evaluative  be  Other methodological problems are  credibility.  paradigms (Cronbach, 1980; and  that can  of the b a s i s for  i n q u i r y and  within  has  the  as  the  basic  attempted to d e f i n e the d i f f i c u l t i e s that inquiry.  to  models  have  to the use of the n a t u r a l i s t i c especially  compatible  Responsive Model (Stake,  with  paradigm.  naturalistic  1975), the  Judicial  Model (Wolf, 1979), the T r a n s a c t i o n a l Model (Rippey, 1973), C o n n o i s s e u r s h i p Model ( E i s n e r , 1975), and (Parlett  &  Hamilton,  p h i l o s o p h i c and Their  1977).  operational  These  ties  with  the I l l u m i n a t i v e Model  five  models  naturalistic  have c l o s e inquiry.  emergence at t h i s time argues s t r o n g l y for the u t i l i t y  naturalistic  i n q u i r y for the  field  of  the  educational  of  evaluation,  26  and  helps  make  the  case  that n a t u r a l i s t i c  i n q u i r y should be  i n v e s t i g a t e d as an a l t e r n a t i v e methodology.  Illuminative Evaluation  The (1977)  illuminative  evaluation  model  (Parlett  &  Hamilton,  was chosen f o r t e s t i n g because i t matched the philosophy  and value system of the i n v e s t i g a t o r . the study of changing multiple  viewpoints  and and  emerging  T h i s model: (1) permitted problems,  perspectives,  (2)  encouraged  (3) focused on program  a c t i v i t i e s and i s s u e s r a t h e r than outcomes and (4) provided a means of studying spontaneous events  for  and s i t u a t i o n s .  [I]lluminative e v a l u a t i o n , takes account of the wider contexts i n which education programs function. I t s primary concern is with description and interpretation rather than measurement and p r e d i c t i o n . I t stands unamb i g u o u s l y w i t h i n the a l t e r n a t i v e methodological paradigm. The aims of i l l u m i n a t i v e e v a l u a t i o n are to study the innovatory program: how i t operates; how i t i s i n f l u e n c e d by the v a r i o u s school s i t u a t i o n s i n which i t i s a p p l i e d ; what those d i r e c t l y concerned regard as i t s advantages and disadvantages; and how students, i n t e l l e c t u a l tasks and academic experiences are most a f f e c t e d . In s h o r t , i t seeks to address and to i l l u m i n a t e a complex array of q u e s t i o n s ( P a r l e t t & Hamilton, 1977, p. 144). Characteristically principal first  stages:  stage  investigator  illuminative  observation,  i s an  exploratory  evaluations  have  i n q u i r y , and e x p l a n a t i o n . phase  during  which  three The the  becomes knowledgeable about the program and people  i n v o l v e d and t r i e s t o understand  and  document  the  day-to-day  27  reality  of  the s e t t i n g or s e t t i n g s under study.  No attempt i s  made t o manipulate, c o n t r o l or e l i m i n a t e s i t u a t i o n s developments.  Faculty,  participants,  planners  persons i n v o l v e d i n the p r o j e c t are observed Documents  are  reviewed  or  program  and any other  and  interviewed.  to o b t a i n an h i s t o r i c a l  p e r s p e c t i v e as  w e l l as a p e r s p e c t i v e on how people regard the i n n o v a t i o n . The is  an  second stage  It  evaluators  and  relevant  decision-makers or information u s e r s .  Narrowing  and  focusing  d e a l i n g with s e v e r a l b a s i c concerns.  What i s  study  means  process  process.  between  the  interactive  i s a narrowing and f o c u s i n g  the purpose of the e v a l u a t i o n ? used?  What  know now?  will  we  How  know a f t e r  will  the  information  be  the e v a l u a t i o n that we do not  What can we do a f t e r the e v a l u a t i o n that we cannot do  now f o r l a c k of information?  What t o p i c s or concerns should  be  selected for intensive investigation? Narrowing  and  focusing  are key elements because programs  are so complex and have so many l e v e l s , goals, There are always more p o t e n t i a l study and  resources  to examine.  and  functions.  t o p i c s than there are time  The a l t e r n a t i v e s , t h e r e f o r e , have to  be narrowed, c l a r i f i e d and r e d e f i n e d . When the a l t e r n a t i v e s have been c l a r i f i e d and d e f i n e d , evaluator  must  determine  e v a l u a t i o n does not have that  function,  that o f f e r example, extent  e v a l u a t i o n procedures. simple,  standardized  the  Illuminative  procedures  for  so the e v a l u a t o r might i n c o r p o r a t e other models  guidelines i f the  study  for operationalizing focuses  to which the program content  the  model.  For  on p a r t i c i p a n t r e a c t i o n s , the was a s s i m i l a t e d  and/or  the  28  change  in  job  Kirkpatrick's  behavior,  (1967) or  the  evaluator  Hamblin's  (1974)  might  incorporate  model.  Both  these  models o f f e r g u i d e l i n e s f o r o p e r a t i o n a l i z i n g the e v a l u a t i o n . The  third  stage  consists  of  seeking general p r i n c i p l e s  u n d e r l y i n g the o r g a n i z a t i o n of the program, s p o t t i n g p a t t e r n s of cause and e f f e c t , broader  and  explanatory  placing  context  individual  ( P a r l e t t & Hamilton,  Table 2 f o r summary of the three Within  the  evaluation,  the  three  findings  1976).  a  (See  stages.)  stage  investigation  within  framework  of  illuminative  can combine four d i f f e r e n t  data  g a t h e r i n g techniques p e r m i t t i n g the program to be examined  from  a  the  number  of  participants resource  angles.  and  persons,  These  events;  (2)  and  covering  many  aspects  research  with  existing  of  are  (1)  evaluation. continuous comments  They record  on  are  with  participants,  administrators;  (3)  questionnaires  the  program;  documents.  an  are of  of  interviews  and  The  d e s c r i b e the data g a t h e r i n g techniques Observat ions  observation  i n more d e t a i l . part  intended  primarily  obvious and l a t e n t  historical  f o l l o w i n g paragraphs  essential  on-going  (4)  events,  to  of  illuminative  to  build-up  add  a  interpretive  f e a t u r e s of the program, and to  uncover t a c i t assumptions and i n t e r p e r s o n a l r e l a t i o n s h i p s . Interviews are used p r i m a r i l y to determine and  views  of i n d i v i d u a l p a r t i c i p a n t s .  processes experienced by d i f f e r e n t  perceptions  D i s c o v e r i n g the view of  p a r t i c i p a n t s i s c r u c i a l to a s s e s s i n g the impact Informal i n t e r v i e w s o f t e n provide unique  the  of the program.  insights  people.  into  program  Illuminative  STAGES  STAGE  STAGE  STAGE  THREE—EXPLANATION This i s the a n a l y s i s and i n t e r p r e t a t i o n phase.  Stages  ACTIVITIES  ONE—OBSERVATION The i n v e s t i g a t o r becomes knowledgeable about the program people involved.  TWO--INOUIRY TrT^ i n v e s t i g a t o r n a r r o w s focuses the study.  Table 2 Evaluation  and  and  - r e v i e w o r d i s c o v e r what i s e x p e c t e d a t t h e o u t s e t -consider the questions, hypotheses or issues already r a i s e d -look f o r p o s s i b l e s t u d i e s t o use as a models -review h i s t o r i c a l documents -form i n i t i a l p l a n of a c t i o n - a n t i c i p a t e key problems, events -consider p o s s i b l e audiences f o r p r e l i m i n a r y and f i n a l reports  - a r r a n g e a c c e s s t o p r o g r a m , n e g o t i a t e pl-an o f a c t i o n -discuss arrangement f o r m a i n t a i n i n g c o n f i d e n t i a l i t y of data, source - i d e n t i f y i n f o r m a n t s and s o u r c e s of p a r t i c u l a r data - s e l e c t or develop q u e s t i o n n a i r e s or s t a n d a r d i z e d procedures i f any -work o u t r e c o r d - k e e p i n g system -make o b s e r v a t i o n s , i n t e r v i e w s , u s e q u e s t i o n n a i r e -keep r e c o r d s o f a c t i v i t i e s and changes  and  - c l a s s i f y raw d a t a ; b e g i n i n t e r p r e t a t i o n s -gather a d d i t i o n a l data, t r i a n g u l a t e data to v a l i d a t e key o b s e r v a t i o n s -search f o r p a t t e r n s of data -seek l i n k a g e s between program a r r a n g e m e n t s , a c t i v i t i e s and outcomes -select illustrations, special intretations -draw t e n t a t i v e i s s u e s , o r g a n i z e a c c o r d i n g t o i s s u e s - d e s c r i b e t h e s e t t i n g where t h e a c t i v i t y occurred -draft reports - d e s c r i b e methods of i n v e s t i g a t i o n - r e v i s e and d i s s e m i n a t e reports  reports  30  Q u e s t i o n n a i r e s and t e s t s are i n c l u d e d to o b t a i n i n f o r m a t i o n that s u s t a i n s or q u a l i f i e s e a r l i e r , Historical  research  using  sources p r o v i d e s i n f o r m a t i o n about The  gathering  of  tentative  background  documentary  before  the  gathered  evaluation  o f t e n suggest  background  of  events.  i n f o r m a t i o n y i e l d s an h i s t o r i c a l was  began.  regarded  by  different  T h i s i n f o r m a t i o n can be  obtained from l e t t e r s , minutes of meetings, data  and  the development  p e r s p e c t i v e of the way the program people  findings.  and  reports.  The  t o p i c s that need i n v e s t i g a t i o n and  expose aspects of the program that otherwise would be missed. The  three stages of i l l u m i n a t i v e e v a l u a t i o n do not f u n c t i o n  s e p a r a t e l y ; they o v e r l a p and are i n t e r r e l a t e d . from stage t o stage occurs as problem c l a r i f i e d and r e d e f i n e d . using  the  data  The  transition  areas become p r o g r e s s i v e l y  Beginning with an e x t e n s i v e data  gathering  techniques  mentioned  base,  above,  i n v e s t i g a t o r s y s t e m a t i c a l l y reduces the scope of the i n q u i r y give  more  c o n c e n t r a t e d a t t e n t i o n to the emerging i s s u e s .  " p r o g r e s s i v e f o c u s i n g " permits unique and u n p r e d i c t e d to be given due weight. and  prevents  the  I t reduces the problem  massive  ( P a r l e t t & Hamilton,  accumulation  the to This  phenomena  of data o v e r l o a d  of unanalyzed m a t e r i a l  1976).  Summary  T h i s chapter evaluation.  illuminated  Although  the  historical  development  e v a l u a t i o n processes have been  of  developed  31  independently,  most  direction--toward  fields  have  developed  accountability.  This  educational  evaluation.  occurred.  in the 1960's,  toward  evaluation  a  qualitative  With the emergence of l a r g e s c a l e U.S. i t was  same  and  While there has been c o n t i n u i t y i n  the development of the e v a l u a t i o n f i e l d , has  the  trend  a c c o u n t a b i l i t y was noted both i n s o c i a l s c i e n c e in  in  found  that  the  classical  change projects  evaluation  approaches were inadequate to d e a l with e v a l u a t i o n q u e s t i o n s and issues  posed  by  those  projects  (Cronbach,  a l t e r n a t i v e to c l a s s i c a l e v a l u a t i o n that arose inquiry  which  ethnography.  had Using  interviewing  and  the  techniques  of  personal o b s e r v a t i o n ,  with  illuminative  the  target  evaluation  model  n a t u r a l i s t i c paradigm permits encourages and provides situations.  multiple  of  of  open-ended  derived (Patton,  has  emerged  from  close  1978).  The  from  the  the studying of changing problems,  viewpoints,  f o r a means  and  t h i s approach r e l i e d on  study  that  in-depth,  One  naturalistic  i t s roots i n the f i e l d s of anthropology  q u a l i t a t i v e data, and d e t a i l e d d e s c r i p t i o n contact  was  1963).  focuses  studying  on program a c t i v i t i e s  spontaneous  events  and  32  CHAPTER I I I  METHODOLOGY  Illuminative  evaluation  study f o r s e v e r a l reasons. based  methodology was s e l e c t e d f o r t h i s  Firstly,  on the assumption that e v a l u a t i o n  needs, i n t e r e s t s , and p e r c e p t i o n s than on measurement c r i t e r i a acknowledges truths.  in-depth  that  there  evaluation  information  instructors  and  interpretations perceptions  while  elicits, is  coordinators  describing  residential  t e s t e d on a s e l e c t e d s i t e . compared  Secondly, i t  investigative  provided  efforts,  and b u i l d s on the by  alike. and  participants, Finally,  differences  the o r i g i n s and context  data in  f o r such  (Guba & L i n c o l n , 1981).  t o determine the s u i t a b i l i t y of t h i s  evaluating  rather  r e a l i t i e s and m u l t i p l e  considers,  similarities  agreements and d i s c r e p a n c i e s In order  of  is  "respond" t o the  the p a r t i c i p a n t s  are m u l t i p l e  that  portray  of  should  established a p r i o r i .  Thus, u n l i k e the m a j o r i t y  illuminative  for  illuminative evaluation  adult  education  methodology,  programs, i t was  Then the r e s u l t s of the t e s t i n g were  with the standards s e t by p r e - s p e c i f i e d  criteria.  If  33  the evidence of s u i t a b i l i t y meets the standards, the methodology can  then  be  deemed  suitable.  c o n t a i n s d e s c r i p t i o n s of  the  The remainder  criteria,  of t h i s chapter  standards,  and  study  site.  Criteria  Three  c r i t e r i a appearing f r e q u e n t l y i n the l i t e r a t u r e were  judged a p p r o p r i a t e to t h i s study.  The l i t e r a t u r e  an e v a l u a t i o n methodology should produce technically  sound, (2) u s e f u l to some  more to the audience  than  criteria,  judge  used  to  suggested  that  i n f o r m a t i o n that i s (1)  audience  and  (3)  worth  i t c o s t s (Grotelueschen, 1980).  These  the s u i t a b i l i t y of t h i s methodology,  w i l l be d e s c r i b e d i n the f o l l o w i n g  paragraphs.  T e c h n i c a l Adequacy Two standards of a t e c h n i c a l l y adequate methodology are the o b j e c t i v i t y and objectivity inquiry  validity.  Of  the  be  or  objective  the  advance...." seems Scriven  to  if  i t simply  data  to  be  (Guba & L i n c o l n ,  'emerges';  i f the  are  1981, p.  124).  are  the  i f i t has no  observations  recorded  that  mentioned,  "For how can an  not  terms  to  specified  The  stem from the meaning given to the term  (1972) has p o i n t e d out  subjective  standards  i s probably the most c o n t r o v e r s i a l .  c a r e f u l c o n t r o l l a i d down a p r i o r i ; made  two  be in  difficulty  objectivity. objective  and  o p p o s i t e s , but they are widely used t o r e f e r t o  34  contrasts  in  qualitative In  two  the  a  quantitative  of  subjects  "'subjective'  biased,  a  sense,  a  matter  unreliable,  opinion,  95-96).  Basically,  and  Scriven  i n d i v i d u a l experiences i s not a  matter  and  a  of  experience  opinion, is  not  r e f e r s to what  'objective'  experience."  means  of  while  refers  to  In the q u a l i t a t i v e biased  or  probably  ' o b j e c t i v e ' means r e l i a b l e ,  f a c t u a l , c o n f i r m a b l e or confirmed, and pp.  quantitative  'subjective'  i n d i v i d u a l subject,  number  sense,  senses:  one.  "occurs to the what  different  so f o r t h " ( S c r i v e n ,  suggested  that  1972,  what  one  n e c e s s a r i l y u n r e l i a b l e , biased,  just  as  what  necessarily  or  a number of i n d i v i d u a l s  reliable,  factual,  and  conf i rmable. Illuminative  evaluation  methodology  n a t u r a l i s t i c paradigm emphasizes the while  evaluation  emphasize  the  illuminative  objectivity  objectivity model,  the  of  the  investigator.  validity. of  the  is  validity.  technically  adequate  instrumentation  and  measure?  mean what we  1978).  c e n t r a l by  getting  they mean? methodology close  data  paradigm the  critical  evaluation  to  (Patton, makes the  the  data,  meaningfulness  employed.  instrument measure what i t purports to  Illuminative  the  I l l u m i n a t i v e methodology emphasizes  c o l l e c t e d and  think  the  confirmable.  It i s concerned with the meaning data  on  In  o b j e c t i v i t y of the data i s of  second standard of a  methodology  of  methodologies based on the c l a s s i c a l  concern; i t should be both f a c t u a l and The  based  issue being  Do  of  Does the the  data  validity  sensitive  to  35  qualitative  distinctions,  developing  p a r t i c i p a n t s , and attempting on the program.  is  not  behavior, but  the  only  program  to e s t a b l i s h a h o l i s t i c p e r s p e c t i v e  1975;  Guba & L i n c o l n ,  legitimate  way  to  by  Denzin  1981;  Patton,  understand  human  i t i s an a l t e r n a t i v e to the d i s t a n c e p r e s c r i b e d by  the dominant c l a s s i c a l paradigm. methodology  with  T h i s c l o s e n e s s to the data suggested  (1971) and others (Campbell, 1978)  empathy  is  on a v a l i d  The  focus i n the  illuminative  r e p r e s e n t a t i o n of what i s happening,  not at the expense of r e l i a b l e measurement, but without a l l o w i n g reliability 1981;  to determine  Parlett  the nature of the data  & Hamilton,  (Guba & L i n c o l n ,  1976).  House (1980) p o i n t e d out that V a l i d i t y i s p r o v i d e d by c r o s s - c h e c k i n g different data sources and by t e s t i n g p e r c e p t i o n s a g a i n s t those of p a r t i c i p a n t s . Issues and questions arise from the people and situations being studied r a t h e r than from the investigator's percept ions .... In c o n s t r u c t i n g e x p l a n a t i o n s , the n a t u r a l i s t looks f o r convergence of h i s data sources and develops sequential, phase-like e x p l a n a t i o n s that assume no event has single causes (p. 280). In  order  to  determine  the  technical  i l l u m i n a t i v e methodology, the data produced and both  valid. factual  and  technical  procedures  be  of  objective  In other words, i t must be shown that the data are confirmable  r e p r e s e n t a t i o n of events. the  must  adequacy  Two  adequacy  as  well  procedures  of  as  give  a  were used to  illuminative  valid  determine  evaluation;  those  are t r i a n g u l a t i o n and continuous o b s e r v a t i o n .  Triangulation  is a  process  of  cross-checking  Cross-checking enables the i n v e s t i g a t o r to determine  findings. i f the data  36  collected  from  Besides  facilitating  increases study,  multiple  the  data  data  sources  confirm  cross-checking,  each  triangulation  from  informal  data sources produced Continuous  interviews,  a  p r o f i l e of the program.  a  continuous  record  remarks.  Much  recording discussions provided  additional  apparent  from  is  also  important  on-going of  with  the and  formal  data  from  events,  valid. i n determining  the  provide  between  transactions  that  participants. might  interviews  and  and  These  not otherwise  be  questionnaire  continuous o b s e r v a t i o n was  triangulation  built  on-site observation involved  provide a v a l i d p i c t u r e of the program. were used i n  these  In t h i s study the i n v e s t i g a t o r  information  more  The  of  and  The process of combining  data that were o b j e c t i v e and  observation  In t h i s  questionnaires,  t e c h n i c a l adequacy, because continuous o b s e r v a t i o n w i l l  responses.  also  c r e d i b i l i t y of data through v a l i d a t i o n .  o b s e r v a t i o n s were cross-checked.  informal  other.  used to  In a d d i t i o n , those data  process.  For  example,  oral  responses were cross-checked with w r i t t e n responses. If data  by  using  collected  illuminative  t r i a n g u l a t i o n and continuous o b s e r v a t i o n the  could  not  evaluation  t e c h n i c a l l y adequate.  be  confirmed  methodology  I f , on the  other  or  validated,  could  not  be  hand,  the  data  the judged were  confirmed through both t r i a n g u l a t i o n and continuous o b s e r v a t i o n , the  methodology  c o u l d be judged t e c h n i c a l l y adequate.  t e c h n i c a l l y adequate methodology p r o v i d e s evidence that o b j e c t i v e and v a l i d as d e f i n e d i n t h i s  section.  Thus, a is  both  37  Utility An  evaluation  r e s u l t s that important,  are and  methodology useful.  The  credible.  l o g i c a l expectation  should  be  results  too  often  time, e f f o r t information, Weiss  why  do not match the may  results  skill  are  that  However,  With a l l the money,  into  the  acquisition  or  of  impact?  suggests s e v e r a l reasons: e v a l u a t i o n r e s u l t s  i n f o r m a t i o n a l needs of decision-makers, r e s u l t s  not be r e l e v a n t to the l e v e l of decision-maker who  them,  the  the r e s u l t s  programming.  ignored.  went  relevant,  i s completed,  does i t g e n e r a l l y have so l i t t l e  (1972)  be  i s that decision-makers w i l l use  the  and  should  Once the e v a l u a t i o n  to make r a t i o n a l d e c i s i o n s about f u t u r e all  f l e x i b l e yet produce  received  r e s u l t s lack c l e a r d i r e c t i o n for f u t u r e programming.  As House (1977) says e v a l u a t i o n s p e r s u a s i o n .... Expecting necessary c o n c l u s i o n s deliver.  But  providethe  if  "can  evaluation  i s to  it  to  expect  cannot  be no more provide  more  produce  than the  c r e d i b l e , the p l a u s i b l e , and  than  acts  compelling evaluation  of and can  necessary,  i t can  the probable'"  (pp.  5-6) . In t h i s study, c r e d i b i l i t y was judging  the  utility  of  an  important  standard  i l l u m i n a t i v e e v a l u a t i o n methodology.  Assurance of c r e d i b i l i t y  in i l l u m i n a t i v e e v a l u a t i o n  best  frequent  obtained  through  p a r t i c i p a n t s as the with  limited  information  credibility  e l i m i n a t e d or strengthened. expose  the  investigator  for  can  and  Thus,  information  i d e n t i f i e d e a s i l y and  Of course, to b i a s e s .  probably  thorough i n t e r a c t i o n with  develops. be  is  such  a  process  either could  While t h i s p o s s i b i l i t y i s  38  undoubtedly biases  r e a l , the i n v e s t i g a t o r  through  observation  i n t h i s study hedged  against  such safeguards as t r i a n g u l a t i o n and continuous  described  previously.  Thus,  triangulation  and  continuous o b s e r v a t i o n are u s e f u l f o r determining both t e c h n i c a l adequacy and  utility.  Another  approach to i n c r e a s e c r e d i b i l i t y  "participant evaluation" using  participant  (Campbell,  1979).  of e v a l u a t i o n s i s  I t i s a move  toward  judgments as p a r t of the e v a l u a t i o n i t s e l f  provide c r e d i b i l i t y  to  checks:  Participants...will usually have a better observational position than... o u t s i d e o b s e r v e r s of a new program. They a c t u a l l y have e x p e r i e n c e d the preprogram c o n d i t i o n s from the same viewing point as they have the s p e c i a l program. Their experience of the program w i l l have been more relevant, direct and valid, less vicarious. Collectively, their greater numerosity will average out observer . i d i o s y n c r a c i e s that might dominate the report of any one ethnographer. While p a r t i c i p a n t s are asked to generate a l o t of data i n program e v a l u a t i o n , rarely are they directly asked to e v a l u a t e the program, to judge the adequacy, to a d v i s e on i t s continuance, discontinuance, d i s s e m i n a t i o n , or m o d i f i c a t i o n . Rather than e v a l u a t i n g programs, p a r t i c i p a n t s are u s u a l l y asked about themselves and their own adequacy. We are thus wasting a l o t of w e l l founded o p i n i o n s (Campbell, Note 3). T h i s study used Campbell's data.  The  approach  to  produce  credible  p a r t i c i p a n t s were asked to e v a l u a t e the program, to  judge i t s adequacy and to a d v i s e on i t s m o d i f i c a t i o n . Two  other  standards  importance of the d a t a . of  data  were  utility  are  the  relevance  In t h i s study, relevance and  determined  d i s c u s s i o n s with the Counc i1 members.  of  program  through planner,  interviews director  and  importance  and  informal  and  Advisory  39  Flexibility utility  i s a l s o an important standard  of i l l u m i n a t i v e e v a l u a t i o n .  important  in  program  Flexibility  evaluation,  and as p l a n n e r s experiment and change t h e i r of  of i l l u m i n a t i v e e v a l u a t i o n to  match  the  evaluation  to  the  to  stay  still &  or  Guttentag, design  same  1975). upon  of  in  A  Within  the  to be  evaluated  personalized  lends  itself  to  stage  framework  observations,  make  evaluation  a  judgment  methodology  standards d e s c r i b e d collected relevant.  by In  above.  various  to  c o n s t r a i n t s on the program.  using  data  interviews,  utility,  provide  illuminative  evidence that met the  To meet the standards s e t , the  means  addition,  assembled  documents. regarding  had  of the  of i l l u m i n a t i v e  areas:  and h i s t o r i c a l  highly  i n advance,  operation  collected  To  the  education.  was  questionnaires  In  required  profile  four  program. not  e v a l u a t i o n , an information from  the  study was not be charted  three  The  o b s e r v e r - p a r t i c i p a n t and  environment of a d u l t this  is  order  s i n c e the course was dependent on the a c t u a l program.  not,  allows  flexible,  close  interaction  informal, personalized course  the  built  observer-instructor  The  does  methodology  i n s u r e s that the program  evaluation  what  priorities.  addition, f l e x i b i l i t y  (Edwards  programs a r e  s i n c e i t does not have p r e s c r i b e d c o n s t r a i n t s .  investigator  stand  extremely  the c h i e f advantages of i l l u m i n a t i v e e v a l u a t i o n i s  flexibility, flexibility  is  for innovative  o f t e n changed as planners l e a r n what works and  One  f o r judging the  the  data  had to be c r e d i b l e , important and methodology  had  to  pose  no  40  Ef f i c iency The  last  criterion  used  to determine the s u i t a b i l i t y  i l l u m i n a t i v e e v a l u a t i o n methodology i s e f f i c i e n c y . e v a l u a t i o n methodology should produce  results  more  1980).  than t h e i r c o s t s (Groteleuschen,  the most d i f f i c u l t measurements.  that  The  i s t r e a t e d almost  treatment admit costs. 1973,  is  tangentially"  illustrative  I  do  not  know  p.  percentage  evaluation  cannot  of  evaluation.  that  be  program On  the  i s taken  made  (Haller,  regarded as an  against  reason  which  are  "when t r e a t e d  1974,  p.  405).  for tangential  " I t embarrasses me  to  somebody  one  else"  to  (Stake,  the  in  should  advance as to what be  expended  1974).  f a c t o r when i t comes time to  On  and  of  the  on  those make  the other hand, e v a l u a t i o n can  i n the f u t u r e  investment program  on  hand, every d o l l a r and hour spent  from other a s p e c t s of the program, and  investment  the  easily  resources  c o s t s become a very important  of  the  and  312).  Decisions  value  worth  anything about the measurement of  I w i l l have to leave  decisions  of  (Haller,  by many e v a l u a t i o n s p e c i a l i s t s :  that  are  to p r e c i s e l y d e f i n e i n terms of standards  U n l i k e t e c h n i c a l adequacy and u t i l i t y ,  following  efficient  This c r i t e r i o n i s  f r e q u e n t l y d e s c r i b e d i n the l i t e r a t u r e , e f f i c i e n c y at a l l ,  An  of  program.  be The  w i l l vary with a c c o u n t a b i l i t y demands the  value  of  reporting  program  performance. Reasonable  costs  for  an  evaluation  can  be  e s t i m a t i n g the s i g n i f i c a n c e of i s s u e s and the l i k e l y the  evaluation.  In a sense,  efficiency  decided  by  impact  of  r e p r e s e n t s the r a t i o of  41  effort  to e f f e c t .  different  Although v a r i o u s e v a l u a t i o n needs w i l l  expenditures  of resources,  entail  some form of e v a l u a t i o n  is  p o s s i b l e w i t h i n any budget. To determine e f f i c i e n c y of a methodology, cost estimates i n terms  of  outlay  (such  as  administrative  needed  (such  examined. take  (such  as  Cost  as s u p p l i e s , space), effort,  program  estimates  time  interviews),  personnel  and  expertise  or i n s t r u c t o r s ) should be  of a c q u i r i n g i n f o r m a t i o n  i n t o account hidden c o s t s .  expenditures  also  should  These include time l o s t to the  program by e v a l u a t i n g , a l t e r n a t i v e use of funds, and human c o s t s such as  invasion  attitudes personnel  and  of  privacy,  reactions,  (Grotelueschen,  dangers  or  of  generating  1980).  creating  pressure  negative on program  These may be compared with the  c o s t s of not e v a l u a t i n g . There w i l l be s i t u a t i o n s i n which i t i s p o s s i b l e to cost be  in  d o l l a r s and others  reasonably  i n which i t i s not.  measured i n d o l l a r s ,  comparable Table  estimate  3 lists  the  ingenuity.  convenient,  Dollars,  generalizable  evaluation costs  the c o s t s to be determined during  the e v a l u a t i o n . following  of  a  When c o s t s can  i t i s u s u a l l y d e s i r a b l e to do  so, although i t sometimes r e q u i r e s a l i t t l e as measuring d e v i c e s , p r o v i d e  assess  and  ( H a l l e r , 1974). the  course  of  Costs i n time and/or money are c o l l e c t e d i n the  general  categories:  personnel,  equipment, p a r t i c i p a n t time and e v a l u a t o r  time.  materials  and  42  Table 3 L i s t of E v a l u a t i o n Costs CATEGORY PERSONNEL Administrative Di r e c t o r Secretary C l e r i c a l Staff Professional Instructors Consultants MATERIALS & EQUIPMENT Supplies Space Equipment PARTICIPANT TIME During program After  program  EVALUATOR TIME Before  program  During  program  Waiting  time  A f t e r program CONTINGENCY COSTS  TIME(hrs.)  COST(dollar)  43  The q u e s t i o n to be answered by the c r i t e r i o n of is  whether  cost.  As  resources,  the  same outcomes c o u l d have been achieved at l e s s  greater  demands  questions  receive closer  of  are  placed  on  limited  consideration.  above  were  used  to  determine  i l l u m i n a t i v e e v a l u a t i o n methodology. the  site  financial  e v a l u a t i o n e f f i c i e n c y w i l l demand and  The c r i t e r i a of t e c h n i c a l adequacy, u t i l i t y described  efficiency  and  the  efficiency  s u i t a b i l i t y of  Evidence was c o l l e c t e d  at  d e s c r i b e d i n the next s e c t i o n and matched a g a i n s t the  standards set out above.  Study S i t e  A r e s i d e n t i a l program was suitable for testing  determined  to  be  particularly  i l l u m i n a t i v e e v a l u a t i o n because i t has some  unique advantages that do not e x i s t  i n other program formats.  (1)  The advantage of detachment from the usual r o u t i n e and the sense of freedom t h i s imparts.  (2)  The advantage of an environmental break which affords a challenge by the new environment to another p a t t e r n of behavior.  (3)  The advantage of c o n c e n t r a t i o n on one work without the usual d i s t r a c t i o n s .  (4)  The advantage integration.  (5)  The advantage of intimacy of students and which r e i n f o r c e s new knowledge.  (6)  The advantage of a community spirit which encourages tolerance and open-mindedness (Schacht, 1960, pp. 2-3).  of  time  for  field  assimilation  of and  tutors  44  The  c h i e f advantage of a r e s i d e n t i a l program over the more  traditional  types  temporarily  from  was  that  of  h i s ongoing  removing  the  participant  responsibilities.  T h i s made i t  p o s s i b l e f o r the e v a l u a t o r to have continuous contact participants.  Continuous contact  that r e l i e s on fieldwork  i s important  with  the  f o r a methodology  techniques.  The J u s t i c e I n s t i t u t e of B r i t i s h Columbia was s e l e c t e d as a site  for this  programs.  since  i t offered  numerous r e s i d e n t i a l  The J u s t i c e I n s t i t u t e as a post-secondary e d u c a t i o n a l  institution network  study  i s a member  of  British  Columbia's post-secondary  of c o l l e g e s and i n s t i t u t e s .  I t p r o v i d e s l e a d e r s h i p and  c o o r d i n a t i o n to support, develop and d e l i v e r training  and  education  a  wide  range  of  programs f o r people working w i t h i n the  f i e l d of j u s t i c e and p u b l i c s a f e t y . to improve the q u a l i t y of j u s t i c e  These programs a r e designed and  public  safety  f o r the  c i t i z e n s of B r i t i s h Columbia. The  Land T i t l e School program of the J u s t i c e I n s t i t u t e was  i d e n t i f i e d as an a p p r o p r i a t e illuminative  evaluation  residential  methodology  program  met Edwards & Guttentag's (1975) c r i t e r i a specified  that  formal  testing  because i t had the unique  r e s i d e n t i a l c h a r a c t e r i s t i c s described previously.  They  for  evaluation  Moreover, i t  f o r formal e v a l u a t i o n . was  program was new, newly changed, or about to  appropriate i f a  change.  The  Land  T i t l e School program met that c r i t e r i o n , because i t was new. addition,  the  program  planner  f o l l o w i n g types of q u e s t i o n s .  was  seeking  answers  to the  Is t h i s program a good idea?  so, what can we do to make i t work as w e l l as p o s s i b l e ?  In  If  I f not,  45  how  can we d e v i s e something b e t t e r , given e x i s t i n g c o n s t r a i n t s ?  (Edwards  Parlett  & Hamilton,  1976).  For those reasons, the Land T i t l e School was  determined  to  a  be  &  Guttentag,  suitable  1975,  G a r s i d e , 1969;  program  to  test  illuminative  evaluation  methodology. The Land T i t l e School program was Institute  of  British  Columbia  M i n i s t r y of Attorney General. personnel  working  b e t t e r understand  for  Although reality The  were  various  of  the  s t a f f weren't o b l i g e d  to  offer  o f t e n the best way felt  training.  theory,  staff  regulations  such  process.  t o expedite i n d i v i d u a l  cases.  courses  awareness  area was  was  to  provide  i n three a r e a s :  and  supplemental  land law theory.  In land  law courses, the l e g a l context f o r the land r e g i s t r y process  law,  Land  T o p i c s i n c l u d e d p r i n c i p l e s of B r i t i s h Columbia  law h i s t o r y , and Land T i t l e The  second  area was  understanding  These  courses  was land  Act.  environmental  T i t l e O f f i c e r e f l e c t s and  wider community.  in  requests f o r e x p l a n a t i o n s .  environmental  The most important  presented.  and  assistance,  goal of the Land T i t l e School program  law  felt  The  registration  Land T i t l e personnel with job enrichment land  Branch,  the more knowledgeable the s t a f f member, the  i t would be to s a t i s f y  The  Title  Land  the l e g a l background of t h e i r work.  part  i t was  Justice  i n the o f f i c e s should have an o p p o r t u n i t y to  which  Director  easier  the  by the  The D i r e c t o r of Land T i t l e s  o f t e n had been requested to i n t e r p r e t procedures  developed  awareness.  The work of a  i s e f f e c t e d by a c t i v i t i e s of the aimed  to  provide  a  of the r e l a t i o n s h i p between the Land T i t l e  broader Office  46  and the environment  i n which  i t operates.  s e s s i o n s were presented on urban history,  operation  As  an  example,  land use, B r i t i s h Columbia  land  of a lawyer's o f f i c e and an a n t h r o p o l o g i c a l  view of l a n d . The  t h i r d area  training  was  successful training into  to  supplemental public  supplemental  essential  completion was  how  not  was  of  to  training.  Supplemental  p r e s c r i b e d job performance or  promotional  exams.  Supplemental  designed to g i v e the p a r t i c i p a n t s g r e a t e r i n s i g h t perform  their  various  tasks.  Examples  t r a i n i n g are l e g a l d e s c r i p t i o n s , documentation and  relations.  Three courses made up the Land T i t l e School program. were  the  overview  The  f o r newly h i r e d c l e r k s p r o v i d e d an  of B r i t i s h Columbia's  heritage.  Land T i t l e  system and  i t s legal  The core l e g a l knowledge course was p r e s e n t e d i n the  week  Intermediate  participants  an  Course.  understanding  The  intensive  program  gave  of the law r e l a t i n g to land and  the l e g a l , s o c i a l and economic i m p l i c a t i o n s of five  They  I n t r o d u c t o r y , Intermediate and Advanced Courses.  three day I n t r o d u c t o r y Course  two  of  land  day Advanced Course c o n c e n t r a t e d on s p e c i f i c  use.  land  The  registry  i ssues. Each course was d i v i d e d sessions.  into a  series  by areas  the  from a  needs  J u s t i c e I n s t i t u t e and conformed  identified  mini-  resource  f a c u l t y , c o n s u l t a n t s , and lawyers.  The m i n i - s e s s i o n content was developed  content  half-day  Each m i n i - s e s s i o n was taught by a d i f f e r e n t  person drawn from u n i v e r s i t y  conducted  of  as  essential--land  assessment to the three  law  theory,  47  environmental The  Land  criteria  an  the  supplemental  School  Secondly,  program  training.  program as o u t l i n e d above met  appropriate  methodology.  program.  have  Title  for  evaluation new  awareness and  site  Firstly,  for  testing  illuminative  the Land T i t l e School was  the program c o o r d i n a t o r was  evaluated.  the  Lastly,  it  was  anxious  a to  offered  on a  would  have  The  first  r e s i d e n t i a l b a s i s which meant that the i n v e s t i g a t o r continuous c o n t a c t with the p a r t i c i p a n t s .  Summary  This  chapter  was  divided  into  two  sections.  s e c t i o n c o n t a i n e d a d e s c r i p t i o n of the c r i t e r i a the s u i t a b i l i t y of i l l u m i n a t i v e e v a l u a t i o n criteria  appearing  frequently  a p p r o p r i a t e to t h i s  study:  efficiency.  literature  The  methodology should produce sound, audience In  (2)  useful  than  the  technical  f o r judging  methodology.  Three  l i t e r a t u r e were judged adequacy,  suggested  i n f o r m a t i o n that  some audience and  that is  utility, an  (1)  evaluation technically  (3) worth more to the  t h i s chapter, the c r i t e r i a were d e f i n e d , standards  the  method f o r c o l l e c t i n g evidence to judge each  was  described.  methodology and v a l i d .  and  i t c o s t s (Grotelueschen, 1980).  and  and  to  in  used  must  In  order  first  to  meet  the  set  Second, the evidence must  be  r e l e v a n t while the methodology remains  criterion  standards,  p r o v i d e evidence that i s both  set,  the  objective  credible,  important,  flexible.  Third,  the  48  evidence must be worth more than the c o s t s . The of  second s e c t i o n of t h i s chapter c o n t a i n e d a d e s c r i p t i o n  the  site  selected  methodology.  A  for testing  residential  illuminative  program  particularly suitable for testing this had  some  formats. more  types  was  that  i n other program  responsibilities.  continuous  contact  Land  This  for testing  is  important  in a  techniques.  an  appropriate  residential  i l l u m i n a t i v e e v a l u a t i o n methodology because  had the unique r e s i d e n t i a l c h a r a c t e r i s t i c s d e s c r i b e d i n  chapter.  The  Land  Title  designed f o r v o l u n t a r y to  those  who  School  was  a  job enrichment. wished a g r e a t e r  residential It  was  insight  These  residential basis. residential  program  the s u i t a b i l i t y  courses  The f a c t and  would  be  offered  that the Land T i t l e  program  intended  duties.  courses designed f o r the program were I n t r o d u c t o r y , Advanced.  this  to  i n t o t h e i r work  than was r e q u i r e d to competently perform assigned  and  made i t  T i t l e School program of the J u s t i c e I n s t i t u t e of  B r i t i s h Columbia was i d e n t i f i e d as  appeal  the  of removing the p a r t i c i p a n t  methodology that r e l i e s on fieldwork  it  because i t  f o r the e v a l u a t o r to have continuous c o n t a c t with the  p a r t i c i p a n t s , because  program  methodology  to be  The c h i e f advantage of a r e s i d e n t i a l program over  t e m p o r a r i l y from h i s ongoing  The  determined  unique advantages that d i d not e x i s t  traditional  possible  was  evaluation  The  Intermediate y e a r l y on a  School  was  a  new made i t an i d e a l s i t e f o r t e s t i n g  of i l l u m i n a t i v e e v a l u a t i o n  methodology.  49  CHAPTER IV  ILLUMINATIVE EVALUATION STRATEGY  T h i s chapter p r o v i d e s evaluation  was  a  employed  discussion in  this  i l l u m i n a t i v e methodology was u t i l i z e d the to  development  a  how  illuminative  study.  Briefly,  natural,  environment  in  I t was used  r e l a t i o n s h i p s that c o u l d be  non-contrived  the  to e v a l u a t e each phase  of the Land T i t l e School program.  d e s c r i b e and understand  in  of  under  observed controlled  conditions. The f i r s t  or p i l o t phase of the Land T i t l e  School  program  represented a t r i a l and e r r o r p e r i o d d u r i n g which new approaches or  procedures  were  tried  revised basis.  During  modifications  occurred.  Intermediate course,  the  the  development The  first  Course (March, 1980).  of  the  pilot  In order  program, course  to  l e d to the development  courses—Introductory  Course  Advanced Course (December, 1980).  For  (November,  was  evaluate  i l l u m i n a t i v e e v a l u a t i o n methodology was  The success of t h i s course pilot  out on a r a t h e r f l e x i b l e and e a s i l y  the this  employed.  of  two  1980)  consistency,  some  the  more  and the three  50  stage  illuminative  methodology  was  o b j e c t i v e of the e v a l u a t i o n of the enough to f u r t h e r develop The  second  or  utilized  pilot  again.  phase  was  The main to  learn  the program.  o p e r a t i o n a l phase of the Land T i t l e  School  program c o n s i s t e d of m o d i f i c a t i o n s of a l l three courses. on  what  was  learned  i n the p i l o t  m o d i f i e d so that they stood the This  final  phase  was  also  Based  courses, these courses were  greatest  chance  evaluated  of  success.  using the i l l u m i n a t i v e  methodology. In order to o p e r a t i o n a l i z e the i l l u m i n a t i v e methodology,  a  number of steps had to be taken at the o b s e r v a t i o n , i n q u i r y , and explanation repeated  stages  for  each  (see  Table  course.  2).  The  These  result  three stages were  was  six  complete  e v a l u a t i o n c y c l e s s i n c e each of the three courses was run twice. In  the  remainder  of  this  chapter,  processes  used  i n each  e v a l u a t i o n c y c l e w i l l be d e s c r i b e d .  Pilot  The started  first  Phase  e v a l u a t i o n c y c l e of the i l l u m i n a t i v e  with the p i l o t  Intermediate  Course.  stage, the i n v e s t i g a t o r became f a m i l i a r Course through with  the  In the o b s e r v a t i o n the  Intermediate  a n a l y s i s of background documentation, d i s c u s s i o n s  program  coordinator  f a m i l i a r i z a t i o n process enabled the i n q u i r y  with  methodology  stage.  and  Land T i t l e D i r e c t o r .  the i n v e s t i g a t o r to  proceed  This to  51  The  inquiry  stage  was  an  e v a l u a t o r and program planner.  interactive  T h i s stage  process  consumed  between  the  major  p o r t i o n of the i n v e s t i g a t o r ' s time because a number of steps had to  be  taken.  For  example,  e v a l u a t i o n process determined, collected.  The  issues  had  to be c l a r i f i e d ,  q u e s t i o n n a i r e s designed and  following  paragraphs  present  d e s c r i p t i o n s of the steps taken d u r i n g t h i s i n q u i r y  Issues  data  detailed  stage.  Clarified  First, the  the f o l l o w i n g i s s u e s were d i s c u s s e d and  clarified--  purpose of the e v a l u a t i o n , what process should be used,  the i n f o r m a t i o n would be used, what t o p i c s for  intensive  investigation.  When  should  these  be  issues  clarified,  the s p e c i f i c e v a l u a t i o n process was  Evaluation  Process  and e v a l u a t i o n can be c a r r i e d out at any of study,  levels  it  was  had  (1)  participant  been  determined.  these  levels  levels.  (1967)  from the lowest r e a c t i o n s , or how  and  level,  Hamblin  (1974).  are:  w e l l they l i k e d  the  program; (2)  learning  or  the  extent  to  which  the program  content was a s s i m i l a t e d ; (3)  In  decided to concentrate on the h i e r a r c h i c a l  i d e n t i f i e d by K i r k p a t r i c k  These l e v e l s , s t a r t i n g  how  selected  E v a l u a t i o n processes can be d i v i d e d i n t o a number of  this  the  behavior change, or the change i n job behavior;  52  (4)  results,  or  the  change  in  organizational  variables. Kirkpatrick  and  Hamblin  e f f e c t c h a i n l i n k i n g the could  break  at  any  r e a c t c o r r e c t l y but to  apply  his  assumed  four l e v e l s .  of i t s l i n k s .  fail  This  on the  For example, a person  could  The  of  the  the  degree  increases  as  difficulty one  the  organization.  is  the  hierarchy  i s climbed,  why.  of these l e v e l s ,  however,  i n c o l l e c t i n g evidence at each  ascends  level  the  hierarchy.  simplest  and  easiest  participant level.  As  the  required  measure a c t u a l program outcomes g e n e r a l l y i n c r e a s e  (Bennett,  results  The  the d i f f i c u l t y and  The  level  the resources  1975).  d i f f i c u l t y o f t e n s t a r t s at the behavior  levels  because  the  about  activities  organization.  of  the  which are used to evaluate those  which  the  or  evaluator  adequate information  at  organization  uses f o r other purposes.  control  those  does over  not the  change  non-training  Furthermore, the levels  a l r e a d y has  If the a p p r o p r i a t e  will  exist  techniques  normally  at i t s d i s p o s a l techniques  i n the o r g a n i z a t i o n , e v a l u a t i o n at the higher  impossible  introduced  for  because  techniques  education  of  this  kind  and  u s u a l l y have  such  p r o d u c t i v i t y measurements or c o s t - b e n e f i t a n a l y s i s don't  be  fail  i s to determine i f the l i n k s i n the  be c o l l e c t e d at any  reaction  to  on  i f they don't where they broke and  of  but  job; or he c o u l d change h i s job  t h i s c o u l d have no e f f e c t evaluator  and  chain  but  Evidence can  a cause  to l e a r n ; or he c o u l d l e a r n ,  learning  chain h o l d and  was  hierarchical  behavior, job  there  be and as  already  levels  will  cannot  be  or t r a i n i n g purposes alone  (Hamblin,  53  1974).  Therefore,  evaluate at every In  the  in  many  cases  it  may  i t was  at the r e s u l t s l e v e l ,  for the Land T i t l e  techniques  for  up  T h e r e f o r e , i t was  impractical  evaluation  decided to  levels: participant  at  concentrate  i m p r a c t i c a l to evaluate Branch the  not  results  on  r e a c t i o n s , l e a r n i n g , and  did  the  first  behavior  procedures This  and and  guidance  Hamblin  three  change.  gave d e t a i l e d examples, and  techniques was  lacking  have  level.  Eesides d e s c r i b i n g the l e v e l s of e v a l u a t i o n f o r t h i s Kirkpatrick  to  level.  case of t h i s study,  set  be  study,  suggested  that c o u l d be used i n most  programs.  i n the l i t e r a t u r e on the  illuminative  e v a l u a t i o n methodology.  Questionnaire In  Design  the  investigator  process  of  discovered  that  should never be c a r r i e d (Bennett,  1975;  t h a t , although is  designing  out  unless  it  has  clear  Stufflebeam,  1967).  objectives  i t ' s p e r m i s s i b l e to c a r r y out such a program,  it  evaluate  i t . However, there are people  d i s a g r e e with both p h i l o s o p h i e s (Hamblin, Rackham,  c l a i m a program  say  to  1978;  theorists  the  Others  impossible  Patton,  some  questionnaires,  1970).  This  study was  1974;  Warr,  guided by the l a t t e r  Bird  who &  authors,  for t h i s program d i d not have any measureable o b j e c t i v e s . It was Title  School  participant known  difficult  about  program  to set measureable o b j e c t i v e s f o r the Land or  r e a c t i o n s and the  specific  sessions  even  at  l e a r n i n g l e v e l s , because so l i t t l e  p a r t i c i p a n t s ' p r e v i o u s s t a t e of l e a r n i n g .  the was In  54  cases the  when o b j e c t i v e s are not best  simply  way  to  in  participants  whether they  whether  they  they  remember  find  from  a  or  program.  most In  the e v a l u a t i o n must adopt open-ended  for  study  contained  mainly  the  techniques.  open-ended  course  job-relevant absence  of  criteria,  Due  of measureable o b j e c t i v e s , the q u e s t i o n n a i r e s  this  be  improved i n  b e h a v i o r a l o b j e c t i v e s which s p e c i f y p r e c i s e e v a l u a t i o n  absence  terms,  the  think t h e i r knowledge has  s p e c i f i c areas and/or the most important point  measureable  of a s s e s s i n g r e a c t i o n s or l e a r n i n g changes may  ask  interesting,  formulated  to  the  developed  questions.  The  q u e s t i o n n a i r e s developed are d e s c r i b e d below. Expectations  Questionnaire  (See  Appendix  A).  q u e s t i o n n a i r e was  used to assess p a r t i c i p a n t e x p e c t a t i o n s  Land T i t l e School  p r i o r to the s t a r t of the program and  conclusion. Rackham  The  (1970,  questionnaire, p.  65),  was  developed used  participants'  expectations  relevance, and  importance of the course.  a  course  with  determining  a  their  set  regarding  to  of  was  different could  divided  resource  be  used  perceived session  a  into  the  used to  learning. on  obtain  at  its  feedback  which are  on  enjoyment,  Participants  a  (See Appendix  approach  important  in  Since  the  scale  end  B).  s e r i e s of m i n i - s e s s i o n s  persons, a q u e s t i o n n a i r e  at  q u e s t i o n n a i r e was  the  r e a c t i o n s to the program.  Mini-session Questionnaires course  of  by Warr, B i r d , and  the u s e f u l n e s s ,  expectations  This  was  of each s e s s i o n .  assess  taught  designed The  participants'  by  that  mini-session  reactions  and  P a r t i c i p a n t s were asked to r a t e each m i n i of  1  to  7  from  "not  very..."  to  55  "extremely..."  on  interest,  i n f o r m a t i o n gained.  relevance  to  job,  and  new  In other words, p a r t i c i p a n t s were asked  to  make d e c i s i o n s about the u s e f u l n e s s of s p e c i f i c content areas i n terms  of  three  important  contexts:  relevance  to  their  e x p e c t a t i o n s , p e r c e i v e d a p p l i c a t i o n to t h e i r work s i t u a t i o n their  previous  knowledge.  In  addition  to  rating  p a r t i c i p a n t s r a t e d the m i n i - s e s s i o n s using open-ended The q u e s t i o n n a i r e , o r i g i n a l l y developed Bird  (1970),  was  adapted  i n v e s t i g a t o r and program administered Final  immediately  determine  immediately how  c o n s i s t e d of  reactions  to  the  following  rating  program  the  felt  as  Questionnaire  completion  questionnaire  of  the  consisting  questionnaire. series  of  The  study  and  by  the  questionnaire  was  of  This questionnaire,  to  was  the  used  program.  assess  gained  to  to The  participant  their  beliefs  work.  A  also included.  (See Appendix D).  F o r t y - f i v e days  participants  several  sections  of  instrument  the  questions  about  the  i n f o r m a t i o n , the e f f e c t of course on p a r t i c i p a n t and p a r t i c i p a n t  responses.  a whole and p a r t i c i p a n t  course,  remainder  open-ended  scales,  Rackham  program,  about  scales  s e r i e s of open-ended q u e s t i o n s was  after  this  The  (See Appendix C ) .  about the relevance of i n f o r m a t i o n  Follow-up  in  Warr,  and  a f t e r each m i n i - s e s s i o n .  participants  instrument  use  coordinator.  Questionnaire  administered  for  by  own  were of  sent  the  final  contained relevance job  s a t i s f a c t i o n with the general program.  a  a of  behavior,  56  Data C o l l e c t i o n Lastly, Course,  in  data  interviews  the  were and  the  started  investigator gave  to c o l l e c t  and  pilot  introduced  the  investigator through  Director  informal used  the  questionnaires. of  Intermediate  study,  Intermediate  observation,  The data  director the  of the p i l o t  through  questionnaires.  program  officially  stage  collected  f o l l o w i n g procedures After  inquiry  Land  Titles  Course,  the  s o l i c i t e d cooperation  and  instructions. A l l p a r t i c i p a n t s were  identity  code  that was  instructed  of  mini-session  participants  the  investigator.  The  day  was  of  end  of each m i n i - s e s s i o n and  course  follow-up q u e s t i o n n a i r e was a f t e r course  These q u e s t i o n n a i r e s  f i n a l q u e s t i o n n a i r e was  the  numerical  completion.  Then  provided  so they c o u l d r a t e each s e s s i o n that was  at  final  a  given to a l l p a r t i c i p a n t s .  questionnaires  a d i f f e r e n t resource person. completed  select  to be used on a l l q u e s t i o n n a i r e s .  the e x p e c t a t i o n s q u e s t i o n n a i r e was packet  to  to  taught  were  A  to  by be  returned to the  administered  on  the  d u r i n g the e v a l u a t i o n s e s s i o n .  The  sent to p a r t i c i p a n t s  Table  f o r t y - f i v e days  4 summarizes the data c o l l e c t i o n  schedule. The  data c o l l e c t i o n step completed the i n q u i r y  i l l u m i n a t i v e methodology. stage enabled  the  The  investigator  steps taken to proceed  during  stage of the  the  inquiry  to the t h i r d and  final  stage of the methodology. In pilot  the  e x p l a n a t i o n stage, a l l the data c o l l e c t e d from the  Intermediate  Course were analyzed  using e i t h e r q u a l i t a t i v e  Table 4 Data C o l l e c t i o n Schedule  TECHNIQUE  SOURCE  H i s t o r i c a l Documents Such as course proposal, brochure.  Program minutes  of  meetings,  letters,  TIME  Director  Two  weeks p r i o r  to  course  course  Expectations Participants This questionnaire assessed p a r t i c i p a n t expectations p r i o r to s t a r t of course. F o r example, w o u l d i t be u s e f u 1 - u s e 1 e s s ; h e l p f u l - u n h e l p f u l ; and important-unimportant.  Prior to start of minis e s s i o n s on t h e f i r s t d a y of course  Mini-sessional This questionnaire c o n c e r n e d i n t e r e s t of s e s s i o n i n f o r m a t i o n r e l e v a n c e t o j o b , l e n g t h o f s e s s i o n , l e v e l of s e s s i o n .  Participants  End o f m i n i - s e s s i o n  Investigator  During  courses  Participants  During  courses  Participants  Final  Participants  45 days following day o f c o u r s e  Observation of p a r t i c i p a n t s during  Interview during  breaks before  course,  and a f t e r  Expectations & Final These questionnaires e x p e c t a t i o n s and t h e i r  at  gained,  breaks.  class.  day o f  course  assessed fulfillment of participants' f e e l i n g s a t t h e end of the c o u r s e .  Follow-up T h i s q u e s t i o n n a i r e was c o n c e r n e d w i t h r e l e v a n c e o f c o u r s e on j o b b e h a v i o r a n d s a t i s f a c t i o n w i t h  of i n f o r m a t i o n , e f f e c t g e n e r a l program.  last  58  or q u a n t i t a t i v e techniques. (n=43), only simple All  This  technique  data,  class  size  obtained  through  n u m e r i c a l l y by p a r t i c i p a n t  enabled  was  small  participants  calculated  for  each  questionnaires identity  code.  the i n v e s t i g a t o r to t r a c e the r a t i n g s of  individual  computer.  the  a n a l y t i c a l procedures were used.  quantitative  only, were arranged  Since  if  required.  rating  scale  Mean  using  a  scores small  These scores were used to develop summary  were  t a b l e top and  trend  charts. Qualitative observations, from  data  obtained  from  and i n t e r v i e w s were typed  questionnaires  were  arranged  questionnaires,  to a i d  analysis.  Data  n u m e r i c a l l y by p a r t i c i p a n t  i d e n t i t y code; thus, each p a r t i c i p a n t ' s comments c o u l d be c r o s s checked with the q u a n t i t a t i v e data. each  mini-session  were combined and typed.  followed f o r each open-ended q u e s t i o n questionnaires.  A l l coded  responses  for  T h i s procedure was  on the f i n a l and  follow-up  The procedure of combining and t y p i n g responses  f a c i l i t a t e d ease of a n a l y s i s and e l i m i n a t e d b i a s e s c r e a t e d by an individual's All  handwriting.  the  information  data  profile  answer the q u e s t i o n s previously. to  collected of  the  were  used  program.  to The  establish  data were used to  posed by the program c o o r d i n a t o r  Is t h i s program a good idea?  make i t work as w e l l as p o s s i b l e ?  an  mentioned  I f so, what can we do  I f not, how can we  devise  something b e t t e r , given e x i s t i n g c o n s t r a i n t s ? When i n t e r p r e t a t i o n of the report  was  written  data  was  completed,  a  final  and sent to a l l p a r t i c i p a n t s , i n s t r u c t o r s ,  59  Land T i t l e r e g i s t r a r s , and A d v i s o r y Committee  members  (Hasman,  Note 4 ) . With the d i s t r i b u t i o n of the r e p o r t , the f i r s t three  stages of the i l l u m i n a t i v e methodology was complete.  first any  c y c l e of the  e v a l u a t i o n c y c l e r e q u i r e d more of  the  subsequent  investigator's  time  c y c l e s because a l l procedures  than  f o r each  stage had to be e s t a b l i s h e d .  The i n v e s t i g a t o r  become  T i t l e work, the p e r s o n n e l , and the  familiar  program.  with  Land  had  The  to  Then, with the c o o p e r a t i o n of the program c o o r d i n a t o r ,  i s s u e s had to be c l a r i f i e d and d e f i n e d and e v a l u a t i o n determined. developed,  Next  Finally  The  operationalization  questionnaires  analyzed.  second  designed,  respectively. cycles,  been  clarified,  third  cycles  amount  to be  collected  and  Since there were no changes  evaluation  processes  time and e f f o r t  i n q u i r y stages of the second  stages  I n t r o d u c t o r y and Advanced  they w i l l be d e s c r i b e d t o g e t h e r .  of  had  of the i l l u m i n a t i v e  q u e s t i o n n a i r e s designed d u r i n g the p i l o t the  data  processes  r e s u l t s had to be r e p o r t e d . and  these  procedures and  s t a r t e d a f t e r development of the p i l o t Courses  quickly  made  during  The i s s u e s had  determined  and  Intermediate Course, so  i n v o l v e d i n the o b s e r v a t i o n and  and  third  cycles  was  decreased  significantly. A  few  modifications,  o b s e r v a t i o n stage.  A  decision  however, was  made  i d e n t i f i c a t i o n code of each p a r t i c i p a n t the  group  second,  size  had  been  there was no b e n e f i t  reduced  were to  made  during  the  d i s c o n t i n u e the  f o r two reasons.  First,  t o a maximum of twenty and  in tracing  individual  responses.  60  Besides  eliminating  needed a l t e r a t i o n s .  the  numerical  For  example,  the  sessional,  follow-up  questionnaires  suggestions  from the program c o o r d i n a t o r , the  Advisory  Council  questionnaire exercise.  members  was  This  were  codes, some q u e s t i o n n a i r e s  modified  replaced  by  an  a  and  result  of  i n v e s t i g a t o r and  (see Appendix E ) .  The e x p e c t a t i o n s  "expectations  warm-up"  q u e s t i o n n a i r e was changed, because the program  c o o r d i n a t o r wanted to e l i m i n a t e some of the The  as  final  modification  provided  both  evaluation  expectations  forms.  data and a group  "warm-up" (See Appendix E ) . After  making  Introductory  the  and  above  Advanced  modifications  f o r both  Courses, the i n v e s t i g a t o r c o l l e c t e d  data by i n t e r v i e w s , o b s e r v a t i o n s , and q u e s t i o n n a i r e s . c o l l e c t i o n methods were the same as those d e s c r i b e d cycle. stage  The  explanation  stage  except  that  the  responses  were  open-ended responses,  so that the i n v e s t i g a t o r  could  Like  the  Course  pilot  Intermediate  w r i t t e n and d i s t r i b u t e d The  during  the  This first  of  a  interpret  the  computer. notes were results.  e v a l u a t i o n , a report was  (Hasman, Note 4 ) .  second and t h i r d c y c l e s through the i l l u m i n a t i v e stages  were then complete. during  i n the f i r s t  i n t e r v i e w s and o b s e r v a t i o n  typed  data  not coded by i d e n t i t y  number and a desk c a l c u l a t o r was used i n s t e a d All  The  followed data c o l l e c t i o n .  followed the same procedures set down  cycle  pilot  As mentioned, the time and e f f o r t  involved  these e v a l u a t i o n c y c l e s was reduced s i g n i f i c a n t l y due to  the procedures e s t a b l i s h e d d u r i n g the f i r s t c y c l e .  61  O p e r a t i o n a l Program  The  last  operational  three e v a l u a t i o n c y c l e s were completed during phase  of  the  Land  Title  School program.  f i n a l c y c l e s through  the i l l u m i n a t i v e stages, the  made  program  were  to  the  itself.  only  In the changes  A f t e r a few minor program  a l t e r a t i o n s , the three courses were i n f i n a l o p e r a t i o n a l Evaluations  of  each course  d u r i n g the f i r s t all  form.  followed the procedures e s t a b l i s h e d  three c y c l e s of the i l l u m i n a t i v e s t a g e s .  Since  three c y c l e s were the same, they w i l l be d e s c r i b e d t o g e t h e r . The  and  the  i n v e s t i g a t o r reconfirmed  proceeded  t o gather data through  and q u e s t i o n n a i r e s .  In due course,  interpreted,  and  the  coordinator,  program  members,  the focus  reported.  the f i n a l  of  the  evaluation  observations, interviews,  the  data  were  organized,  Upon p r e s e n t a t i o n of the r e p o r t t o participants  and  Advisory  Council  three c y c l e s of the i l l u m i n a t i v e stages were  completed.  Summary  T h i s chapter provided evaluation  methodology  a  discussion  was  employed  of in  how this  i l l u m i n a t i v e e v a l u a t i o n methodology was u t i l i z e d three courses of the Land course  ran  in  both  a  Title pilot  School and  an  illuminative study.  The  to evaluate a l l  program. operational  Since  each  form, s i x  62  e v a l u a t i o n c y c l e s of the  three-stage  illuminative  methodology  were used. In  the  first  methodology, time was observation,  inquiry,  third cycles  of  observation  cycle  and  illuminative  explanation.  methodology,  inquiry  the procedures had  the  stages.  been e s t a b l i s h e d  During  time  was  during  efforts  on  those  stages.  r e q u i r e d somewhat l e s s time. which reduced the amount of The  last  encompassed  three  the  three  T h i s was  majority  explanation all  of  stage.  six cycles.  the The  the second in  the  to the f a c t  that  the  first  cycle.  i n v e s t i g a t o r d i d not  explanation  due  and  stage  also  to s m a l l e r c l a s s s i z e  data.  cycles course  of  the  illuminative  operational  c y c l e s , no changes were made to the the  The  stages:  reduced  T h i s i s due  Since only minor a l t e r a t i o n s were made, the focus  evaluation  e q u a l l y d i v i d e d between the three  the  and  of  program.  evaluation  investigator's  time  was  a n a l y s i s took about the  stages In  these  procedures, spent same  on  time  so the for  63  CHAPTER V  RESULTS  The  literature  reviewed  f o r t h i s study suggested  e v a l u a t i o n methodology should produce technically  sound,  (2)  more to the audience  useful  evaluation  used in  to  that  assess  relation  I I I were s e l e c t e d f o r t h i s  the  i s (1)  to some audience and (3) worth  than i t c o s t s (Groteleuschen, 1980).  c r i t e r i a d e s c r i b e d i n Chapter Evidence  information  that an  suitability  of  These study.  illuminative  to these c r i t e r i a w i l l be presented i n  t h i s chapter.  T e c h n i c a l Adequacy  A t e c h n i c a l l y adequate e v a l u a t i o n methodology w i l l evidence  which  meets the standards of o b j e c t i v i t y and v a l i d i t y  as d e f i n e d i n Chapter adequate,  the  produce  III.  methodology  In order to must  be  produce  judged data  technically that  can  be  64  confirmed.  In other words, the burden of proof moves  i n v e s t i g a t o r to the i n f o r m a t i o n Illuminative  evaluation  of data from m u l t i p l e cross-check and v a l i d  and  sources  confirm  information.  methodology encouraged and  perspectives  in  collection order  to  T h i s would ensure o b j e c t i v e  Procedures  confirming  results  observation.  The evidence presented  support  technical  the  the  itself.  results.  were  from  used  in  triangulation in  the  this  study  and  continuous  paragraphs  for  below  adequacy of the i l l u m i n a t i v e e v a l u a t i o n  methodology.  Triangulation T r i a n g u l a t i o n was  used e x t e n s i v e l y i n t h i s study to p r o v i d e  evidence of o b j e c t i v i t y and  validity.  Triangulation f o r c e s the observer to combine multiple data sources, research methods, and theoretical schemes in the inspection and a n a l y s i s . . . . 11 f o r c e s him to s i t u a t i o n a l l y check the v a l i d i t y of h i s c a u s a l propositions....It directs the observer to compare h i s s u b j e c t ' s theories of behavior with his emerging t h e o r e t i c a l scheme.... (Denzin, 1971, p. 177). The  first  example  of  triangulation  involved  q u a l i t a t i v e and q u a n t i t a t i v e data from each quantitative  data  consisted  of  mean  cross-checking  mini-session.  scores  from each m i n i -  s e s s i o n while the q u a l i t a t i v e data c o n s i s t e d of typed from each m i n i - s e s s i o n .  The  responses  As a r e s u l t of c r o s s - c h e c k i n g these  two  types of data, the r a t i n g s of each m i n i - s e s s i o n were confirmed. Additionally, and  comments  from  the f i n a l q u e s t i o n n a i r e confirmed  f u r t h e r v a l i d a t e d the m i n i - s e s s i o n data.  65  Besides  cross-checking  and  confirming,  the q u a l i t a t i v e data  were used f o r i n t e r p r e t i n g q u a n t i t a t i v e data, cannot t e l l why instance,  several  v a r i a b l e "new same  someone  high.  The  a  session  participants  information  session  were low.  rated  rated  gained"  The  while  because  high  or  sessions  statistics low.  low  the m a j o r i t y  i n v e s t i g a t o r wondered why  For  on  the  rated  the  the  ratings  f o l l o w i n g comments i l l u m i n a t e d the problem.  This material was just covered R e g i s t r a r ) before t h i s course.  by  (Mr.  Everybody 'jogs'. I am aware of s t r e s s and h e a l t h hazards and I would think most of the c l a s s would be. In  another  instance,  a s e s s i o n r a t e d p o o r l y on  "information  gained"  qualitative  information,  there  problems  were  qualitative  but  high the  with  information  on  and  Without  the  "relevance."  investigator  content  revealed  "interest"  and/or  the  would  not  know i f  instructor.  The  problem:  I've waited two weeks e s p e c i a l l y f o r t h i s l e s s o n and was thoroughly d i s a p p o i n t e d . I think this specific area could prove b e n e f i c i a l to us a l l and gone i n t o some depth. The level of 'presentation' we received was f a r below that which c o u l d prove u s e f u l and r e l e v a n t . In  addition  qualitative difficult noted that  to  information  to capture  cross-checking provided  recognizing  reports"  richness  i n a q u a n t i t a t i v e summary. these  "immediately l e g i t i m i z e s the evaluative  a  and  (p.  functions  of  interpretation, of  description  Campbell  (1975)  qualitative  data  ' n a r r a t i v e h i s t o r y ' p o r t i o n of most 9).  He  suggested the  importance of  66  q u a l i t a t i v e data be given formal r e c o g n i t i o n i n the p l a n n i n g  and  execution  are  of  evaluations.  uninterpretable  without  this,  i n t e r p r e t e d with more" (p. A  second  responses  example  on  "Evaluation  of  variation  the  triangulation  and met,  see  The  P a r t i c i p a n t s were day  better  asked  hoping  Three  used.  The  expectations comments  to  write  down  their  Then, on the f i n a l from the f i r s t  If t h e i r e x p e c t a t i o n s were  were asked to e x p l a i n .  of  second v a r i a t i o n c o n s i s t e d of  of c l a s s .  i f they were f u l f i l l e d .  wrote "I'm  were  type s c a l e to open-ended  asked to check t h e i r e x p e c t a t i o n s  they  be  comparison  c o n s i s t e d of comparing pre-course  e x p e c t a t i o n s on the f i r s t were  was  questionnaire  f i n a l questionnaire.  the f o l l o w i n g .  they  would  pre- and p o s t - e x p e c t a t i o n s q u e s t i o n n a i r e s .  using a semantic d i f f e r e n t i a l on  most  9).  v a r i a t i o n s of the e x p e c t a t i o n s first  and  studies  For example, one  not to be bored."  Comments  from  q u e s t i o n n a i r e r e f l e c t e d that the p a r t i c i p a n t had  day, day not  participant the  final  not been bored:  I came i n t o t h i s course expecting to be bored and was surprised about the competence of the i n s t r u c t o r s and the time they took to e x p l a i n the answers to a l l our q u e s t i o n s . In some cases, p a r t i c i p a n t s had c e r t a i n e x p e c t a t i o n s as a r e s u l t of  talking  illustrates  to how  former  participants.  participants  can  e x p e c t a t i o n s and have them changed by the  The come  following with  comment erroneous  course.  Did not have e x p e c t a t i o n s but was more informative and i n t e r e s t i n g than I was l e d to b e l i e v e by p r e v i o u s students.  67  The  t h i r d variation consisted  expectations  orally  on  of p a r t i c i p a n t s the  first  day  p a r t i c i p a n t ' s e x p e c t a t i o n s were noted and day,  participants  wrote  whether  F i n a l l y , a f t e r comparing the  the  ones,  typed  differences.  the  I t was  course would be  participants found  that  job t r a i n i n g as  their  class. On  Each  the  final  e x p e c t a t i o n s had written  were  many  of  typed.  their  fulfilled.  presenting  been  responses  asked  to  to  note  any  participants  felt  the  r e f l e c t e d i n t h i s comment:  Thought i t would be more i n l i n e with my work duties, but because of the way the system (L.T.O.) i s s t r u c t u r e d , I can see why i t was more general. A  third  interviews. check  The  saying one A  final  information. the  that to  another he  to  triangulation  example Two  from  determine  o r a l l y and  involved  of  the if  another t h i n g  the in  triangulation  participants  should be h i r e d by Land T i t l e  a l l employees.  responses from the  crossThe were  writing. unsolicited to  say  Intermediate Course promotional t o l d one  to o f f e r h i s  exam.  instructor  mini-session  These u n s o l i c i t e d comments f u r t h e r courses.  from  participants  investigator  helped them pass t h e i r several  used to  involved  i n f o r m a t i o n gained from the p i l o t  instance,  data  mini-sessions.  p a r t i c i p a n t s c a l l e d the  p l u s t h e i r notes had In  data  checked  thing  of  information from i n t e r v i e w s was  quantitative  investigator  that  example  confirmed  68  Continuous o b s e r v a t i o n If one way t o e s t a b l i s h the v a l i d i t y of data use of t r i a n g u l a t i o n , another observations. reasons  i s through the  way i s through the use of repeated  E i s n e r (1975) makes the p o i n t that  why  "One  of the  i t i s important... to have extended c o n t a c t with an  educational situation  i s to  be  able  c h a r a c t e r i s t i c s that are a t y p i c a l .  to  recognize  events  One needs s u f f i c i e n t  or  time i n  a s i t u a t i o n to know which q u a l i t i e s c h a r a c t e r i z e i t and which do not"  (p.  218). Thus, v a l i d i t y  i s , to some extent, a f u n c t i o n  of the amount of time and e f f o r t which the i n v e s t i g a t o r in  repeated  and  continuous  i n v e s t i g a t o r be able  to  observation.  differentiate  Not  typical  invests  only w i l l the from  atypical  s i t u a t i o n s , or i d e n t i f y p e r v a s i v e q u a l i t i e s which c h a r a c t e r i z e a situation,  but  he  will  also  know when to give c r e d i t  o c c a s i o n a l i d i o s y n c r a t i c o b s e r v a t i o n which n e v e r t h e l e s s great  i n s i g h t and meaning  to the carries  (Guba, Note 2 ) .  Continuous o b s e r v a t i o n and e x t e n s i v e c o n t a c t s are hallmarks of  i l l u m i n a t i v e e v a l u a t i o n methodology.  p r o v i d e s a v a r i e t y of collected  by  any  information other  means.  that  Continuous o b s e r v a t i o n could  Through  i n v e s t i g a t o r d i s c o v e r e d the mood of the group. gained  through  not  have  observation, The  was  different.  For  the  information  o b s e r v a t i o n was passed on to upcoming  for the mood of each group  been  speakers,  example,  one  group d i d n ' t ask any q u e s t i o n s during p r e s e n t a t i o n s , although at coffee  breaks  they asked many q u e s t i o n s .  Upon q u e s t i o n i n g the  p a r t i c i p a n t s , the i n v e s t i g a t o r d i s c o v e r e d that the p a r t i c i p a n t s were  h e s i t a n t to i n t e r r u p t the speaker because they  felt  i t was  69  e i t h e r rude or d i s r u p t i v e . passed on t o succeeding The of  the  group.  alerted  the group was  speakers.  i n v e s t i g a t o r a l s o was a b l e t o observe  the  development  Some p a r t i c i p a n t s were shy or unaccustomed t o a  p a r t i c i p a t i v e group.  to  T h i s i n f o r m a t i o n about  When t h i s was observed,  the program c o o r d i n a t o r .  the  investigator  These i n s i g h t s were passed on  ensuing speakers who used the i n f o r m a t i o n t o h e l p the process  along; f o r example, the speaker might use an warm-up  exercise.  or  Through continuous o b s e r v a t i o n and e x t e n s i v e  c o n t a c t , the i n v e s t i g a t o r was alleviate  " i c e breaker"  misconceptions  able  about  to  observe  group  trends,  the program and check how the  program was being r e c e i v e d . The above evidence supports the t e c h n i c a l adequacy i l l u m i n a t i v e e v a l u a t i o n methodology. were  confirmed  and  validated  Since the q u a l i t a t i v e data  by  methodology met the standards of  of the  the q u a n t i t a t i v e data,  objectivity  and  validity  this as  d e f i n e d i n Chapter I I I .  Utility  In  order t o meet the c r i t e r i o n of u t i l i t y ,  evaluation  methodology  useful, v a l i d evidence  that  must  information. meets  the  c r e d i b i l i t y , and f l e x i b i l i t y following  evidence  remain This  flexible  methodology  standards as d e f i n e d  the i l l u m i n a t i v e while producing should  of r e l e v a n c e , i n Chapter  produce  importance, I I I . The  i s presented i n support of the c r i t e r i o n of  70  utility.  Relevance  and Importance  Evidence of data relevance to the program planner in  the  pilot  types  of  programming  Intermediate Course  courses,  shown  changes made as a r e s u l t of the  findings.  For example, i n  subsequent  d e s c r i p t i v e brochures were d i s t r i b u t e d ; c l a s s s i z e was  reduced; breaks  is  time a l l o c a t i o n s f o r m i n i - s e s s i o n s were  scheduled;  sought.  and  varied;  more  group d i s c u s s i o n s and f i e l d t r i p s were  used r a t h e r than s t r a i g h t introduced;  were  l e c t u r e ; more a u d i o - v i s u a l  aids  were  resource persons with t e a c h i n g experience were  These changes made as a r e s u l t of the e v a l u a t i o n s  were  used to f u r t h e r develop the program. W r i t t e n and o r a l r e p o r t s by the i n v e s t i g a t o r were to  members  of  the  Land  Title  School A d v i s o r y C o u n c i l .  members found the i n f o r m a t i o n p r o v i d e d was u s e f u l f u t u r e d i r e c t i o n s and p o l i c y f o r the Land T i t l e For i n s t a n c e , discussion  in  the  investigator  the  pilot  reported  Intermediate  i n f o r m a t i o n , the C o u n c i l recommended a  resource speakers,  qualitative people. so  t h e i r courses.  that As  and  i n determining  lack  Course. maximum  of  class  Based on that class  size  of  discussion.  i n f o r m a t i o n was r e l e v a n t and u s e f u l to the  Participant they  evidenced  the  comments  could  p a r t i c i p a n t s d i d not understand mini-sessions  The  School program.  the  twenty which they f e l t would f a c i l i t a t e c l a s s The  important  work  were  forwarded  to  use the i n f o r m a t i o n to improve  through  comments,  a  number  of  the r e l a t i o n s h i p between c e r t a i n that  they d i d .  The p a r t i c i p a n t s  71  found  it  difficult  practice.  to  bridge  the  When an i n s t r u c t o r was  gap  between  made aware of that  he t r i e d to e x p l a i n the r e l e v a n c e or r e l a t i o n s h i p . as  a  result  of  participant  comments,  and  difficulty, Furthermore,  several  requested t o u r s through the Land T i t l e O f f i c e might  theory  instructors  i n order that  they  b e t t e r understand the needs of the p a r t i c i p a n t s .  Credibility Assurance  of c r e d i b i l i t y was  obtained through  involvement.  In order to o b t a i n p a r t i c i p a n t and decision-maker  involvement i n  t h i s e v a l u a t i o n , i t was  necessary to gain  confidence  demonstrating  i n t h e i r o p i n i o n s and w i l l i n g n e s s to act  interest  their  by  on t h e i r a d v i c e . The p a r t i c i p a n t s were asked to e v a l u a t e the course specific,  detailed  through  comments and suggestions r e g a r d i n g changes,  a d d i t i o n s / s u b t r a c t i o n s , and m o d i f i c a t i o n s of each m i n i - s e s s i o n . The p a r t i c i p a n t s were t o l d that the course was  designed to  their  important  needs.  appropriate, on  the  If  i t was  it their  questionnaires.  was  not  relevant,  r e s p o n s i b i l i t y to respond The  p a r t i c i p a n t s were encouraged  following  suit and/or  accordingly  i s an example of the  way  to e v a l u a t e the program:  Please r e f l e c t on your experiences of the past week when answering the following items. Be c a n d i d i n e x p r e s s i n g your f e e l i n g s , whether they are p o s i t i v e or n e g a t i v e . Make your comments very s p e c i f i c f o r they w i l l h e l p us tremendously when we plan the next c o u r s e . In  addition,  participants  were  given  copies  of  the  e v a l u a t i o n r e p o r t , so they would have t a n g i b l e evidence that the  72  information  they  generated  was  being  example, s e v e r a l courses were developed When  read  and - used.  from p a r t i c i p a n t  For  ideas.  that o c c u r r e d , i t was mentioned t o subsequent groups. A l l  t h i s helped e s t a b l i s h the c r e d i b i l i t y of the e v a l u a t o r utility  and the  of the e v a l u a t i o n .  The  credibility  of  the study f o r the program c o o r d i n a t o r  and Advisory C o u n c i l members was enhanced by i n v o l v i n g the decision-making decisions  process.  concerning  evaluation.  the  them  in  For example, they were i n v o l v e d i n nature,  Involvement of those  purpose,  persons  and  methods  encouraged  keep informed by reading r e p o r t s and a t t e n d i n g  them  of to  meetings.  Flexibility Flexibility  i s inherent i n most methodologies  n a t u r a l i s t i c paradigm. this  study  i s can  based  on the  However, the q u e s t i o n t o be answered  in  the i l l u m i n a t i v e e v a l u a t i o n methodology be  f l e x i b l e yet produce t e c h n i c a l l y adequate and u s e f u l data. One  of the c h i e f  methodology  advantages  is its flexibility,  constraints.  of  illuminative  f o r i t does not have p r e s c r i b e d  The i l l u m i n a t i v e methodology a l l o w s f l e x i b i l i t y i n  data c o l l e c t i o n techniques, types of data used changes  to  evaluation  name  a  few.  and  Evidence of f l e x i b i l i t y  programming i s presented  below. Data C o l l e c t ion Techniques. techniques  were u s e d — o b s e r v a t i o n ,  h i s t o r i c a l documents. suggested  A number  of  data  collection  i n t e r v i e w , q u e s t i o n n a i r e and  The i l l u m i n a t i v e  evaluation  methodology  how these techniques c o u l d be used and encouraged the  73  use of a l l f o u r . for a s p e c i f i c example,  However, c e r t a i n techniques  stage of the methodology than were  historical  o b s e r v a t i o n stage, insight  and  were  used  documents  because  During to  used  of  the  others.  only  investigator  as  well  i n t e r v i e w s and  as  to  the gain  background  the nature, purpose, and  Interviews  For  during  needed  project's  the i n q u i r y stage,  determine  evaluation.  were  the  understanding  development.  were more s u i t a b l e  and  meetings  focus of the  observations  and  q u e s t i o n n a i r e s were used e x c l u s i v e l y to c o l l e c t data during each course.  The  investigator  use  of  these  varied  f l e x i b i l i t y as w e l l as a  and c o n f i r m i n g  experimenting  with  and  evaluation  modified  had  to  get  open-ended q u e s t i o n s .  focusing  process  r e f i n e and  questionnaires pilot  was  were  Intermediate  pilot  instruments,  had  as  utilizing  u s e l e s s and  much  cross-checking  used to r e j e c t improve  critical  the Since  those  redesigned  cut  By  investigator the  objectives,  the  i n f o r m a t i o n as p o s s i b l e by narrowing  and  those q u e s t i o n s that were that  were  useful.  The  and m o d i f i e d at the end of the  Course and again at  the  completion This r e f i n i n g  that only the most r e l e v a n t data would be  Director  during  neither  Then a s i f t i n g ,  For example, a f t e r the p i l o t Title  clear  I n t r o d u c t o r y and Advanced Courses.  ensured  the  subsequent m o d i f i c a t i o n s .  questionnaires.  program nor the m i n i - s e s s i o n s investigator  of  F l e x i b i l i t y was  q u e s t i o n n a i r e design and  developed  means  allowed  results.  Q u e s t i o n n a i r e Development. original  techniques  Intermediate  of  the  process  collected.  Course, the  Land  wanted to d e l e t e the q u e s t i o n which r e f e r r e d to  74  p a r t i c i p a n t s ' perceived  relevance  purpose  was  he  felt  of  the course  not  that those q u e s t i o n s  b e l i e v i n g the course relevance  should  questions  questionnaires.  Intermediate be gained  It  course.  job t r a i n i n g but  be  job  were  was  the  might mislead  Another  expectations.  of  the p a r t i c i p a n t s i n t o Thus,  eliminated  decided  in  of  the  review  concerned  of  the  through a warm-up e x e r c i s e p r i o r to the s t a r t  Data  pilot  Course that i n f o r m a t i o n of p r i o r e x p e c t a t i o n s  program thereby  e l i m i n a t i n g one  C o l l e c t ion  data c o l l e c t i o n had Intermediate  p r e s e n t a t i o n " and  Methods.  i t was  could  of  the  form. In some instances the method of  to be m o d i f i e d .  Course,  job  subsequent  change  during  the  job enrichment,  relevant.  example  Since  For example, in  important  the  to know i f the  "length of s e s s i o n " were  pilot  " l e v e l of  appropriate  to  the  participants.  The  i n v e s t i g a t o r t r i e d to measure those v a r i a b l e s  statistically,  but  the i n f o r m a t i o n obtained  example,  was  not u s e f u l .  For  the r a t i n g s of " l e n g t h of s e s s i o n " were mid-range with  no s i g n i f i c a n t v a r i a n c e ; that i s , the m i n i - s e s s i o n s were n e i t h e r too long nor too s h o r t .  The  also  the  clustered  around  r a t i n g s of " l e v e l of mid-point  although  variance.  Moreover, the p a r t i c i p a n t s made no  to c l a r i f y  those r a t i n g s .  The  investigator,  however,  because  Therefore, during  the  the  I'm  always  hardest  running  Intermediate  written  some  comments  Course  part was  around  i n v e s t i g a t o r changed the data  pilot  there was  heard comments that seemed to  c o n t r a d i c t the mid-range r a t i n g s : "The seated  presentation"  from  the  to  stay  office."  collection  method  r a t i n g s c a l e s to  75  o b s e r v a t i o n s and i n t e r v i e w s i n order to o b t a i n v a l i d and data.  Through  that  observations  mini-sessions  participants  useful  and i n t e r v i e w s , i t was d i s c o v e r e d  were  too  long.  needed more breaks.  More  importantly,  In subsequent courses,  rating  s c a l e s f o r " l e n g t h of s e s s i o n " and " l e v e l of p r e s e n t a t i o n " eliminated,  since  Information  on  they  "level"  provided and  no  "length"  useful was  information.  obtained  i n f o r m a l i n t e r v i e w s and o b s e r v a t i o n s i n a l l succeeding Types  of  data.  The data c o l l e c t e d  were  through courses.  i n t h i s study were not  l i m i t e d to one type; both q u a l i t a t i v e and q u a n t i t a t i v e data were collected. produce  The q u a n t i t a t i v e format  summaries  of  enabled  mini-sessions  These data were used f o r comparing  the i n v e s t i g a t o r  quickly  individual  to  and a c c u r a t e l y . mini-sessions  as  w e l l as f o r comparing the r a t i n g s of two d i f f e r e n t groups on the same  mini-session.  courses.  Comparisons  For example, the p i l o t  could  a l s o be made of e n t i r e  Intermediate  compared with the o p e r a t i o n a l Intermediate Although  Course  could  be  Course.  reading and summarizing numerous lengthy  responses  to open-ended q u e s t i o n s was a very time-consuming procedure, the qualitative  data  gave  participants' perceptions. participant  perceptions  the The from  investigator following  the  pilot  These composite p e r c e p t i o n s g i v e f a r more f e e l i n g than numerical  is  insight a  composite  Intermediate depth,  into of  Course.  richness  ratings.  I really enjoyed t h i s course and obtained much valuable knowledge. It will give me more confidence i n my day-to-day work. I r e a l i z e the work i n v o l v e d i n p l a n n i n g t h i s course f o r us and I think it's terrific. I t never hurts t o l e a r n  and  76  more about your job. The more knowledgeable I am about my job, the more I ' l l enjoy i t . Result b e t t e r work! I f e e l we -should have a course l i k e t h i s once a year. Let me know the data. Bravo!! Besides  providing  insight,  open-ended  questions  advantages that are u s e f u l f o r t h i s  study.  questions  of  permitted  ventilation  have  First,  opinions  in  an open-ended response.  did  not  exactly  p a r t i c i p a n t s were asked jobs.  The  match how  their  feelings. their  exact  I f they had been asked  simply check items, they might have f e l t that  open-ended  participant  P a r t i c i p a n t s were given the o p p o r t u n i t y to express  other  forced  into  attitudes.  r e l e v a n t the  course  f o l l o w i n g are some examples of t h e i r  to  responses  For i n s t a n c e , was  to  their  responses:  Dealt too much on h i s own p o i n t of view; got impression he was t r y i n g to f l o g h i s book.  the  T h i s l e c t u r e i s not r e l e v a n t to our jobs as i t was n e i t h e r a p r e s e n t a t i o n of new m a t e r i a l nor an in-depth treatment of known i n f o r m a t i o n . Although the t h e o r i e s i n v o l v e d were r e l e v a n t and p o s s i b l y i n t e r e s t i n g the r e l e v a n c e seemed too f a r removed from our own e x p e r i e n c e s . Second, open-ended q u e s t i o n s produced responses the  evaluator's  unanticipated  attention  when  q u e s t i o n n a i r e s were  the  to  course  which  a s i t u a t i o n or outcome that was  developed  and/or  designed.  I had no idea of the problems faced by my c l e r k s i n other o f f i c e s . It h i g h l i g h t e d my them.  weaknesses so now  I can  fellow improve  The  c a l i b r e of the l e c t u r e r s was e x c e l l e n t .  All  knowledge that a person  gains  over  his/her  drew was when  77  life will e f f e c t what and who that person i s or becomes i n that a l l persons continue through t h e i r l i f e to change. That life i s not sometimes seems!  as simple or as b o r i n g as i t  T h i r d , open-ended q u e s t i o n s d i d not possible example,  answers i f you  impressions  of  as  would  wanted the  closed  to  know  limit  response  about  the  range  questions.  participants'  of For  salient  program, an open-ended q u e s t i o n asking f o r  impressions i s b e t t e r than a c h e c k l i s t  of p o s s i b l e  responses:  If you were to r e o r g a n i z e the course, what would you change, leave the same, e t c . ? Explain. Program methodology,  Changes. the  Due  program  i n v a l i d a t i n g the study. in  a  developing  lunch  itself  program.  changes were made.  For  time.  In  modified  as  not  the  several  to  facilitate  programming  field  trips,  the  being  to s i t f o r long p e r i o d s of  evaluation  results  of  the  (n=43)  Since the s i z e seemed t o  d i s c u s s i o n , c l a s s e s were reduced to a maximum of 20. of  to  pilot  was  too  i n t e r a c t i o n among p a r t i c i p a n t s and between  p a r t i c i p a n t s and i n s t r u c t o r s .  flexibility  and  These changes  accustomed  Intermediate Course showed that the group s i z e large  important  a r e s u l t of the  were a few of the changes. were  without  i s extremely  breaks",  i t quite d i f f i c u l t  addition,  of the e v a l u a t i o n  be  example,  "stretch  were made because p a r t i c i p a n t s students and found  could  Intermediate Course,  More  breaks  flexibility  This f l e x i b i l i t y  e v a l u a t i o n of the p i l o t  longer  to  methodology,  these  changes  program but d i d not e f f e c t the v a l i d i t y of t h i s  inhibit  Due to the  improved the  study.  The above evidence supports the u t i l i t y of the i l l u m i n a t i v e  78  e v a l u a t i o n methodology. while producing defined  Since  useful, valid  the methodology remained information,  i t met  the  flexible standards  in Chapter I I I .  Ef f ic iency  An e f f i c i e n t e v a l u a t i o n methodology should the  r e c i p i e n t s than i t c o s t s .  measure f o r a l l e v a l u a t i o n s for  the  illuminative  e f f i c i e n t , the q u e s t i o n  This  i s not an easy c r i t e r i o n  r e q u i r e time and  evaluation  to be answered by t h i s study was  whether  i n v e s t i g a t o r i s r e s p o s i b l e to The  of the c r i t e r i o n of  It was invested  impossible by  bookkeeping  various  As  most  out  of  the  evidence i s presented  people  dollar during  have  value this  Table  the  study,  increased  alternative,  to  5 was  time  for  the  the costs  developed to  g r a p h i c a l l y summarize the r e l a t i v e amount of time spent on stage  of  the  illuminative  evaluation  development of the Land T i t l e School that  the  greatest  i n v e s t e d i n the p i l o t pilot costs.  Intermediate  in  efficiency.  would  an  at l e s s c o s t , because  the  following  to a t t a c h a  involved  unnecessarily.  make  to  order judged  the  support  In  to  be  same outcomes c o u l d have been achieved  available.  money.  methodology  the  resources  be worth more to  amount  of  Intermediate Course  As the time spent was  was  methodology during  program.  time  and,  Course. high due  each  This table  shows  t h e r e f o r e money, The to high  cost  the  of  was the  developmental  reduced so were the c o s t s .  By  the  Table 5 Effort and Time Spent on Illuminative Stages PILOT OBSERVATION  PROGRAM INQUIRY  OPERATIONAL EXPLANATION  OBSERVATION  PROGRAM  INQUIRY  EXPLANATION  INTERMEDIATE COURSE  5 WEEKS  2 1/2 WEEKS  3 WEEKS  2 1/2 WEEKS  INTRODUCTORY COURSE  •  ADVANCED COURSE  3 WEEKS  2 1/2 WEEKS  APPROXIMATE EFFORT:  A P P R O X I M A T E TIME:  = 1/2 WEEK =1  = MINIMAL  =AVERAGE  =MAXIMUM  WEEK =1  1/2 WEEKS =2  WEEKS  80  time of the o p e r a t i o n a l program, the c o s t s had material  and  program.  Based on the experience  should expect  supplies  costs  The  remained constant throughout i n t h i s study, an  to spend between four and  course of t h i s k i n d .  stabilized.  the  investigator  f i v e weeks on a one week  I n v e s t i g a t o r time  i s divided  between  the  a c t i v i t i e s of the o b s e r v a t i o n , i n q u i r y and e x p l a n a t i o n stages of the  methodology  Table 5  is  (see  divided  Table  2).  according  The  to  f o l l o w i n g e x p l a n a t i o n of  the  three  stages  of  the  i l l u m i n a t i v e methodology. The  observation  involved  stage  considerable  coordinator  time  and  of  the  investigator  time,  no p a r t i c i p a n t  time.  i n v e s t i g a t o r became f a m i l i a r with historical  documents  coordinator. School  and  after  the  pilot  i n subsequent courses,  until  i t reached a s t a b i l i z e d  this  c o n t e x t , a person  Intermediate minimal  level.  should expect  through  the  program  with  the  Intermediate  was  program  mainly  with  f o r t h i s stage. time  Course  In t h i s stage, the  program  familiar  i n v e s t i g a t o r r e q u i r e d l e s s time table,  the  discussions  Being thoroughly  program,  pilot  Land  Title  Course,  As noted on  reduced  the the  significantly  Based on the experience i n to spend about one week or  l e s s on t h i s phase. The required  i n q u i r y stage of the considerable  C o u n c i l member time this  stage  determined, Participants  that  investigator,  Intermediate coordinator  (approximately two weeks). the i s s u e s were c l a r i f i e d ,  questionnaires were  pilot  involved  designed in  the  and data  It  Course and was  also  Advisory during  e v a l u a t i o n process data  collected.  collection  phase.  81  Participant course  time involvement was approximately  since  only  questionnaires.  minor  The  alterations  time spent  the same f o r each  were  made  the  to  the  made  to  Intermediate  Course.  questionnaires,  so  Only  minor  alterations  the i n v e s t i g a t o r as w e l l as  c o o r d i n a t o r and C o u n c i l members d i d not i n v e s t much time. no  fact  e v a l u a t i o n design and procedures had been e s t a b l i s h e d  d u r i n g the p i l o t were  the  on t h i s stage during the p i l o t  I n t r o d u c t o r y and Advanced Courses was reduced due that  to  changes  were  made  to  the  evaluation  Since  procedures  or  q u e s t i o n n a i r e s d u r i n g the o p e r a t i o n a l program, the time i n v e s t e d remained s t a b l e . The  explanation  secretarial pilot  stage  involved  time (approximately  Intermediate  both  two weeks).  investigator T h i s stage  of  and the  Course r e q u i r e d s i g n i f i c a n t l y more time than  subsequent courses  f o r two reasons:  (1) the c l a s s s i z e was l a r g e  (n=43); and (2) a l l the data  from q u e s t i o n n a i r e s were  coded  student  In  the c l a s s  identity  number.  the f o l l o w i n g courses  by  s i z e was reduced t o a maximum of twenty p a r t i c i p a n t s which meant a reduction student  i n the  coding  information.  amount  was  of  dropped  These  two  data for  changes  collected. i t did resulted  not in  In  provide u s e f u l both  s e c r e t a r i a l and i n v e s t i g a t o r time to approximately Both  qualitative  d u r i n g t h i s study. information time.  Coding,  involved  In comparison,  information  was  and  quantitative typing  and  data  addition,  reduced  one week. were  analyzing  collected qualitative  considerable  s e c r e t a r i a l and i n v e s t i g a t o r  coding  analyzing  and  the  quantitative  r a p i d s i n c e only mean scores were c a l c u l a t e d .  82  The a c t u a l a n a l y s i s time was the same f o r a l l the courses the p i l o t  Intermediate Course  (approximately one week).  Although the q u a l i t a t i v e collecting,  i n f o r m a t i o n r e q u i r e d more time f o r  coding and a n a l y z i n g than q u a n t i t a t i v e i n f o r m a t i o n ,  i t was more r e l e v a n t  and  program  and Advisory C o u n c i l members.  coordinator  support of t h i s statement ("Relevance evidence utility  and  to  the  i s presented i n  Importance")  of  this  resource  the  Evidence i n  Utility  chapter.  i n the U t i l i t y  people,  section  In a d d i t i o n ,  section  support  the  of the more c o s t l y q u a l i t a t i v e d a t a . above  evidence  c r i t e r i o n of e f f i c i e n c y .  from  important  i n "Types of Data"  The  aspect  after  was  presented  The evidence  of the e f f i c i e n c y c r i t e r i o n . the  data  determined  by  cost  perspective.  in  support  presented  is  the  only  one  I t addresses the c r i t e r i o n The  value  of  cost  was  the i n v e s t i g a t o r based on comments from A d v i s o r y  C o u n c i l members, p a r t i c i p a n t s , and resource persons impartial  of  not  by  an  person.  Summary  In  order  evaluation  to determine  methodology  education  programs,  program.  This  illuminative Evidence  for  i t was  chapter  evaluation  the s u i t a b i l i t y of the i l l u m i n a t i v e evaluating tested  contained methodology  on the on  residential the Land T i t l e  results the  of  selected  adult School testing site.  from the t e s t i n g was compared with the standards set by  83  the p r e - s p e c i f i e d  c r i t e r i a c o n t a i n e d i n Chapter  summarizes  criteria  the  determine  the  and  suitability  standards of  III.  that  Table  were  illuminative  used  6 to  evaluation  methodology. The (1)  results  indicated  that:  The data were confirmed through t r i a n g u l a t i o n  and  continuous o b s e r v a t i o n . (2)  The  data  credible, (3)  collected  by  various  means  were  important and r e l e v a n t .  The data c o l l e c t e d c o u l d not have  been  obtained  by l e s s c o s t l y methods. Thus, the evidence of s u i t a b i l i t y this  study.  satisfied  the standards set by  Criteria  CRITERIA  Technical-  and  Standards  Used  STANDARD  Adequacy  Objectivity Validity  For  Table 6 S u i t a b i l i t y of  Determining  Illuminative  E v a l u a t i o n Methodology  TECHNIQUE  and  T r i a n g u l a t i o n p r o v i d e s evidence of o b j e c t i v i t y and validity by cross-checking q u a l i t a t i v e and q u a n t i t a t i v e d a t a , by a i d i n g i n t e r p r e t a t i o n o f q u a n t i t a t i v e d a t a , and by c o m p a r i n g p r e - p o s t q u e s t i o n n a i r e r e s u l t s . Continuous Observation provides evidence of validity through repeated o b s e r v a t i o n . I t a l s o p r o v i d e s i n f o r m a t i o n u n a t t a i n a b l e by any o t h e r means, i.e. mood o f g r o u p o r d e v e l o p m e n t o f g r o u p .  Ut i 1 i t y  Efficiency  Relevance and Importance  Written and oral r e p o r t s . T h e s e were u s e d t o p r o v i d e i n f o r m a t i o n f o r d e c i s i o n making, f o r r e v i s i o n of m i n i - s e s s i o n m a t e r i a l , and for providing participants w i t h e v i d e n c e t h a t t h e i n f o r m a t i o n t h e y g e n e r a t e d was used.  Cred i b i 1 i ty  This was insured by involvement. A d v i s o r y c o m m i t t e e members were i n v o l v e d i n decision-making; p a r t i c i p a n t s e v a l u a t e d the c o u r s e g i v i n g d e t a i l e d comments and s u g g e s t i o n s r e g a r d i n g changes.  F1 ex i b i1 i t y  Flexibility of this m e t h o d o l o g y must be m a i n t a i n e d w h i l e p r o d u c i n g t e c h n i c a l l y a d e q u a t e and u s e f u l d a t a . F l e x i b i l i t y was d e m o n s t r a t e d by c h a n g e s made in data collection t e c h n i q u e s , q u e s t i o n n a i r e d e v e l o p m e n t , t y p e s o f d a t a u s e d and p r o g r a m changes.  Cost/benefit  Time and  effort  Comparison Cost of maker. —  of  data  i n v e s t e d by  various people  during  c o s t s of a n a l y z i n g q u a l i t a t i v e collection  compared  to u t i l i t y  this  versus of  data  study.  quantitative as  determined  data. by  the  decision  CO  85  CHAPTER VI  SUMMARY, CONCLUSIONS, IMPLICATIONS AND  RECOMMENDATIONS  Summary  For  many  evaluating  years,  their  educational  adult  programs.  evaluation  which d e r i v e s  its  educators have been i n t e r e s t e d i n most  formal  s t u d i e s have used the c l a s s i c a l  paradigm  methodology  Until  from  However,  there has  t h i s type  ( P a r l e t t & Hamilton, 1976;  and  paradigm,  experimental  requires  number  movement.  of  These  of  Stake,  1978)  an a l t e r n a t i v e paradigm r e l a t e d to  social  emerged. a  This  Smith, 1976;  alternative,  fundamentally  evaluation  models  the  have  time argues s t r o n g l y for the u t i l i t y  Their  evaluation  paradigm. emerged  models have c l o s e p h i l o s o p h i c  t i e s with the n a t u r a l i s t i c paradigm.  naturalistic  different  methodology from that used with the c l a s s i c a l A  psychology.  been i n c r e a s i n g r e s i s t a n c e to e v a l u a t i o n s  a movement to use  anthropology has  recently,  and  from  operational  emergence  of n a t u r a l i s t i c  the  at  this  i n q u i r y for  86  the  field  of  naturalistic  education,  and  helps  make  i n q u i r y should be i n v e s t i g a t e d  the  as  case  an  that  alternative  methodology. Illuminative  evaluation  was  s e l e c t e d f o r t h i s study from  the emergent models because i t i s r e l a t i v e l y new the  naturalistic  paradigm.  and  based  I t i s a dynamic e v a l u a t i o n  process  which i s not t i e d to a s i n g l e treatment, predetermined goals outcomes,  but  rather  focuses  on  the  1978).  important  innovative  often  changed  as  for  planners  or  a c t u a l o p e r a t i o n s of a  program over a p e r i o d of time (Patton, i n program e v a l u a t i o n ,  on  experiment  T h i s i s extremely programs  and  change  are their  priorities. R e s i d e n t i a l programs are p a r t i c u l a r l y the  value  of  the  suitable for  testing  i l l u m i n a t i v e e v a l u a t i o n methodology because  t h i s type of program has some unique c h a r a c t e r i s t i c s that do not exist  in  programs  other differ  program  temporarily  This  example,  residential  removed  from  their  ongoing  makes i t p o s s i b l e f o r the i n v e s t i g a t o r  to have continuous contact with contact  For  from most t r a d i t i o n a l types of programs because  the p a r t i c i p a n t s a r e responsibilities.  formats.  the  participants.  Continuous  i s important f o r a methodology that r e l i e s on f i e l d w o r k  techniques. The Land T i t l e School of the J u s t i c e I n s t i t u t e Columbia  was  for t e s t i n g  identified illuminative  of  British  as an a p p r o p r i a t e r e s i d e n t i a l program evaluation  u n d e r l y i n g assumptions (see Table the Land T i t l e School program.  methodology  1) f i t  because i t s  the e v a l u a t i o n needs of  The methodology:  87  (1)  allows  f o r the  study of open, changing  systems  and emergent problems; (2)  encourages  the  representation  of  multiple  viewpoints and value p e r s p e c t i v e s ; (3)  focuses  on  program a c t i v i t i e s and i s s u e s r a t h e r  than outcomes; (4)  provides a means of studying spontaneous  events,  s i t u a t i o n s , and c r i s e s ; (5) To  i s s e n s i t i v e to the context and s e t t i n g . determine the s u i t a b i l i t y of t h i s methodology,  c o l l e c t e d during the study was compared with the by  the  pre-specified c r i t e r i a :  efficiency.  I t was f e l t  was  technically  more t o some audience  standards  set  t e c h n i c a l adequacy, u t i l i t y and  that f o r the purpose of t h i s study, the  i l l u m i n a t i v e e v a l u a t i o n methodology should that  evidence  produce  information  sound, u s e f u l t o some audience than  i t c o s t s (Groteleuschen,  and worth  1980).  Conclusions  Evidence  of the degree to which i l l u m i n a t i v e e v a l u a t i o n met  the t e c h n i c a l adequacy,  utility  and  efficiency  c o l l e c t e d d u r i n g the Land T i t l e School program. as  interviews,  collect  the  qualitative  questionnaires,  evidence. techniques  The  and  evidence  criteria  was  Techniques  such  o b s e r v a t i o n s were used to was  analyzed  using  to determine whether the methodology met  88  the standards set  by  section  descriptive  provides  c o l l e c t e d on each  the  criteria.  The  remainder  interpretations  of  this  of the evidence  criterion.  T e c h n i c a l Adequacy Two major c r i t i c i s m s of i l l u m i n a t i v e e v a l u a t i o n naturalistic  inquiries  appear i n the l i t e r a t u r e .  i n t e r p r e t a t i o n s cannot be o b j e c t i v e . cannot  be  valid.  In response  any e v a l u a t i o n study—whether naturalistic  and  other  (1) Personal  (2) D e s c r i p t i v e  studies  to the c r i t i c i s m of o b j e c t i v i t y :  i t conforms to  paradigm—requires  the  classical  s k i l l e d human judgment.  or  Human  judgment i s necessary at every stage of any study whether i t be descriptive 1978).  For  constructing  or  experimental  example,  it is  questionnaires,  choosing s t a t i s t i c a l  treatment,  (Guba used  &  Lincoln, in  choosing  administering interpreting  1981; Patton, samples,  questionnaires, statistical  data  and p r e s e n t i n g f i n d i n g s . Responses to the second c r i t i c i s m of v a l i d i t y are presented in the f o l l o w i n g  quotes:  a methodology, whether d e s c r i p t i v e or i n f e r ential, experimental or non-experimental, can seldom obtain valid results unless closely associated with s u b s t a n t i v e knowledge of the process being studied. (Bennett & Lumsdaine, 1975, p. 20). E v a l u a t i o n data are never c l e a r c u t and a b s o l u t e : studies are always flawed i n some way, and there are always q u e s t i o n s of r e l i a b i l i t y and v a l i d i t y . E r r o r - f r e e instruments do not and cannot e x i s t i n the measurement of complex human, social, behavioral, and p s y c h o l o g i c a l phenomena (Patton, 1978, p. 180).  89  Although the process of techniques This  is  process  findings.  collecting  time consuming and expensive, helps  Once  ensure  the  information  Each  sufficient  technique  to  available.  cause But  its  a  various  i t i s worthwhile. adequacy  interpretation  contains  rejection  when  technical  through  of  its  if  series  greatly  b i t of e r r o r , perhaps  that of  is  were  bits  all  of  that  was  evidence  are  t r i a n g u l a t e d and a l l evidence tends i n the same d i r e c t i o n , direction al.,  assumes  greater  believability  (Guba, 1968;  from  attained  by  historical  interviews.  cross-checking documents,  Webb et  observation No  validity  and c o n f i r m i n g data  collected  observations,  questionnaires  through  triangulation  and  continuous  can be completely o b j e c t i v e and v a l i d as p o i n t e d  out i n the d i s c u s s i o n above. to  However, by  cross-check,  confirm  and  using  evidence  presented  standards of o b j e c t i v i t y and  a  validate  c r i t i c i s m s of "lack of o b j e c t i v i t y and v a l i d i t y " the  and  procedures.  study  techniques  and  The data c o l l e c t e d u s i n g the above techniques were  v a l i d a t e d and confirmed  From  that  1966). In t h i s study a c e r t a i n l e v e l of o b j e c t i v i t y  was  the  has been confirmed by two or more  techniques, the u n c e r t a i n t y of reduced.  data  are  variety  findings, reduced.  i n Chapter V, t h i s study met  validity.  of  the  90  Utility The  evidence  illuminative produce  evaluation  useful  evaluation  i n Chapter V demonstrated methodology  results.  Since  methodologies  Therefore, through  presented  to  be  phases and  model based on u n i f o r m i t y and e v a l u a t i o n design was and  was  not  formulated  modified  as  also  techniques  progressed.  Due  d i d not e f f e c t  of  and  static, flexible.  both  that the  collecting  i n t e r a c t e d with instruments  description  judgment, decision-making,  data.  qualitative  and  are  presented  qualitative  i n v e s t i g a t o r can  descriptive  and  validating  the data  build  information  of and  what  happened  greatly  on  that  have  a number of weaknesses.  be learned by asking p a r t i c i p a n t s to like  A  aids  utility.  In c o n t r a s t , q u a n t i t a t i v e data are easy to code and  Scales  changes  u s e f u l , meaningful r e p r e s e n t a t i o n of what happened.  comprehensive  but  the  continuously  of the model, these  confirming  is  study,  collection  the advantage of o b t a i n i n g  perspectives  unfolds  were r e f i n e d as the program  collecting  Besides  emergent i n s i g h t s by a  evaluator  from a v a r i e t y of p e r s p e c t i v e s  V.  q u a n t i t a t i v e data, many  this  the v a l i d i t y or o b j e c t i v i t y of the  benefits  Chapter  evolved  In  Data  to the f l e x i b i l i t y  q u a n t i t a t i v e data  gives  be  i n advance but  the  and  from  yet  not  should  rigidity.  decision-makers.  in  useful  are  flexible  s t r a t e g i e s i s more u s e f u l than a  p a r t i c i p a n t s and  The  programs  be  the  a model such as i l l u m i n a t i v e e v a l u a t i o n that  successive  evolved  could  that  analyze  I t i s d o u b t f u l that much can rate  their  perceptions.  5-4-3-2-1 tend to obscure f a c t s of f e e l i n g , not  to  91  clarify  them.  Some p a r t i c i p a n t s use the s c a l e backwards; others  make a p o l i c y of "never g i v i n g a 5 or a 1".  In a d d i t i o n ,  simply  knowing that outcomes are high, low or d i f f e r e n t does not r e v e a l much about what t o do about them. The  r e s u l t s of the e v a l u a t i o n  relevance,  importance  from the e v a l u a t i o n s coordinator  and  and  credibility.  people.  of  coordinator,  Advisory C o u n c i l  investigator  spent c o n s i d e r a b l e the  the  wide  standards  First,  required  time and  establishing  rapport  feedback p r o c e s s .  the  program  members and the i n v e s t i g a t o r .  involvement  The and  Although  costly,  they  f o r t h i s study f o r three reasons.  the  investigator  participants  in  credibility  and  significantly  enhanced by  participants. evaluation,  r e s u l t s were  range of p e o p l e - - p a r t i c i p a n t s ,  those processes were time consuming and, t h e r e f o r e were important  program  utilized.  involvement  coordinating  of  The r e s u l t s obtained  Because  Meeting the p r e - s p e c i f i e d u t i l i t y a  standards  were c r e d i b l e to the i n v e s t i g a t o r ,  resource  c r e d i b l e , they were  met the u t i l i t y  the  established  evaluation  technical  rapport and i n v o l v e d  process.  adequacy  checking  of  out  the  example, findings  information  In a d d i t i o n , with p a r t i c i p a n t  the i n v e s t i g a t o r c o u l d  For  with  the were the  involvement i n the  be reasonably sure  that  the  f i n d i n g s r e f l e c t e d the i n s i g h t s and judgments of the group. Second,  coordinator  and C o u n c i l  ensure the r e l e v a n c e and c r e d i b i l i t y them.  The  coordinator  investigator  and  Advisory  of the data  discussed Council  member involvement helped  the  members,  collected for  findings helped  with them  the draw  92  implications  and  recommendations  f o r a c t i o n from the data  and  monitored the r e s u l t s of m o d i f i c a t i o n s made on the b a s i s of  the  evaluat i o n . Third, members  the  ensured  Utilization crucial  involvement the  of  rationale  for  of  the  evaluation  act i o n . . . unless  the  utilization  information  indicator  of  of  coordinator  and  the  collected.  collected value is  of  data  from  evaluations  evaluation.  that i t provides  is a  "The  basic  i n f o r m a t i o n for  i t gains s e r i o u s h e a r i n g when program  are made, i t f a i l s  Council  decisions  in i t s major purpose" (Weiss, 1972,  p.  318).  Ef f i c iency Illuminative  evaluation  involve i n v e s t i g a t o r s considerable no  for  financial  studies  seemingly  are c o s t l y , because inordinate  expense to the program.  durations  at  T h i s study  was  different. Much time was  study.  The  spent  in the  investigator  developmental  needed  to  establishing  rapport,  feedback p r o c e s s . developing  a  In  and  coordinating  addition,  the  considerable  of  this  the s u s t a i n i n g and  credible--  involvement time  was  and spent  s p e c i f i c e v a l u a t i o n plan because the i l l u m i n a t i v e  methodology  lacked  evaluation  activities.  procedural  methodology and a reason  This  guidelines is  a  major  to  facilitate weakness  of  the the  f o r high developmental c o s t s .  Although the development and initial  stages  initiate  r e l a t i o n s h i p s that made the e v a l u a t i o n p o s s i b l e  the  they  design c o s t s were high  c y c l e s of the methodology, they became l e s s  during costly  93  in  subsequent c y c l e s  not  f o r two reasons.  One, the i n v e s t i g a t o r d i d  have t o spend as much time as i n the  stage.  initial  developmental  Two, the i n v e s t i g a t o r became more p r o f i c i e n t i n c a r r y i n g  out  evaluation  the  illuminative  methodology  a c t i v i t i e s each time the program c y c l e d stages.  appears  Thus,  to  be  the  illuminative  more  efficient  through  evaluation  for  on-going  r e s i d e n t i a l programs than f o r "one-shot" programs. This data  study produced mainly q u a l i t a t i v e  are  both  difficult  But  even  judgment i s e x e r c i s e d making.  For  data.  under  Q u a l i t a t i v e data can  these  occasionally  occurs i n which comments c o n t r a d i c t numbers. slowly;  Seventeen  people w i l l  eighteen w i l l  really t e l l either  say that  an e v a l u a t o r ?  too  slow  circumstances,  the data r e t a i n s  example,  or  i t s value a  each other i n  say that  too  fast,  but  the  that  decision-  phenomenon"  almost  equal  that  What does t h i s  the  program  was  the design needs to  I t might a l s o t e l l the  there i s too l i t t l e ongoing feedback  main weakness of q u a l i t a t i v e data i s that  descriptions program  during  are often  preparation  a  long  and i n v o l v e d .  A  the r e s u l t a n t  decision-maker  planner does not always have time t o read a long  selective is  i f careful  session. The  in  be  the program moved too  i t was too r a p i d .  Probably not  also  for  "balancing  p r o v i d e more time f o r i n d i v i d u a l i z a t i o n . resource person that  Qualitative  to analyze and r e q u i r e more time f o r  a n a l y s i s than q u a n t i t a t i v e confusing.  data.  for a decision.  i n the information  potential  source  or  report  Thus, the i n v e s t i g a t o r has to be  presented.  The  selection  process  of b i a s which can harm the v a l i d i t y of  94  results. The cost of the benefits.  The  opinions  and  process.  To  evaluation and  information  Advisory  participants.  Encouraging feelings enhance  reports,  was  Council the  made this,  so  be  compared  useful  to  members, participants  them  feel  they  were  The  part given  qualitative  and u s e f u l to the resource  to  coordinator both  act.  The  Advisory  found the data  decision-making  outweighed the  its  to  in  program and  give  their  of the planning copies  of  the  of  Council  evaluation  information  members  r e l e v a n t , u s e f u l , and  the  information  was  people, f o r i t enabled them  and program p l a n n i n g .  costs  the  instructors  to improve t h e i r s e s s i o n s by p r o v i d i n g adequate which  with  they c o u l d see both t h e i r c o n t r i b u t i o n  the program i n t o t a l i t y .  relevant  must  b e n e f i t s of using q u a l i t a t i v e data presented  Chapter V showed the coordinator,  methodology  and  on  program  important  for  Thus, the b e n e f i t s methodology  so i t s  the methodology i s not h i g h l y e f f i c i e n t ,  the l o s s  e f f i c i e n c y was adequate f o r t h i s program. Although in e f f i c i e n c y In  i s balanced by gains  considering  more important  i n the other  the importance of the c r i t e r i a , factor in evaluation  many e v a l u a t i o n s t u d i e s gather dust. useful  and u t i l i z e d ,  two  criteria.  utility  than e f f i c i e n c y ,  i s a far  f o r f a r too  I f the e v a l u a t i o n  i t is inefficient  is  not  r e g a r d l e s s of the a c t u a l  cost. Based on the i n t e r p r e t a t i o n of the f i n d i n g s of t h i s the  illuminative  evaluating  study,  e v a l u a t i o n methodology i s judged s u i t a b l e f o r  residential  adult  education  programs,  for  the  95  evidence  collected  met  the  standards  of t e c h n i c a l adequacy,  u t i l i t y and e f f i c i e n c y . An what  important q u e s t i o n  conditions  does  remains  an  evaluation  n a t u r a l i s t i c paradigm provide Neither  the  literature  for  investigators.  Under  methodology based on the  the best guidance f o r e v a l u a t i o n s ?  nor  this  on  either  study  has  shown  methodologies  based  naturalistic—is  i n t r i n s i c a l l y b e t t e r than the o t h e r .  that  paradigm—classical  or  The f i n a l  choice between methodologies i n "any i n q u i r y or e v a l u a t i o n ought to  be  made  on  the  basis  of  assumptions...and the phenomenon being 1981,  p.  the  best  studied"  fit  between  (Guba & L i n c o l n ,  56).  I m p l i c a t i o n s and Recommendations  Implications The  f i n d i n g s of t h i s study have the f o l l o w i n g i m p l i c a t i o n s  for • r e s e a r c h e r s illuminative the value  and  practitioners  evaluation  methodology  desiring  to  as a means of  use  the  determining  of a program.  (1) The degree  of  f i t between  the  assumptions  of  the  i l l u m i n a t i v e e v a l u a t i o n methodology and the program's e v a l u a t i o n needs  is  methodology. particularly  an  important Illuminative  suitable  consideration evaluation  in  choosing  methodology  f o r e v a l u a t i n g a d u l t education  this is  programs  96  which have complex goals that are d i f f i c u l t and  thus  defy  quantitative  developmental  costs  are  high,  methodology i s more e f f i c i e n t (2)  The  the  involved  Because the  illuminative  with  initial  evaluation  when used with on-going programs.  i n v o l v e d , s i n c e the more  increase  decision-makers  a p r o j e c t , the more they are apt to  utilize  information. (3) The  p a r t i c i p a n t s should  maximize  the  them, the  i n v e s t i g a t o r can  will  precisely  b e n e f i t s of using t h i s type of e v a l u a t i o n  i f decision-makers are are  measurement.  to d e f i n e  reflect (4)  utility  the  a l s o be  of t h i s type of e v a l u a t i o n . be  i n s i g h t s and  reasonably  develop good i n t e r v i e w and to data  involved  By i n v o l v i n g  sure that the  findings  observation  t h i s methodology need to skills,  f o r these  skills  collection.  (5) I n v e s t i g a t o r s need to become aware of t h e i r own They should  to  judgments of the group.  I n v e s t i g a t o r s d e s i r i n g to use  are c r i t i c a l  actively  t r y to be understanding and  biases.  open to d i f f e r i n g  points  of view, at the same time a v o i d i n g c o l l u s i o n or over-involvement which tend to c r e a t e  biases.  Recommendations Based  on  the  study completed and  recommendations are presented  reported,  the  f o r those d e s i r i n g to  following  do  further  r e s e a r c h or those d e s i r i n g to employ t h i s methodology. (1)  Further  tasks, questions,  work  needs  to  be  done to develop  specific  a c t i t i v i e s and/or procedures which c o u l d  implementation of each  stage  of  the  illuminative  guide  evaluation  97  methodology. weak  on  Because the i l l u m i n a t i v e e v a l u a t i o n methodology i s  specifying evaluation a c t i v i t i e s ,  and i t a l s o d i m i n i s h e s Procedural  i t i s open f o r abuse  the p o s s i b i l i t y of g e n e r a l i z i n g r e s u l t s .  a c t i v i t i e s such as those presented  i n Table  2  could  be used and/or f u r t h e r r e f i n e d . (2) become  Educational adult  programs  educators  naturalistic  approach  assimilated  into  which  could to  current  be  prepare  expanded  evaluation. adult  individuals to  include  to the  T h i s approach c o u l d be  education  curriculum  as  an  a l t e r n a t i v e to the c l a s s i c a l approach. (3)  Further  understanding  studies  of the i l l u m i n a t i v e  evaluation  studies  were  accessible  more  should  using  be  done  evaluation  illuminative  and/or  to  build  up  the  methodology.  If  e v a l u a t i o n methodology  published  more  frequently,  i n v e s t i g a t o r s would be able to determine the s u i t a b i l i t y of t h i s methodology f o r other (4)  Further  types of programs.  evaluations  of  residential  programs  using  i l l u m i n a t i v e e v a l u a t i o n methodology need to be done i n order contribute  to  residential  program  generalization t h i s study small n's.  the  understanding formats.  of  this  Further  methodology  studies  will  to for aid  of r e s u l t s , f o r there are obvious l i m i t a t i o n s to  such as evaluator b i a s , l i m i t e d g e n e r a l i z a b i l i t y , and In a d d i t i o n , the i n v e s t i g a t o r played a dual r o l e  of  program c o o r d i n a t o r and e v a l u a t o r . (5) suitability evaluating  Further of other  studies  could  illuminative types  of  be  done  to  evaluation  adult  education  determine  the  methodology  for  programs.  Can  98  elements  of  formats?  What m o d i f i c a t i o n s  The  this  findings  become more evaluation.  methodology  in  creative They  c r e a t i v i t y may take.  this and  also  be  applied  to  other  program  need to be made to the methodology? study c h a l l e n g e resourceful  present  in  adult their  educators to approach  a model of one d i r e c t i o n  to this  99  REFERENCES  Apps, J . Problems i n c o n t i n u i n g education. H i l l , 1979.  New  York:  McGraw-  Bale,  R.L. & M o l i t o r , J.A. The Mountain-Plains Educat ion and Economic Development Program: Success or F a i l u r e ? , 1978. (ERIC Document Reproduction S e r v i c e No. ED 153 073).  Bass,  B.M. & Vaughan, J.A. Training i n i n d u s t r y : The management of l e a r n i n g . Belmont, CA: Wadsworth Publishing Co., 1966.  Beckhard, R. How to plan and conduct workshops and New York: A s s o c i a t i o n Press, 1956. Bennett, C. Up the h i e r a r c h y . (March, A p r i l ) , 7-12.  conferences.  J o u r n a l of E x t e n s i o n ,  1975, _1_3  Bennett, C.A. & Lumsdaine, A.A. S o c i a l program evaluation: Definitions and i s s u e s . In Bennett, C.A. & Lumsdaine, A.A. (eds.) Evaluat i on and exper iment: Some critical issues in assessing s o c i a l programs. New York: Academic Press, 1975. B e r n s t e i n , I.N. & Freeman, H.E. Academic and entrepreneur i a l research. New York: R u s s e l l Sage Foundation, 1975. Blackwell, B.L. evaluation. 175-186.  & Bolman, W.M. The p r i n c i p l e s and problems of Community Mental Health J o u r n a l , 1977, _1_3 ( 2 ) ,  Blaney, J.P. & McKie, D. Knowledge of conference objectives and e f f e c t on l e a r n i n g . Adult Education, 1969, J_9 (2), 981 05. Bogdan, R. & T a y l o r , S.J. I n t r o d u c t i o n to q u a l i t a t i v e r e s e a r c h methods. New York: John Wiley & Sons, 1975. Bunch, M.B. Summat ive evaluat ion of Mountain-Plains Volume 11. 1976. (ERIC Reproduction S e r v i c e No. ED 150 3987"] B r i n g l e , R.R. E f f e c t s of human r e l a t i o n s l a b o r a t o r y t r a i n i n g on flexibility and a t t i t u d e s toward s u p e r v i s i o n . (Doctoral d i s s e r t a t i o n , Oregon State U n i v e r s i t y , 1967). Campbell, D.T. A s s e s s i n g the impact of planned s o c i a l change. In Lyons, G.M. (ed.) Soc i a l research and p u b l i c p o l i c i e s . Hanover, N.H.: P u b l i c A f f a i r s Center, Dartmouth C o l l e g e ,  100  1 975. Campbell, D.T. Comparative  Degrees of freedom and the case P o l i t i c a l S t u d i e s , 1975 (2) 178-193.  study.  Campbell, D.T. & S t a n l e y , J.C. Experimental and experimental d e s i g n s . Chicago: Rand McNally, 1963.  quasi -  Conrad, R.W. Summative e v a l u a t i o n of Mountain-Plains Volume III. 1976"! CERIC Document Reproduction S e r v i c e NO. ED 150 399). Cox, W.F., J r . Development and implementation of a training program for educat i o n a l research and developmental p e r s o n n e l , F i n a l Report. 1974. (ERIC Document Reproduction S e r v i c e No. ED 095 157). Cronbach, L . J . Course improvement through e v a l u a t i o n . C o l l e g e Record, 1963, 64^ 672-683.  Teachers  Cronbach, L . J . (ed.) Toward reform of program e v a l u a t i o n . F r a n c i s c o : Josey-Bass P u b l i s h e r s , 1980.  San  Davis, L. & McCallon, E. P l a n n i n g , conducting, and e v a l u a t i n g workshops. A u s t i n , Texas: L e a r n i n g Concepts, 1974. Deantonio, E. E v a l u a t i o n of the workshop, 1971 -1972, 1973. S e r v i c e No. ED 089 147).  community school relations (ERIC Document Reproduction  Densmore, M.L. An e v a l u a t i v e a n a l y s i s of s e l e c t e d university conference programs conducted at K e l l o g g Center f o r C o n t i n u i n g E d u c a t i o n , Michigan State U n i v e r s i t y . (Doctoral d i s s e r t a t i o n , Michigan S t a t e U n i v e r s i t y , 1965). Denzin, N.K. The l o g i c of n a t u r a l i s t i c 1971, (50) 166-182.  inquiry.  S o c i a l Forces,  D e v l i n , L.E. The i n f l u e n c e of d i r e c t e d i n s t r u c t i o n on in a conference setting. (Unpublished M.A. U n i v e r s i t y of Chicago, 1966). D i c k i n s o n , G. systems.  & Lamoureux, M.E. Evaluating educative Adult Educat ion, 1975, xxv, (2), 81-89.  learning Thesis, temporary  Edelbach, R.D. A study of the i n f l u e n c e of r e s i d e n t i a l and nonresidential skill training programs on trainee selfesteem, o c c u p a t i o n a l i d e n t i f i c a t i o n , and work m o t i v a t i o n . (Doctoral dissertation, Rutgers University, The State U n i v e r s i t y of New J e r s e y , 1973). Edwards W. & Guttentag, M. Experiments and e v a l u a t i o n s : A r e examination. In Bennett, C.A. & Lumsdaine, A.A. (eds.) E v a l u a t ion and experiment: Some c r i t i c a l issues in  101  assessing  soc i a l programs.  New  York: Academic Press,  1975.  E i s n e r , E.W. The perceptive eye: Toward the reformaton of educational evaluation. Stanford: Stanford Evaluation Consortium, December 1975. E r i c k s o n , F. Some approaches to inquiry in school-community ethnography. Anthropology and Education, 1977(8). Fienberg, S.E. The c o l l e c t i o n and in educational research. 1977(8), 50-57..  a n a l y s i s of ethnographic data Anthropology and Education,  Freeman, H.E. & Solomon, M.A. Evaluat ion studies review annual, volume 6. Beverly H i l l s : Sage P u b l i c a t i o n s , 1981. Garside, D. value.  Short-term r e s i d e n t i a l c o l l e g e s : T h e i r o r i g i n s and Studies i n Adult Education, 1969, J_ (1), 2-30.  George, C. & Green, D. Summary of outcome evaluat ion report for preparing educat i o n a l training consultants: skills training (PETC-I), 1976. (ERIC Document Reproduction S e r v i c e No. ED 134 627). Grotelueschen, A.D. Program e v a l u a t i o n . In Knox, A.B. & Associates. Developing, a d m i n i s t e r i n g and e v a l u a t i n q a d u l t educat ion. San F r a n c i s c o : Jossey-Bass P u b l i s h e r s , 1 980') . Guba, E.G. Toward a methodology of naturalistic educational evaluat ion. CSE Monograph Evaluation. Los Angeles: Center for the Evaluation, 1978. Guba,  E.G. & L i n c o l n , Y.S. Effective F r a n c i s c o : Jossey-Bass P u b l i s h e r s , 1981.  inquiry Series Study  evaluation.  Guttentag, M. & Saar, S. Evaluat ion studies review volume 2. Beverly H i l l s : Sage P u b l i c a t i o n s , 1977.  in in of San  annual,  Haller, E.J. Cost a n a l y s i s f o r e d u c a t i o n a l program e v a l u a t i o n . In Popham, W.J. Educat i o n a l evaluat ion: Current applications. Berkeley: McCutchan P u b l i s h e r s , 1974. Halverson, M.B. Facing the r e a l i t i e s : Some conference planning principles. Adult Leadership, 1974, 2 (2), 47-49. Halverson, M.B. & Thiesse, J. r e g i o n a l conference approach. Years, 1979, m (3), 4 - 6 . Hamblin, A.C. Evaluat ion McGraw-Hill, 1974. Hamilton,  D.  Making  and sense  P r a c t i c a l e v a l u a t i o n and the L i f e l o n g Learning: The Adult  control of  of  training.  curriculum  London:  evaluation.  In  1 02  Shulman, L. (Ed.). I I . : F.E. Peacock,  Review of Education, 1977.  Vol.  5.  Itasca,  Havelock, R.G. & Havelock, M.C. Preparing knowledge linking change agents in educat i o n : A m a t e r i a l s and training development p r o j e c t . F i n a l Report, 1 9 7 1 . (ERIC Document Reproduction S e r v i c e No. ED 0 5 6 2 5 7 ) . Houle, C O . R e s i d e n t i a l cont i n u i n g educat i o n . Notes and Essays on Education for A d u l t s # 7 0 . P u b l i c a t i o n s in Continuing Education. New York: Syracuse U n i v e r s i t y Publications, 1971.  House, E. The conscience of e d u c a t i o n a l e v a l u a t i o n . C o l l e g e Record, 1 9 7 2 , 7 3 ( 3 ) , 4 0 5 - 4 1 4 .  Teachers  House, E. The l o g i c of evaluative agreement. Los Center f o r the Study of E v a l u a t i o n , UCLA, 1 9 7 7 .  Angeles:  House, E. Evaluating with Publications, 1 9 8 0 .  validity.  Beverly  Hills:  Sage  J e n k i n s , R.L., III. Summative evaluat ion of the MountainPlains community development component. An af feet ive evaluation report, 1976. [ERIC Document Reproduction S e r v i c e No. ED 1 5 0 3 6 9 ) . Katz,  D.S. & Morgan, R.L. A holistic strategy for the formative e v a l u a t i o n of e d u c a t i o n a l programs. In Borich, G.D. (ed.), E v a l u a t i n g e d u c a t i o n a l programs and products. Englecliffs, N.J.: Educational Technology Publications, 1 974.  K i r k p a t r i c k , D.L. E v a l u a t i o n of t r a i n i n g . In Craig, R.L. & Bittel, L.R. (eds.) T r a i n i n g and development handbook. New York: McGraw-Hill, 1 9 6 7 . Kuhn, T. The s t r u c t u r e of scientific U n i v e r s i t y of Chicago Press, 1 9 7 0 .  revolutions.  Chicago:  Lacognata, A.A. A comparison of the e f f e c t i v e n e s s of a d u l t resident i a l and non-resident i a l learning situations. Chicago: Center f o r the Study of L i b e r a l Education for Adults, 1 9 6 1 . L i p p i t t , R. S t u d i e s in e x p e r i m e n t a l l y created autocratic democratic groups. University of Iowa Studies C h i l d w e l f a r e , 1 9 4 0 , J_6 ( 3 ) , 45-198. L u t z , T.W. & Ramsey, M.A. methods i n education.  and in  The use of anthropological field Educat i o n a l Researcher, 1 9 7 4 ( 3 ) , 5-  9.  M i l e s , W.R.  Illuminative  evaluation  for  formative  decision-  103  making.  E v a l u a t i o n Review, 1981, 5 (4), 479-499.  Miller, H.L. Teaching and l e a r n i n g i n a d u l t e d u c a t i o n . York: Macmillan Co., 1964.  New  M i l o z a r e k , G.J. F i e l d t e s t and outcome milestone report f o r prepar ing educational training consultants: Consulting (PETC-II), 1976. (ERIC Document Reproduction S e r v i c e No. ED 126 157). Newcomb, T. Rinehart,  P e r s o n a l i t y and & Winston, 1943.  soc i a l change.  New York: H o l t ,  P a r l e t t , M. & Hamilton, D. E v a l u a t i o n as i l l u m i n a t i o n : A new approach to the study of innovatory programs. In G l a s s , G.V. (ed.) Evaluat ion s t u d i e s : Review Annual, Volume I. Beverly H i l l s , Ca.: Sage, 1976. Parlett, M. & Hamilton, D. approach to the study Hamilton, D. et a l . London: Macmillan, 1977.  E v a l u a t i o n as i l l u m i n a t i o n : A new of innovatory programmes. In (Eds.) Beyond the numbers game.  Parlett, M. & King, J.G. Concentrated study:A pedagogic innovation observed. London: S o c i e t y f o r Research i n t o Higher Education, 1971. P a t t i s o n , R.M. (ed.) C o u n s e l l i n g educationally disadvantaqed adults. Indianapolis: Indiana State Department of P u b l i c I n s t r u c t i o n , D i v i s i o n of Adult Education, 1968. (ERIC Document Reproduction S e r v i c e No. ED 023 015). Patton, M.Q. Utilization-focused Sage, 1978.  evaluat ion.  Beverly  Hills:  Peterson, K.P. Influence of e v a l u a t i v e conditions and p r e conference contact on p a r t i c i p a n t s * e v a l u a t i o n of a conference. (Masters thesis, University of British Columbia, 1971). Posavac, E . J . & case s t u d i e s . Rippey, R.M. Berkeley, Rist,  R.C. school.  Carey, R.G. Program e v a l u a t i o n : Methods and Englewood C l i f f s : P r e n t i c e - H a l l , 1980.  (ed.) S t u d i e s in transactional Ca: McCutcheon, 1973.  evaluation.  Ethnographic techniques and the study of an urban Urban Education, 1975(10), 86-108.  Roberts, E.R. & Holmes, A. The p r o f e s s i o n a l development s e r i e s for work experience educat ion. Evaluat ive r e p o r t . 1971. (ERIC Reproduction S e r v i c e No. ED 124 756). Schacht,  R.H.  Weekend l e a r n ing i n the U.S.A.  Notes and Essays  104  on Education for A d u l t s #29. Chicago: Center for the Study of L i b e r a l Education f o r A d u l t s , 1960. S c r i v e n , M. The methodology of evaluation. In Stake, R.E. (ed.) Curriculum evaluation. AERA Monograph S e r i e s on Curriculum E v a l u a t i o n ( V o l . O, Chicago: Rand McNalley, 1967. Scriven, M. O b j e c t i v i t y and subjectivity in educational research. In Thomas, L.G. (Ed.) P h i l o s o p h i c a l r e d i r e c t i o n of educat i o n a l r e s e a r c h . 71st Yearbook, Part I, National Scoiety for the Study of Education. Chicago: The U n i v e r s i t y of Chicago Press, 1972. Scruggs, J.A. Florida migratory c h i I d compensatory program s t a t e conference (September 1975). Evaluation report, 1976. (ERIC Reproduction S e r v i c e No. ED 148 507). Smallegan, M. A comparison of two t r a i n i n g formats f o r persons with varying i n t e r p e r s o n a l needs. Adult Educat ion, 1971, 2J_ (3) , 1 66-1 76. Smith, M.L.; G a b r i e l , R.; Schott, J . & Podia, W.L. Evaluation of the effects of outward bound. In G l a s s , G.V. (ed.) E v a l u a t i o n S t u d i e s Review Annual, Volume I. Beverly H i l l s : Sage, 1976. Stake, R.E. The countenance of educational Teachers C o l l e g e Record, 1967, 68, 523-540.  evaluation.  Stake, R.E. Evaluation design, instrumentation, data c o l l e c t i o n , and a n a l y s i s of data. In Worthen, B.R. & Sanders, J.R. Educat i o n a l evaluat ion: Theory and p r a c t i c e . Worthington, Ohio: C h a r l e s A. Jones P u b l i s h i n g Co., 1973) . Stake, R.E. Evaluating the a r t s in e d u c a t i o n : approach. Columbus, OH: M e r r i l l , 1975. Stake, R.E. The case study method in E d u c a t i o n a l Researcher, 1978, 7 (Feb.),  A  social 5-8.  responsive inquiry.  Stake, R.E.; Brown, C.; Hoke, G.; Maxwell, G.; & Friedman, J . Evaluat ing a r e g i o n a l envi ronmental learning system. Urbana: Center for I n s t r u c t i o n a l Research and Curriculum E v a l u a t i o n , 1979. S t e e l e , S.M. Program e v a l u a t i o n : A broad of E x t e n s i o n , 1970 (summer), 5-17.  definition.  Journal  Stewart, CW. A study of the r e s u l t s of a program of c o n t i n u i n q education for protestant clergy. Bloomfield Hills, Michigan: I n s t i t u t e for Pastoral Studies, 1965. (ERIC Document Reproduction S e r v i c e No. ED 021 190).  105  Stufflebeam, D.L. The use of and abuse of e v a l u a t i o n i n T i t l e III. Theory i n t o P r a c t i c e , 1967, 6 (June), 126-133. Stufflebeam, D.L. & Webster, W.J. An a n a l y s i s of a l t e r n a t i v e approaches to e v a l u a t i o n . In Freeman, H.E. & Solomon,' M.A. ( e d s ) . Evaluat ion s t u d i e s review annual, volume 6. Beverly H i l l s : Sage P u b l i c a t i o n s , 1981. Sutton, E.W. A n a l y s i s of research on s e l e c t e d aspects of a d u l t education. (Doctoral dissertation, Florida State U n i v e r s i t y , 1969). Torrence, P.E. The Adult Leadership,  Tuskegee experiment 1966, J_5 (3), 83-84;  in adult 96.  training.  T o u z e l , T.J. The s t a b i l i t y of a t t i t u d e s in a l o n g i t u d i n a l study resulting from a summer conference on individualized instruction. (Doctoral d i s s e r t a t i o n , The U n i v e r s i t y of Tennessee, 1975). V a l l a , D.C. An e v a l u a t i v e study of three workshops f o r the aging i n nine planning d i s t r i c t s of V i r g i n i a . (Doctoral dissertation, Virginia Polytechnic I n s t i t u t e and State U n i v e r s i t y , 1975). Warr,  P.B.; B i r d , M.W. & Rackham, N. Evaluat ion of management training. London: Gower Press, 1970.  Webb, et a l . Unobtrusive Measures. Co., 1966.  Chicago:  Rand  McNally  &  Weiss, C.H. E v a l u a t i n g e d u c a t i o n a l and s o c i a l a c t i o n programs: A t r e e f u l l of owls. In Weiss, C. (ed.). Evaluating act ion programs. Boston: A l l y n & Bacon, 1972. Wientge, K.M. & Lahr, J.K. The i n f l u e n c e of soc i a l c l i m a t e on adult achievement. The Impact of a r e s i d e n t i a l experience on l e a r n i n g and a t t i t u d e change of a d u l t students e n r o l l e d in an evening c r e d i t class. St. Louis: Washington University, 1966. [ERIC Document Reproduction S e r v i c e ED 01 1 371). Wohllenben, A. The pattern conferences. (Unpublished Chicago, 1965).  of anxiety in residential M.A. Thesis, University of  Wolf, R.L. The use of judicial evaluation methods i n the formulation of e d u c a t i o n a l p o l i c y . Educat i o n a l Evaluat ion and P o l i c y A n a l y s i s , 1979, 57 (1), 19-28. Worthen, B.R. A look at the mosaic of educat i o n a l accountabi1ity. Portland: Northwest Regional Laboratory, 1974.  evaluation Educational  106  REFERENCE NOTES  Note 1 R u s n e l l , D. D e c i s i o n s in the Unpublished manuscript, 1978.  design  of  evaluation.  Note  2 Guba, E.G. Toward a methodology of n a t u r a l i s t i c i n q u i r y in educational evaluation. Unpublished manuscript, February 1, 1978.  Note  3 Campbell, D.T. Q u a l i t a t i v e knowing i n a c t i o n Unpublished manuscript, 1979.  Note 4 Hasman, R.M. Land T i t l e Unpublished manuscript, 1980.  School  Evaluation  research. Report.  1 07  APPENDIX A  Expectat ions Q u e s t i o n n a i r e  LAND  TITLE  SCHOOL  P l e a s e g i v e us y o u r f r a n k r e a c t i o n s a n d o p i n i o n s ; t h e y will h e l p us e v a l u a t e t h i s c o u r s e a n d i m p r o v e f u t u r e p r o g r a m s . A l l information i s confidential a n d w i l l be u s e d o n l y t o improve f u t u r e programs. We w o u l d be g r a t e f u l i f y o u w o u l d u s e a n ID number o f y o u r c h o i c e on t h i s f o r m as i t w i l l h e l p us to a n a l y z e t h e r e s u l t s , P l e a s e u s e t h e same ID number o n a l l f o r m s . ID Please give appropriate  t  y o u r o p i n i o n o f t r a i n i n g by c i r c l i n g t h e number i n e a c h o f t h e o p i n i o n s c a l e s b e l o w .  Example:  ,  I  r  extremely  1  2  1  very  3 fairly  1  5 —  —  'n between  i  fairly  1  -V  6 very  7  ,  1 I extremely  -  COMPLICATED  SIMPLE  complicated  1  2  3  4  5  6  7  s imple  unprac t i c a l  1  2  3  4  5  6  7  practical  accurate  1  2  3  4  5  6  7  i n a c c u r a te  dull  1  2  3  4  5  6  7  interes  difficult  1  2  3  4  5  6  7  easy  helpful  1  2  3  4  5  6  7  unhelpful  fast  1  2  3  4  5  6  7  slow  important  1  2  3  4  5  6  7  unimportant  will  be?  How  useful  do y o u t h i n k 1 useless  How  enjoyable  2  this 3  do y o u t h i n k  1 2 d i d n ' t enjoy i t v e r y much  3  training 4  this 4  5  training 5  6  7 useful  will 6  be?  7 enjoyed i t v e r y much  ting  109  LAND T I T L E  SCHOOL  P l e a s e g i v e us y o u r f r a n k r e a c t i o n s and o p i n i o n s ; t h e y w i l l help us e v a l u a t e t h i s c o u r s e and i m p r o v e f u t u r e c o u r s e s . All i n f o r m a t i o n i s c o n f i d e n t i a 1 and w i l l be u s e d o n l y t o improve f u t u r e c o u r s e s . ID  #  B a s e d on y o u r e x p e r i e n c e o f t h e p a s t 2 w e e k s , p l e a s e g i v e y o u r o p i n i o n o f t h e Land T i t l e S c h o o l by c i r c l i n g t h e a p p r o p r i a t e number in e a c h o f t h e o p i n i o n s c a l e s below. E xamp1e:  1 extremely  2 very too  3 fairly  CS 5 Tn fairly between  6 very too  short  LAND T I T L E  7 extremely  long  SCHOOL  Complicated  1  2  3  fr  5  6  7  Simple  Impractical  1  2  3  fr  5  6  7  P r a c t i ca  Dull  1  2  3  fr  5  6  7  Interest ing  1  2  3  fr  5  6  7  Easy  Helpful  1  2  3  fr  5  6  7  Not  Fast  1  2  3  fr  5  6  7  S  Important  1  2  3  fr  5  6  7  Un i m p o r t a n t  5  6  D i f f i cu  How  1  useful  t  was  this  training  1  Useless How  enjoyable  was  this  2  program?  3  training  I 2 Didn't Enjoy It V e r y Much  1  3  fr  Usef u 1  program?  fr  5  6  7  Enjoyed It V e r y Much  1  Helpful ow  1 10  APPENDIX B  Mini -Session  Questionnaire  ID #  Name o f S e s s i o n :  the number a  ^  e  °  f  l  l  0  W  i  n  g  l  t  e  m  ^  s  a  Placing  circle  around  I found thl« (••ilon to be:  I-.,..  9 , 5 , i  n  between • hort  1.  1  2  long  3  ^  6  2  3  p i c k e d up d u r i n g  ij  S  6  Taught me little I d i d n ' t know  3.  7 Enjoyed i t very much  Amount o f new i n f o r m a t i o n 1  5  Taught me a lot  R e l e v a n c e o f s e s s i o n t o own j o b 1  2  3  h  5  6  Not very relevant  7 Very r e l e v a n t  Length o f s e s s i o n Too  5.  session  7  1  long  2  3  if  s  6  7 Too  short  Level of presentation 1  Complicated  2  3  Comments & S u g g e s t i o n s :  '4  5  6  7  Simple  7  extremely  Enjoyment o f s e s s i o n Didn't enjoy i t very much  2.  6  l«TrTyvery  APPENDIX C  Final  Questionnaire  1 13  ID # LAND TITLE SCHOOL EVALUATION OF TOTAL PROGRAM  I have found t h i s program: chaotic  1  2  3  k  5  6  7  well-ordered  unstimulating  1  2  3  it  5  6  7  stimulating  unimportant  1  2  3  it  5  6  7  uninteresting  1  2  3  5  6  7  interesting  I learned nothing  1  2  3  it  5  6  7  I learned a lot  not relevant to my job  1  2  3  it  5  6  7  relevant to my job  1.  Which o f your e x p e c t a t i o n s fulfilled?  o f t h i s program were  2.  Which o f your e x p e c t a t i o n s unfulfilled?  o f t h i s program were  3.  What i n f o r m a t i o n s k i l l g a i n e d most v a l u a b l e t o you?  t h r o u g h t h i s program i s  Do you f e e l t h i s program w i l l Yes No  e f f e c t your work?  How much e f f e c t ? i little effect Please  2  3 ' fr _o^  5  =0 s o _ s o  6 7 rouch effect  e x p l a i n your answer:  What were the major s t r e n g t h s  o f t h i s program?  What were t h e major weaknesses o f t h i s program?  Did you r e c e i v e enough i n f o r m a t i o n on the c o n t e n t of t h i s program b e f o r e coming? Yes ; No ,. I f no, would i t have been h e l p f u l t o have information? Yes ; No . What s u g g e s t i o n s  this  do you have f o r f u t u r e programs?  a.  Number o f s e s s i o n s  b.  Length o f each s e s s i o n  c.  S u b j e c t s t o be covered  d.  Changes you would make  e.  Other  Based on your e x p e r i e n c e s o f t h e past two weeks, would you come t o t h e next l e v e l c o u r s e . Yes  ;  No  .  Comment  11 6  APPENDIX D  Follow-Up Q u e s t i o n n a i r e  LAND  TITLE  SCHOOL  FOLLOW-UP  P l e a s e g i v e us y o u r f r a n k r e a c t i o n s and o p i n i o n s ; t h e y w i l l h e l p us e v a l u a t e t h i s p r o g r a m a n d i m p r o v e f u t u r e ones. A l l i n f o r m a t i o n i s c o n f i d e n t i a l and w i l l be used o n l y t o improve f u t u r e programs. '••Ie w o u l d b e g r a t e f u l i f y o u w o u l d u s e t h e s a n e chosen d u r i n g t h e course.  I D number ID f  Please give your opinion o f t r a i n i n g by c i r c l i n g t h e a p p r o p r i a t e number i n e a c h o f t h e o p i n i o n s c a l e s b e l o w . Example:  |  3  extremely  very  fairly  i —© — i in between  5 fairly  COMPLICATED  1  2  3  accurate dull  _5  6  difficult  extremely  7  simule  7  practical  7  inaccurate  7_ i n t e r e s t i n g easy  7  unhelpful  fast  7  s low  imoortant  7  unimportant  _5  u s e f u l was t h i s  1  enjoyable  6  training? 2  3  4  useless How  r  very  7  helpful  How  6  SIMPLE  comlicated unpractical  1  was t h i s  6  7  useful  training?  1 2 didn't enjoy i t v e r y much  3  4 enjoyed i t v e r y much  1 18  Has  this  How much  program  e f f e c t e d your work?  ;  HO  effect: 1  2  3  little effect Please  YES  explain  4  5  so-so  your  7  6  much effect  answer:  V.'hat i n f o r m a t i o n o r s k i l l v a l u a b l e t o you?  gained through t h i s  'las y o u r o u t l o o k t o w a r d y o u r Please explain:  j o b changed?  Do y o u h a v e a n y s u g g e s t i o n s f o r f u t u r e  program  YES  courses?  i s most  APPENDIX E  Revised Q u e s t i o n n a i r e s  Mini-session  Final  Follow-up  Expectat ions  JUSTICE INSTITUTE OF B.C.  LAND TITLE SCHOOL Name o f s e s s i o n : P l e a s e r a t e t h i s s e s s i o n on t h e f o l l o w i n g i t e m s by p l a c i n g c i r c l e a r o u n d t h e a p p r o p r i a t e n u m b e r . P l e a s e be c a n d i d i n e x p r e s s i n g your f e e l i n g s , whether they a r e p o s i t i v e or negative. Note t h ed e f i n i t i o n s below.  Example: I  found  1 extremely  this  session  2 veTy  3 fairly  t o be: ^ 5 I R f a i r l y between  6 very  . extremely 7  long  short Definitions:  hadn't heard I t before heard I t before but didn't understand  holds your a t t e n t i o n captures your imagination stimulates  INTEREST o f s e s s i o n 1 no i n t e r e s t NEW INFORMATION  applicable p e r t iner.t appropriate  t o me: 2  3  gained  during  1 2 gained l i t t l e new i n f o r m a t i o n 3.  RELEVANCE  HEW INFORMATION  INTEREST  3  much i n t e r e s t  session: 4  5  6 7 gained a l o t o£ new i n f o r m a t i o n  I n terms of my j o b , I found the i n f o r m a t i o n from today's s e s s i o n t o be:  0  hiqhly useful  useful  not u s e f u l  b)  p r e v i o u s l y NOT known  both o l d and new  previously known  Comm en t s :  FINAL EVALUATION LAND TITLE SCHOOL  P l e a s e r e f l e c t o n y o u r e x p e r i e n c e s o f t h e p a s t week when answering t h e f o l l o w i n g items. Make y o u r comments v e r y specific. Y o u r c o m m e n t s w i l l h e l p u s t r e m e n d o u s l y when we p l a n t h e n e x t c o u r s e .  Definitions:  INTEREST  RELEVANCE  holds your a t t e n t i o n captures your imagination stimulates  1.  applicable pertinent appropriate  How r e l e v a n t d o y o u now c o n s i d e r t h i s your p o s i t i o n ?  entire  not relevant  very relevant  P e r s o n a l l y how i n t e r e s t i n g w a s t h i s 1  2  3  4  5  course? 6  7  not very interesting  Did t h e c o n t e n t expectations?  very  Please  course t o  very interesting  o f t h e course  little  e x p l a i n your  agree  moderately  w i t h your  original  very much  answer:  What i n f o r m a t i o n g a i n e d t h r o u g h w i l l be most v a l u a b l e t o you?  this  course  do y o u f e e l  How  have you judged v a l u e  ( i n q u e s t i o n M)?  (Please check one)  Most p r a c t i c a l use Most  remembered'  Most  revealing  Most  interesting  Other  (Please explain)  What w e r e t h e m a j o r s t r e n g t h s o f t h i s e n t i r e specific.  What w e r e t h e m a j o r w e a k n e s s e s o f t h i s e n t i r e specific.  Please  g i v e any a d d i t i o n a l  comments a n d / o r  course?  course?  Be  Be  suggestions.  123  LAND TITLE SCHOOL FOLLOW-UP  1.  What i s your present p o s i t i o n ?  2.  How  3.  What course d i d you attend?  4.  Have you shared with your f e l l o w workers the handouts and i n f o r m a t i o n presented at the course? Yes No_  long have you held t h i s  I f yes under what s e t t i n g c o f f e e , etc.)  position?  ( i . e . s t a f f meetings,  over  5.  How  6.  Did the i n f o r m a t i o n presented enable you to s o l v e problems or meet s i t u a t i o n s on your job which p r e v i o u s l y you had not been a b l e to do on your own? Yes No  o f t e n have you used the handouts from the course?  Explain:  7.  Did you f i n d the i n f o r m a t i o n presented i n the course u s e f u l i n your d a i l y work? Yes No  8.  If you were to r e o r g a n i z e the course, what would you change, leave the same, etc.? Explain.  E x p e c t a t i o n s Warm-Up  Names a r e important  t o a l l of us.  I t f e e l s c o m f o r t i n g to be  addressed by name i n a s t r a n g e environment.  Having  one-person  i n t r o d u c e h i m / h e r s e l f i s f i n e but t h e r e a r e few people who remember even h a l f  will  the names mentioned.  Using d o u b l e - f o l d e d sheets o f paper o r o l d computer cards as desk-top name c a r d s I s more u s e f u l than the s t i c k - o n type of name t a g . Have each person p r i n t  ( i n b o l d l e t t e r s ) t h e name they  want t o be c a l l e d on both s i d e s o f t h e c a r d . put on the i n s i d e  Have each  person  "Only f o r you t o see" the c o m p l e t i o n of these  sentences: -What I'd r e a l l y l i k e  t o do r i g h t now i s . . .  -I hope t h i s c o u r s e won't be... -What I would l i k e t o l e a r n i n t h i s c o u r s e While  this information i s confidential  includes...  a t t h i s stage, you may ask  v o l u n t e e r s l a t e r on i n the s e s s i o n t o share i t with the c l a s s . I t i s a technique t o h e l p members f o c u s on t h e i r e x p e c t a t i o n s , t h e i r present f e e l i n g s  and t h e i r hopes.  n o t i o n t h a t you c a r e about presence  i n the room.  these f e e l i n g s  I t might a l s o convey the and a r e aware o f t h e i r  

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0055871/manifest

Comment

Related Items