UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Educational evaluation : two theoretical models in a corporate based application 1998

You don't seem to have a PDF reader installed, try download the pdf

Item Metadata

Download

Media
ubc_1998-0193.pdf [ 5.49MB ]
ubc_1998-0193.pdf
Metadata
JSON: 1.0078149.json
JSON-LD: 1.0078149+ld.json
RDF/XML (Pretty): 1.0078149.xml
RDF/JSON: 1.0078149+rdf.json
Turtle: 1.0078149+rdf-turtle.txt
N-Triples: 1.0078149+rdf-ntriples.txt
Citation
1.0078149.ris

Full Text

EDUCATIONAL EVALUATION: TWO THEORETICAL MODELS •' IN A CORPORATE BASED APPLICATION By GORDON W. BARRETT B.Ed., The U n i v e r s i t y of B r i t i s h Columbia A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS i n THE FACULTY OF GRADUATE STUDIES (Department of Language Education) We accept t h i s t h e s i s as conforming to trye/required s t a n d a r d THE UNIVERSITY OF BRITISH COLUMBIA • . APRIL 1998 © G O R D O N WILLIAM BARRETT, 1998 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. Department of L/^fJ(~>Uyi^£ £•~P vc:/rT'&Aj The University of British Columbia Vancouver, Canada Date g2 S~ /9Pfttl- 19<?g DE-6 (2/88) A b s t r a c t 11 The Provus Discrepancy E v a l u a t i o n Model (1973) and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model (1973) are examined a g a i n s t the backdrop of two e v a l u a t i o n s t h a t were conducted by u n s k i l l e d e v a l u a t o r s i n a co r p o r a t e based s e t t i n g by a l a r g e c o r p o r a t i o n . D i f f e r e n c e s between the two t h e o r e t i c a l models and the two co r p o r a t e e v a l u a t i o n s r e v e a l e d t h a t there are f a c t o r s , not c o n s i d e r e d i n the t h e o r e t i c a l models, which can impact t h e i r e f f e c t i v e n e s s when p r a c t i c a l l y a p p l i e d . The Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model were i n some ways a p p r o p r i a t e and both committees would have b e n e f i t e d from u t i l i z i n g s i m i l a r e v a l u a t i o n models. F a i l u r e of these two committees to address s i g n i f i c a n t aspects of formal e v a l u a t i o n might have been remedied by the a p p l i c a t i o n of formal e v a l u a t i o n models. E d u c a t i o n a l e v a l u a t i o n models a l s o have s i g n i f i c a n t "gaps" i n c l u d i n g p e r s o n a l investment of committee members, co r p o r a t e agendas, co s t and f i n a n c i a l impacts, b i a s and s t a l l p o i n t s (the p o i n t where the e v a l u a t i o n model ceases to be e f f e c t i v e i n the c u r r e n t c o n t e x t ) . Corporate e v a l u a t i o n s are an ongoing process and r e q u i r e d i f f e r e n t types of e v a l u a t i o n models, depending upon c u r r e n t need ( S t a l l P o i n t Theory). I l l Two e v a l u a t i o n s t h a t were conducted by task f o r c e committees i n a c o r p o r a t e s e t t i n g were examined. One task f o r c e committee examined how c o r p o r a t e t r a i n i n g was being c a r r i e d out and the second task f o r c e committee examined adherence to how corpor a t e p o l i c y was being c a r r i e d out. Using a formal e v a l u a t i o n model would have p r o v i d e d s t r u c t u r e , o b j e c t i v e c l a r i f i c a t i o n , and g r e a t e r confidence i n the r e s u l t s and recommendations of the corpor a t e e v a l u a t i o n s . The development of e d u c a t i o n a l e v a l u a t i o n models would be enhanced by c o n s i d e r i n g the needs of the end users of the models, making the models more dynamic by i n c r e a s i n g f l e x i b i l i t y f o r g e n e r a l / s p e c i f i c a p p l i c a t i o n and acknowledging the models' l i m i t a t i o n s . i v TABLE OF CONTENTS A b s t r a c t i i Table of Contents i v L i s t of Tables i x L i s t o f F i g u r e s x Acknowledgements x i D e d i c a t i o n x i i CHAPTER ONE: INTRODUCTION • 1 Statement of the Problem 1 T h e o r e t i c a l Models Examined 2 Discrepancy E v a l u a t i o n Model 2 C.I.P.P. E v a l u a t i o n Model • 3 O r i g i n s of the Corporate E v a l u a t i o n Committees . 4 Claims T r a i n i n g E v a l u a t i o n Task Force Committee 5 L i t i g a t i o n Management E v a l u a t i o n Task Force Committee 6 P r e - E v a l u a t i o n s C o n s i d e r a t i o n s 6 I n t e r n a l E v a l u a t i o n Advantages 7 I n t e r n a l E v a l u a t i o n Disadvantages 8 E x t e r n a l E v a l u a t i o n Advantages . 9 E x t e r n a l E v a l u a t i o n Disadvantages 10 A Combination Approach 11 S i g n i f i c a n c e of the Study 12 V S i g n i f i c a n c e to Ed u c a t i o n 12 S i g n i f i c a n c e to Business 13 Two Questions f o r C o n s i d e r a t i o n 13 L i m i t a t i o n s of This Study . 14 Summary 17 CHAPTER TWO: LITERATURE ON EDUCATIONAL EVALUATION.. 19 H i s t o r y o f E v a l u a t i o n i n Educat i o n 19 Edu c a t i o n E v a l u a t i o n Models Used i n This Study . 26 Provus Discrepancy E v a l u a t i o n Model 26 Advantages 2 6 L i m i t a t i o n s 27 Stufflebeam et a l C.I.P.P. E v a l u a t i o n Model .... 28 Advantages 2 8 L i m i t a t i o n s 30 Other Educa t i o n E v a l u a t i o n Models. Considered . . . 31 T y l e r i a n E v a l u a t i o n Approach 31 Hammond's E v a l u a t i o n Approach 32 U.C.L.A. E v a l u a t i o n Model 33 Stake's Countenance E v a l u a t i o n Model 34 R a t i o n a l e f o r S e l e c t i n g the Provus and Stufflebeam et a l . E v a l u a t i o n Models 35 Edu c a t i o n E v a l u a t i o n Research Sources 36 R e s u l t s of Documentation Search 37 Summary 37 v i CHAPTER .THREE:- METHOD ' 38. Task Force Members 38 Role of the P a r t i c i p a n t Observer 39 P a r t i c i p a n t s i n the Claims T r a i n i n g E v a l u a t i o n Task Force Committee I n v e s t i g a t i o n 40 C r i t e r i a f o r Focus Group S e l e c t i o n 41 Task Force Procedures • 42 Methodology: Claims T r a i n i n g E v a l u a t i o n Task Force Committee - H i s t o r i c a l Background 43 Methodology: Claims T r a i n i n g E v a l u a t i o n Task Force Committee - Current S i t u a t i o n 44 L i t i g a t i o n Management E v a l u a t i o n Task Force Committee 4 8 Data C o l l e c t i o n 55 Summary . . . 58 CHAPTER FOUR: RESULTS 59 E v a l u a t i o n Model A p p l i c a t i o n Comparison 60 Stage I - Programme Design 60 Stage II - Programme Operation 66 Stage I I I - Programme I n t e r i m Products .... 70 Stage IV - Programme Terminal Products .... 74 Stage V - Programme Cost • 79 Stufflebeam et a l (1977) C.I.P.P. E v a l u a t i o n Model 81 v i i Context E v a l u a t i o n •• 81 Input E v a l u a t i o n . 82 Process Evaluation- . 88 Product E v a l u a t i o n . .• 92 Summary 94 CHAPTER FIVE: SUMMARY, CONCLUSIONS AND RECOMMENDATIONS 96 Summary .' 97 I d e n t i f i c a t i o n of Programme O b j e c t i v e s and P l a c i n g Them In Context 97 Examination of Programme S t r u c t u r e and Supporting I n f r a s t r u c t u r e .. ... 99 A n t i c i p a t i o n of B a r r i e r s to Success 101 O r i e n t a t i o n of Programme Towards.Results .. 102 Conclusions 102 B e n e f i t s of Using Provus and Stufflebeam's Models : ' 102 F i n a n c i a l Comparison f o r Cost Containment . 106 Committees' E f f o r t s F a c i l i t a t e d by Using a Formal E v a l u a t i o n Model 107 Developing Foundations of E v a l u a t i o n s 115 E v a l u a t i o n Approaches Must Be F l e x i b l e i n P r a c t i c e 122 S t a l l P o i n t s ; 123 v i i i Recommendations . . . . 125 Ep i l o g u e 129 Impact of E x t e r n a l I n f l u e n c e s on E v a l u a t i o n . Model E f f e c t i v e n e s s 129 E x t e r n a l I n f l u e n c e s A c t i n g upon the Claims T r a i n i n g E v a l u a t i o n Task Force Committee . . 129 E x t e r n a l I n f l u e n c e s A c t i n g upon the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee 133 References 138 L i s t of Tables i x Table 1. U.C.L.A. and C.I.P.P. E v a l u a t i o n Models - Comparative Process 34 2. Comparison - Provus Discrepancy E v a l u a t i o n Model, Stage I, and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee Mandate .65 3. Comparison - Provus Discrepancy E v a l u a t i o n Model, Stage I I , and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee Mandate 69 4. * B u l l e t i n CDB894 F a i l u r e : Reasons For F a i l u r e 91 L i s t of F i g u r e s x Fig u r e 1. L i t i g a t i o n Management E v a l u a t i o n Task Committee Weighting of E v a l u a t i o n Process 118 2. Claims E v a l u a t i o n Task Force Committee Weighting of E v a l u a t i o n Process 120 ACKNOWLEDGEMENTS x i I would l i k e to thank my wife, Lynda, f o r a l l of the help and support t h a t she has given me throughout my s t u d i e s . She kept me going when I was f r u s t r a t e d - and overwhelmed at times. She has helped our f a m i l y by t a k i n g . on more than her f a i r share of the l o a d while h e l p i n g me to balance work and U n i v e r s i t y . I would a l s o l i k e to thank my c h i l d r e n , Rachel and W i l l i a m , f o r g i v i n g me t h e i r understanding when I was away at c l a s s e s f o r a good p a r t of t h e i r l i v e s . My Grandmother, L i l i a n B a r r e t t s t a r t e d me o f f on my graduate work many years ago and while she i s not a l i v e today I am c e r t a i n t h a t she i s watching as t h i s now comes to a c l o s e . F i n a l l y , I would l i k e to thank Dr. Joe Belanger f o r h i s p a t i e n c e and support while we t r i e d to reshape my business mind back i n t o an e d u c a t i o n a l focus on a r e g u l a r b a s i s . I am w r i t i n g t h i s acknowledgement on Good F r i d a y . I t i s a reminder t h a t with every end t h e r e i s always a new b e g i n n i n g . The promise of a g l o r i o u s tomorrow wi t h every new moment has been given to us i f we are prepared to stand s t e a d f a s t i n f a i t h . Pax Gordon W i l l i a m B a r r e t t A p r i l 10, 1998 X l l DEDICATION I would l i k e to d e d i c a t e t h i s t h e s i s to my wife, Lynda and to my parents, Gordon and Georgia. I wrote the Acknowledgement on Good F r i d a y , l i t t l e knowing t h a t my mother would pass away on E a s t e r Day; the day of the' R e s u r r e c t i o n . In the Acknowledgement I spoke of the beginnings and endings t h a t we encounter on a d a i l y b a s i s . I t i s w r i t t e n t h a t God i s the Alpha and the Omega, the beginn i n g and the end. My mother's e a r t h l y l i f e has come to an end and now she begins a new journey through P a r a d i s e . She was always proud of her f a m i l y and we were proud t h a t she was our mother. Gordon B a r r e t t A p r i l 20, 1998 CHAPTER ONE INTRODUCTION 1 The q u e s t i o n t h i s study addresses i s what s t r e n g t h s and weaknesses are r e v e a l e d i n two co r p o r a t e e v a l u a t i o n s when they are judged post-hoc u s i n g two e v a l u a t i o n models designed f o r e d u c a t i o n a l e v a l u a t i o n , the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . Context, Input, Process, Product Model. A r e l a t e d q u e s t i o n i s what kinds of r e v i s i o n s i n the e d u c a t i o n a l e v a l u a t i o n models do a p p l i c a t i o n s i n a corpora t e s e t t i n g suggest? This study examines these two t h e o r e t i c a l e d u c a t i o n a l e v a l u a t i o n models and compares them to two e v a l u a t i o n models th a t were, apparently, i n t u i t i v e l y developed by e v a l u a t i o n committees f o r e v a l u a t i o n programmes w i t h i n a l a r g e c o r p o r a t i o n . An examination of the two t h e o r e t i c a l e v a l u a t i o n models and the two i n t u i t i v e l y developed e v a l u a t i o n models i d e n t i f i e s s i m i l a r i t i e s and d i f f e r e n c e s which l e a d to a b e t t e r understanding of the e v a l u a t i o n process t h a t f u n c t i o n e d w i t h i n the context of t h i s study. Statement of the Problem This study examines the processes used by the two i n t e r n a l e v a l u a t i o n committees i n order to expl o r e the d i f f e r e n c e s and s i m i l a r i t i e s between e v a l u a t i o n models t h a t are developed i n t u i t i v e l y by u n t r a i n e d e v a l u a t o r s and 2 t h e o r e t i c a l e d u c a t i o n a l e v a l u a t i o n models. Such a comparison should p r o v i d e i n s i g h t s f o r both business and . education. T h e o r e t i c a l Models Examined Using Worthen and Sanders (1987) t e x t , " E d u c a t i o n a l E v a l u a t i o n : A l t e r n a t i v e Approaches and P r a c t i c a l G u i d e l i n e s " , and M i l l e r and S e l l e r ' s (1990) "C u r r i c u l u m P e r s p e c t i v e s and P r a c t i c e s " as a guide, s e v e r a l t h e o r e t i c a l e v a l u a t i o n models were c o n s i d e r e d f o r comparison to the i n t u i t i v e models developed by the C o r p o r a t i o n . The two t h e o r e t i c a l models that''were. chosen f o r t h i s study were the Provus (1973) Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . (1973) C.I.P.P. E v a l u a t i o n Model. Discrepancy E v a l u a t i o n Model The Provus (1973) Discrepancy E v a l u a t i o n Model was chosen s p e c i f i c a l l y because i t addresses e v a l u a t i o n as a "continuous i n f o r m a t i o n management process designed to serve as ^the watchdog of program management' and the handmaiden of a d m i n i s t r a t i o n i n the management of program development through sound d e c i s i o n making" (Provus, 1973, p. 186). The Provus Discrepancy E v a l u a t i o n Model suggests t h a t e v a l u a t i o n i s a process which should pass through f i v e developmental stages: D e f i n i t i o n , I n s t a l l a t i o n , , Process ( i n t e r i m - products) , Product and C o s t - b e n e f i t a n a l y s i s (optional) . 3 The Provus Discrepancy E v a l u a t i o n Model i s concerned with c l e a r l y d e f i n i n g o b j e c t i v e s and then d e v e l o p i n g a p l a n to achieve those o b j e c t i v e s . The Discrepancy E v a l u a t i o n Model, simply put, examines the process of o b j e c t i v e attainment and measures the gaps between what the intended outcomes and the a c t u a l outcomes are i n order t h a t changes may be made i n the program to narrow the gap. This e v a l u a t i o n model i s concerned with i n p u t s , outputs, a l l o c a t e d resources and o b j e c t i v e s . The emphasis i s upon de v e l o p i n g an o p e r a t i o n a l p l a n which r e l i e s upon c l e a r l y d e f i n e d s p e c i f i c o b j e c t i v e s . For the p r e c e d i n g reasons, the Provus Discrepancy E v a l u a t i o n Model was chosen as one of the' two e d u c a t i o n a l e v a l u a t i o n models to apply i n a c o r p o r a t e s e t t i n g . C.I.P.P. E v a l u a t i o n Model The second model chosen was the Stufflebeam et a l . (1973) C.I.P.P. E v a l u a t i o n Model. T h i s model i s concerned with f o u r developmental stages: Context, Input, Process and Product (hence the acronym C.I.P.P.). The focus of the Stufflebeam et al.(1973) C.I.P.P. E v a l u a t i o n Model, u n l i k e Provus's o b j e c t i v e o r i e n t e d model, i s towards making the e v a l u a t i o n model a management-like t o o l with the aim of a s s i s t i n g with the d e c i s i o n making pr o c e s s . Stufflebeam et a l . (1973, p. 129) d e s c r i b e t h i s model as "the process of d e l i n e a t i n g , o b t a i n i n g , and 4 p r o v i d i n g u s e f u l i n f o r m a t i o n f o r j u d g i n g d e c i s i o n a l t e r n a t i v e s " . The l i t e r a t u r e on e v a l u a t i o n d i d not r e v e a l s t u d i e s which drew p a r a l l e l s between Provus's and Stufflebeam et a l . ' s t h e o r e t i c a l models and business management concepts. This study has chosen the Provus and Stufflebeam et a l . e v a l u a t i o n models to use i n a comparison•between t h e o r e t i c a l e v a l u a t i o n models and i n t u i t i v e e v a l u a t i o n models which are co r p o r a t e based and are both f i e l d developed and t e s t e d . T h i s comparison w i l l address t h i s study's q u e s t i o n with r e s p e c t to i d e n t i f y i n g some of the gaps between theory and p r a c t i c a l a p p l i c a t i o n of e d u c a t i o n a l e v a l u a t i o n models i n a co r p o r a t e s e t t i n g i n order to l e a r n from these s i m i l a r i t i e s and d i f f e r e n c e s . O r i g i n s of the Corporate E v a l u a t i o n Committees The e v a l u a t i o n committees w i t h i n the c o r p o r a t i o n were s t r u c k f o r the purpose of e v a l u a t i n g the c o r p o r a t i o n ' s Claims T r a i n i n g Department and the c o r p o r a t i o n ' s l i t i g a t i o n budgeting p r o c e s s . T h i s study w i l l f i r s t examine the o r i g i n s of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and then the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee. The work of these committees w i l l be used as a v e h i c l e to t e s t the e f f e c t i v e n e s s of the Provus Discrepancy E v a l u a t i o n , Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model i n a co r p o r a t e s e t t i n g . 5 Claims T r a i n i n g E v a l u a t i o n Task Force Committee I n i t i a l d i s c u s s i o n s with the Claims T r a i n i n g E v a l u a t i o n Task Force Committee chairman r e v e a l e d t h a t the c o r p o r a t i o n ' s T r a i n i n g Department had been r e c e i v i n g c r i t i c i s m about being out of touch with the needs i n the f i e l d , t h a t the method of c e n t r a l i z e d t r a i n i n g was f i n a n c i a l l y i n a p p r o p r i a t e g i v e n the r e g i o n a l nature of the c o r p o r a t i o n and t h a t the t r a i n i n g process i t s e l f was out of date given changes i n a d u l t e d u c a t i o n methodology and m u l t i - media technology. My f i e l d notes of May 5, 1994 captured the comments of a s e n i o r manager who was the c o r p o r a t i o n ' s Deputy C h i e f Underwriter. These comments r e f l e c t t h a t there was concern with r e s p e c t to core s k i l l t r a i n i n g and the use of technology w i t h i n the c o r p o r a t i o n ' s e d u c a t i o n programme. A sampling of these comments i s l i s t e d below: ... t h a t claims t r a i n i n g as i t e x i s t e d i s out of touch with f i e l d the process of insurance claims has not changed s i n c e the 1930's and 40's, i . e . , the way we do business has not changed technology (as)- the k e r n e l of insurance, has not • changed emphasize that t h i s i s the message of the C h i e f Underwriter and the former C h i e f Underwriter. 6 Operating from t h i s need to respond to the p e r c e i v e d concerns- by s t a f f and management, an e v a l u a t i o n committee was st r u c k , under the sponsorship of a s e n i o r manager, to determine the f u t u r e of the Claims T r a i n i n g Department. L i t i g a t i o n Management E v a l u a t i o n Task Force Committee The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee Was s t r u c k f o r the purpose of e v a l u a t i n g the c o r p o r a t i o n ' s l i t i g a t i o n budgeting p r o c e s s . The budgeting process which was i n p l a c e at the time of t h i s study was e s t a b l i s h e d i n order to c o n t r o l the expense of l i t i g a t i o n by p r o j e c t i n g the f u t u r e c o s t s of defending l e g a l a c t i o n s . Senior management, supported by data c o l l e c t e d by the c o r p o r a t i o n ' s r e g i o n a l managers, f e l t t h a t by a c c u r a t e l y f o r e c a s t i n g and a s s e s s i n g the co s t of defending l e g a l a c t i o n s , these a c t i o n s c o u l d be s e t t l e d e a r l i e r f o r l e s s expense by i n c o r p o r a t i n g p r o j e c t e d defence c o s t s i n t o the r i s k a n a l y s i s p r o c e s s . A l s o , by i n c o r p o r a t i n g a budgeting process f o r l i t i g a t i o n , i t was f e l t t h a t defence counsel would be more l i k e l y to h o l d themselves accountable f o r p r o v i d i n g an expected l e v e l of s e r v i c e r a t h e r than i n c r e a s i n g c o r p o r a t e expenses through unnecessary delays and l i t i g a t i o n i n e f f i c i e n c i e s . P r e - e v a l u a t i o n s C o n s i d e r a t i o n s Senior managers i n the c o r p o r a t i o n i n each circumstance had to decide whether the e v a l u a t i o n s would be conducted / 7 e i t h e r by an i n t e r n a l e v a l u a t i o n committee or by e x t e r n a l c o n s u l t a n t s , or i f i t would be of value to conduct both an i n t e r n a l and e x t e r n a l e v a l u a t i o n i n order to have a complete, well-rounded e v a l u a t i o n . Senior management would have to take i n t o c o n s i d e r a t i o n the advantages and disadvantages with u s i n g e i t h e r approach or a combination of approaches. Some of these advantages and disadvantages - i n c l u d e l e v e l s of b i a s , e x p e r t i s e (or l a c k of e x p e r t i s e ) i n the s u b j e c t area being evaluated, time frames f o r completion of the e v a l u a t i o n and the o v e r a l l c o s t of each of the v a r i o u s o p t i o n s a v a i l a b l e . I n t e r n a l E v a l u a t i o n Advantages With an i n t e r n a l e v a l u a t i o n , e v a l u a t o r s having an a p p r e c i a t i o n f o r the s u b j e c t b e i n g ' e v a l u a t e d c o u l d be seen as an advantage. Having a c l o s e f a m i l i a r i t y w i t h the s u b j e c t matter should allow f o r g r e a t e r i n s i g h t . While there are elements of b i a s , when conducting i n t e r n a l e v a l u a t i o n s there are advantages to be had from the i n t e r a c t i o n between e v a l u a t o r s who share a common i n t e r e s t . As s t a t e d by Garaway (1995, p.87): In p a r t i c i p a t o r y e v a l u a t i o n , the p o o l of p a r t i c i p a n t s tends to be a s m a l l e r group. While there i s s t i l l p l u r a l i s m , with the concomitant problem of c o n f l i c t i n g i n t e r e s t s , there i s the added aspect of i n t e r a c t i o n which somewhat spreads the r o l e of a d j u d i c a t i o n among the p a r t i c i p a n t s . 8 From my experience as a manager, u t i l i z i n g i n t e r n a l s t a f f f o r conducting e v a l u a t i o n s i s c o s t e f f e c t i v e . Only i n s p e c i a l circumstances would the purpose of an e v a l u a t i o n outweigh the expense of u s i n g e x t e r n a l c o n s u l t a n t s . In other words, the s i t u a t i o n would need to be of great importance f o r the c o s t of a.dual e v a l u a t i o n to be warranted. I n t e r n a l e v a l u a t i o n s are c o s t - e f f e c t i v e i n t h a t s t a f f employees can be seconded to conduct e v a l u a t i o n s and the c o s t to the c o r p o r a t i o n i s l i m i t e d to the l o s s of those employees' p r o d u c t i v i t y . A wide range of p o i n t s of view may be brought i n t o such an e v a l u a t i o n by drawing upon the d i v e r s e nature of the c o r p o r a t i o n which means t h a t the e v a l u a t o r s c o u l d be drawn from the l a r g e p o o l of employees a v a i l a b l e from any area of the c o r p o r a t i o n . Again, as i n d i c a t e d by Garaway(1995, p. 98): P a r t i c i p a t o r y e v a l u a t i o n c r e a t e s a l e a r n i n g environment i n which e v a l u a t i o n f i n d i n g s are processed and accumulated by end-users i n the very process of t h e i r b e i n g gathered. In other words, an i n t e r n a l e v a l u a t i o n has the c a p a c i t y to e f f e c t ongoing change as the e v a l u a t i o n u n f o l d s which may be of c o n s i d e r a b l e advantage to an o r g a n i z a t i o n . I n t e r n a l E v a l u a t i o n Disadvantages There are s e v e r a l disadvantages to u t i l i z i n g an i n t e r n a l e v a l u a t i o n p r o c e s s . S t a n l e y and Hopkins (1972, p. 4) s t a t e d : The extent to which a measurement or e v a l u a t i o n i s s u b j e c t i v e i s the degree to which p e r s o n a l b i a s and p r e j u d i c e can i n f l u e n c e s c o r e s . I t i s d e s i r a b l e to i n c r e a s e the o b j e c t i v i t y of t e s t i n g , i n t e r v i e w i n g , r a t i n g , and s i m i l a r e n t e r p r i s e s that o f t e n are pursued q u i t e s u b j e c t i v e l y . As the Barnes C r a i g & A s s o c i a t e s r e p o r t (1993, p. 6) s t a t e d : T a l l e y g r a n d i s c r e d i t e d w i t h saying, xWar i s too s e r i o u s a matter to be e n t r u s t e d to the m i l i t a r y . ' So too, l i t i g a t i o n has become f a r too s e r i o u s a matter to be l e f t i n the hands of Lawyers. This study c o n s i d e r e d the r o l e of i n t e r n a l e v a l u a t o r s who were drawn from st a k e h o l d e r groups which may have had v e s t e d i n t e r e s t s i n the outcome of the e v a l u a t i o n . For p o l i t i c a l reasons i n t e r n a l e v a l u a t i o n s should be used . c a r e f u l l y as p e r s o n a l and departmental agendas may i n f l u e n c e the outcome of the e v a l u a t i o n . Even with the best of i n t e n t i o n s and a l a c k of p o l i t i c a l i n f l u e n c e , i n t e r n a l e v a l u a t o r s may simply be too c l o s e to the s u b j e c t being e v a l u a t e d and t h e r e f o r e b l i n d to s i g n i f i c a n t e v a l u a t i o n c r i t e r i a . They may a l s o l a c k e v a l u a t i o n e x p e r t i s e . E x t e r n a l E v a l u a t i o n Advantages There are advantages to u s i n g an e x t e r n a l e v a l u a t i o n p r o c e s s . One advantage i s t h a t s p e c i f i c s p e c i a l t i e s of e x t e r n a l e v a l u a t o r s c o u l d be s e l e c t e d i n order to u t i l i z e t h e i r e x p e r t i s e i n a p a r t i c u l a r area when such e x p e r t i s e i s not a v a i l a b l e i n t e r n a l l y . I 10 O u t l i n e d below are the s e r v i c e s o f f e r e d i n the Barnes C r a i g & A s s o c i a t e s r e p o r t (1993, p. 1). L i c e n s e d a d j u s t i n g s t a f f i s drawn from both the l e g a l community and insurance i n d u s t r y . T h i s enables B.C. & A. to e c o n o m i c a l l y d e l i v e r [ s i c ] h i g h q u a l i t y , i n n o v a t i v e s o l u t i o n s by f o c u s i n g the a p p r o p r i a t e e x p e r t i s e on every p r o j e c t and case. E x t e r n a l e v a l u a t i o n s , i n c o n t r a s t to i n t e r n a l e v a l u a t i o n s , w i l l g e n e r a l l y p r o v i d e an unbiased, u n i n f l u e n c e d p e r s p e c t i v e to the e v a l u a t i o n . In my experience, when an e v a l u a t i o n i s being conducted i n an area where the i s s u e s are v o l a t i l e and emotional, an e x t e r n a l e v a l u a t i o n process i s able to p r o v i d e an a i r of f a i r n e s s and d e f l e c t h o s t i l i t i e s from the p a r t i e s who have commissioned the e v a l u a t i o n . In t h i s sense, the e v a l u a t o r p a r t i a l l y assumes the r o l e of mediator i n the e v a l u a t i o n p r o c e s s . As i n d i c a t e d by Garaway (1995, p. 87): In e v a l u a t i o n c a r r i e d out by the p a r t i c i p a n t s , the e x t e r n a l e v a l u a t o r goes beyond being the primary i n v e s t i g a t o r and p a r t i c i p a n t observer to becoming a f a c i l i t a t o r . As f a c i l i t a t o r , h i s aim i s to help t r a n s f o r m a f a i r l y n a t u r a l process ( e v a l u a t i o n being something we a l l undertake, p e r s o n a l l y , ' c o n t i n u a l l y ) i n t o a b r o a d l y u t i l i z a b l e p r o c e s s . E x t e r n a l E v a l u a t i o n Disadvantages There are s e v e r a l disadvantages to an e x t e r n a l e v a l u a t i o n . E x t e r n a l e v a l u a t i o n s have a tendency to be expensive and the r e s u l t s can sometimes be c h a l l e n g e d on the b a s i s of the e x t e r n a l e v a l u a t o r s ' a p p a r e n t l y not being 11 f a m i l i a r w i t h the p a r t i c u l a r nuances of the o b j e c t being e v a l u a t e d . While the e x t e r n a l e v a l u a t o r s h o l d no a l l e g i a n c e , t h e i r n o n - p a r t i s a n r o l e may be c r i t i c i s e d on the b a s i s t h a t a l a c k of i n t i m a t e knowledge of the e v a l u a t i o n o b j e c t and i t s impact on the r e s t of the c o r p o r a t i o n may r e s u l t i n a m i s i n t e r p r e t a t i o n of the data. T h i s d e f i c i e n c y i s i d e n t i f i e d i n the Claims T r a i n i n g P r o j e c t Proposal ( A p r i l 6, 1994, p. 3) where the p r o j e c t team members are d e s c r i b e d as having " f i r s t hand experience of the i n f o r m a t i o n being generated and able to ask questions as a p p r o p r i a t e " . A Combination Approach A t h i r d o p t i o n i s to conduct both an i n t e r n a l and e x t e r n a l e v a l u a t i o n to o f f s e t the d e f i c i e n c i e s of both e v a l u a t i o n methods. This o p t i o n i s not u s u a l l y v i a b l e due to the expense i n v o l v e d . D i s c u s s i o n s with the Claims T r a i n i n g E v a l u a t i o n Task Force Committee chairman r e s u l t e d i n the f o l l o w i n g summary from my notes on the s t r e n g t h s and weaknesses of a dual e v a l u a t i o n . These notes were not shown to the chairman f o r approval Or comment. In order to warrant the conducting of a dual e v a l u a t i o n , the i s s u e s must be s e r i o u s and the p o t e n t i a l r e s u l t s must have-a wide-ranging impact on the C o r p o r a t i o n . A dual e v a l u a t i o n w i l l be both c o s t l y i n terms of p r o d u c t i v i t y and f i n a n c i n g , but the r e s u l t s are u s u a l l y v e r y s o l i d , w ith l i t t l e opening f o r c r i t i c i s m i f they are conducted i n a competent f a s h i o n . The u s u a l p r a c t i c e would be to conduct an i n t e r n a l e v a l u a t i o n i n i t i a l l y and f o l l o w t h a t by an e x t e r n a l e v a l u a t i o n to c o n f i r m the r e s u l t s of the i n t e r n a l e v a l u a t i o n . The e x t e r n a l e v a l u a t o r s can 12 draw upon the experience and f i n d i n g s of the i n t e r n a l e v a l u a t i o n once they have conducted t h e i r e x t e r n a l e v a l u a t i o n i n order to. determine v a r i a n c e s and explore the reasons f o r the v a r i a n c e s . The dual e v a l u a t i o n a l s o has the e f f e c t of a c t i n g as a b u f f e r f o r management i n t h a t i t w i l l d e f l e c t c r i t i c i s m . ( A p r i l 8, 1994) S i g n i f i c a n c e of the Study This study examines two t h e o r e t i c a l e d u c a t i o n a l e v a l u a t i o n models, the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model, to determine what b e n e f i t s , i f any, c o u l d be had i f they were a p p l i e d i n a corpora t e s e t t i n g . By s t u d y i n g these models i n a corpor a t e s e t t i n g weaknesses and s t r e n g t h s of the models w i l l be h i g h l i g h t e d . By s t u d y i n g these q u a l i t i e s , e d u c a t i o n a l e v a l u a t o r s w i l l be i n a b e t t e r p o s i t i o n to improve upon the design of e v a l u a t i o n models. The e v a l u a t o r s i n the co r p o r a t e s e t t i n g w i l l be able to weigh the advantages of u s i n g a formal e v a l u a t i o n process to enhance co r p o r a t e e v a l u a t i o n s . S i g n i f i c a n c e to Educ a t i o n . E d u c a t i o n would b e n e f i t from the knowledge gained of the p r a c t i c a l needs and concerns of b u s i n e s s . As w e l l , a f u r t h e r i n g of the understanding of t h e o r e t i c a l models by an examination of any gaps t h a t e x i s t between the i n t u i t i v e and t h e o r e t i c a l models w i l l be i n s t r u c t i v e . By s t u d y i n g these gaps e d u c a t i o n a l e v a l u a t o r s w i l l be i n a p o s i t i o n to enhance the p r a c t i a l a p p l i c a t i o n p o t e n t i a l of e v a l u a t i o n models by a v o i d i n g 13 p i t f a l l s and d e s i g n i n g e v a l u a t i o n models which add value f o r the users of the models. S i g n i f i c a n c e to Business. The b u s i n e s s world would b e n e f i t from the a p p l i c a t i o n of t h e o r e t i c a l models, which may be t a i l o r e d to s p e c i f i c needs, without having to " r e - i n v e n t the wheel". As Robbins and Stuart-Kotze (1986, p. 24), s t a t e i n t h e i r management theory t e x t : because theory and p r a c t i c e are o f t e n d i v e r g e n t , management i s both a normative (what should be done) and d e s c r i p t i v e (what i s a c t u a l l y done) pr o c e s s . In other words, the p r a c t i c a l i t y of the day-to-day job o f t e n takes p r i o r i t y over t h e o r e t i c a l "how t o ' s " . However, by t a k i n g t h e o r e t i c a l e v a l u a t i o n models and l e a r n i n g about the s i m i l a r i t i e s and gaps between what i s expected to occur and what i s a c t u a l l y o c c u r r i n g , r e l e v a n t " p i e c e s " may be s i f t e d from the process i n order to enhance the success p o t e n t i a l of the e v a l u a t i o n s t h a t would otherwise be s o l e l y i n t u i t i v e . Two questions f o r c o n s i d e r a t i o n 1. Is there a need f o r t h i s study? In business as i n education there i s an ongoing need to improve e f f i c i e n c y and to f o r e c a s t r e s u l t s a c c u r a t e l y . As s t a t e d by Robbins and Stuart-Kotze, (1986, p. 179): O r g a n i z a t i o n s whose management can develop accurate f o r e c a s t s of e x t e r n a l and i n t e r n a l f a c t o r s have a d i s t i n c t advantage over t h e i r l e s s s u c c e s s f u l competitors. I f the v a r i a t i o n management i s attempting to p r e d i c t f o l l o w s some e s t a b l i s h e d p a t t e r n or r e l a t i o n s h i p , there are f o r e c a s t i n g techniques that can be v a l u a b l e . 14 'By l o o k i n g at the v a r i a n c e s between i n t u i t i v e e v a l u a t i o n models and t h e o r e t i c a l e d u c a t i o n a l e v a l u a t i o n models i t i s p o s s i b l e f o r those d i f f e r e n c e s and s i m i l a r i t i e s to be ex p l o r e d and new l i n e s o f thought developed f o r e d u c a t i o n a l e v a l u a t i o n purposes. The cor p o r a t e based e v a l u a t i o n p r a c t i c e s can be expected to show weaknesses i n the e d u c a t i o n a l e v a l u a t i o n , models by v i r t u e of the f a c t t h a t gaps become apparent when a p p l i e d to d i f f e r e n t p o p u l a t i o n s and s e t t i n g s ; i n a d d i t i o n the education models may o f f e r a p e r s p e c t i v e that has not been apparent to b u s i n e s s . 2. How i s the study d e l i m i t e d ? T h i s study examines o n l y two t h e o r e t i c a l e d u c a t i o n a l e v a l u a t i o n models and two i n t u i t i v e c o r p o r a t e based e v a l u a t i o n s . T h i s study has not e x p l o r e d the p o s s i b l e e f f e c t s of an e v a l u a t i o n team upon the a p p l i c a t i o n of e v a l u a t i o n models, nor has i t c o n s i d e r e d the p o s s i b l e i n f l u e n c e of committee composition upon r e s u l t s . • While there i s a s t r o n g business component to t h i s study, there w i l l be no d i s c u s s i o n of business p r o c e s s e s . I t i s not the i n t e n t of t h i s study to develop s t r a t e g i e s f o r business management. L i m i t a t i o n s of This Study Two i n t u i t i v e models were developed w i t h i n a l a r g e 15 c o r p o r a t i o n and there c o u l d be v a r i a n c e s among l a r g e c o r p o r a t i o n s or between l a r g e c o r p o r a t i o n s and s m a l l e r f i r m s . Given d i f f e r e n t types of business, d i f f e r e n t types of e v a l u a t i o n processes may be r e q u i r e d . The p a r t i c i p a n t observer r o l e may c r e a t e an element of b i a s i n t h i s study. I f t h i s study were to be r e p l i c a t e d u s i n g an-' i m p a r t i a l observer, d i f f e r e n t o b s e r v a t i o n s may be made. At times, my dual r o l e as p a r t i c i p a n t and observer was i n c o n f l i c t . As a p a r t i c i p a n t i n b o t h committees, I had c e r t a i n r e s p o n s i b i l i t i e s which c o u l d have i n f l u e n c e d my o b j e c t i v i t y as an observer. In the Claims T r a i n i n g E v a l u a t i o n Task Force Committee, I co-authored the f i n a l r e p o r t and I acted as chairman of the p r e s e n t a t i o n committee. As chairman, I l e d the p r e s e n t a t i o n committee i n d e l i v e r i n g the r e s u l t s to the Manager of Human Resources Development. In t h i s r o l e I was expected to defend the work of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and i n p a r t i c u l a r the methodology, r e s u l t s and c o n c l u s i o n s . In the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee I authored the r e p o r t and was a member of the p r e s e n t a t i o n committee. In both cases, I had a p e r s o n a l and p r o f e s s i o n a l i n t e r e s t i n the success of the committees' r e s u l t s . However, t h i s c o n f l i c t d i d not a r i s e i n e i t h e r committee u n t i l a f t e r the data had been c o l l e c t e d and •16 analyzed by the f u l l membership of each committee. During the course of each e v a l u a t i o n , I was not aware of the r o l e t h a t I would take i n a u t h o r i n g r e p o r t s or p r e s e n t i n g f i n d i n g s . The f i n a l r e p o r t s of each committee were submitted f o r review by the committee members p r i o r to b e i n g made p u b l i c . The rank and p o s i t i o n of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and L i t i g a t i o n Management E v a l u a t i o n Task Force Committee members may have p l a y e d a r o l e i n the f u n c t i o n i n g of each committee. Given d i f f e r e n t committee members with d i f f e r e n t s t a t u s w i t h i n the c o r p o r a t i o n , the r e s u l t s may have been d i f f e r e n t . The time frames allowed f o r each committee may have i n f l u e n c e d the approach taken by each committee. For example, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee was g i v e n approximately three months w i t h i n which to conduct i t s e v a l u a t i o n of the Claims T r a i n i n g Department and p r e s e n t the r e s u l t s to s e n i o r management. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, on the other hand, was under no such time r e s t r i c t i o n . In f a c t i t had no time r e s t r i c t i o n at a l l . V a r y i n g the time frames by e i t h e r extending or r e s t r i c t i n g the time p e r i o d s allowed f o r e v a l u a t i o n may have produced d i f f e r e n t r e s u l t s . The method used f o r data c o l l e c t i o n was t h a t of p a r t i c i p a n t observer. The p a r t i c i p a n t observer r o l e may 17 have i n f l u e n c e d the workings of both the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee. A l s o , as p a r t i c i p a n t observer, i t was p o s s i b l e to move too e a s i l y from my r o l e as p a r t i c i p a n t observer i n t o my r o l e as an a c t i v e p a r t i c i p a n t on the committees. There may have been an impact on the f u n c t i o n i n g o f both committees by e x t e r n a l p e r s o n a l and p o l i t i c a l i n f l u e n c e s . The impact of e x t e r n a l i n f l u e n c e s w i l l be d i s c u s s e d i n Chapter F i v e . T h i s f a c t o r i s s i g n i f i c a n t when examining e v a l u a t i o n programmes i n corporat e s e t t i n g s and i s not taken i n t o account i n e i t h e r the Provus Discrepancy E v a l u a t i o n Model or the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model. The p o t e n t i a l i s s i g n i f i c a n t f o r corporate based e v a l u a t i o n s to be a f f e c t e d by p e r s o n a l and corporate agendas and other e x t e r n a l f a c t o r s which may or may not be apparent to e v a l u a t i o n committees. T h i s aspect underscores the whole of the e v a l u a t i o n p r o c e s s . I t i s a f a c t o r which p o t e n t i a l l y c o u l d put the members of an e v a l u a t i o n committee at r i s k and, as w i l l be d i s c u s s e d , can app a r e n t l y shape the focus of the e v a l u a t i o n p r o c e s s . Summary This study w i l l use two t h e o r e t i c a l e d u c a t i o n a l e v a l u a t i o n models, the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model. These models are a p p l i e d r e t r o s e p e c t i v e l y i n a c o r p o r a t e s e t t i n g i n order to r e v e a l s t r e n g t h s and weaknesses of the models. These s t r e n g t h s and weaknesses w i l l be c o n s i d e r e d from the p e r s p e c t i v e of what can be added to the general knowledge of e d u c a t i o n a l e v a l u a t i o n . In a d d i t i o n , these s t u d i e s w i l l be used to p r o v i d e i n s i g h t i n t o enhancing e v a l u a t i o n s conducted i n a corpo r a t e s e t t i n g . 19 CHAPTER TWO LITERATURE ON EDUCATIONAL EVALUATION Chapter Two w i l l p r o v i d e a b r i e f overview of the h i s t o r y of E d u c a t i o n E v a l u a t i o n with a p a r t i c u l a r emphasis upon the Ed u c a t i o n E v a l u a t i o n Models of the 1960's and 1970's. Models from t h i s time p e r i o d were chosen as i t was the b e g i n n i n g of a new era i n e d u c a t i o n a l e v a l u a t i o n . T h i s b r i e f h i s t o r y i s p r e s e n t e d as a h i g h l e v e l overview and i s intended to p r o v i d e ' g e n e r a l background i n f o r m a t i o n . The two models used i n t h i s study, the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model, w i l l be d i s c u s s e d b r i e f l y i n order to h i g h l i g h t t h e i r s t r e n g t h s and weaknesses. Other e v a l u a t i o n models w i l l a l s o be b r i e f l y o u t l i n e d . H i s t o r y of E v a l u a t i o n i n E d u c a t i o n E v a l u a t i o n , as a component of the e d u c a t i o n process, was not p r e v a l e n t u n t i l the l a t t e r h a l f of the n i n e t e e n t h century. A c c o r d i n g to Worthen and Sanders (1987, p. 12), e v a l u a t i o n i n American education p r i o r to the mid n i n e t e e n t h century was e s s e n t i a l l y n o n - e x i s t e n t . The d i r e c t i o n taken by education was i n f l u e n c e d by r e l i g i o u s or p o l i t i c a l p o l i c i e s as opposed to competency, need, e t c . As s t a t e d i n Worthen and Sanders (1987, p. 12) : P r i o r to 1837, p o l i t i c a l and r e l i g i o u s b e l i e f s d i c t a t e d most education c h o i c e s . Communities were 20 happy to a t t r a c t and hold' teachers, r e g a r d l e s s of t h e i r competence, and i f a teacher d i d prove incompetent i n those days-, formal e v a l u a t i o n was r e l a t i v e l y p o i n t l e s s anyway - the school j u s t c l o s e d f o r l a c k of students. The foundation of data c o l l e c t i o n f o r e d u c a t i o n a l purposes has been a t t r i b u t e d to Henry Barnard, Horace Mann and W i l l i a m Torrey H a r r i s . They worked f o r the s t a t e e d u c a t i o n departments of Massachusetts and Connecticut, as w e l l as the U n i t e d S t a t e s E d u c a t i o n Bureau, where they developed data c o l l e c t i o n processes f o r amassing i n f o r m a t i o n which c o u l d be used to make d e c i s i o n s r e g a r d i n g e d u c a t i o n a l p o l i c y . Worthen and Sanders (1987, p. 12) i n d i c a t e t h a t d u r i n g the p e r i o d 1838 to 1850, "Horace Mann submitted 12 annual r e p o r t s to the Board of Ed u c a t i o n of the Commonwealth of Massachusetts". These r e p o r t s c o n t a i n e d concerns with r e s p e c t to a number of areas, ranging from o u t s i d e s u p e r v i s i o n to the s e l e c t i o n or c o n s t r u c t i o n of c u r r i c u l u m m a t e r i a l s . No mention was made as to how the data were c o l l e c t e d , a c c o r d i n g to Worthen and Sanders. During the l a t t e r p o r t i o n of the n i n e t e e n t h century, Joseph R i c e conducted an assessment of l a r g e s c h o o l systems across the U n i t e d - S t a t e s . T h i s assessment was conducted with the aim of e s t a b l i s h i n g - whether or not h i s theory o f the i n e f f i c i e n t use of sch o o l time was c o r r e c t . As Worthen; and Sanders (1987, p. 13) s t a t e : 21 He used these data to support h i s p r o p o s a l s f o r r e s t r u c t u r i n g s p e l l i n g i n s t r u c t i o n . His t e s t s of a r i t h m e t i c , on the other hand, r e v e a l e d l a r g e d i f f e r e n c e s among sc h o o l s ; consequently Rice proposed the s e t t i n g up of s t a n d a r d i z e d examinations. Edward Lee Thorndike, at the b e g i n n i n g of the Twentieth Century, proposed t h a t measuring developmental change was an important f a c e t of the education p r o c e s s . Measurement took the form of t e s t i n g , which became the accepted means of e v a l u a t i n g s c h o o l s . Guba and L i n c o l n (1989, p 24) make the f o l l o w i n g comments on the use of t e s t i n g : The u t i l i t y of t e s t s f o r school purposes was w e l l r e c o g n i z e d by l e a d e r s h i p p e r s o n n e l . The N a t i o n a l E d u c a t i o n A s s o c i a t i o n appointed a committee i n 1904 to study the use of t e s t s i n c l a s s i f y i n g c h i l d r e n and determining t h e i r p r o g r e s s ; the a s s o c i a t i o n appointed three a d d i t i o n a l committees by 1911. In 1912, the f i r s t s c hool d i s t r i c t Bureau of Research was e s t a b l i s h e d i n New York C i t y . Worthen and Sanders (1987, p. 13) s t a t e : The t e s t i n g movement was i n f u l l swing by 1918, with i n d i v i d u a l and group t e s t s b e i n g developed f o r use i n many e d u c a t i o n a l and p s y c h o l o g i c a l d e c i s i o n s . Though the e a r l y school system surveys had r e l i e d mainly on c r i t e r i o n - r e f e r e n c e d t e s t s to gather group i n f o r m a t i o n i n school s u b j e c t areas, the 1920's saw the emergence of norm-referenced t e s t s developed f o r use i n measuring i n d i v i d u a l performance l e v e l s . In the f o r t y - f i v e year p e r i o d between 1920 and 1965, g r e a t e r emphasis and a focus upon the development of t e s t i n g 22 processes o c c u r r e d . As Worthen and Sanders (1987, p. 14) s t a t e : The development of s t a n d a r d i z e d achievement t e s t s f o r use i n l a r g e - s c a l e t e s t i n g programs was a n a t u r a l outgrowth of t h i s t r e n d (State-wide t e s t i n g ) . During t h i s p e r i o d , " e v a l u a t i o n " was most o f t e n used to mean the a s s i g n i n g of grades or summarizing of student performance on t e s t s . Ralph T y l e r proposed, i n 1932, an o b j e c t i v e s - b a s e d approach to e d u c a t i o n a l e v a l u a t i o n and developed measurement t o o l s to support h i s concept. Wolf (1987, p. 3) observes, "In the 1930's, l a r g e l y as a r e s u l t of work by Ralph T y l e r , e v a l u a t i o n was being f o r m a l l y c o n c e p t u a l i z e d and a f l e d g l i n g technology was developed." The o b j e c t i v e - b a s e d approach i n v o l v e d d e f i n i n g o b j e c t i v e s and examining outcomes to determine whether or not o b j e c t i v e achievement was s u c c e s s f u l . A c c o r d i n g to Worthen and Sanders (1987), the 1940's and 1950's were e s s e n t i a l l y a p e r i o d where p r e v i o u s e d u c a t i o n a l e v a l u a t i o n concepts were a p p l i e d . O b j e c t i v e s f o r e d u c a t i o n were debated, agreed upon and implemented. Throughout the mid 1950's and 1960's e d u c a t i o n a l e v a l u a t i o n , which was based upon T y l e r ' s approach, was f u r t h e r developed. Bloom et a l . p u b l i s h e d i n 1956, Taxonomy of E d u c a t i o n a l O b j e c t i v e s : Handbook I: C o g n i t i v e Domain. "Bloom's Taxonomy", i s r e f e r e n c e d by Worthen and Sanders (1987, p. 16) and focussed upon d e f i n i n g : 23 i n e x p l i c i t d e t a i l a h i e r a r c h y of t h i n k i n g s k i l l s a p p l i c a b l e to v a r i o u s content areas. T h i s document continues to be a standard t o o l both i n t e s t i n g and i n c u r r i c u l u m development; d e s i g n and e v a l u a t i o n . A major development i n the U n i t e d States i n 1964 was the enactment of the C i v i l Right's Act which was f o l l o w e d by the passage of the Elementary and Secondary Educ a t i o n Act (ESEA) i n 1965. A c c o r d i n g to House,(1995, p. 15): P r i o r to 1965, e v a l u a t i o n was a minor a c t i v i t y , a s i d e l i n e engaged in.by academics as e x t r a c o n s u l t i n g work. Then came the Great S o c i e t y L e g i s l a t i o n i n the U n i t e d S t a t e s . With the passage of the Elementary and Secondary Educ a t i o n A c t i n 1965, e v e r y t h i n g changed. Senator Robert Kennedy i n s i s t e d t h a t an e v a l u a t i o n amendment be at t a c h e d to the education b i l l , and e v a l u a t i o n became a f e d e r a l mandate that spread to other s o c i a l programs. The e f f e c t of these a c t s was an outpouring of money • i n t o education to support education programmes f o r disadvantaged youths and e d u c a t i o n a l r e s e a r c h . As a c o n t r o l over t h i s expenditure of funds, e v a l u a t i o n programmes became mandatory i n order to h o l d educators and r e s e a r c h e r s accountable f o r how the funds were spent. As Worthen and Sanders (1987, p. 17) s t a t e : T r a n s l a t e d i n t o o p e r a t i o n a l terms, t h i s meant t h a t thousands of educators were f o r the f i r s t time r e q u i r e d to spend t h e i r time e v a l u a t i n g t h e i r own e f f o r t s . P r o j e c t e v a l u a t i o n s mandated by s t a t e and f e d e r a l governments have s i n c e become standard p r a c t i c e , with e v a l u a t i o n emerging as a p o l i t i c a l t o o l to c o n t r o l the expenditure of p u b l i c funds. - 24 The l a t e 19'60's was a t r a n s i t i o n p e r i o d where educators had to develop e x t e n s i v e e v a l u a t i o n programmes i n order to comply with the f e d e r a l mandates of 1964 and 1965. Wolf (1987, p. 4) d e s c r i b e s t h i s p e r i o d as one where; The p o l i t i c a l p o p u l a r i t y of the e v a l u a t i o n requirement q u i c k l y spread to other s o c i a l l e g i s l a t i o n so, by- the end of the 1960's, i t was commonplace to r e q u i r e s y s t e m a t i c annual e v a l u a t i o n s of s o c i a l programs. Worthen and Sanders (1987, p. 17) i d e n t i f y a poignant quote.of Guba (1967, p. 312) t h a t comments upon the e f f e c t i v e n e s s of the e v a l u a t i o n s t h a t arose from the p a s s i n g Of t h i s l e g i s l a t i o n . None of these product e v a l u a t i o n s w i l l g ive the F e d e r a l Government the data i t needs to review the general T i t l e I I I program and to decide how the program might be reshaped to be more e f f e c t i v e . (Guba, 1967, p. 312) The above quote i d e n t i f i e d a grave concern t h a t the then c u r r e n t e v a l u a t i o n programmes were inadequate to evaluate education programmes and p r o v i d e meaningful data with which to s a t i s f y the f e d e r a l mandate. O p e r a t i o n a l l y , there were no"adequate g u i d e l i n e s f o r e v a l u a t o r s , which l e d to e v a l u a t i o n s being c r e a t e d and conducted by i n e x p e r i e n c e d people. Wolf (1987, p. 4) f u r t h e r comments t h a t : The p e r i o d from 1965 to the e a r l y 1970's was one of c o n s i d e r a b l e c o n f u s i o n . A great d e a l of a c t i v i t y o c c u r r e d under the heading of e v a l u a t i o n . Much of the work was h i g h l y q u e s t i o n a b l e . 25 From 1967 to 1973 new s t r a t e g i e s f o r e v a l u a t i o n were formulated, and s e v e r a l e v a l u a t i o n models were developed. Greater emphasis was p l a c e d upon e v a l u a t i o n , and e v a l u a t i o n as a f i e l d of study was, by n e c e s s i t y , c r e a t e d . With v a r i o u s e v a l u a t i o n models s u r f a c i n g , debate arose as to the a p p r o p r i a t e n e s s and general a p p l i c a t i o n of these models. In 1967 the U n i t e d States government c r e a t e d the Centre "for the Study of E v a l u a t i o n . T h i s was followed, i n 1972, by the c r e a t i o n of the N a t i o n a l I n s t i t u t e of E d u c a t i o n (NIE). Worthen and Sanders (1987, p. 19) s t a t e t h a t the: NIE focused one of i t s r e s e a r c h programs on e v a l u a t i o n i n education, s u p p o r t i n g f i e l d r e s e a r c h t h a t added to our knowledge of e v a l u a t i o n methodology, and a l s o funded r e s e a r c h to adapt methods and techniques from other d i s c i p l i n e s f o r use i n e d u c a t i o n a l e v a l u a t i o n . In the years f o l l o w i n g the c r e a t i o n of NIE, there has been a c o n s i d e r a b l e amount of r e s e a r c h and development of e d u c a t i o n a l e v a l u a t i o n . E d u c a t i o n a l e v a l u a t i o n has become a f i e l d of study i n i t s own r i g h t , and i t continues to explore new methodologies and a p p l i c a t i o n s . As Worthen and Sanders (1987, p. 20) observe, e d u c a t i o n a l e v a l u a t i o n : must continue to grow and adapt to changing c o n d i t i o n s and demands. The r e s u l t i n g , decrease i n demand f o r e v a l u a t i o n of l a r g e s c a l e , f e d e r a l l y supported e d u c a t i o n a l programs has l e d some commentators to make gloomy p r e d i c t i o n s about the f u t u r e of e v a l u a t i o n i n education and r e l a t e d areas. 26 The s h i f t from l a r g e s c a l e e v a l u a t i o n s to e v a l u a t i o n s at the l o c a l l e v e l i s one method of adapting e d u c a t i o n a l e v a l u a t i o n to meet the needs' of l o c a l governments and s c h o o l boards. Wolf (1987, p. 5) summarizes the then c u r r e n t s t a t e of e d u c a t i o n a l e v a l u a t i o n i n c o n t r a s t to the l a t e 1960's to mid 1980's below. E v a l u a t i o n i s now seen to be a more open and ongoing process intended to y i e l d i n f o r m a t i o n t h a t w i l l l e a d to the improvement of e d u c a t i o n a l programs. This l a t t e r view i s i n marked c o n t r a s t to the view of 10-20 years ago when i t was f e l t t h a t the r e s u l t s of e v a l u a t i o n s t u d i e s would be the most important determinant of continued support. In my o p i n i o n , e d u c a t i o n a l e v a l u a t i o n w i l l continue to be an i n t e g r a l p a r t of the "business of e d u c a t i o n " as long as the demands f o r r e s p o n s i b l e e d u c a t i o n and j u d i c i a l use of p u b l i c funds remain p o l i t i c a l and p u b l i c concerns. E d u c a t i o n E v a l u a t i o n Models Used i n Thi s Study As s t a t e d i n Chapter One,.the two e v a l u a t i o n models chosen f o r t h i s study are the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model. Each model has advantages as w e l l as l i m i t a t i o n s . Provus Discrepancy E v a l u a t i o n Model Advantages. The Provus Discrepancy E v a l u a t i o n Model has the advantage of u s i n g a s t r a i g h t f o r w a r d concept which i s b a s i c to e v a l u a t i o n . That i s , i t c l e a r l y i d e n t i f i e s what 27 i t i s t h a t w i l l be e v a l u a t e d by c o n c e n t r a t i n g upon d e f i n i n g o b j e c t i v e s . The balance of the e v a l u a t i o n i s then focussed upon a comparison of a c t u a l outcomes with s t a t e d o b j e c t i v e s . T h i s i s a very c l e a n concept t h a t i s easy to f o l l o w and should produce d e f i n i t i v e ' r e s u l t s . T h i s overview o v e r s i m p l i f i e s the a c t u a l model, but does capture the essence of the model's major advantages. L i m i t a t i o n s . The Provus Discrepancy E v a l u a t i o n Model i s l i m i t e d i n t h a t i t has been c r i t i c i z e d f o r being too narrow i n f o c u s . Worthen and Sanders, (1987, p. 7 3 ) , I n d i c a t e t h a t e v a l u a t i o n models such as the Provus Discrepancy E v a l u a t i o n Model with an o b j e c t i v e s o r i e n t e d approach have t h e i r c r i t i c s . They summarize these concerns i n nine p o i n t s as l i s t e d below. The o b j e c t i v e s - o r i e n t e d e v a l u a t i o n approach: 1) l a c k s a r e a l e v a l u a t i v e component ( f a c i l i t a t i n g measurement and assessment.of o b j e c t i v e s r a t h e r than r e s u l t i n g i n e x p l i c i t judgements of m e r i t or worth). 2) l a c k s standards to judge the importance of observed d i s c r e p a n c i e s between o b j e c t i v e s and performance l e v e l s . 3) N eglects the value of the o b j e c t i v e s themselves. 4) Ignores, important a l t e r n a t i v e s t h a t should be c o n s i d e r e d i n p l a n n i n g an e d u c a t i o n a l program. 5) Neglects t r a n s a c t i o n s that occur w i t h i n the program or a c t i v i t y b e i n g e v a l u a t e d . 6) Neglects the context i n which the e v a l u a t i o n takes p l a c e . 28 7) Ignores important outcomes other than those covered by the o b j e c t i v e s (the unintended outcomes of the a c t i v i t y ) . 8) Omits evidence of programme value not r e f l e c t e d i n i t s own o b j e c t i v e s . 9) Promotes a l i n e a r , i n f l e x i b l e approach to e v a l u a t i o n . Given Worthen and Sanders' (1987) comments, i t i s apparent t h a t t h i s type of e v a l u a t i o n model has severe l i m i t a t i o n s which c o u l d a f f e c t the value of the r e s u l t s o btained. In my study, the Provus Discrepancy E v a l u a t i o n Model i s c o n s i d e r e d from the p e r s p e c t i v e of i t s a p p l i c a t i o n i n a business s e t t i n g . What may be seen as l i m i t a t i o n s f o r use i n education may have p r a c t i c a l advantages when used i n an environment where the focus i s g e n e r a l l y upon o b j e c t i v e attainment. Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model Advantages. The Stufflebeam et a l . (1972) C.I.P.P. E v a l u a t i o n Model i s a management-oriented e v a l u a t i o n approach. Worthen and Sanders (1987) i n d i c a t e t hat t h i s approach i s designed to serve the needs of management i n the d e c i s i o n making p r o c e s s . The focus of t h i s approach i s such that e v a l u a t i o n can s t a r t at the o u t s e t of a programme and p r o v i d e ongoing i n f o r m a t i o n which w i l l a i d i n the development of the programme. A c c o r d i n g to Worthen and Sanders (1987), t h i s approach takes advantage of 29 o p p o r t u n i t i e s as they a r i s e and allows management to make informed d e c i s i o n s at the time those d e c i s i o n s need to be made. A p a r t i c u l a r advantage of u s i n g the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model i s t h a t i t p l a c e s the o b j e c t i v e s of an e v a l u a t i o n i n context. T h i s allows f o r a more complex e v a l u a t i o n to be conducted. That i s , r e l e v a n t data may be c o l l e c t e d to support questions which have a g r e a t e r complexity. As Worthen and Sanders (1987, p. 84) s t a t e : The C.I.P.P. Model, i n p a r t i c u l a r , i s a simple h e u r i s t i c t o o l t h a t helps the e v a l u a t o r generate p o t e n t i a l l y important questions to be addressed i n an e v a l u a t i o n . The management-oriented approach to e v a l u a t i o n supports e v a l u a t i o n of every component of an e d u c a t i o n a l program as i t operates, grows, or changes. I t s t r e s s e s the t i m e l y use of feedback by decision-makers so t h a t e d u c a t i o n i s not l e f t to f l o u n d e r or proceed u n a f f e c t e d by updated knowledge about needs, resources, new developments i n education, the r e a l i t i e s of day to day o p e r a t i o n s , or the consequences of p r o v i d i n g education i n any given way. An example of t h i s process o c c u r r e d i n the L i t i g a t i o n . Management E v a l u a t i o n Task Force Committee proceedings. The committee a s c e r t a i n e d t h a t there was no compliance with B u l l e t i n CDB894 and then used t h a t i n f o r m a t i o n to r e f o c u s i t s mandate. By r e f o c u s s i n g i t s mandate, the L i t i g a t i o n Management E v a l u a t i o n Task Force assumed ownership of the development of a new l i t i g a t i o n management s t r a t e g y . 30 L i m i t a t i o n s . T h i s type of management-oriented e v a l u a t i o n model i s l i m i t e d i n t h a t i t serves the needs of the d e c i s i o n makers and may r e s t r i c t or impede the e v a l u a t o r ' s e x p l o r a t i o n of other i s s u e s t h a t a r i s e through the course of the e v a l u a t i o n . While some of these p o t e n t i a l i s s u e s may be important, they w i l l be overlooked i n favour of complying with the o b j e c t i v e s and d i r e c t i o n s of the d e c i s i o n makers. This leads to a second weakness i n t h a t t h i s type of e v a l u a t i o n may be s u b j e c t to p o l i t i c a l or p e r s o n a l agendas which c o u l d shape the outcome of an e v a l u a t i o n . There i s concern i n t h i s study t h a t t h i s v a r i a b l e i s a c t i v e due to the presence of p o l i t i c a l and p e r s o n a l i n f l u e n c e s w i t h i n and surrounding both the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and L i t i g a t i o n Management E v a l u a t i o n Task Force Committee. This v a r i a b l e was not s p e c i f i c a l l y t e s t e d and would t h e r e f o r e act as a l i m i t a t i o n on the g e n e r a l i z a b i l i t y of t h i s study. T h i s i s s u e w i l l be d e a l t w i t h f u r t h e r i n Chapter Three. Another l i m i t a t i o n i s the c o s t f a c t o r r e l a t e d to conducting an e v a l u a t i o n of t h i s type i n i t s e n t i r e t y . As Worthen and Sanders (1987, p. 85) s t a t e , " i f f o l l o w e d i n i t s e n t i r e t y , the management-oriented approach can r e s u l t i n c o s t l y and complex e v a l u a t i o n s " . 31 Other Educa t i o n E v a l u a t i o n Models Considered The f o l l o w i n g e d u c a t i o n a l e v a l u a t i o n models were co n s i d e r e d f o r use i n t h i s study. 1. T y l e r i a n E v a l u a t i o n Approach 2. Hammond's E v a l u a t i o n Approach 3. U.C.L.A. E v a l u a t i o n Model 4. Stake's Countenance Model Each model w i l l be b r i e f l y d i s c u s s e d i n order to p r o v i d e a background of a l t e r n a t i v e e v a l u a t i o n models. T y l e r i a n E v a l u a t i o n Approach. Ralph T y l e r , d u r i n g the l a t e 1930's and e a r l y 1940's, developed an e d u c a t i o n a l e v a l u a t i o n model. T h i s e d u c a t i o n e v a l u a t i o n model was intended to look at programme o b j e c t i v e s and determine whether or not they had been a t t a i n e d . T h i s model i s very l i n e a r , which makes i t simple to understand and apply. I t c o n s i s t s of seven b a s i c s t e p s . Worthen and Sanders (1987, p. 63) i d e n t i f y these seven steps below: 11 E s t a b l i s h broad goals or o b j e c t i v e s . 2. C l a s s i f y the goals or o b j e c t i v e s . 3. Define o b j e c t i v e s i n b e h a v i o u r a l terms. 4. F i n d s i t u a t i o n s i n which achievement of o b j e c t i v e s can be shown. 5. Develop or s e l e c t measurement techniques. 6. C o l l e c t performance data. 32 7. Compare performance data with b e h a v i o u r a l l y s t a t e d o b j e c t i v e s . In comparing o b j e c t i v e s w i t h performance outcomes/ i t i s p o s s i b l e to determine the success or f a i l u r e of the programme. E a r l y i d e n t i f i c a t i o n of v a r i a n c e s between o b j e c t i v e s and performance outcomes allows f o r ongoing • m o d i f i c a t i o n s to be made i n order to enhance the p o t e n t i a l f o r programme success. Hammond's E v a l u a t i o n Approach. R. Hammond (1973) developed an e v a l u a t i o n approach which f o l l o w e d the T y l e r i a n approach very c l o s e l y with one e x c e p t i o n / i n n o v a t i o n . Hammond's e v a l u a t i o n model c o n s i s t s o f s i x s t e p s . Worthen and Sanders (1987, p. 68) i d e n t i f y these steps as l i s t e d below. 1. D e f i n i n g the program. 2. D e f i n i n g the d e s c r i p t i v e v a r i a b l e s (using h i s cube). 3. S t a t i n g o b j e c t i v e s . 4. A s s e s s i n g performance. 5. A n a l y z i n g r e s u l t s . 6. Comparing r e s u l t s with o b j e c t i v e s . The major d i f f e r e n c e between Hammond's E v a l u a t i o n Model and the T y l e r i a n Approach i s th a t Hammond has added a t h i r d dimension to the e v a l u a t i o n approach. Hammond's cube (as . 3 3 noted i n step 2 above) i s a t o o l which may be used by the e v a l u a t o r to generate a number of questions t h a t may be exp l o r e d i n the e v a l u a t i o n . In essence, i t c o n s i s t s of three dimensions ( i n s t r u c t i o n , i n s t i t u t i o n and b e h a v i o u r a l o b j e c t i v e s ) which are broken down i n t o s m a l l e r d i v i s i o n s . Where these s m a l l e r d i v i s i o n s i n t e r s e c t w i t h i n the three dimensional cube r e l a t i o n a l questions are generated or suggested. I f a l l f a c t o r s apply to a given e v a l u a t i o n , a maximum of n i n e t y c e l l s are a v a i l a b l e to generate questions about r e l a t i o n s h i p s . The number of f a c t o r s a v a i l a b l e are reduced i n each dimension i n accordance with t h e i r a p p l i c a b i l i t y to the e v a l u a t i o n undertaken. Worthen and Sanders (1987, p. 68) s t a t e t h a t the value o f Hammond's e v a l u a t i o n approach i s as "a v a l u a b l e h e u r i s t i c t o o l the e v a l u a t o r can use i n a n a l y z i n g the successes and f a i l u r e s of an e d u c a t i o n a l a c t i v i t y i n a c h i e v i n g i t s o b j e c t i v e s " . U.C.L.A. E v a l u a t i o n Model. M.C. A l k i n (1969) developed an e v a l u a t i o n model t h a t c l o s e l y p a r a l l e l s the Stufflebeam et a l . (1973) C.I.P.P. E v a l u a t i o n Model. Worthen and Sanders (1987, p. 81) make the f o l l o w i n g o b s e r v a t i o n . Both the C.i.P.P. and U.C.L.A. frameworks f o r e v a l u a t i o n appear t o be s e q u e n t i a l , but the developers have s t r e s s e d t h a t such i s not the case. For example, the e v a l u a t o r would not have to complete an input e v a l u a t i o n or a systems assessment i n order to undertake one of the other types of e v a l u a t i o n l i s t e d i n the framework. 34 While the a p p l i c a t i o n o f the U.C.L.A. E v a l u a t i o n Model may not n e c e s s a r i l y be l i n e a r , the a c t u a l s t r u c t u r e i s e s s e n t i a l l y the same as the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model. Table 1 prese n t s a comparison of the stages of both the U.C.L.A. and Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Models. Table 1 • U.C.L.A . and C. I-.P.P. E v a l u a t i o n Models - Comparative Processes Stage U.C.L.A. Stage C.I.P.P. Process Process 1 Systems assessment 1 Context e v a l u a t i o n 2 Program p l a n n i n g 2 Input e v a l u a t i o n 3 Program implementation 4 Program improvement 3 Process e v a l u a t i o n 5 Program c e r t i f i c a t i o n 4 Product e v a l u a t i o n The b a s i c d i f f e r e n c e between the models i s t h a t the U.C.L.A. E v a l u a t i o n Model does not r e q u i r e one stage to be completed b e f o r e p a s s i n g on to the next. I t allows the e v a l u a t o r to " c y c l e " from one stage to another, c o n t i n u a l l y r e v i s i n g and reviewing as the need a r i s e s . Stake's Countenance E v a l u a t i o n Model. R.E. Stake (1967) developed h i s concept t h a t e v a l u a t i o n s should c o n s i s t 35 of two b a s i c components, " D e s c r i p t i o n " and "Judgement". These two components were proposed as the two countenances of e v a l u a t i o n . The purpose of t h i s model was to p r o v i d e the e v a l u a t o r with a t o o l f o r o r g a n i z i n g and a n a l y z i n g data. Worthen and Sanders (1987, p. 130) d e s c r i b e the workings of the Stake's Countenance Model below. The e v a l u a t o r would analyze i n f o r m a t i o n i n the d e s c r i p t i o n matrix by l o o k i n g at the congruence between i n t e n t s and o b s e r v a t i o n s , and by l o o k i n g at dependencies (contingencies) of outcomes on t r a n s a c t i o n s and antecedents, and of t r a n s a c t i o n s on antecedents. Judgements would be made by a p p l y i n g standards to the d e s c r i p t i v e data. Stake's Countenance Model p r o v i d e s the e v a l u a t o r w i t h a conceptual framework as opposed to an e v a l u a t i o n formula. R a t i o n a l e f o r S e l e c t i n g the Provus and Stufflebeam et a l . E v a l u a t i o n Models The Provus and Stufflebeam et a l . e v a l u a t i o n models were chosen f o r t h i s study f o r the f o l l o w i n g reason. I wanted to use e v a l u a t i o n models t h a t were not complex i n s t r u c t u r e i n order to keep the study simple. The more dimensions t h a t are added to t h i s study the more complex i t c o u l d become and t h i s might c l o u d some b a s i c i s s u e s . The T y l e r model was too l i n e a r and the balance of the models co n s i d e r e d added too many dimensions f o r t h i s study to c o n s i d e r . The Provus and Stufflebeam et a l . e v a l u a t i o n models seemed to f i t i n w e l l with the c o r p o r a t e environment 36 and both of these models seemed to be capable of p r o v i d i n g i n f o r m a t i o n t h a t c o u l d be of value i n a co r p o r a t e s e t t i n g . E d u c a t i o n E v a l u a t i o n Research Sources S e v e r a l sources were c o n s u l t e d to determine i f s i m i l a r s t u d i e s to mine e x i s t and to i d e n t i f y c r i t i q u e s of the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model. A s i d e from d i r e c t access to the t e x t u a l m a t e r i a l , as l i s t e d i n the b i b l i o g r a p h y , an i n t e r n e t search was conducted. T h i s search used Yahoo as the search engine and was conducted through the U.B.C. L i b r a r y Web Page. The search parameters were r e s t r i c t e d to the f o l l o w i n g key words and phrases. 1. Provus 2. Stufflebeam 3. Discrepancy E v a l u a t i o n 4. C.I.P.P. 5. E v a l u a t i o n 6. E d u c a t i o n 7. Educa t i o n E v a l u a t i o n . In t o t a l , one hundred and twenty-four records were searched and h i t s o c c u r r e d i n one hundred and t h i r t e e n of these r e c o r d s . On examination of the one hundred and t h i r t e e n h i t s , there were no i n s t a n c e s of s t u d i e s which were of a s i m i l a r nature to my t h e s i s . There were s i x h i t s under 37 the Provus search and three h i t s under Stufflebeam. None of these nine h i t s appeared from t h e i r synopses to c r i t i q u e the models or t h e i r a p p l i c a t i o n s , but were i n s t a n c e s where the models were r e f e r e n c e d i n r e l a t i o n to other s t u d i e s . Only one h i t was achieved under Discrepancy E v a l u a t i o n . This h i t was Discrepancy E v a l u a t i o n For E d u c a t i o n a l Program Improvement and Assessment,(1928)[sic] by Malcolm Provus. i R e s u l t s of Documentation Search While the search I conducted was by no means exhaustive, i t i s reasonable to conclude t h a t t h i s study appears to be unique i n i t s f o c u s . Summary Chapter Two has p r e s e n t e d a review of the h i s t o r y o f ' e d u c a t i o n a l e v a l u a t i o n .and has d i s c u s s e d the models used i n t h i s study. Some p o t e n t i a l a l t e r n a t i v e models were a l s o d i s c u s s e d and p o t e n t i a l sources of f u r t h e r i n f o r m a t i o n through an i n t e r n e t search were presented. 38 CHAPTER THREE METHOD Chapter Three w i l l d i s c u s s the methodology used i n t h i s study. The methodology w i l l be presented from two p e r s p e c t i v e s . The f i r s t p e r s p e c t i v e w i l l examine the methodologies used by both the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee. The second p e r s p e c t i v e w i l l present the method of data c o l l e c t i o n t h a t I used i n order to allow f o r p o t e n t i a l r e p l i c a t i o n o f t h i s study. Chapter Three w i l l a l s o d i s c u s s the makeup of the committees i n v o l v e d i n t h i s study and my r o l e as . a p a r t i c i p a n t observer. The methodologies used by both committees w i l l be presented and r e l a t e d to the two e d u c a t i o n a l models chosen f o r t h i s - s t u d y , the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model. Task Force Members Two e v a l u a t i o n task f o r c e s w i t h i n a l a r g e c o r p o r a t i o n , the Claims T r a i n i n g and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committees, were examined.. E l e v e n members ( s i x managers and f i v e f i e l d s t a f f ) and a s e c r e t a r y , who would act as r e c o r d e r only, comprised the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. The nine member L i t i g a t i o n Management E v a l u a t i o n Task Force Committee was comprised of 39 f i v e c l aims managers, one claims examiner, three lawyers, and a c l e r k who would act as r e c o r d e r o n l y . Role of the P a r t i c i p a n t Observer .In both committees I f u n c t i o n e d as a p a r t i c i p a n t observer and d i d not take an a c t i v e r o l e i n the a c t u a l d e s i g n of the e v a l u a t i o n , but p a r t i c i p a t e d i n t h e i r a c t i v i t i e s . Both committees were ad v i s e d i n t h e i r i n i t i a l meetings t h a t I would be a c t i n g as a p a r t i c i p a n t observer f o r the development of my t h e s i s and th a t I would a l s o be a c o n t r i b u t i n g member of each committee. As a p a r t i c i p a n t i n both committees I was expected to take an a c t i v e r o l e i n the workings of the committees.. In the Claims T r a i n i n g E v a l u a t i o n Task Force Committee I' co- authored the f i n a l r e p o r t and i n the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee I authored the f i n a l r e p o r t . I acted as chairman of the p r e s e n t a t i o n committee f o r the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. As chairman I l e d the p r e s e n t a t i o n team i n d e l i v e r i n g . t h e r e s u l t s to the Manager of Human Resources Development. In this, r o l e I was expected to defend the work of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and i n p a r t i c u l a r the methodology, r e s u l t s and c o n c l u s i o n s . I was a member of the p r e s e n t a t i o n team L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, but I d i d not assume a l e a d r o l e . 40 The g u i d e l i n e s f o r the o p e r a t i o n of each committee v a r i e d s l i g h t l y , but i t was understood by a l l committee members th a t t i t l e or rank w i t h i n the c o r p o r a t i o n d i d not p l a y a r o l e i n the committee. For example, at the i n i t i a l meeting of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee on October 18, 1993 the Chairman of the committee set out the "Rules of Conduct" as f o l l o w s : 1) Rank has no p r i v i l e g e . . We are a l l equal p a r t n e r s . 2) No i d e a i s a bad i d e a . 3) A l l suggestions w i l l be approached with an open mind. Only c o n s t r u c t i v e c r i t i c i s m w i l l be allowed. 4) We w i l l remain n e u t r a l i n the face of v e s t e d i n t e r e s t s . 5) C r e a t i v i t y w i l l be r e s p e c t e d . 6) D e c i s i o n making by consensus. 7) A l l are e q u a l l y r e s p o n s i b l e f o r the success of the p r o j e c t . ' P a r t i c i p a n t s i n the Claims T r a i n i n g E v a l u a t i o n Task Force Committee I n v e s t i g a t i o n The Claims t r a i n i n g E v a l u a t i o n Task Force Committee operated under the sponsorship of a s e n i o r manager who commissioned the p r o j e c t . The Task Force team members c o n s i s t e d of an o f f i c e manager, a claims manager (the p a r t i c i p a n t o b s e r v e r ) , and a r e g i o n a l manager as f i e l d 41 management r e p r e s e n t a t i v e s . A l s o r e p r e s e n t i n g management •was a head o f f i c e support manager, an o r g a n i z a t i o n development manager and a systems manager who would act as f a c i l i t a t o r . The f i v e s t a f f members who made up the balance of the committee r e p r e s e n t e d a c r o s s s e c t i o n of f i e l d s t a f f members who were d i r e c t l y impacted by Claims T r a i n i n g . A s e c r e t a r y was a l s o i n v o l v e d as r e c o r d keeper. The focus groups and i n d i v i d u a l s i n t e r v i e w e d were s e l e c t e d by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee to ensure c r o s s d i v i s i o n a l , r e g i o n a l and geographic r e p r e s e n t a t i o n . They were chosen to r e p r e s e n t each of the i n t e r e s t e d groups who were a f f e c t e d by the t r a i n i n g which was•conducted by the Claims T r a i n i n g Department. C r i t e r i a For Focus Group S e l e c t i o n The Claims T r a i n i n g E v a l u a t i o n Task Force Committee decided t h a t there would be one focus group chosen as a r e p r e s e n t a t i v e of each work group a f f e c t e d by the Claims T r a i n i n g Department. Each focus group c o n s i s t e d of 15 members and each s e l e c t e d member was sent an E-mail which o u t l i n e d , i n general terms, what the purpose of the committee was, but d i d not p r o v i d e s p e c i f i c questions or g u i d e l i n e s with r e s p e c t to what would be asked i n each focus group s e s s i o n . The focus groups were balanced f o r gender, r e g i o n a l r e p r e s e n t a t i o n and l e n g t h of s e r v i c e . 42 Each focus group was asked the same questions f o r c o n s i s t e n c y between groups. The members of the focus groups were seated i n a conference room and asked s p e c i f i c a l l y prepared questions i n a roundtable format. T h e i r answers were captured by i n d i v i d u a l committee members i n p r i v a t e notes and by the committee s e c r e t a r y . The s e c r e t a r y ' s notes were documented i n MS Word, p r i n t e d and p r o j e c t e d on an overhead p r o j e c t o r f o r the p a r t i c i p a n t s to view d u r i n g each focus group s e s s i o n . Each focus group s e s s i o n was h e l d to a two hour i n t e r v i e w p e r i o d and each group was i n t e r v i e w e d o n l y once. Task Force Procedures The p r o j e c t team chairman announced i n the f i r s t meeting of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee on A p r i l 8, 1994 t h a t the best method of o b t a i n i n g an accurate assessment of the claims t r a i n i n g s i t u a t i o n was to review the h i s t o r i c a l documentation w h i c h ' e x i s t e d r e g a r d i n g p r e v i o u s assessments of the Claims T r a i n i n g Department and the recommendations t h a t were made. No other o p t i o n s were d i s c u s s e d and the Claims T r a i n i n g E v a l u a t i o n Task Force Committee Chairman set the d i r e c t i o n f o r the committee. F o l l o w i n g the h i s t o r i c a l review, the p r o j e c t team r e c e i v e d a s e r i e s o f p r e s e n t a t i o n s by managers who e i t h e r had a v e s t e d i n t e r e s t i n the f u t u r e of the Claims T r a i n i n g Department, or who were a f f e c t e d d i r e c t l y by the 43 product of the Claims T r a i n i n g Department. In a d d i t i o n , the p r o j e c t team i n t e r v i e w e d the r e c i p i e n t s of t r a i n i n g by the Claims T r a i n i n g Department by way of focus group s e s s i o n s . The p r e s e n t a t i o n s and focus group s e s s i o n s were conducted w i t h i n the context of the three o b j e c t i v e s p r e s c r i b e d i n s e n i o r management's mandate f o r the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. The three o b j e c t i v e s s e t . out by s e n i o r management are l i s t e d below. 1. To determine the best o r g a n i z a t i o n a l s t r u c t u r e f o r p r o v i d i n g the t r a i n i n g . 2. To determine what claims t r a i n i n g i s needed. 3. To determine who c o u l d best d e l i v e r what claims t r a i n i n g . Methodology: Claims T r a i n i n g E v a l u a t i o n Task Force Committee - H i s t o r i c a l Background Six documents were reviewed by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee p r i o r to the commencement of the p r e s e n t a t i o n s and i n t e r v i e w s . These documents were s e l e c t e d by the chairman of the Task Force f o r review by i t s members - i n order to p r o v i d e an h i s t o r i c a l perspective.. The documents reviewed by t h i s Task Force were: 1. Claims Manpower Planning: T r a i n i n g Sub-committee, January 1991. 2. S t r a t e g y f o r Claims T r a i n i n g Redevelopment: Gateway Systems S e r v i c e s , June 1991. 3. Claims Task Force on T r a i n i n g & Development: Report No. 1, August 1991. 44 4. Claims Task Force on T r a i n i n g & Development: F i n a l Report, November 1991. 5. J o i n t Task Force Report on Claims Issues, February 1992. 6. Corporate T r a i n i n g Consortium I n i t i a t i v e , September 1993. Methodology: Claims T r a i n i n g E v a l u a t i o n Task Force Committee - Current S i t u a t i o n In t o t a l , the p r o j e c t team heard p r e s e n t a t i o n s from ten managers who were d i r e c t l y i n v o l v e d with claims t r a i n i n g at a departmental l e v e l and i n t e r v i e w e d approximately e i g h t y s t a f f and managers through the focus group p r o c e s s . The ten managers who were i n v o l v e d at a departmental l e v e l were chosen f o r t h e i r s p e c i f i c d i r e c t involvement i n Claims T r a i n i n g . T h e i r p o s i t i o n s p l a c e d them i n a r o l e of e i t h e r b e i n g r e s p o n s i b l e f o r the d e l i v e r y of e f f e c t i v e t r a i n i n g (not the a c t u a l d e l i v e r y ) or f o r the support of Claims T r a i n i n g through the p r o v i s i o n of resources or p l a n n i n g s t r a t e g i e s . For example, corpor a t e f i n a n c i a l r e s u l t s may be dependent upon the e f f e c t i v e n e s s and q u a l i t y of the t r a i n i n g r e c e i v e d by s t a f f . The importance of e f f e c t i v e t r a i n i n g on claims s e v e r i t i e s ( d o l l a r s spent per claim) was summed up by one s e n i o r manager. Given h i s p o s i t i o n as a s e n i o r manager, his' comment c a r r i e d a great d e a l of weight with the committee 45 and c o u l d have i n f l u e n c e d the committee's p e r s p e c t i v e . The s e n i o r manager, (May 5, 1994) s t a t e d : S e v e r i t y c o n t r o l ( c o n t r o l l i n g the number of d o l l a r s spent per claim) • important, but to dwell on i t i s wrong focus . I f you do i t p r o p e r l y , you w i l l get s e v e r i t y under c o n t r o l . Lot of pre s s u r e on us. F i r s t three months of t h i s year were a d i s a s t e r . A p r i l was good. From t h i s p e r s p e c t i v e , i t was not the t r a i n i n g i t s e l f t h a t was important but r a t h e r the outcome of t r a i n i n g . T h i s i s t y p i c a l of the p e r s p e c t i v e s of the ten managers i n v o l v e d i n c l aims t r a i n i n g at a departmental l e v e l . These managers were r e s u l t s o r i e n t e d r a t h e r than process o r i e n t e d . These managers were not focussed upon the "how's" and the "why's" of t r a i n i n g , but r a t h e r on the r e s u l t s . T h e i r concerns and i n t e r e s t s come from a corpora t e or macro p e r s p e c t i v e , where t h e i r focus i s upon r e s u l t s . They were not concerned with managing at a micro l e v e l which i n v o l v e s the a c t u a l workings of claims t r a i n i n g . From t h e i r viewpoint, they were able to see the e f f e c t of claims t r a i n i n g upon the c o r p o r a t i o n ' s f i n a n c i a l p i c t u r e and t h e i r concerns would have a d i f f e r e n t emphasis than the concerns of those people i n v o l v e d with the d e l i v e r y of claims t r a i n i n g . In c o n t r a s t to the p r e s e n t a t i o n s made by the ten managers who were i n v o l v e d i n claims t r a i n i n g at a departmental l e v e l , a sample c r o s s s e c t i o n of the r e c i p i e n t s of t r a i n i n g by the Claims T r a i n i n g Department was 46 i n t e r v i e w e d through the focus group p r o c e s s . As r e c i p i e n t s of t r a i n i n g i t was expected t h a t they would be more process o r i e n t e d than r e s u l t s o r i e n t e d as they were d i r e c t l y impacted by the d e l i v e r y of t r a i n i n g . The data o b t a i n e d were reviewed upon c o l l e c t i o n a f t e r each focus group/interview s e s s i o n by the e n t i r e Claims T r a i n i n g E v a l u a t i o n Task Force Committee i n order to ensure t h a t by examining the data f o r trends, the weaknesses or s t r e n g t h s of the c u r r e n t s t a t e of claims t r a i n i n g c o u l d be immediately i d e n t i f i e d . Upon c o n c l u s i o n of the data c o l l e c t i o n process f o r each group, the data were analyzed and ge n e r a l i s s u e s and s p e c i f i c issues'were i d e n t i f i e d by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee which l e d to a s e r i e s o f recommendations and s u p p o r t i n g r a t i o n a l e b e i n g developed. The data were c o l l e c t e d between May 6 and May 20, 1994. This e v a l u a t i o n process was very focussed and the Claims T r a i n i n g E v a l u a t i o n Task Force Committee was v e r y cognizant of i t s t i m e - l i m i t e d mandate. The p r o j e c t team was r e q u i r e d to make a p r e s e n t a t i o n of i t s f i n d i n g s to s e n i o r management by June, 1994. The p r e s e n t a t i o n s to the Claims T r a i n i n g E v a l u a t i o n Task Force Committee by the ten managers who were d i r e c t l y i n v o l v e d with Claims T r a i n i n g at a departmental l e v e l were or g a n i z e d i n such a f a s h i o n t h a t the schedule c o u l d be completed w i t h i n the fewest number of days p o s s i b l e . The 47 p r e s e n t a t i o n s were completed w i t h i n two s e s s i o n s . The process of c a p t u r i n g i n f o r m a t i o n i n v o l v e d p e r s o n a l note- t a k i n g , as w e l l as n o t e - t a k i n g by the s e c r e t a r y on a l a p t o p computer. The notes from the la p t o p computer were p r i n t e d and d i s t r i b u t e d to the Claims T r a i n i n g E v a l u a t i o n Task Force Committee members. These notes were then compared with the pe r s o n a l notes made by the committee members at the time of the i n t e r v i e w s . T h i s was an i n v o l v e d process but the l e v e l o f d e t a i l was c o n s i d e r e d important and necessary by the committee to ensure accuracy. The committee as a whole amended or c o r r e c t e d the notes on the la p t o p computer to cr e a t e the " o f f i c i a l " notes immediately upon c o n c l u s i o n of each p r e s e n t a t i o n . I t was f e l t t h a t t h i s method would a l l o w the p r o j e c t team to c o l l e c t accurate i n f o r m a t i o n by revi e w i n g the p r e s e n t a t i o n i n as short a time-frame as p o s s i b l e a f t e r the p r e s e n t a t i o n was made. This method of n o t e - t a k i n g was used on a t r i a l b a s i s d u r i n g the i n i t i a l p r e s e n t a t i o n s and upon completion of the p r e s e n t a t i o n s r e c e i v e d endorsement from the p r o j e c t team f o r use i n the focus group p r e s e n t a t i o n s as w e l l as a l l f u r t h e r meetings. As a p a r t i c i p a n t observer, I maintained p e r s o n a l notes and c o n t r i b u t e d those notes to the group s e s s i o n , which u l t i m a t e l y formed p a r t of the formal p r o j e c t team notes. A l l members of the p r o j e c t team were aware t h a t I was p a r t i c i p a t i n g i n t h i s process as a p a r t i c i p a n t observer, but that had no apparent i n f l u e n c e upon the f u n c t i o n i n g of the committee. I made t h i s assumption based on the f a c t t h a t except f o r the i n i t i a l meeting, the s u b j e c t of my r o l e i n y the study was never d i s c u s s e d again by any of the committee members. The flow of each meeting d i d not appear to be hampered by my presence which l e d me to b e l i e v e ' t h a t the p e r c e p t i o n of my presence on the committee was focussed upon what I c o u l d c o n t r i b u t e as opposed to what I was o b s e r v i n g . My r o l e as p a r t i c i p a n t observer posed no t h r e a t to the members of the committee as the r e s u l t s of my study would have no d i r e c t i n f l u e n c e upon the outcome of the e v a l u a t i o n nor, on a more p e r s o n a l l e v e l , the f u t u r e of t h e i r c a r e e r s . I attended a l l focus group s e s s i o n s with the e x c e p t i o n of two focus group s e s s i o n s which were h e l d concurrent w i t h s e s s i o n s t h a t I was a t t e n d i n g . L i t i g a t i o n Management E v a l u a t i o n Task Force Committee The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee f u n c t i o n e d i n a d i f f e r e n t manner from t h a t of the Claims- T r a i n i n g Task Force. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee was concerned w i t h an e v a l u a t i o n of the c o r p o r a t i o n ' s h a n d l i n g of the l i t i g a t i o n budgeting process as o u t l i n e d i n an i n t e r n a l d i r e c t i v e , B u l l e t i n CDB894. The p r o j e c t team f o r t h i s task f o r c e was s p e c i f i c a l l y chosen. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, i n a s i m i l a r ' f a s h i o n to the Claims 49 T r a i n i n g E v a l u a t i o n Task Force Committee, operated under the sponsorship of a s e n i o r manager. U n l i k e the Claims T r a i n i n g E v a l u a t i o n Task Force Committee, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee was under no s t r i c t time c o n s t r a i n t . The emphasis was to be on a q u a l i t y e v a l u a t i o n with the outcome being p r a c t i c a l s o l u t i o n s to the p e r c e i v e d problem. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee operated under loose r e s t r i c t i o n s . The Committee began on October 18, 1993 and culminated i n June 1996 with the r o l l o u t of a c o r p o r a t e l i t i g a t i o n management s t r a t e g y programme. T h i s document had the " b l e s s i n g " of the o f f i c e of the A t t o r n e y General. The r e s u l t was the development of new g u i d e l i n e s f o r the h a n d l i n g of procedures, r e l a t i o n s h i p s and the c r e a t i o n of a new r e p o r t i n g format. The o r i g i n a l mandate to evaluate B u l l e t i n CDB894 consequently r e s u l t e d i n i t s replacement a l t o g e t h e r with an e n t i r e l y new d i r e c t i o n i n corporate focus. The chairman of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee ad v i s e d me i n a p r i v a t e c o n v e r s a t i o n near the b e g i n n i n g of the p r o j e c t t h a t he was s e l e c t e d by s e n i o r management f o r h i s e x p e r t i s e i n l i t i g a t i o n claims h a n d l i n g . . He a l s o a d v i s e d t h a t the members of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee were s p e c i f i c a l l y s e l e c t e d by the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee chairman on the b a s i s of 50 having a known e x p e r t i s e i n the f i e l d o f l i t i g a t i o n c l aims h a n d l i n g . The p r o j e c t team was made up of one o f f i c e manager as the chairman, a head o f f i c e claims manager ( s p e c i a l i z i n g i n l a r g e d o l l a r , h i g h p r o f i l e c l a i m s ) , f i v e claims managers (one of which was the p a r t i c i p a n t o b s e r v e r ) , one claims examiner, three lawyers and a c l e r k . These members were s e l e c t e d on the b a s i s of having been r e c o g n i z e d as people with s t r o n g l i t i g a t i o n backgrounds. A s s i s t i n g the committee was a s e c r e t a r y and the Claims t r a i n i n g r e p r e s e n t a t i v e would act as f a c i l i t a t o r . (I would r e p r i s e my r o l e as p a r t i c i p a n t observer under the same c o n d i t i o n s as i n the Claims T r a i n i n g E v a l u a t i o n Task Force Committee.) The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee by v i r t u e of i t s composition was a very powerful committee. I t combined e x p e r t i s e and experience i n such a manner t h a t the composition of the committee c o u l d have had an e f f e c t on the nature of the r e s u l t s and the development of the e v a l u a t i o n p r o c e s s . In other words, the p o s i t i o n a l power and s p e c i a l i z e d knowledge of the committee members may have been an i n f l u e n c i n g f a c t o r upon the e v a l u a t i o n . However, with r e s p e c t to the o r i g i n a l mandate of the committee, which was to determine whether or not there was compliance with B u l l e t i n CDB894, the e v a l u a t i o n r e q u i r e d a simple "yes or no" response from the s u b j e c t group. The composition of the committee, i n t h i s i n s t a n c e , would not l i k e l y have a f f e c t e d • 51 t h i s outcome but may have p l a y e d a r o l e i n the f u t u r e workings of the committee. The L i t i g a t i o n Management Committee f i r s t met on! ! • 'iv.' •••• October 18,. 1993 to d i s c u s s the scope of the p r o j e c t and develop a methodology which would be s u i t a b l e f o r f u l f i l l i n g the mandate. This r e s u l t e d i n a r e d e f i n i n g of the mandate and c l a r i f i e d o b j e c t i v e s . From the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee minutes of November 1, 1993, item number f i v e s t a t e s : I t was agreed t h a t defence counsel must have a c l e a r understanding of what the C o r p o r a t i o n ' s e x p e c t a t i o n s are f o r h a n d l i n g l i t i g a t e d f i l e s . Such e x p e c t a t i o n s should a l s o encompass the a d j u s t e r s ' d u t i e s . I t was. f e l t t h a t the e x p e c t a t i o n s c o u l d be presented i n the form of a c o n t r a c t . a n d w i l l be addressed as a separate i s s u e , but i n c o n j u n c t i o n with the reworking of CDB894. The above quote was the f i r s t i n d i c a t i o n t h a t the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee had changed i t s focus from simply determining whether or not there was compliance with B u l l e t i n CDB894 to d e v e l o p i n g a new l i t i g a t i o n management s t r a t e g y . Having r e d e f i n e d i t s purpose, the mandate and o b j e c t i v e s , the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee's next task was to develop an e v a l u a t i o n methodology which would a s s i s t the committee i n a c h i e v i n g i t s new mandate. 52 The L i t i g a t i o n Management Committee decided t h a t the best method of o b t a i n i n g an accurate assessment of the s t a t e of the c u r r e n t l i t i g a t i o n management p r a c t i c e s was to review B u l l e t i n CDB894 and i n t e r v i e w s t a f f members and defence co u n s e l . The committee a l s o conducted a review of the l i t e r a t u r e of the r e f e r e n c e m a t e r i a l a v a i l a b l e on the i s s u e of l i t i g a t i o n management. Defence counsel and s t a f f members were asked to respond to the f o l l o w i n g questions l i s t e d below. 1. Do you comply with B u l l e t i n 894? 2. What about B u l l e t i n 894 works or doesn't work? 3. Are the requirements of B u l l e t i n 894 necessary? I f not, what are the a l t e r n a t i v e s ? 4. What can be done to improve B u l l e t i n 894? Seven members of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee were s e l e c t e d to i n t e r v i e w seven s e n i o r lawyers who acted as defence counsel f o r the c o r p o r a t i o n . The seven s e n i o r lawyers were i n d i v i d u a l l y i n t e r v i e w e d i n one-on-one h a l f hour s e s s i o n s . Each i n t e r v i e w was conducted by one committee member i n t e r v i e w i n g one lawyer. Each of the seven i n t e r v i e w s was conducted by a d i f f e r e n t committee member. The lawyers were asked the four questions l i s t e d above. I t was f e l t t h a t by i n t e r v i e w i n g s e n i o r counsel, the committee would be able to take advantage of both t h e i r 53 experience and e x p e r t i s e with the l i t i g a t i o n management proce s s . In t o t a l , f o r t y - n i n e documents/reports were reviewed and the committee conducted i n d i v i d u a l i n t e r v i e w s with seven defence counsel and group i n t e r v i e w s with approximately f i f t y s t a f f members. The data were ob t a i n e d between October 18, 1993 and November 4, 1993 and d i s c u s s e d w i t h i n two weeks of c o l l e c t i o n . The r e s u l t s of t h i s data c o l l e c t i o n process were used immediately as p a r t of the formative e v a l u a t i o n of the committee's mandate. In other words, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee used the data upon c o l l e c t i o n to e s t a b l i s h whether or not there was compliance w i t h B u l l e t i n CDB894. T h i s allowed the committee to develop a " p i c t u r e " of what the f i n a l r e s u l t s would look l i k e and e s t a b l i s h an understanding so t h a t when the l a s t of the data were c o l l e c t e d , the c o n c l u s i o n was a l r e a d y e v i d e n t . The committee was t h e r e f o r e able to broaden the scope of i t s i n v e s t i g a t i o n i n order to make informed recommendations with regard to f u t u r e l i t i g a t i o n management procedures and defence counsel- e x p e c t a t i o n s . During' the course of the committee's i n v e s t i g a t i o n i n t o l i t i g a t i o n ' f i l e management, the committee.was asked to develop "canned defence p l a n s " . As a r e s u l t , a sub- committee was s t r u c k and a defence p l a n f o r f i b r o m y a l g i a (a n o n - s p e c i f i c s o f t t i s s u e i n j u r y c o n d i t i o n diagnosed by p a i n 54 response to 12. or more of 18 " t r i g g e r p o i n t s " . A confirmed d i a g n o s i s of f i b r o m y a l g i a c o u l d command a l a r g e monetary award i n court) was developed. In c o n j u n c t i o n with the development of sub-committees, a head i n j u r y committee was a l s o formed as p a r t of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee's mandate. The members of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, i n c l u d i n g me, were not d i r e c t l y i n v o l v e d with these sub- committees. The r e s u l t of the e v a l u a t i o n process was t h a t the o r i g i n a l mandate of the committee was r e f o c u s s e d i n t o an examination of the e n t i r e l i t i g a t i o n management process r a t h e r than the o r i g i n a l narrow focussed examination of B u l l e t i n CDB894. The methodologies used i n both task f o r c e s are r e p r e s e n t a t i v e of two e v a l u a t i o n processes, the review of the l i t e r a t u r e and the focus group e v a l u a t i o n . The review of the l i t e r a t u r e i n v o l v e d r e a d i n g a r t i c l e s t h a t were w r i t t e n by e x t e r n a l sources which were a v a i l a b l e to the p u b l i c . As w e l l , the review i n v o l v e d an examination of i n t e r n a l r e p o r t s which had l i m i t e d c i r c u l a t i o n w i t h i n the c o r p o r a t i o n . These r e p o r t s were not g e n e r a l l y a c c e s s i b l e w i t h i n the c o r p o r a t i o n , nor were they a v a i l a b l e to the p u b l i c . The essence of the focus group e v a l u a t i o n process i s to conduct p e r s o n a l i n t e r v i e w s that, as s t a t e d by Worthen 55 and Sanders, (1987, p. 108), "allows c l a r i f i c a t i o n and p r o b i n g " . I n t e r v i e w i n g i s h e l p f u l i n determining, by c a r e f u l assessment, the values and b e l i e f s of the people who are d i r e c t l y a f f e c t e d by the outcome of the e v a l u a t i o n . The review of the l i t e r a t u r e approach c o n s i s t s of an exhaustive search of a l l r e l e v a n t documents p e r t a i n i n g to the s p e c i f i c s u b j e c t area and then i n t e r v i e w i n g people who are d i r e c t l y a f f e c t e d by the process a g a i n s t the background of the h i s t o r i c a l documentation. The d i f f e r e n c e between the two approaches i s that the former seeks to determine the c u r r e n t need whereas the l a t t e r seeks to p l a c e the c u r r e n t s i t u a t i o n w i t h i n an h i s t o r i c a l context. Data C o l l e c t i o n The method used f o r data c o l l e c t i o n was t h a t of p a r t i c i p a n t observer. A c t i n g i n the r o l e of p a r t i c i p a n t observer had disadvantages and l i m i t a t i o n s . As Cousins (1996, p. 20) observed: The most s i g n i f i c a n t c o n t r i b u t i o n of the present study i s i t s evidence that, indeed, the l e v e l of r e s e a r c h e r involvement i n p a r t i c i p a t o r y e v a l u a t i o n does make a d i f f e r e n c e . S p e c i f i c a l l y , while the experience was g e n e r a l l y p o s i t i v e f o r the s c h o o l - based r e s e a r c h committee members, the r e s e a r c h e r ' s f u l l p a r t n e r s h i p r o l e may have l e d to the establishment of u n r e a l i s t i c e x p e c t a t i o n s of the committee and i t s r e p o r t . Everyone i n v o l v e d i n both the Claims T r a i n i n g E v a l u a t i o n Task Force and L i t i g a t i o n Management E v a l u a t i o n Task Force Committees was aware'of my r o l e . a s p a r t i c i p a n t 56 observer and there was a s h o r t t r a n s i t i o n p e r i o d which had to be undergone be f o r e the group was comfortable w i t h t h a t r o l e . For example, the simple act of t a k i n g notes c o u l d make some members f e e l uneasy. The committee members c o u l d be apprehensive i n not knowing whether the notes b e i n g taken were f o r the purposes of the committee or the study. T h i s s e t t l i n g - i n p e r i o d was s h o r t - l i v e d i n t h i s study as the committees became more i n v o l v e d and focussed on the e v a l u a t i o n s themselves. As evidence f o r t h i s , I observe from my f i e l d notes that there was no r e f e r e n c e to my r o l e as a p a r t i c i p a n t observer a f t e r the f i r s t meeting of the committees. Consequently, the p a r t i c i p a n t observer aspect, i n t h i s study, a p p a r e n t l y faded i n t o the background very q u i c k l y i n the minds of the other e v a l u a t o r s . As a p a r t i c i p a n t observer I had to be aware t h a t i t was very easy to focus on the p a r t i c i p a t i o n r o l e i n the e v a l u a t i o n r a t h e r than on the o b s e r v a t i o n r o l e , and the data c o u l d be skewed due to a l a c k of c l e a r f o c u s . I t was a d e l i c a t e balance. As Cousins(1996, p. 23) suggests: I f p a r t i c i p a t o r y e v a l u a t i o n i s to become a v i a b l e approach to s u p p o r t i n g o r g a n i z a t i o n a l d e c i s i o n - making processes and enhancing o r g a n i z a t i o n a l l e a r n i n g c a p a c i t y , a much more r e a l i s t i c p e r c e p t i o n of what i s e n t a i l e d i s needed. Cousins(1996, p. 23) r a i s e s an i n t e r e s t i n g q u e s t i o n with r e s p e c t to the t r a d e - o f f between q u a l i t y and quick r e s u l t s 57 with r e s p e c t to s u p p o r t i n g p a r t i c i p a t o r y e v a l u a t i o n and the p a r t i c i p a n t observer when he asks: W i l l d i s s a t i s f a c t i o n with lame r e s u l t s from t e c h n i c a l l y i n f e r i o r "quick and d i r t y " s t u d i e s a d v e r s e l y a f f e c t such a t t r a c t i o n or enhance a d m i n i s t r a t o r s ' p r o p e n s i t y to r e s p e c t more t e c h n i c a l l y sound and c o s t l y p r o j e c t s ? The simple answer c o u l d be t h a t whether or not an o r g a n i z a t i o n chooses to use p a r t i c i p a t o r y e v a l u a t i o n and the p a r t i c i p a n t observer w i l l l a r g e l y depend upon the o r g a n i z a t i o n making, a business d e c i s i o n with r e s p e c t to the needs of the o r g a n i z a t i o n at the time t h a t the e v a l u a t i o n i s commissioned. In the Claims T r a i n i n g E v a l u a t i o n Task Force Committee, I c o l l e c t e d data by way of t a k i n g e x t e n s i v e p e r s o n a l notes and-assembling copies of a l l documents • ( p r i n t e d and e l e c t r o n i c ) . The notes were intended to capture my o b s e r v a t i o n s and the main p o i n t s b e i n g d i s c u s s e d i n wording as c l o s e as p o s s i b l e to what was being used. The balance of the documents was c o l l e c t e d with the i n t e n t of having a copy of e v e r y t h i n g generated by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee ( i n c l u d i n g d r a f t s ) i n order to c r e a t e an accurate body of documentation. In the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee the data c o l l e c t i o n process was d i f f e r e n t . I t c o n s i s t e d mostly of a c o l l e c t i o n of documents with an 58 emphasis upon e l e c t r o n i c e-mails, minutes and notes. Some pe r s o n a l notes were taken, but these were not e x t e n s i v e . T h i s L i t i g a t i o n Management E v a l u a t i o n Task Force Committee met i r r e g u l a r l y , and much of the work of the committee was conducted by committee members i n i s o l a t i o n . The meetings were used to update committee members on the s t a t u s of v a r i o u s i n d i v i d u a l t a s k s . Summary A b r i e f h i s t o r y of e d u c a t i o n a l e v a l u a t i o n was d i s c u s s e d i n order to p r o v i d e the reader with an h i s t o r i c a l context w i t h i n which to p l a c e the development of the e d u c a t i o n a l e v a l u a t i o n models used i n t h i s study. There are c e r t a i n aspects of each task f o r c e ' s methodology which seem to p a r a l l e l the t h e o r e t i c a l models of Provus and Stufflebeam et a l . very c l o s e l y . The methodologies used by the two c o r p o r a t e task f o r c e s were o u t l i n e d as w e l l as the methodology used by the p a r t i c i p a n t observer f o r data c o l l e c t i o n . Chapter Four w i l l p r e s e n t the data c o l l e c t e d i n t h i s study which w i l l be used to e x p l o r e the r e l a t i o n s h i p s between the t h e o r e t i c a l e v a l u a t i o n models of Provus and Stufflebeam e t . a l . and the two c o r p o r a t e based e v a l u a t i o n s . The d i s c u s s i o n and i n t e r p r e t a t i o n of the r e s u l t s w i l l be presented i n Chapter F i v e . CHAPTER FOUR 59 RESULTS The two cor p o r a t e e v a l u a t i o n s examined i n the study, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, were analyzed u s i n g two models designed f o r the e v a l u a t i o n of cu r r i c u l u m : the Provus (1973) Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . (1973) C.I.P.P. E v a l u a t i o n Model. In a d d i t i o n , the records of the p r o j e c t s were e x p l o r e d from the p o i n t o f view of the i n f l u e n c e of e x t e r n a l f a c t o r s . From the p e r s p e c t i v e of the e v a l u a t i o n models themselves, comparisons were made to i l l u s t r a t e how the p r a c t i c a l a p p l i c a t i o n of each model may a f f e c t the outcome of e v a l u a t i o n s . The two e d u c a t i o n a l e v a l u a t i o n models were a p p l i e d r e t r o s p e c t i v e l y to two e v a l u a t i o n s t h a t were conducted i n a corp o r a t e s e t t i n g i n order to explore, h y p o t h e t i c a l l y , how . these e v a l u a t i o n s may have b e n e f i t e d from the a p p l i c a t i o n of such models d u r i n g the e v a l u a t i o n p r o c e s s . N e i t h e r p r o j e c t conducted i t s e v a l u a t i o n s by u t i l i z i n g a formal e v a l u a t i o n p r o c e s s . The e f f e c t o f f a c t o r s o u t s i d e the e v a l u a t i o n models was a l s o examined to determine p o s s i b l e i n f l u e n c e s upon both the a p p l i c a t i o n of the models and the 60 i n t e r p r e t a t i o n • o f the data c o l l e c t e d . The data c o l l e c t e d i n t h i s study are presented from both p e r s p e c t i v e s . As w i l l - be d i s c u s s e d below, The Provus (1973) Discrepancy E v a l u a t i o n Model was h e l p f u l f o r a s s e s s i n g approaches taken by two e v a l u a t i o n committees and the impact t h a t good p l a n n i n g or the l a c k of i t has on the q u a l i t y and u s e f u l n e s s of e v a l u a t i o n s . The Stufflebeam et a l . (1973) C.I.P.P. E v a l u a t i o n Model adds to Provus's focus by examining the context in. which the e v a l u a t i o n s take p l a c e . However, both models f a i l e d to account f o r the i n f l u e n c e t h a t e x t e r n a l f a c t o r s may have upon e v a l u a t i o n s . As w e l l as the e f f e c t of e x t e r n a l f a c t o r s , there i s no i n d i c a t i o n by e i t h e r Provus or Stufflebeam et a l . t h a t any c o n s i d e r a t i o n has been given to the p o s s i b i l i t y t h a t the a p p l i c a t i o n of e i t h e r model may not r i g i d l y f o l l o w the e v a l u a t i o n g u i d e l i n e s e x a c t l y as set out due to e x t e r n a l f a c t o r s : E v a l u a t i o n Model A p p l i c a t i o n Comparison The Provus (1973) Discrepancy E v a l u a t i o n Model c o n s i s t s of f i v e stages. Stage I - Programme Design. The f i r s t stage of the Provus D i s c r e p a n c y - E v a l u a t i o n Model, d e f i n i n g the problem to be s t u d i e d i n terms of the programme design, was h e l p f u l i n understanding the dynamics of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee and the importance of c l e a r l y i d e n t i f y i n g programme o b j e c t i v e s . The Provus 61 Discrepancy E v a l u a t i o n Model was l e s s h e l p f u l i n the Claims T r a i n i n g E v a l u a t i o n Task Force Committee, where there was a conspicuous l a c k of c l e a r l y d e f i n e d o b j e c t i v e s . Nonetheless, The Provus Discrepancy E v a l u a t i o n Model was i n s t r u c t i v e because of the c o n t r a s t between the absence of clear- o b j e c t i v e s i n the Claims T r a i n i n g E v a l u a t i o n Task Force Committee with the t h e o r e t i c a l need f o r c l e a r o b j e c t i v e s as o u t l i n e d i n the Provus Discrepancy E v a l u a t i o n Model. Stage I of the Provus (1973) Discrepancy E v a l u a t i o n Model r e q u i r e s t h a t the programme design be c l e a r l y l a i d out i n terms of three dimensions: Input, Process and Output. This f i r s t stage, a c c o r d i n g to Provus, i s c r u c i a l i n c r e a t i n g a foundation f o r the e v a l u a t i o n i n t h a t i t i s necessary to i d e n t i f y what the intended o b j e c t i v e s are, the p l a n f o r a c h i e v i n g those o b j e c t i v e s and what success should look l i k e i f those o b j e c t i v e s are achieved. Provus suggested t h a t i n Stage I the a n a l y s i s of the problem i s paramount i n t h a t without a p r o p e r l y d e f i n e d problem there i s an u n c l e a r r e f e r e n c e upon which to base the remainder of the a n a l y s i s . For the Claims T r a i n i n g E v a l u a t i o n Task Force Committee, the f i r s t stage of the Provus Discrepancy E v a l u a t i o n Model suggested t h a t the committee would f i n d some d i f f i c u l t y completing i t s task e f f e c t i v e l y . T h i s 62 p r e d i c t i o n was made based on the d i s c r e p a n c y between the need f o r c l e a r l y d e f i n e d o b j e c t i v e s , as per the Provus Discrepancy E v a l u a t i o n Model, and the l a c k o f c l e a r l y d e f i n e d o b j e c t i v e s as was evidenced i n the Claims T r a i n i n g E v a l u a t i o n Task Force'Committee. Although a mandate to evaluate claims t r a i n i n g was given to the Claims T r a i n i n g E v a l u a t i o n Task Force Committee by s e n i o r management, t h i s mandate d i d not p r o v i d e s p e c i f i c o b j e c t i v e s with r e s p e c t to e v a l u a t i n g . t h e claims t r a i n i n g programme. The mandate was given to the Claims T r a i n i n g E v a l u a t i o n Task Force Committee to p r o v i d e g e n e r a l recommendations to the Claims D i v i s i o n and the Human Resources D i v i s i o n concerning the f u t u r e o f claims t r a i n i n g . Rather than evaluate Claims T r a i n i n g w i t h i n the scope of i t s performance, the recommendations were to focus on the c l i e n t s ' needs and s p e c i f y the f o l l o w i n g : 1. How claims t r a i n i n g needs can best be addressed. 2. What claims t r a i n i n g i s needed. 3. Who should d e l i v e r the claims t r a i n i n g . These o b j e c t i v e s c o u l d be used to determine the m a r k e t a b i l i t y of the the n - c u r r e n t programme but added l i t t l e to e s t a b l i s h i n g whether or not the Claims T r a i n i n g Programme had performed e f f e c t i v e l y . The a c t u a l intended performance outcomes of the Claims T r a i n i n g Programme were not s t a t e d . 63 The broad mandate was n e i t h e r questioned nor f u r t h e r d e f i n e d by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. The Committee spent no time determining what the mandate s p e c i f i c a l l y meant with r e s p e c t to the performance of the Claims T r a i n i n g Department, but r a t h e r focussed on the product being d e l i v e r e d . In the f i r s t meeting, a c c o r d i n g to the Provus Discrepancy E v a l u a t i o n Model, i t would be expected t h a t the problem would be i d e n t i f i e d , c l a r i f i e d , or at l e a s t d i s c u s s e d and a p l a n of a c t i o n developed with r e s p e c t to how the e v a l u a t i o n would be conducted. Instead, at the f i r s t meeting of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee, a great d e a l of time was spent d i s c u s s i n g how changes would a f f e c t the members of the Task Force Committee who were d i r e c t l y i n v o l v e d with Claims T r a i n i n g . As I observed i n my notes of the f i r s t meeting A p r i l 3, 1994, the s e s s i o n moved along slowly, c r o s s i n g o l d ground s e v e r a l times with most of the members being c a u t i o u s . T h i s l a c k of c l e a r focus l e d to s e v e r a l d i s c u s s i o n s i n v o l v i n g emotional i s s u e s surrounding the need f o r change. For example, from my f i e l d notes of A p r i l 8, 1994/ one member whose former group would be a f f e c t e d by changing the Claims T r a i n i n g Department was very w i l l i n g to express h i s views - and i t was obvious t h a t he had s t r o n g emotions surrounding the changes and the manner i n which the changes would take p l a c e . T h i s had a s t r o n g 64 i n f l u e n c e on the committee and made the committee s e n s i t i v e to the impact of change upon Claims T r a i n i n g Department t r a i n e r s . While l i k e l y unintended, the focus of the e v a l u a t i o n stayed away from performance e v a l u a t i o n , which would have r e f l e c t e d on the i n d i v i d u a l t r a i n e r s . The e v a l u a t i o n was t h e r e f o r e d i r e c t e d towards the product b e i n g d e l i v e r e d r a t h e r than the d e l i v e r y of the product. In e f f e c t , the t r a i n e r s were p r o t e c t e d from c r i t i c i s m by the focus of the e v a l u a t i o n b e i ng on the product. Conversely, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee had a very c l e a r mandate. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, a p p a r e n t l y i n t u i t i v e l y , was able to f o l l o w the Provus Discrepancy E v a l u a t i o n Model p r e c i s e l y . Research by the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee at the outset l e d to a review of Claims D i v i s i o n B u l l e t i n 894 which l a i d out o b j e c t i v e s , procedures and expected b e n e f i t s of the programme. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee's mandate was to determine compliance by B.I. ( B o d i l y Injury) A d j u s t e r s and s t a k e h o l d e r s to B u l l e t i n CDB894 and i t s e f f e c t i v e n e s s . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee spent the f i r s t meeting both i n p l a c i n g the e v a l u a t i o n i n an h i s t o r i c a l p e r s p e c t i v e and i n d e f i n i n g the problem to be examined. As my f i e l d notes 65 i n d i c a t e , " I t was not necessary to d e f i n e the design c r i t e r i a as the c r i t e r i a were s p e c i f i e d i n Claims D i v i s i o n B u l l e t i n 8 94." Table 2 Comparison - Provus Discrepancy E v a l u a t i o n Model, Stage I, and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee Mandate Personal F i e l d Notes L i t i g a t i o n Management E v a l u a t i o n Task Force Mandate It was not necessary to d e f i n e the design c r i t e r i a as the c r i t e r i a were s p e c i f i e d i n B u l l e t i n CDB894. The mandate of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee was to reduce claims a l l o c a t e d expenses and s e v e r i t i e s by: - c l a r i f y i n g and l i m i t i n g involvement of e x t e r n a l defense counsel i n the settlement p r o c e s s . - a r r a n g i n g e a r l i e r p o t e n t i a l end dates ( d i s c o v e r y and t r i a l ) f o r the l e g a l process, thus promoting e a r l i e r s e ttlement. - implementing a process by which counsel and a d j u s t e r w i l l agree to a documented and budgeted course of action- f o r each new l i t i g a t e d f i l e r e q u i r i n g more than c l o s u r e of p l e a d i n g s . - d e f e r r i n g the i n i t i a l counsel review of complete f i l e contents u n l e s s and u n t i l necessary. - p r o v i d i n g f u r t h e r t o o l s f o r defense counsel e v a l u a t i o n and a u d i t . 66 • Table 2 above o u t l i n e s the Mandate a s . i t was presented, by Senior Management, to the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee. This w e l l d e f i n e d o b j e c t i v e accords with the Provus Discrepancy E v a l u a t i o n Model and l a y s the groundwork for.. Stage I I , Programme Operation.- A f t e r having d e f i n e d an o b j e c t i v e i t i s c r u c i a l t h a t a s i m i l a r l y w e l l d e f i n e d implementation p l a n be designed along with an e f f e c t i v e e v a l u a t i o n process which must be i n p l a c e i n order to e s t a b l i s h a c c u r a t e l y whether or not the r e s u l t s a t t a i n e d are congruent with the o r i g i n a l o b j e c t i v e s . Stage II -Programme Operation. Stage II of the Provus Discrepancy E v a l u a t i o n Model compares the c u r r e n t o p e r a t i o n s of the programme being e v a l u a t e d with the o r i g i n a l o b j e c t i v e s and procedures s e t out i n the design c r i t e r i a . I t i s necessary to examine the r e s u l t s a t t a i n e d a g a i n s t the o r i g i n a l o b j e c t i v e s . A c c o r d i n g to Provus, o n l y by e s t a b l i s h i n g whether or not congruence e x i s t s between the o r i g i n a l o b j e c t i v e s and the a t t a i n e d r e s u l t s can an e v a l u a t i o n e f f e c t i v e l y e s t a b l i s h the success or f a i l u r e of a programme. Stage II of the Provus Discrepancy E v a l u a t i o n Model, Programme Operation, was not f o l l o w e d by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. Instead of f o l l o w i n g steps s i m i l a r to those suggested by Provus, which would have 67 r e q u i r e d the conducting of a comparison between the intended o b j e c t i v e s and the a c t u a l outcomes of Claims T r a i n i n g , the Claims T r a i n i n g E v a l u a t i o n Task Force Committee chose to examine c u r r e n t Claims T r a i n i n g students and s t a k e h o l d e r groups through the use of focus groups and i n d i v i d u a l i n t e r v i e w s to determine what the e x p e c t a t i o n s were of Claims T r a i n i n g . As s t a t e d i n my f i e l d notes of A p r i l 8, 1994: Given the expense of m a i n t a i n i n g a c e n t r a l i z e d programme and a demand from the f i e l d f o r t r a i n i n g upon s p e c i f i c day-to-day i s s u e s , the q u e s t i o n t h a t s u r f a c e d was whether or not there was a b e t t e r way to conduct s t a f f t r a i n i n g . The Claims T r a i n i n g e v a l u a t i o n was p r e d i c a t e d upon an assumption of a p e r c e i v e d problem, Claims T r a i n i n g ' s l a c k of adding value to the students' development, and sought to d e f i n e a l t e r n a t e methods of d e l i v e r y which i n c l u d e d an emphasis upon c e n t r a l i z e d t r a i n i n g versus d e c e n t r a l i z e d t r a i n i n g . T h i s movement d i r e c t l y i n t o the i n v e s t i g a t i o n of a p e r c e i v e d problem and a movement away from the examination of a c t u a l versus intended outcomes l e f t the Claims T r a i n i n g E v a l u a t i o n Task Force Committee open to c r i t i c i s m w i t h r e s p e c t to the outcome of the e v a l u a t i o n . On the other hand, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee c r e a t e d a simple f o u r - q u e s t i o n template (Table 3, r i g h t hand column) which was designed to e l i c i t answers from a s e l e c t i o n of B o d i l y I n j u r y A d j u s t e r s and 68 s t a k e h o l d e r s i n t e r v i e w e d which would e s t a b l i s h compliance with B u l l e t i n CDB894, the o f f i c i a l p o l i c y b u l l e t i n which set out a l i t i g a t i o n budgeting procedure. This i s s i m i l a r to the suggestion i n the Provus Discrepancy E v a l u a t i o n Model which e s t a b l i s h e s a f o u n d a t i o n f o r e v a l u a t i o n s by i d e n t i f y i n g what the intended outcomes are and comparing them to the a c t u a l outcomes. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee ev a l u a t e d the e f f e c t i v e n e s s of B u l l e t i n CDB894 through the design of a simple e v a l u a t i o n q u e s t i o n n a i r e to determine whether or not there was compliance. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee used a small sampling of s t a f f members and s t a k e h o l d e r s to answer the four questions, o u t l i n e d i n Table 3 below, to p r o j e c t the p o t e n t i a l of whether or not there was g e n e r a l compliance on a corporate-wide b a s i s . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee followed, a p p a r e n t l y i n t u i t i v e l y , the procedures suggested i n Stage II of the Provus Discrepancy E v a l u a t i o n Model wherein Programme Design was examined i n terms of input and process dimensions. The c u r r e n t o p e r a t i o n at t h a t time was compared with the design c r i t e r i a by the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee i n order to determine compliance. • ' 69 Table 3 Comparison - Provus Discrepancy E v a l u a t i o n Model, Stage I I , and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee. Mandate Personal F i e l d Notes L i t i g a t i o n Management Ev a l u a t i o n ' Task Force Committee F i n a l Report Page 7 The c u r r e n t o p e r a t i o n was The L i t i g a t i o n Management compared to the Committee dec i d e d t h a t the o b j e c t i v e s and procedures best method of o b t a i n i n g an as l a i d out i n CDB894. accurate assessment of the s t a t e of the c u r r e n t l i t i g a t i o n management p r a c t i c e s was to review CDB894 and i n t e r v i e w BI A d j u s t e r s and Defence Counsel. The Committee would' a l s o conduct a review of the l i t e r a t u r e of the r e f e r e n c e m a t e r i a l a v a i l a b l e on the i s s u e of l i t i g a t i o n management. Defence Counsel and BI A d j u s t e r s were asked to respond to the f o l l o w i n g q u e s t i o n s : 1. Do you comply with CDB8 94? 2. What about CDB8 94 works or doesn't work? .3. Are the requirements of CDB894 necessary? I f • , not, what are the a l t e r n a t i v e s ? 4. What can be done to /• improve CDB894? In other words, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee i d e n t i f i e d what the intended outcomes were 70 by c o n f i r m i n g the requirements of B u l l e t i n CDB894 and compared them to the a c t u a l outcomes i n order to e s t a b l i s h whether or not a v a r i a n c e e x i s t e d . Stage I I I - Programme I n t e r i m Products. Stage I I I of the Provus Discrepancy E v a l u a t i o n Model r e q u i r e s an examination of the process and the s p e c i f i c outcomes. The focus of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee turned from the ou t s e t towards conducting a needs assessment and not towards d e f i n i n g the process and intended outcomes of claims t r a i n i n g . As was evidenced by the methodology used by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee appeared to make the assumption that claims t r a i n i n g was d y s f u n c t i o n a l and not meeting the needs of the r e c i p i e n t s . One r e s u l t of t h i s assumption was th a t the questions put to the focus groups s t e e r e d the answers towards what the Claims T r a i n i n g E v a l u a t i o n Task Force Committee appeared to f e e l was r i g h t or wrong about the Claims T r a i n i n g Department. A second r e s u l t was t h a t the responses recorded by the Committee, when i n t e r p r e t e d post i n t e r v i e w , appeared to have been i n f l u e n c e d by those assumptions. The f o l l o w i n g examples of questions used i l l u s t r a t e the apparent b i a s i n the e v a l u a t i o n d e s i g n . 71 - Is t h i s s p e c i f i c to the BIT ( B o d i l y I n j u r y T r a i n i n g ) Programme or i s i t t r a i n i n g as w e l l as BI's i n g e n e r a l . [Questions whether the p e r c e i v e d problem • r e s t s with t r a i n i n g or the a d j u s t e r s themselves] Is t h i s a r e v i s i o n of the BIT programme? - O r g a n i z a t i o n s t r u c t u r e - does i t matter? - Would i t be d i f f e r e n t i f l e a r n i n g c e n t r e was out of o f f i c e ? - What claims t r a i n i n g i s needed? - T r a i n i n g i n advance of need - i s t h i s a problem? - Should there be a new, s p e c i a l i z e d course? - Would there be some value to a networking set-up? - D e c e n t r a l i z i n g or r e g i o n a l i z i n g - no t r a v e l time - do you have any comments on t h i s ? - What i f , i n s t e a d of going to HO you had a r e g i o n a l o f f i c e - would t h a t p r o v i d e the same th i n g ? - R e g i o n a l i z a t i o n - s h o u l d t r a i n i n g be at o f f i c e s i t e , or o f f s i t e w i t h i n region? - Is what they teach worthwhile? - Do t r a i n e e s come out of the t r a i n i n g programme with the b a s i c knowledge/theory they need? The types of questions asked were not those which would compare the outcome of the r e s u l t s of claims t r a i n i n g with the intended outcomes. Rather, these questions took the respondents i n t o a d i r e c t i o n which would r e d e f i n e claims 72 t r a i n i n g . While these questions were v a l i d from the p e r s p e c t i v e of determining a new focus f o r claims t r a i n i n g they were premature and were the type of questions which c o u l d be asked a f t e r e s t a b l i s h i n g whether or not claims t r a i n i n g had met i t s o r i g i n a l o b j e c t i v e s . In other words, these types of questions were more a p p r o p r i a t e f o r d e s i g n i n g a new d i r e c t i o n of claims t r a i n i n g a f t e r determining t h a t a change of focus was r e q u i r e d . The Provus Discrepancy E v a l u a t i o n Model r e q u i r e s t h a t a c t u a l outcomes be compared to intended outcomes, which would focus an e v a l u a t i o n i n an e n t i r e l y d i f f e r e n t d i r e c t i o n from t h a t taken by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. T h i s stage of the Provus Discrepancy E v a l u a t i o n Model underscores the need f o r d e v e l o p i n g c l e a r l y d e f i n e d obj e c t i v e s . Ralph T y l e r i n h i s a r t i c l e Changing Concepts of E d u c a t i o n a l E v a l u a t i o n (1986, pp. 53-55) i d e n t i f i e s four c r i t e r i a f o r d e v e l o p i n g o b j e c t i v e s . They are summarized as f o l l o w s : 1. O b j e c t i v e s should be i n harmony with the e d u c a t i o n a l p h i l o s o p h y of the s c h o o l . 2. Relevance and a p p r o p r i a t e n e s s of the o b j e c t i v e to the s u b j e c t matter. 3. The o p p o r t u n i t y the l e a r n e r has to use what he or she i s l e a r n i n g . 73 4. The ap p r o p r i a t e n e s s of the o b j e c t i v e to the needs, i n t e r e s t s and presen t development of the p a r t i c u l a r students f o r whom the program i s planned. T y l e r (1986, p.55) makes the f o l l o w i n g comment with r e s p e c t to the importance of c l e a r l y d e f i n i n g o b j e c t i v e s . The process of e v a l u a t i n g the o b j e c t i v e s o f a proposed e d u c a t i o n a l program i s l a r g e l y t h a t of reminding those r e s p o n s i b l e f o r the development of the program t h a t these four c r i t e r i a should be c a r e f u l l y c o n s i d e r e d . The r e s u l t s of the focus.group s e s s i o n s formed the b a s i s of the recommendations of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. For example, from the May 6, 1994 focus group s e s s i o n the f o l l o w i n g sample of quotations made by the respondents a l s o i l l u s t r a t e the focus of the e v a l u a t i o n . - We c o u l d have e a s i e r access to r e s o u r c e s . - B i g g e s t concern re i n i t i a l t r a i n i n g — too • f a r away from a c t u a l work we do. - H i r i n g someone from the o u t s i d e who doesn't know claims process, t h e i r t r a i n i n g i s not s p e c i f i c to the problem at hand. - Lots of "war s t o r i e s " . - L e a r n i n g through PC - no. - When you are at HO, they don't know what i s going on i n claims needs to be more of a "t o g e t h e r " working r e l a t i o n s h i p . 74 - Claims t r a i n i n g - v e r y e n t e r t a i n i n g ! Perhaps we needed more d i r e c t i o n . In c o n t r a s t to the procedures used by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee i n t e r v i e w s , which were conducted with B.I. ( B o d i l y Injury) A d j u s t e r s and s t a k e h o l d e r s , were s p e c i f i c a l l y d i r e c t e d towards determining whether a v a r i a n c e e x i s t e d between the intended outcomes and the a c t u a l outcomes of the a p p l i c a t i o n of B u l l e t i n CDB894. The questions developed i n Stage II were used and the i n t e r v i e w e e s were allowed to e l a b o r a t e upon t h e i r answers. As the L i t i g a t i o n Management E v a l u a t i o n Task Force Report i n d i c a t e s on page 4, " i t r a p i d l y became apparent t h a t there was v i r t u a l l y no compliance" with B u l l e t i n CDB894 and, i n ge n e r a l , t h a t the working r e l a t i o n s h i p between Defence Counsel and B.I. ( B o d i l y Injury) A d j u s t e r s was s t r a i n e d . These r e s u l t s were presented i n the f i n a l r e p o r t to s e n i o r management and recommendations were made to terminate B u l l e t i n CDB894 with the i n t e n t i o n of implementing a new l i t i g a t i o n management pr o c e s s . Stage IV - Programme Terminal Products. In Stage I I I of the Provus Discrepancy E v a l u a t i o n Model, Programme I n t e r i m Products, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee focussed upon needs assessments r a t h e r than d e f i n i n g the process and d e f i n i n g the intended outcomes of , 75 claims t r a i n i n g as i s r e q u i r e d by the Provus Discrepancy E v a l u a t i o n Model. The Claims T r a i n i n g E v a l u a t i o n Task Force Committee began e v a l u a t i n g the Claims T r a i n i n g programme output design a g a i n s t the e x p e c t a t i o n s of the students and s t a k e h o l d e r s . While t h i s r e f l e c t s Stage IV of the Provus Discrepancy E v a l u a t i o n Model and would come towards the end of the process, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee moved i n t o t h i s stage almost immediately. The f i r s t i n t e r v i e w s took p l a c e on the A p r i l 20, 1994, 12 days a f t e r the formation of the Task Force. The f i r s t focus group s e s s i o n was h e l d on May 6, 1994, 28 days a f t e r the formation o f the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. The f o l l o w i n g excerpt from the Focus Group F a c i l i t a t o r ' s Guide ( A p r i l 6, 1994) c r y s t a l i z e s the d i r e c t i o n i n which the Claims T r a i n i n g E v a l u a t i o n Task Force Committee chose to take t h i s e v a l u a t i o n . The f a c i l i t a t o r may want to use pr o b i n g q u e s t i o n s a s s o c i a t e d with the three o b j e c t i v e s to c l a r i f y i n p u t . What Claims t r a i n i n g i s needed? Probes: What would be d i f f e r e n t from the way i t i s now? For whom (CR's [Claims R e p r e s e n t a t i v e s ] , CA's [Claims A d j u s t e r s ] , BI's [ B o d i l y I n j u r y A d j u s t e r s ] ) ? What would add, d e l e t e , or change concerning content, amount, or t i m i n g ( a p p r o p r i a t e t i m i n g f o r the v a r i o u s stages of BI or CA development)? WHY? Is the c u r r e n t t r a i n i n g r e l e v a n t to the job, e f f e c t i v e / n o t e f f e c t i v e , up-to-date? (Any comments around l e a r n i n g methods: l e c t u r e , i n t e r a c t i v e , case study, on-the-job, etc.?) 76 What i s the best o r g a n i z a t i o n s t r u c t u r e to p r o v i d e the t r a i n i n g ? Probes: which d i v i s i o n or department would i t be best to have own the claims t r a i n i n g (eg. HR-hrd [Human Resources-Human' Resources Development]; C l a i m s - F i e l d , head o f f i c e , MD R&T [ M a t e r i a l Damage-Research & T r a i n i n g ] ) ? WHY? Who should d e l i v e r what claims t r a i n i n g ? Probes: HRD T r a i n e r s , UM's [Unit Managers], e x t e r n a l c o n s u l t a n t s / t r a i n e r s , d i s t a n c e education, PC based, degree or type t h a t c o u l d be c e n t r a l i z e d / d e c e n t r a l i z e d ? WHY? The f o l l o w i n g excerpts are t y p i c a l of the responses to the questions which were put to the focus groups by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. They f u r t h e r i l l u s t r a t e the d i r e c t i o n i n which the focus groups were headed and h i g h l i g h t the l a c k of focus on a c l e a r mandate. May 6, 1994-BI Focus Group: - Claims T r a i n i n g i s too f a r removed from the f i e l d . • - Current system was good f o r i n i t i a l t r a i n i n g but there i s a need to pr o v i d e t r a i n i n g / a s s i s t a n c e on r e a l l i f e f i l e s i n day to day o p e r a t i o n . "XX", to the b e s t of h i s schedule, helped i n t h i s area. - We've been r e a c t i v e and a p r o a c t i v e t r a i n i n g approach i s necessary. - Wednesday mornings s e t a s i d e f o r t r a i n i n g . - E a s i e r access to resources r e q u i r e d . - User f r i e n d l y access to i n f o r m a t i o n . 77 The d i v e r s i t y of the responses i n these s e s s i o n s , as o u t l i n e d above, i s an i n d i c a t o r of the l a c k of c l e a r focus f o r the e v a l u a t i o n with r e s p e c t to whether or not the Claims T r a i n i n g Department had achieved i t s o b j e c t i v e s . The d i r e c t i o n t h a t the Claims T r a i n i n g E v a l u a t i o n Task Force Committee took with the data g a t h e r i n g process was based upon assumptions t h a t were made about the successes or f a i l u r e s of the Claims T r a i n i n g Department. The format of the focus group s e s s i o n s was aimed at examining processes and e s t a b l i s h i n g whether or not the Claims T r a i n i n g Department was meeting the c u r r e n t needs of the s t a f f . To t h i s end, the focus group process was designed, i n t e n t i o n a l l y or not, to f u l f i l the assumptions made by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. As the above answers show, the r e s u l t s appear to have been gathered through the use of l e a d i n g questions to achieve s p e c i f i c outcomes. The Claims T r a i n i n g E v a l u a t i o n Task Force Committee focussed upon the f u l f i l m e n t of needs. The Provus Discrepancy E v a l u a t i o n Model, however, focussed upon i d e n t i f y i n g the o b j e c t i v e s (intended outcomes). There i s a fundamental o p e r a t i o n a l d i f f e r e n c e i n approach between the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and the Provus Discrepancy E v a l u a t i o n Model. The Claims' T r a i n i n g E v a l u a t i o n Task Force Committee took the f u l f i l m e n t of 78 c u r r e n t needs as being the measure- of success. Needs which e x i s t at the time o f . t h e e v a l u a t i o n , while p o t e n t i a l l y i n t e r e s t i n g , are i r r e l e v a n t i n determining whether or not the programme o b j e c t i v e s achieved what they set out to do. ' The emphasis should be, as the Provus Discrepancy E v a l u a t i o n Model d i c t a t e s , upon programme o b j e c t i v e s and whether, or not they were achieved. The Claims t r a i n i n g e v a l u a t i o n moved towards conducting a needs assessment, e s t a b l i s h i n g a benchmark of the then c u r r e n t a b i l i t y of the Claims T r a i n i n g Department to s e r v i c e i t s .-customers adequately, and c r e a t e d a v i s i o n of what the f u t u r e might look l i k e . .However, i t d i d not e v a l u a t e the Claims T r a i n i n g Department's success at a c h i e v i n g i t s o b j e c t i v e s as those o b j e c t i v e s were never d i s c u s s e d nor e x p l o r e d . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee had no d i f f i c u l t y i n f o l l o w i n g through i n i t s a p p a r e n t l y i n t u i t i v e m i r r o r i n g of the Provus Discrepancy E v a l u a t i o n Model. The questions t h a t the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee developed were very s p e c i f i c and were c l e a r l y focussed upon determining whether or not there was compliance with B u l l e t i n CDB894. There was no : obvious b i a s i n the questions and the r e s u l t s were simple to i n t e r p r e t as they e s s e n t i a l l y o n l y r e q u i r e d a "yes" or "no" type answer. 79 There were major d i f f e r e n c e s between the manner i n which both committees proceeded with t h e i r e v a l u a t i o n s . These d i f f e r e n c e s may have developed due to the d i f f e r e n c e s i n t h e i r mandates as w e l l as the membership of each committee. In other words, d i f f e r e n t mandates may r e q u i r e d i f f e r e n t approaches. The members of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee were made up of a c r o s s s e c t i o n of s t a f f from the c o r p o r a t i o n and were l e d by a chairman with Head O f f i c e experience. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee c o n s i s t e d o f members of the c o r p o r a t i o n from a very s p e c i a l i z e d area w i t h f i e l d experience. The d i f f e r i n g . m a n d a t e s and p e r s p e c t i v e s c o u l d have i n f l u e n c e d the development of the two d i f f e r e n t approaches taken. Stage V - Programme Cost. No c o s t a n a l y s i s was made by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee but recommendations were advanced to examine the c o s t b e n e f i t s of m a i n t a i n i n g permanent in-house t r a i n i n g versus " j u s t i n time" t r a i n i n g which c o u l d be p r o v i d e d by c o n t r a c t i n g with o u t s i d e c o n s u l t a n t s . T h i s recommendation was developed d u r i n g the Claims T r a i n i n g E v a l u a t i o n Task Force Committee's wrap up s e s s i o n on May 31, 1994. I t , among other recommendations, was formulated through a s y n t h e s i s of trends and o b s e r v a t i o n s . T h i s recommendation was put . 80 forward as a s e r i o u s suggestion t h a t would r e q u i r e f u r t h e r development. While the Claims T r a i n i n g E v a l u a t i o n Task Force Committee recommended, a p p a r e n t l y as an'afterthought, t h a t a c o s t a n a l y s i s be conducted, t h i s i s a c t u a l l y a planned p a r t of the process as s e t out by Provus. I t i s not a process to be r e f e r r e d to and passed along to some f u t u r e committee or e v a l u a t i o n team but i s r a t h e r an i n t e g r a l p a r t of the e v a l u a t i o n p r o c e s s . An e v a l u a t i o n u s i n g the Provus Discrepancy E v a l u a t i o n Model i s not complete u n t i l the c o s t a n a l y s i s has been conducted. While a c o s t / b e n e f i t a n a l y s i s was not performed by the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, recommendations were made th a t one be completed i n order to "measure the e f f e c t i v e n e s s of p r o c e d u r a l change." These recommendations were made almost as- an a f t e r t h o u g h t d u r i n g the composition of the f i n a l r e p o r t . There was no evidence i n the committee minutes or my notes t h a t t h i s was an e a r l i e r c o n s i d e r a t i o n . As the author of the f i n a l r e p o r t , I added the c o s t / b e n e f i t a n a l y s i s recommendation to the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee report 1. I i n c l u d e d the c o s t / b e n e f i t recommendations as I was aware that t h i s was a necessary p a r t of an e v a l u a t i o n from p r e v i o u s t r a i n i n g . 81 Stufflebeam et a l . (1973) C.I.P.P. E v a l u a t i o n Model The Provus Discrepancy E v a l u a t i o n Model serves to i l l u s t r a t e the importance of d e f i n i n g c l e a r o b j e c t i v e s i n order to focus an e v a l u a t i o n . S i m i l a r l y , the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model e s t a b l i s h e s e v a l u a t i o n g u i d e l i n e s which underscore the need to have a c l e a r l y focussed e v a l u a t i o n by p l a c i n g the e v a l u a t i o n i n context and examining the input, processes and product of the e v a l u a t i o n (C.I.P.P.). Both the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee were examined from the p e r s p e c t i v e of the Stufflebeam et a l . Model to determine i f p a r a l l e l s e x i s t between theory and p r a c t i c e . Context E v a l u a t i o n . Stufflebeam et a l . , i n the C.I.P.P. E v a l u a t i o n Model, take the p o s i t i o n t h a t the f i r s t stage of the e v a l u a t i o n should be to p l a c e the o b j e c t being e v a l u a t e d i n c ontext. T h i s i n v o l v e s i d e n t i f y i n g and a s s e s s i n g needs w i t h i n the context of the e v a l u a t i o n and d e f i n i n g problems which underscore the needs. The Claims T r a i n i n g E v a l u a t i o n Task Force Committee d i d conduct an u n s t r u c t u r e d needs assessment through the focus group process, from May 06, 1994 to May 20, 1994 but d i d not p l a c e i t i n context as the a c t u a l programme o b j e c t s were not examined. The needs assessment was based upon examining the responses made d u r i n g the focus group process to i d e n t i f y . 82 those responses t h a t r e l a t e d d i r e c t l y to the needs of the respondents. The process undertaken by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee to determine needs c o u l d not be c o n s i d e r e d a formal needs assessment but i t does i n d i c a t e t h a t a needs assessment was c o n s i d e r e d . There were leading- questions to guide the focus groups but the format of the s e s s i o n s was f r e e f l o w i n g i n order to e s t a b l i s h a comfortable atmosphere and encourage c r e a t i v e thought. On the other hand, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee d i d i d e n t i f y the standards expected by examining B u l l e t i n CDB894 and p l a c e d them w i t h i n an o p e r a t i o n a l context. As B u l l e t i n CDB894 was a p u b l i s h e d document, i t saved the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee a c o n s i d e r a b l e amount of time i n t h a t i t was very c l e a r what was to be e v a l u a t e d and what the expected r e s u l t s would be. Input E v a l u a t i o n . The second stage, Input E v a l u a t i o n , i d e n t i f i e s and assesses system c a p a b i l i t i e s , a v a i l a b l e i n p u t s t r a t e g i e s , and designs f o r implementing the s t r a t e g i e s . In simple terms, the system i n which the o b j e c t being e v a l u a t e d and i s expected to operate i s examined to determine i f the e x i s t i n g i n f r a s t r u c t u r e i s capable of s u p p o r t i n g the programme being e v a l u a t e d . The i n f r a s t r u c t u r e i n c l u d e s l o g i s t i c a l i s s u e s d e a l i n g with p h y s i c a l needs as w e l l as s u p p o r t i n g programmes and o p e r a t i o n a l i s s u e s which c o n s i s t 83 of programme design and implementation s t r a t e g i e s . I f these are not a v a i l a b l e , then there i s l i t t l e p o i n t , i n c o n t i n u i n g the e v a l u a t i o n . I f the system i s i n c a p a b l e of a l l o w i n g f o r implementation to take p l a c e , then a c c o r d i n g to Stufflebeam et a l . , the e v a l u a t i o n would cease at t h i s p o i n t . The Claims T r a i n i n g E v a l u a t i o n Task Force Committee d i d assess the a v a i l a b i l i t y of resources and i d e n t i f i e d implementation s t r a t e g i e s by r e c o u n t i n g the t h e n - e x i s t i n g s t r u c t u r e and methods of d e l i v e r y of the Claims T r a i n i n g Department. This was accomplished at the o u t s e t of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee on May 5, 1994. As an example, i n an i n t e r v i e w on May 5, 1994 with a s e n i o r manager, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee captured the f o l l o w i n g comments. -We are i n a time of r e s t r a i n t - but don't l e t s t a f f i n g i s s u e s i n f l u e n c e where your recommendations are going. I f we need more s t a f f , we w i l l take i t under advisement. Don't t h i n k t h a t j u s t because i t r e q u i r e s a d d i t i o n a l r esources, you shouldn't recommend. Up to us to f i n d r e s o u r c e s - t o do i t . -To have two departments doing t e c h n i c a l t r a i n i n g i n i s o l a t i o n doesn't h e l p the o v e r a l l , e f f o r t . Should be a c l o s e r l i n k . Needs more i n t e g r a t i o n from t r a i n i n g p e r s p e c t i v e . These recommendations were process o r i e n t e d and d i d not r e f l e c t an e v a l u a t i o n of the Claims T r a i n i n g Department of t h a t time, but r a t h e r sought to f i l l i n gaps between s t a f f needs and d e l i v e r y . Three examples of the nine 84 recommendations from page 10 of the Claims T r a i n i n g Report i l l u s t r a t e the focus on r e d e s i g n as opposed to e v a l u a t i o n . 1. That Claims E d u c a t i o n S e r v i c e s become p a r t of the Claims D i v i s i o n as p a r t of a new department t i t l e d "Claims T r a i n i n g and Research". 2. That the Manager, Claims T r a i n i n g and Research, r e p o r t d i r e c t l y to the V i c e P r e s i d e n t , Claims. T h i s Department would be r e s p o n s i b l e f o r i n t e g r a t i n g and meeting a l l t r a i n i n g needs w i t h i n the Claims D i v i s i o n . 9. That each Claim Centre have a Claims O f f i c e T r a i n i n g L i a i s o n person, whose r e s p o n s i b i l i t i e s w i l l i n c l u d e the co- o r d i n a t i o n of the t r a i n i n g needs of a l l Work Groups w i t h i n the Claim Centre, p. 10 The Stufflebeam et a l . C.I.P.P. Model i s concerned with e v a l u a t i o n , not the r e - d e s i g n or r e - e n g i n e e r i n g of the o b j e c t b e i ng e v a l u a t e d . T h i s leads to a fundamental d i f f e r e n c e between the Claims T r a i n i n g E v a l u a t i o n Task Force Committee approach and Stufflebeam et a l . The Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model takes a c l i n i c a l approach and seeks to diagnose by e s t a b l i s h i n g whether or not the intended outcomes were achieved w i t h i n the context t h a t the programme being e v a l u a t e d operated. On the other hand, the 85 Claims T r a i n i n g E v a l u a t i o n Task Force Committee assumed that a problem e x i s t e d ( i . e . , t r a i n i n g not meeting needs) and sought to cure i t by d e v e l o p i n g recommendations f o r change which would r e s u l t i n a "new" t r a i n i n g department being designed. T h i s was complete a l l the way from what should be taught, to whom i t should be taught r i g h t up to the r e p o r t i n g s t r u c t u r e of the newly designed department. In the end, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee never d i d address whether or not the Claims T r a i n i n g Department had met i t s o b j e c t i v e s . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee a l s o conducted a loose system a n a l y s i s . As a l l members of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee had worked f o r the c o r p o r a t i o n and f u n c t i o n e d w i t h i n i t s c o r p o r a t e s t r u c t u r e f o r s e v e r a l years they were i n t i m a t e l y f a m i l i a r with the then e x i s t i n g c o r p o r a t e s t r u c t u r e . Whether or not the s t r u c t u r e was capable of s u p p o r t i n g the implementation of B u l l e t i n CDB894 was never d i s c u s s e d . T h i s absence of c o n s i d e r a t i o n would appear to run c o n t r a r y to the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model. I t may be, however, t h a t the c l o s e n e s s of the e v a l u a t o r s to the s i t u a t i o n b e i n g e v a l u a t e d c o u l d allow f o r some unspoken " g i v e n s " which are i m p l i c i t and may not be apparent to a n e u t r a l observer from o u t s i d e the Claims D i v i s i o n nor p o s s i b l e had the e v a l u a t i o n been conducted by 8 6 an e x t e r n a l e v a l u a t i o n team. This aspect of advantages/disadvantages of i n t e r n a l or e x t e r n a l e v a l u a t o r s i s not c o n s i d e r e d by e i t h e r the Provus or Stufflebeam et a l . E v a l u a t i o n Models. This i s i n t e r e s t i n g because i f the e v a l u a t i o n had been conducted by e v a l u a t o r s who d i d not have •an i n t i m a t e understanding of the s t r u c t u r e they would have had to complete a d e t a i l e d a n a l y s i s . Being f a m i l i a r . i s advantageous i n t h a t i t saves time but important i s s u e s c o u l d be overlooked. The Stufflebeam et a l . model, used i n a s t r i c t a p p l i c a t i o n , does not take i n t o c o n s i d e r a t i o n e x t e r n a l f a c t o r s which may have an impact upon the s u p p o r t i n g i n f r a s t r u c t u r e . Therefore, while the s t r u c t u r e may e x i s t to support implementation, implementation does not take p l a c e . The c a p a b i l i t y i s there, but circumstance does not allow i t to happen. •In the case of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, f o r example, while the s t r u c t u r e was s u f f i c i e n t to support the i n i t i a t i v e , there were e x t e r n a l f a c t o r s which competed f o r the time of the people expected to implement the programme. Consequently, non-compliance was encouraged due to a n a t u r a l p r i o r i t i z i n g of f u n c t i o n s . I f the programme was not g i v e n a h i g h p r o f i l e to emphasize- i t s h i g h p r i o r i t y , l i t t l e or no compliance c o u l d be expected. On the s u r f a c e , t h e r e f o r e , the s t r u c t u r e appeared adequate to those c l o s e to the s t r u c t u r e , but time and p o s i t i o n p r i o r i t y may have been 87 c o n t r i b u t i n g f a c t o r s to non-compliance. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee determined t h a t while the system was capable of s u p p o r t i n g the implementation of B u l l e t i n CDB894, i t was the concensus of the Task Force t h a t the implementation procedures were cumbersome and would encourage non-compliance. Below are three examples of t h i s p r e d i c t i o n which come from the L i t i ' g a t i o n Management E v a l u a t i o n Task Force Committee meeting minutes of October 19, 1993: - R a m i f i c a t i o n s of going over budget not i d e n t i f i e d / e x p l a i n e d to defence c o u n s e l / a d j u s t e r s - A c c o u n t a b i l i t y without a u t h o r i t y - Poor implementation or f o l l o w up Very l i t t l e time was spent i n t h i s area due to the f a m i l i a r i t y of the Task Force with the c o r p o r a t i o n ' s f i e l d s t a f f c a p a b i l i t i e s and support i n f r a s t r u c t u r e . In f a c t , there was o n l y a p a s s i n g mention of t h i s " a n a l y s i s " . As an e-mail to a committee member from a s t a f f member who was surveyed i l l u s t r a t e s : i n theory i t i s a great idea, but i n p r a c t i c e d i d n ' t work, perhaps t h i s i s because the corp. l e t the t h i n g drop t h a t i t never, got past the "working out the bug stage", maybe i f we i n s i s t e d i t be done on a l l f i l e s , and updated as the f i l e p rogressed, i t may have developed i n t o a good p l a n n i n g and c o s t c o n t r o l t o o l . (E-mail, 19th of October, 1993) In essence, as r e q u i r e d by the Input E v a l u a t i o n stage of Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model, the system was 88 capable of the implementation of B u l l e t i n CDB894 and s u s t a i n i n g i t , but, as Stufflebeam et a l . f a i l to take i n t o account, the l a c k o f commitment, follow-up and the p o t e n t i a l i n f l u e n c e of e x t e r n a l f a c t o r s l e d to non-compliance. T h i s f a i l u r e to take committment, followup and the i n f l u e n c e of e x t e r n a l f a c t o r s i n t o c o n s i d e r a t i o n i s a l i m i t a t i o n of the Stufflebeam et a l C.I.P.P. E v a l u a t i o n Model. Process E v a l u a t i o n . In the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model, Process E v a l u a t i o n i n v o l v e s the p r e d i c t i o n of d e f e c t s i n the p r o c e d u r a l design or implementation s t r a t e g i e s and maintains a r e c o r d of p r o c e d u r a l events and a c t i v i t i e s . The Claims T r a i n i n g E v a l u a t i o n Task Force Committee made no p r e d i c t i o n of programme de s i g n flaws. T h i s absence of p r e d i c t i o n i s a r e f l e c t i o n of the single-minded d i r e c t i o n t h a t the Claims T r a i n i n g E v a l u a t i o n Task Force Committee took i n assuming t h a t a needs assessment would i l l u m i n a t e what was or was not working with Claims T r a i n i n g . The meeting minutes and my f i e l d notes of A p r i l 8, 1994 r e f l e c t t h a t there was no evidence to support t h a t a p r e d i c t i o n of programme de s i g n flaws was c o n s i d e r e d . No m o n i t o r i n g of p o t e n t i a l p r o c e d u r a l b a r r i e r s took p l a c e ; t h e r e f o r e , the Claims T r a i n i n g E v a l u a t i o n Task Force Committee was b l i n d to a l t e r n a t e courses f o r the outcome, p o s i t i v e or negative, o f Claims T r a i n i n g . 89 On the other hand, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee d i d make p r e d i c t i o n s as to the success of compliance with B u l l e t i n CDB894 and s t a t e d a probable cause. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee meeting minutes of October 18, 1993 i n d i c a t e t h a t the d i s c u s s i o n s c e n t r e d upon w h y • B u l l e t i n CDB894 f a i l e d and l i s t e d the f o l l o w i n g d i r e c t quote of the p r e d i c t i o n s of why B u l l e t i n CDB894 f a i l e d i n the October 18, 1993 committee minutes. - drew b a t t l e l i n e s between a d j u s t e r s and defence counsel - Defence counsel f e l t too much emphasis on budget $$$'s - Defence c o u y s e l f e l t they would be l o c k e d i n t o budget when unable to p r e d i c t v a r i a b l e s . - R a m i f i c a t i o n s o f going over Budget not i d e n t i f i e d / e x p l a i n e d to Defence Counsel/ A d j u s t e r s - C o n f l i c t of message - C o n f l i c t as to when Budget/Planning process to take p l a c e , i . e . , b e f o r e or a f t e r Examinations f o r D i s c o v e r y . - A c c o u n t a b i l i t y without A u t h o r i t y - B u l l e t i n b e i n g i g n o r e d - U n r e a l i s t i c time frames - In-house Counsel exempt - No r a t i o n a l e f o r i s s u i n g b u l l e t i n - Defence Counsel (external) f e l t t h ere was a' hidden agenda - A l i e n a t e d Defence Councel from A d j u s t e r s - Process take focus away from f i l e r e s o l u t i o n - Poor implementation and/or follow-up - No c o n s i d e r a t i o n f o r other programs [Note:- the above t y p o g r a p h i c a l e r r o r s are as they appear i n the o r i g i n a l document] The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee f e l t t h a t there would be l i t t l e i f any compliance 90 with B u l l e t i n CDB894. As s t a t e d above, there was no r a t i o n a l e f o r i s s u i n g the b u l l e t i n and with c o n f l i c t i n g messages the procedures, were j u s t too cumbersome to be e f f e c t i v e . T h i s p r e d i c t i o n of non-compliance arose from a roundtable d i s c u s s i o n on October 18, 1993 of the e f f e c t i v e n e s s of B u l l e t i n CDB894, and r e l i e d h e a v i l y upon the experience of those committee members who had been i n v o l v e d with the c o r p o r a t i o n ' s success r a t e of s i m i l a r l y complex programmes. In the past, programmes had been developed by head o f f i c e departments f o r implementation i n the f i e l d . As w e l l c o n s t r u c t e d as they were, they o f t e n f a i l e d to have a f i e l d p e r s p e c t i v e or c o n s i d e r a t i o n of f i e l d p r i o r i t i e s . Given l a c k of follow-up, b u y - i n and ownership from the f i e l d s t a f f , these head o f f i c e programmes o f t e n r e c e i v e d a lower p r i o r i t y from the s t a f f whose time and demands were d r i v e n by s e r v i c i n g the p u b l i c and e n s u r i n g good s t a t i s t i c a l r e s u l t s . The head o f f i c e programmes had to compete wi t h the day to day b u s i n e s s demands wit h which the f i e l d s t a f f have to contend. In other words, the e x p e c t a t i o n t h a t s t a f f and defence counsel would comply wi t h the implementation of a budgeting process as o u t l i n e d i n B u l l e t i n CDB894 was compared to what was a c t u a l l y o c c u r r i n g i n the f i e l d . The c o n c l u s i o n was that B u l l e t i n CDB894 had f a i l e d , and a number of reasons f o r f a i l u r e , i n a d d i t i o n to those p r e d i c t e d , were i d e n t i f i e d and are r e f e r e n c e d on page 91 f i v e of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee's f i n a l r e p o r t and are l i s t e d i n Table 4 below. I n t e r e s t i n g l y , these complaints r e v o l v e around g e n e r a l working r e l a t i o n s h i p s r a t h e r than B u l l e t i n CDB894. The Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model assumes t h a t an adequate i n f r a - s t r u c t u r e must be i n p l a c e , but f a i l s to c o n s i d e r other f a c t o r s as i n d i c a t e d above. In t h i s case, the s t r u c t u r e was f i n e but other f a c t o r s prevented implementation. TABLE 4 B u l l e t i n CDB894 F a i l u r e : Reasons For F a i l u r e Defence Counsel B o d i l y I n j u r y A d j u s t e r s - A d j u s t e r f a i l i n g to p r o v i d e adequate i n i t i a l i n s t r u c t i o n s on assignment of f i l e s to defence c o u n s e l . - Defence counsel f a i l i n g to f o l l o w i n s t r u c t i o n s . - Defence counsel not f o c u s i n g upon e a r l y settlement o p p o r t u n i t i e s . - The constant changing of a d j u s t e r s a s s i g n e d to f i l e s . - r e s i s t a n c e from defence counsel to a g g r e s s i v e l y manage l i t i g a t e d f i l e s . - A d j u s t e r s making unreasonable demands upon defence counsel; chambers (Court) a p p l i c a t i o n s , J u n i o r counsel being a s s i g n e d to f i l e s . - Lack of communication with B.I. A d j u s t e r s ( B o d i l y I n j u r y A d j u s t e r s ) - Changing counsel d u r i n g • the course of l i t i g a t i o n . - Having to pay defence counsel to review the f i l e m a t e r i a l a f t e r a new counsel has been assig n e d due to changes w i t h i n the defence f i r m . - l a c k of communication with'defence c o u n s e l . 92 T h i s follow-up s e s s i o n r e l i e d upon p e r s o n a l anecdotes as w e l l as the s t a f f and defence counsel i n t e r v i e w s . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee p r e d i c t i o n and the s p e c i f i c i n t e r v i e w questions which were put to those i n t e r v i e w e d l e d to a very focussed e v a l u a t i o n which r e s u l t e d i n savings i n both time and e f f o r t . In essence, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee c l e a r l y d e f i n e d what needed to be evaluated, p r e d i c t e d what the outcome would be and then implemented an e v a l u a t i o n procedure to t e s t the p r e d i c t i o n . T h i s focus prevented a wasting of time and e f f o r t by a l l i n v o l v e d w i t h the e v a l u a t i o n . Product E v a l u a t i o n . The Claims T r a i n i n g E v a l u a t i o n Task Force Committee c o u l d make no comparison between a c t i v i t i e s recorded and a c t u a l programme o b j e c t i v e s because the a c t u a l programme o b j e c t i v e s were never examined or d e f i n e d . Rather, recommendations.were made based upon the assumption that an examination of the ' p e r s o n a l needs of the stak e h o l d e r s and r e c i p i e n t s of Claims T r a i n i n g would, c o n s t i t u t e an e v a l u a t i o n of the Claims T r a i n i n g Department. The o n l y measurement t h a t took p l a c e i n v o l v e d a comparison between outcomes and e x p e c t a t i o n s . The recommendations t h a t were made were based s o l e l y upon t h i s comparison. Based upon the assumption t h a t the c u r r e n t Claims T r a i n i n g Department was not e f f e c t i v e due to the l a c k .of s a t i s f a c t i o n 93 of needs, a new s t r u c t u r e was proposed. While the recommendations may or may not have been b e n e f i c i a l , the e v a l u a t i o n i t s e l f i s open to p r o c e d u r a l q u e s t i o n . The recommendations were made with the i n t e n t i o n t h a t f u t u r e committees would take over ownership of the problem w i t h r e s p e c t to implementation. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee went beyond making t h i s s i n g l e recommendation to recommend t h a t the whole l i t i g a t i o n management process be re-engineered. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, while r e c o g n i z i n g t h a t i t c o u l d not take on the f u l l r e s p o n s i b i l i t y f o r a complete overhaul, d i d c r e a t e sub-committees to look at s p e c i f i c aspects of the l i t i g a t i o n management processes t h a t had been i d e n t i f i e d as being h i g h p r i o r i t y i s s u e s . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, i n the f a l l of 1994, debated the scope of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee mandate. A f t e r some debate, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee decided to c r e a t e sub- committees to explore i s s u e s t h a t s u r f a c e d d u r i n g the e v a l u a t i o n which were important enough to be addressed but which were not d i r e c t l y t i e d to the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee's mandate. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee f i n a l r e p o r t (page 8) made recommendations t h a t s p e c i a l i z e d defence plans 94 be c r e a t e d as an o f f s h o o t of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee mandate. During the course of the committee's i n v e s t i g a t i o n i n t o l i t i g a t i o n f i l e management, the committee was asked to develop "canned defence p l a n s " . As a r e s u l t , a sub-committee was s t r u c k and a defence p l a n f o r f i b r o m y a l g i a (a c o n d i t i o n which i s being a s s o c i a t e d with t r a u m a t i c i n j u r y and r e c e i v i n g h i g h d o l l a r awards i n court) i s being developed. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee took on the r e s p o n s i b i l i t y of r e - e n g i n e e r i n g the l i t i g a t i o n f i l e h a n d l i n g process, d e v e l o p i n g both new and simpler forms and procedures. The changes were implemented i n a t e s t s i t u a t i o n and f o r m a l l y adopted corporate-wide. Regular follow-up was conducted t o make m o d i f i c a t i o n s where r e q u i r e d to ensure s i m p l i c i t y and compliance. N e i t h e r Provus nor Stufflebeam take i n t o account the e x t e r n a l i n f l u e n c e s or p e r s o n a l agendas which may impact an e v a l u a t i o n . By not a d d r e s s i n g these i s s u e s both models- are e x c l u d i n g the p o t e n t i a l impact t h a t e x t e r n a l i n f l u e n c e s may have upon the a p p l i c a t i o n of t h e i r e v a l u a t i o n models. Summary The r e s u l t s p resented i n Chapter Four i l l u s t r a t e the e f f e c t i v e n e s s of a p p l y i n g two t h e o r e t i c a l e v a l u a t i o n models to two e v a l u a t i o n s conducted i n a c o r p o r a t e s e t t i n g . The Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P'. E v a l u a t i o n Model were examined, stage by stage, and compared to the o p e r a t i o n a l stages of both the Claims 95 T r a i n i n g E v a l u a t i o n Task Force Committee and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee e v a l u a t i o n s . T h i s comparison r e v e a l e d t h a t the Claims T r a i n i n g E v a l u a t i o n Task Force Committee d i d not f o l l o w the p a t t e r n of e i t h e r t h e o r e t i c a l model. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee's e v a l u a t i o n was very s i m i l a r to those of Provus and Stufflebeam et a l . T h i s study a l s o i d e n t i f i e d gaps i n the e f f e c t i v e n e s s of both the e d u c a t i o n a l e v a l u a t i o n models t h a t c o u l d have had an impact on the outcome of the e v a l u a t i o n s had they been a c t u a l l y a p p l i e d . Chapter F i v e w i l l d i s c u s s the r e s u l t s of t h i s study and c o n s i d e r the b e n e f i t s of u s i n g formal e d u c a t i o n a l e v a l u a t i o n models. In p a r t i c u l a r , the b e n e f i t s of u s i n g the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.l.P.P. E v a l u a t i o n Model w i l l be examined w i t h i n the context of t h i s study. D i f f e r e n c e s and s i m i l a r i t i e s between the two t h e o r e t i c a l e d u c a t i o n a l e v a l u a t i o n models and the two corpora t e based e v a l u a t i o n s w i l l a l s o be e x p l o r e d . CHAPTER FIVE SUMMARY, CONCLUSIONS AND RECOMMENDATIONS 96 In Chapter Four, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee e v a l u a t i o n s were assessed w i t h i n the frameworks p r o v i d e d by the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model. There were d i f f e r e n c e s and s i m i l a r i t i e s between the t h e o r e t i c a l e v a l u a t i o n models and the manner i n which these two c o r p o r a t e based e v a l u a t i o n s were a p p l i e d . Chapter F i v e w i l l summarize the r e s u l t s of t h i s study, d i s c u s s the r e l e v a n c e of these f i n d i n g s i n the c o n c l u s i o n and make recommendations f o r f u r t h e r study. Two thoughts to c o n s i d e r when re a d i n g Chapter F i v e are: 1. How the two e v a l u a t i o n committees c o u l d have b e n e f i t e d from u s i n g an e v a l u a t i o n model such as e i t h e r the Provus Discrepancy E v a l u a t i o n Model or the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model. 2. There are concerns with r e s p e c t to what appears to be d e f i c i e n c i e s i n the p r a c t i c a l a p p l i c a t i o n of both Provus's and Stufflebeam et a l . ' s t h e o r e t i c a l models. Two p e r c e i v e d weaknesses t h a t have been i d e n t i f i e d are " S t a l l P o i n t s " and the "Impact of E x t e r n a l Forces on E v a l u a t i o n Model E f f e c t i v e n e s s . " 97 Summary . I d e n t i f i c a t i o n of Programme O b j e c t i v e s and P l a c i n g Them i n Context. The Provus Discrepancy E v a l u a t i o n Model r e q u i r e s t h a t the o b j e c t i v e s of the programme being e v a l u a t e d be c l e a r l y d e f i n e d . These programme o b j e c t i v e s , must be i d e n t i f i e d p r i o r • t o the e v a l u a t i o n proceeding- i n order t h a t the e v a l u a t o r s have no q u e s t i o n i n t h e i r minds as to what i t i s t h a t they are e v a l u a t i n g . T h i s i s necessary to p r o v i d e a proper focus on the e v a l u a t i o n . . I f the o b j e c t i v e s are u n c l e a r or m i s i n t e r p r e t e d , the e v a l u a t i o n w i l l be m i s d i r e c t e d and the.value of the r e s u l t s o f the e v a l u a t i o n would be q u e s t i o n a b l e . By p r o p e r l y a p p l y i n g the Provus Discrepancy E v a l u a t i o n Model an e v a l u a t i o n team would be able to concentrate upon determining whether or not the programme being e v a l u a t e d had achieved what i t - s e t . o u t to accomplish. For example, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee d i d not spend time s p e c i f i c a l l y i d e n t i f y i n g and c l a r i f y i n g the o b j e c t i v e s of claims t r a i n i n g . .This l e d the committee to make assumptions about claims t r a i n i n g and conduct i t s e v a l u a t i o n i n such a manner as to support or r e j e c t those assumptions. Given t h a t the assumptions made may or may not r e f l e c t the o b j e c t i v e s of claims t r a i n i n g , 98 the c o n c l u s i o n s reached by the e v a l u a t i o n are weak at best and h i g h l y s u b j e c t to c r i t i c i s m . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, on the other hand, was p r o v i d e d with s p e c i f i c w r i t t e n o b j e c t i v e s with r e s p e c t to determining compliance with B u l l e t i n CDB894. T h e • o b j e c t i v e s were very c l e a r and the e v a l u a t i o n was focussed upon whether or not there was compliance. The r e s u l t s of the e v a l u a t i o n were compared to the o r i g i n a l o b j e c t i v e s and there can be a measure of confidence with the c o n c l u s i o n s . The i n t e r e s t i n g p o i n t to note w i t h the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee i s t h a t no formal e v a l u a t i o n model was a p p l i e d and i t appears t h a t t h i s committee, by circumstance, " f e l l i n t o " a process t h a t was s i m i l a r to a formal e v a l u a t i o n model. The Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model p l a c e s the e v a l u a t i o n i n an o p e r a t i o n a l context and assesses needs w i t h i n t h a t context. By p l a c i n g the e v a l u a t i o n i n context, i t i s p o s s i b l e to compare a c t u a l outcomes wi t h intended outcomes. An e v a l u a t i o n team u t i l i z i n g the Stufflebeam et a l . "Context E v a l u a t i o n " stage w i l l not o n l y be able to i d e n t i f y c l e a r l y the programme o b j e c t i v e s as suggested by Provus, but w i l l a l s o be able to p l a c e the design o b j e c t i v e s w i t h i n a c o n t e x t u a l framework which would d e l i n e a t e the o b j e c t i v e s from the outcomes. This w i l l a llow the e v a l u a t i o n team to determine not o n l y what the o b j e c t i v e s 99 were, but whether they are synchronous with the i n t e n t o f the programme. For example, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee d i d not c l e a r l y i d e n t i f y the o b j e c t i v e s of the• Claims T r a i n i n g Department nor d i d i t p l a c e the programme i n the context w i t h i n which i t had to operate. There was no examination of the i n t e n t of the programme. This l a c k of c l e a r o b j e c t i v e s and context l e d to what appeared t o be erroneous assumptions being made about the nature o f , t h e committee's mandate and subsequently, a m i s d i r e c t i o n of the . e v a l u a t i o n . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, on the other hand, was pr o v i d e d ' w i t h c l e a r o b j e c t i v e s . The committee members were aware t h a t compliance or non-compliance with B u l l e t i n CDB894 had f i n a n c i a l i m p l i c a t i o n s as B u l l e t i n CDB894 was a co r p o r a t e d i r e c t i v e t h a t set out a procedure f o r the l i t i g a t i o n management p r o c e s s . Again, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, a p p a r e n t l y i n t u i t i v e l y , f o l l o w e d an e v a l u a t i o n process without a p p l y i n g a formal e v a l u a t i o n model. Examination of Programme S t r u c t u r e and Supporting I n f r a s t r u c t u r e . By examining the s t r u c t u r e of the programme b e i n g e v a l u a t e d i n terms of i t s o p e r a t i o n a l context i n comparison to the programme de s i g n c r i t e r i a , i t 100 i s p o s s i b l e to determine whether or not the programme i n f r a s t r u c t u r e i s capable of s u p p o r t i n g the programme. Stufflebeam et a l . a s s i s t the e v a l u a t i o n team by h i g h l i g h t i n g t h i s as a "Process E v a l u a t i o n " stage. The. b e n e f i t s of c l e a r l y d e f i n i n g the programme implementation s t r u c t u r e are twofold. F i r s t l y , as Stufflebeam et a l . suggest, i f the s u p p o r t i n g i n f r a s t r u c t u r e i s found to be l o g i s t i c a l l y inadequate, the e v a l u a t i o n team need not proceed f u r t h e r . T h i s c o n c l u s i o n c o u l d p o t e n t i a l l y l e a d to the i n f r a s t r u c t u r e b e i n g m o d i f i e d to ensure adequate resources i n order to allow the programme to proceed and enhance the p o s s i b i l i t y of the o b j e c t i v e s having an o p p o r t u n i t y to be developed. Secondly, as Provus i n d i c a t e s , the programme o p e r a t i o n i s compared to the standard o f the programme de s i g n to ensure t h a t there i s congruence•with the de s i g n c r i t e r i a . In t h i s case, the s u p p o r t i n g i n f r a s t r u c t u r e may be l o g i s t i c a l l y adequate, but i f the design of the s u p p o r t i n g i n f r a s t r u c t u r e i s such t h a t i t causes the programme to s t r a y from the i n t e n t of the o b j e c t i v e s then once again the e v a l u a t i o n should cease at t h i s p o i n t . The programme should be terminated or at the very l e a s t , the i n f r a s t r u c t u r e should be m o d i f i e d to allow the de s i g n c r i t e r i a to be supported. 101 A n t i c i p a t i o n of B a r r i e r s to Success. As Stufflebeam r • et a l . purport, i t i s an important f e a t u r e of e v a l u a t i o n to i examine and a n t i c i p a t e what may be b a r r i e r s to success and where they may l i e w i t h i n the e v a l u a t i o n p r o c e s s . The b e n e f i t here f o r an e v a l u a t i o n team i s t h a t without l i m i t i n g the scope of the e v a l u a t i o n i t d i r e c t s the e v a l u a t o r s towards l i k e l y problem areas t h a t , i f confirmed, would a l l o w the e v a l u a t i o n to be terminated. In other words, the whole e v a l u a t i o n process c o u l d be shortened. For example, the Claims T r a i n i n g E v a l u a t i o n . Task Force Committee d i d not make an attempt to i d e n t i f y p o t e n t i a l b a r r i e r s and as a r e s u l t , t h e i r e f f o r t s were spread out i n a number of d i r e c t i o n s . The end r e s u l t was a very broad approach to the e v a l u a t i o n i n order to determine whether or not the programme was s u c c e s s f u l . On the other hand, the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee d i d a n t i c i p a t e b a r r i e r s to the programme being e v a l u a t e d and t h i s l e d the e v a l u a t i o n team to focus i t s a t t e n t i o n upon the p e r c e i v e d b a r r i e r and e i t h e r prove or d i s p r o v e i t s theory. The theory, i n t h i s case, was proved and the whole e v a l u a t i o n ' p r o c e s s was shortened. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee was able to make more e f f i c i e n t use of time, resources and money by s h o r t e n i n g the e v a l u a t i o n process through the 102 a n t i c i p a t i o n and v e r i f i c a t i o n of p o t e n t i a l b a r r i e r s to the success of the programme being e v a l u a t e d . O r i e n t a t i o n of Programme Towards R e s u l t s . Both the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model are r e s u l t s o r i e n t e d . Provus examines the r e s u l t s i n r e l a t i o n to the s t a t e d programme o b j e c t i v e s . R e s u l t s were important to both the Claims T r a i n i n g E v a l u a t i o n Task Force and L i t i g a t i o n Management E v a l u a t i o n Task Force Committees. The r e s u l t s of. the Claims T r a i n i n g E v a l u a t i o n Task Force Committee were the product of an e v a l u a t i o n based upon assumptions of q u e s t i o n a b l e value and are t h e r e f o r e themselves q u e s t i o n a b l e . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee r e s u l t s c o u l d be compared with s p e c i f i c o b j e c t i v e s and would add value to the e v a l u a t i o n . The process, once again, was due more to l u c k and circumstance than to d e s i g n . .. S i m i l a r l y , Stufflebeam-et a l . examine the r e s u l t s from a c o n t e x t u a l p e r s p e c t i v e . In other words, does the programme achieve what i t s t a t e s t h a t i t set out to accomplish? T h i s r e f l e c t s upon the need to have c l e a r l y d e f i n e d o b j e c t i v e s to determine the success of the r e s u l t s . C onclusions B e n e f i t s of u s i n g Provus and Stufflebeam et a l . E v a l u a t i o n Models. Both committees c o u l d have enjoyed 103 d e f i n i t e b e n e f i t s had they used e i t h e r the Provus or Stufflebeam et a l . E v a l u a t i o n Models. Both committees would have r e a l i z e d b e n e f i t s simply by the a p p l i c a t i o n of any e v a l u a t i o n model, r e g a r d l e s s of i t s type as both committees l a c k e d a formal e v a l u a t i o n s t r u c t u r e and seemingly proceeded through t h e i r e v a l u a t i o n s by t r i a l and e r r o r . The Claims T r a i n i n g E v a l u a t i o n Task Force Committee, i n p a r t i c u l a r , s t r a y e d from formal e v a l u a t i o n p r i n c i p l e s and the b a s i s f o r i t s c o n c l u s i o n i s weak, which b r i n g s the r e s u l t s i n t o q u e s t i o n . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, a p p a r e n t l y i n t u i t i v e l y , f o l l o w e d a process t h a t was s i m i l a r to formal e v a l u a t i o n and i t s c o n c l u s i o n s are ' seemingly w e l l founded and s o l i d . While the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee d i d not use a.formal e v a l u a t i o n process, and seemed to a r r i v e at a s i m i l a r outcome to what c o u l d have been expected had a formal process been used, i t was more l i k e l y due to l u c k and circumstance than due to d e s i g n . A formal e v a l u a t i o n process would have p r o v i d e d a s o l i d framework t h a t would have minimized the L i t i g a t i o n Management E v a l u a t i o n Task Force committee's r e l i a n c e upon l u c k . Some of the b e n e f i t s t h a t the Claims T r a i n i n g E v a l u a t i o n Task-Force Committee and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee c o u l d have 104 r e a l i z e d from u t i l i z i n g the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model are: i d e n t i f i c a t i o n . o f programme o b j e c t i v e s and p l a c i n g them i n context, examination of programme s t r u c t u r e and s u p p o r t i n g i n f r a s t r u c t u r e , a n t i c i p a t i o n of b a r r i e r s to success, o r i e n t a t i o n of programme towards r e s u l t s and f i n a n c i a l comparison f o r . c o s t containment. N e i t h e r the Claims T r a i n i n g E v a l u a t i o n Task Force Committee nor the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee f o r m a l l y examined the s u p p o r t i n g i n f r a s t r u c t u r e as suggested by Provus and Stufflebeam et a l ; T h i s important stage of the formal e v a l u a t i o n process was given o n l y a p a s s i n g c o n s i d e r a t i o n by the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee (presumably because there was an i n h e r e n t f a m i l i a r i t y between the committee members and the i n f r a s t r u c t u r e w i t h i n which they worked on a d a i l y b a s i s ) and the. Claims T r a i n i n g E v a l u a t i o n Task Force Committee d i d not c o n s i d e r i t at a l l . Both committees would have b e n e f i t e d from u s i n g formal e v a l u a t i o n processes, such as the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model, which would have examined t h i s stage i n d e t a i l . Had. t h i s stage been f o r m a l l y examined i t may have r e v e a l e d whether or not there were other f a c t o r s which were 105 not c o n s i d e r e d t h a t c o u l d have i n h i b i t e d the e f f e c t i v e n e s s of e i t h e r programme. For e v a l u a t i o n teams u s i n g e i t h e r model, the importance of examining the r e s u l t s i n r e l a t i o n to the o b j e c t i v e s i s underscored by the outcomes of both the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee. The Claims T r a i n i n g E v a l u a t i o n Task Force Committee d i d not c l e a r l y i d e n t i f y the programme o b j e c t i v e s nor d i d i t . r e f e r e n c e i t s c o n c l u s i o n s with o b j e c t i v e s and the r e s u l t s were recommendations which have q u e s t i o n a b l e value i n r e l a t i o n to the success or f a i l u r e of the programme t h a t was e v a l u a t e d . Future c o r p o r a t e plans may r e l y upon those recommendations t h a t c o u l d have p o t e n t i a l l y n e g a t i v e r e s u l t s . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, on the other hand, had a c l e a r l y d e f i n e d o b j e c t i v e and compared the r e s u l t s d i r e c t l y a g a i n s t t h a t o b j e c t i v e . I t s c o n c l u s i o n with r e s p e c t to the success or f a i l u r e of the programme i t e v a l u a t e d was t h e r e f o r e b e t t e r grounded than that of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. I t i s evident t h a t f o r an e v a l u a t i o n to be e f f e c t i v e the importance of having c l e a r l y d e f i n e d o b j e c t i v e s , which i s i d e n t i f i e d by Provus and Stufflebeam et a l . as a c r u c i a l element of a s u c c e s s f u l e v a l u a t i o n , must be s t r e s s e d . The 106 comparison of the r e s u l t s achieved w i t h the s t a t e d o b j e c t i v e s i s a l s o a c r i t i c a l component i n determining' the accuracy of the e v a l u a t i o n ' s c o n c l u s i o n s . F i n a n c i a l Comparison For Cost Containment. Both Provus and Stufflebeam et a l . have, as f i n a l elements of t h e i r models, a component d e a l i n g with c o s t a n a l y s i s . They, are r e l a t i v e l y s i m i l a r w i t h r e s p e c t to comparing the c o s t of the programme being e v a l u a t e d w i t h i n d u s t r y standards or s i m i l a r programmes. T h i s process may be termed "benchmarking". I t i s important i n t h a t r e g a r d l e s s of the o r i g i n s of the programme being evaluated, b u s i n e s s d e c i s i o n s have to be made with r e s p e c t to c o n t i n u i n g or t e r m i n a t i n g programmes. In many cases, the b a s i s f o r c o n t i n u i n g or t e r m i n a t i n g a programme i s f i n a n c i a l . Both the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee c o n s i d e r e d but d i d not conduct a f i n a n c i a l a n a l y s i s . Both committees recommended t h a t a f i n a n c i a l a n a l y s i s be conducted. The problem here, though, i s t h a t i f a f u t u r e f i n a n c i a l a n a l y s i s were conducted, i t would l i k e l y not i n v o l v e the o r i g i n a l committee members and there would be a l o s s of c o n t i n u i t y . Other p o t e n t i a l problems with both committees are a l a c k of completeness to the e v a l u a t i o n and a l a c k of 107 a c c o u n t a b i l i t y f o r the implementation of recommendations th a t are made without a f i n a n c i a l a n a l y s i s . For e v a l u a t i o n teams, there i s a r e a l b e n e f i t t h a t may be had by f o l l o w i n g e i t h e r the Provus or the Stufflebeam et a l . E v a l u a t i o n Models with r e s p e c t to c o s t a n a l y s i s . ' The models emphasize the importance of f i n a n c i a l concerns t h a t • are addressed at the c o n c l u s i o n of the e v a l u a t i o n . p r o c e s s . The p l a c i n g of a f i n a n c i a l assessment at the end of an e v a l u a t i o n i l l u s t r a t e s t h a t r e g a r d l e s s . o f the success or f a i l u r e of a programme through a comparison of o b j e c t i v e s and r e s u l t s , the. d e c i s i o n to continue or terminate a programme may be a f i n a n c i a l one. I t i s the e v a l u a t i o n team's re s p o n s i b i l i t y , - * u n l e s s otherwise d i r e c t e d , to complete the e v a l u a t i o n by conducting a f i n a n c i a l a n a l y s i s . The e v a l u a t i o n team i s i n a good p o s i t i o n to conduct such an e v a l u a t i o n or give input to a f i n a n c i a l e v a l u a t i o n committee a f t e r having conducted an in-dept h programme e v a l u a t i o n . F a i l i n g to conduct a f i n a n c i a l a n a l y s i s , as was the case with both the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and.the L i t i g a t i o n .Management E v a l u a t i o n Task Force Committee, leaves an e v a l u a t i o n incomplete.. Committees' E f f o r t s . F a c i l i t a t e d by u s i n g a Formal E v a l u a t i o n Model Some of the ways i n .which the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and the L i t i g a t i o n 108 Management E v a l u a t i o n Task Force Committee c o u l d have f a c i l i t a t e d t h e i r e f f o r t s by u s i n g a formal e v a l u a t i o n model are t h a t a formal e v a l u a t i o n model p r o v i d e s s t r u c t u r e , suggests methodology, helps with d e t e r m i n a t i o n of r e s u l t s and a i d s with the making of recommendations. Provides S t r u c t u r e . Both committees c o u l d have f a c i l i t a t e d the o p e r a t i o n of t h e i r committees and i n c r e a s e d e f f i c i e n c i e s by u s i n g a formal e v a l u a t i o n model such as the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model. The f i r s t b e n e f i t t h a t i s evident i s t h a t a formal e v a l u a t i o n model p r o v i d e s a s t r u c t u r e and d i r e c t i o n f o r an e v a l u a t i o n . By mapping out a p l a n f o r how an e v a l u a t i o n w i l l take p l a c e , the e v a l u a t o r s would be b e t t e r able to focus upon the c r u c i a l elements of the e v a l u a t i o n without d i s t r a c t i o n , m i s p l a c e d resources and wasted e f f o r t . Wandering through an e v a l u a t i o n by d e s i g n i n g i t "on the go" i s i n e f f i c i e n t , r i s k y and leaves both the process and the r e s u l t s open to c r i t i c i s m and c h a l l e n g e . For example, n e i t h e r the Claims T r a i n i n g E v a l u a t i o n Task Force Committee nor the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee used a formal e v a l u a t i o n model. Had a formal e v a l u a t i o n model such as the Provus Discrepancy E v a l u a t i o n Model or the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model been used, i t would have been apparent at the ou t s e t t h a t the o b j e c t i v e s of the programme 109 must be c l e a r l y i d e n t i f i e d . F a i l i n g to do t h i s l e d the Claims T r a i n i n g E v a l u a t i o n Task Force Committee to s t r a y from e v a l u a t i n g the programme towards e v a l u a t i n g the s u i t a b i l i t y of the product of the programme, and making recommendations f o r product change. The committee d i d not address whether the programme, as designed, was s u c c e s s f u l i n accomplishing what i t intended to do g i v e n the o r i g i n a l o b j e c t i v e s . Using a formal e v a l u a t i o n model c o u l d have prevented t h i s s i t u a t i o n from a r i s i n g and p r o v i d e d a focus to the e v a l u a t i o n . In a l i k e f a s h i o n , the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee c o u l d have had a s i m i l a r b e n e f i t from u s i n g a formal e v a l u a t i o n model. While the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee seemingly f o l l o w e d the general o u t l i n e of a formal e v a l u a t i o n , i t d i d so on what appears to have been an i n t u i t i v e l e v e l as opposed to a conscious e f f o r t . There was never a mention of f o l l o w i n g a s p e c i f i c e v a l u a t i o n p l a n or even s e t t i n g out a; formal process p r i o r to the e v a l u a t i o n commencing. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee seems to have been " l u c k y " . The mandate of the committee was very simple and s p e c i f i c with the o b j e c t i v e s of the programme being p u b l i s h e d as B u l l e t i n CDB894. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee may have had s i m i l a r d i f f i c u l t i e s to the Claims 110 T r a i n i n g E v a l u a t i o n Task. Force Committee i f the o b j e c t i v e s had not been s p e c i f i e d i n B u l l e t i n CDB894. A formal e v a l u a t i o n model would have guided the e v a l u a t i o n process i n an o r g a n i z e d f a s h i o n without r e l y i n g upon i n t u i t i o n as the r u l e . Methodology. This study suggests t h a t the r e s u l t s of an e v a l u a t i o n can be brought i n t o q u e s t i o n and p o t e n t i a l l y rendered of no value i f the methodology of the e v a l u a t i o n can be shown to be flawed. In other words, evidence of the e x i s t e n c e of an accept a b l e e v a l u a t i o n methodology i s a c r u c i a l element of the e v a l u a t i o n p r o c e s s . The more sound and accepted the methodology i s , the more confidence t h a t may be given to the r e s u l t s . I t must be remembered t h a t the second p a r t of methodology i s not j u s t i t s process, but a l s o i t s a p p l i c a t i o n w i t h i n a p a r t i c u l a r context. Using a formal e v a l u a t i o n model would have helped both the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and L i t i g a t i o n Management E v a l u a t i o n Task Force Committee e v a l u a t i o n s over p a r t of t h i s m e t h o d o l o g i c a l h u r d l e . In p a r t i c u l a r , the Claims T r a i n i n g E v a l u a t i o n Task Force Committee s u f f e r e d f o r not having used an accepted methodology. For example, d u r i n g the p r e s e n t a t i o n of r e s u l t s on June 6, 1994, I was the chairman of the p r e s e n t a t i o n committee. The Claims T r a i n i n g E v a l u a t i o n Task Force Committee f i n a l r e p o r t was c h a l l e n g e d upon i t s I l l methodology with the i n t e n t . o f undermining the recommendations by e s t a b l i s h i n g t h a t they were of no value due to a f a u l t y m e t h o d o l o g i c a l process, and while I was unable to r e c o r d the questions asked, my r e c o l l e c t i o n and comments i n an.e-mail to other committee members post p r e s e n t a t i o n r e f l e c t t h a t i t was a very d i r e c t c h a l l e n g e . In my r o l e as chairman, i t f e l l upon me to defend the e v a l u a t i o n . I t was a d i f f i c u l t process as the methodology used was l o g i c a l g i v e n the approach t h a t the Claims T r a i n i n g E v a l u a t i o n Task Force Committee took, but i t was not sound i n the face of formal e v a l u a t i o n models. C r i t i c s a n a l y z i n g the Claims T r a i n i n g E v a l u a t i o n Task Force Committee e v a l u a t i o n a g a i n s t the backdrop of the Provus and Stufflebeam et a l . E v a l u a t i o n Models should have been able to p u l l i t a p a r t . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, on the other hand, was working w i t h a l e s s complex e v a l u a t i o n . That i s , i t s mandate was to determine whether there was compliance w i t h B u l l e t i n CDB894. While i t d i d not u t i l i z e a formal e v a l u a t i o n process, the e v a l u a t i o n process i t d i d use was both l o g i c a l and s i m i l a r to what might have been used had a formal e v a l u a t i o n model been co n s i d e r e d . I t begs the q u e s t i o n as to whether or not the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee i n t u i t i v e l y used a method s i m i l a r to a formal e v a l u a t i o n 112 process or whether they f e l l i n t o i t , as there was r e a l l y no other l o g i c a l method of d e a l i n g with the i s s u e at hand. I t was p o s s i b l e t h a t the experience l e v e l and t e c h n i c a l e x p e r t i s e of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee members l e d the committee to approach the e v a l u a t i o n i n a s i m i l a r f a s h i o n to problems and events t h a t are d e a l t with on a d a i l y b a s i s . That i s , there c o u l d be a g e n e r a l a p p l i c a t i o n of e x i s t i n g b u s i n e s s coping s k i l l s , coupled with common sense, and " l u c k " which l e d to the success of L i t i g a t i o n Management E v a l u a t i o n Task Force e v a l u a t i o n . The " l u c k " component i s d i f f i c u l t to measure or e x p l a i n but the nature of the c o r p o r a t i o n ' s b u s i n e s s i s one t h a t focuses upon r i s k t a k i n g . Risk t a k i n g r e l i e s upon working very hard at being " l u c k y " . E v a l u a t i o n committees can l e a r n from t h i s examination of methodology. S i m i l a r to the importance of c l e a r l y d e f i n i n g the o b j e c t i v e s of the programme b e i n g evaluated, the soundness of the methodology used i s extremely important i n determining the success of the whole e v a l u a t i o n p r o c e s s . The methodology s e c t i o n i s the underpinning of the e v a l u a t i o n . Determination of R e s u l t s . Both the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and L i t i g a t i o n Management E v a l u a t i o n Task Force Committee c o u l d have b e n e f i t e d by the r e a l i z a t i o n t h a t w h ile i t i s necessary to be r e s u l t s 113 o r i e n t e d , the focus of the e v a l u a t i o n should not be upon the r e s u l t s . The r e s u l t s are what they are and i f the e v a l u a t i o n process i s sound the r e s u l t s w i l l n a t u r a l l y flow from the e v a l u a t i o n . In other words, i f the o b j e c t i v e s are c l e a r and the e v a l u a t i o n methodology i s unshakeable, the r e s u l t s w i l l be able to withstand c r i t i c i s m . I f there i s a weakness i n an e v a l u a t i o n i t w i l l be i n the e v a l u a t i o n process i t s e l f as opposed to the r e s u l t s . Recommendations. Recommendations are made by • examining the r e s u l t s and r e f e r e n c i n g the o r i g i n a l mandate. Both the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee made recommendations. The Claims T r a i n i n g E v a l u a t i o n Task Force Committee recommendations were broad i n nature and flowed from a c o m p i l a t i o n of the r e s u l t s of the focus groups. There was a g e n t l e anonymity f o r the Claims T r a i n i n g E v a l u a t i o n Task Force Committee members as the recommendations c o u l d be shown to be r e f l e c t i o n s of the w i l l of a l a r g e r group. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, on the other hand, made s p e c i f i c recommendations upon an i n t e r p r e t a t i o n of the r e s u l t s of the e v a l u a t i o n . T h i s committee took f u l l r e s p o n s i b i l i t y f o r i t s .114 recommendations. • The s t r e n g t h ' o f i t s recommendations came from d i r e c t l y a d d r e s s i n g i t s o b j e c t i v e s . In t h i s study, the recommendations of each e v a l u a t i o n committee were p r o v i d e d to the " D e c i s i o n Makers" who gave the e v a l u a t i o n committees t h e i r o r i g i n a l mandates. Whether or not the recommendations were "weighted" or implemented was beyond the scope of the e v a l u a t i o n committees and was s t r i c t l y a d e c i s i o n made by s e n i o r management. They d i d , however, form a v a l u a b l e p i e c e of the p u z z l e f o r the d e c i s i o n makers, as would be expected, and the more confidence t h a t c o u l d be had i n the e v a l u a t i o n process, the more weight t h a t c o u l d be a f f o r d e d to the recommendations. The end r e s u l t , though, wi t h r e s p e c t to implementation or a c t i o n upon recommendations, c o u l d be i n f l u e n c e d by f a c t o r s which may or may not be known to the e v a l u a t i o n committee. As i s ev i d e n t i n both the Provus and Stufflebeam et a l . E v a l u a t i o n Models, the process ends f o r the e v a l u a t i o n committee wi t h a f i n a n c i a l a n a l y s i s . T h i s b r i n g s the e v a l u a t i o n process to a c l o s e from the p e r s p e c t i v e of the e v a l u a t i o n committees, but i m p l i e s t h a t any f u r t h e r e v a l u a t i o n w i l l be conducted by the people r e s p o n s i b l e f o r commissioning the e v a l u a t i o n . In the two e v a l u a t i o n s at hand, very few of the recommendations made by the Claims T r a i n i n g E v a l u a t i o n Task Force Committee were implemented, whereas the' 115 recommendations of the L i t i g a t i o n Management E v a l u a t i o n Task- Force Committee were implemented i n t h e i r e n t i r e t y . T h i s may have been a r e f l e c t i o n upon the confidence t h a t the d e c i s i o n makers had i n the e v a l u a t i o n s themselves, or due to other f a c t o r s . In any case, however, n e i t h e r of the committees was p r i v y to the d e c i s i o n m a k e r s ' . r a t i o n a l e . Developing Foundations of E v a l u a t i o n T h i s study r e v e a l s t h a t e v a l u a t i o n s should c o n s i d e r the d i s t r i b u t i o n of time and e f f o r t i n conducting an e v a l u a t i o n . I t a l s o suggests t h a t there should be a g r e a t e r emphasis on the s e t t i n g up of. an e v a l u a t i o n r a t h e r than upon the d e t e r m i n a t i o n of r e s u l t s . In other words, as Provus and Stufflebeam et a l . ' suggest, o b j e c t i v e c l a r i f i c a t i o n should be the f i r s t step i n any e v a l u a t i o n . Consequently, o b j e c t i v e c l a r i f i c a t i o n should be of primary importance as i t l a y s the foundation- f o r the balance of an e v a l u a t i o n . T h i s i s r e f l e c t e d i n the e v a l u a t i o n s conducted by both the Claims T r a i n i n g E v a l u a t i o n Task Force Committee and . L i t i g a t i o n Management E v a l u a t i o n Task Force Committee. Both the Provus Discrepancy E v a l u a t i o n Model and the Stufflebeam et a l . C.I.P.P. E v a l u a t i o n Model emphasize the need to develop c l e a r o b j e c t i v e s at the o u t s e t of an e v a l u a t i o n . Without a c l e a r focus at the b e g i n n i n g of an e v a l u a t i o n , the balance of the e v a l u a t i o n i s weakened. This concept i s demonstrated i n both the Claims T r a i n i n g 116 E v a l u a t i o n Task Force and L i t i g a t i o n Management E v a l u a t i o n Task Force e v a l u a t i o n s . The Claims T r a i n i n g E v a l u a t i o n Task Force Committee d i d not have c l e a r l y d e f i n e d o b j e c t i v e s and the r e s u l t s were q u e s t i o n a b l e . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, on the other hand, had a s p e c i f i c o b j e c t i v e and the r e s u l t s were s o l i d as they c o u l d be compared d i r e c t l y to the o b j e c t i v e s . The importance of having a s o l i d f oundation to an e v a l u a t i o n by having c l e a r l y d e f i n e d o b j e c t i v e s cannot be understated. T h i s study,based on two cases, seems to c o n f i r m the Provus and Stufflebeam et a l . a s s e r t i o n t h a t the g r e a t e s t emphasis i n an e d u c a t i o n a l e v a l u a t i o n should be upon o b j e c t i v e development and the methodological s t r u c t u r e . By extension, t h i s c o u l d be a p p l i e d to a l l e v a l u a t i o n s as demonstrated by the two c o r p o r a t e e v a l u a t i o n s examined i n t h i s study. The e v a l u a t i o n pyramid i n F i g u r e 1 demonstrates the p r o p o r t i o n a l importance and amount of e f f o r t t h a t should be expended i n the development of an e v a l u a t i o n . O b j e c t i v e c l a r i f i c a t i o n forms the foundation of an e v a l u a t i o n . The formation of a m e t h o d o l o g i c a l process should b u i l d upon the o b j e c t i v e s and so on through the e v a l u a t i o n h i e r a r c h y . T h i s i s s i m i l a r to the emphasis t h a t the Provus and Stufflebeam et a l . E v a l u a t i o n Models p l a c e on o b j e c t i v e development. I f 117 there i s a weakness at any l e v e l , then e v e r y t h i n g above t h a t l e v e l would see t h a t weakness mag n i f i e d . As t h a t weakness becomes magnified, i t flows a l l the way through the balance of the e v a l u a t i o n and renders each hig h e r l e v e l suspect and open to c r i t i c i s m . The re v e r s e i s a l s o t r u e ; the more s o l i d the lower l e v e l s are, the more confidence t h a t can be had i n the higher l e v e l s . F i g u r e 1 below i l l u s t r a t e s my p o s i t i o n with r e s p e c t to t h i s p r i n c i p l e as a p p l i e d to the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee. The e v a l u a t i o n pyramid c o u l d be a p p l i e d to both the Claims T r a i n i n g E v a l u a t i o n Task Force and L i t i g a t i o n Management E v a l u a t i o n Task Force Committees. Each committee passed through the v a r i o u s l e v e l s of the pyramid i n ascending order, but with a d i f f e r e n t emphasis f o r each committee. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee e v a l u a t i o n proceeded through the l e v e l s of the e v a l u a t i o n pyramid as o u t l i n e d i n Fig u r e 1 below. As suggested by the Provus and Stufflebeam et a l . E v a l u a t i o n Models the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee p l a c e d a g r e a t e r emphasis upon o b j e c t i v e c l a r i f i c a t i o n , methodology and so on up through the pyramid, which r e s u l t e d i n a g r e a t e r emphasis upon the f a c t based foundation and p r o v i d e d a s o l i d b a s i s f o r i n t e r p r e t i n g r e s u l t s and making recommendations. 118 F i g u r e 1. L i t i g a t i o n Management E v a l u a t i o n Task Force Committee Weighting of E v a l u a t i o n Processes. The amount of weight p l a c e d upon each of the e v a l u a t i o n stages i s i l l u s t r a t e d by the p r o p o r t i o n a l b l o c k s of the pyramid. The g r e a t e s t emphasis was p l a c e d upon o b j e c t i v e c l a r i f i c a t i o n which formed the foundation of t h i s e v a l u a t i o n . The f a c t based stages are o b j e c t i v e whereas the i n t e r p r e t a t i v e stages are s u b j e c t i v e i n nature. The Claims T r a i n i n g E v a l u a t i o n Task Force Committee p l a c e d l i t t l e or no emphasis on o b j e c t i v e c l a r i f i c a t i o n and chose to concentrate upon data c o l l e c t i o n , i n t e r p r e t a t i o n of r e s u l t s and making recommendations. By not c o n c e n t r a t i n g upon o b j e c t i v e c l a r i f i c a t i o n , c o n t r a r y to the emphasis 119 p l a c e d on o b j e c t i v e development by Provus and Stufflebeam et a l . , the Claims T r a i n i n g E v a l u a t i o n Task Force Committee based i t s e v a l u a t i o n upon assumptions which arose from an e x p e c t a t i o n t h a t a focus upon r e s u l t s would h i g h l i g h t problems w i t h i n the Claims T r a i n i n g Department. By overemphasising the needs of the r e c i p i e n t s of claims t r a i n i n g and f o c u s i n g upon i n t e r p r e t a t i o n of the r e s u l t s and recommendations, r a t h e r than i d e n t i f y i n g the o b j e c t i v e s of claims t r a i n i n g , the Claims T r a i n i n g . E v a l u a t i o n Task Force Committee developed an e v a l u a t i o n process t h a t was m e t h o d o l o g i c a l l y u n s t a b l e by t h e o r e t i c a l e d u c a t i o n a l e v a l u a t i o n p r o c e s s e s . T h i s emphasis would produce an i n v e r t e d pyramid as i l l u s t r a t e d i n F i g u r e 2 below. Given the l e s s e r emphasis upon o b j e c t i v e c l a r i f i c a t i o n , a l l l e v e l s above t h i s stage w i l l see t h a t weakness magnified. The r e s u l t would be t h a t recommendations made i n t h i s s i t u a t i o n would be open to q u e s t i o n as the foun d a t i o n of the e v a l u a t i o n would not be s o l i d . The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee demonstrates•this p r i n c i p l e i n a s i m p l i s t i c form. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee e v a l u a t i o n o b j e c t i v e s were very c l e a r , as they were s e t out i n B u l l e t i n CDB894. There c o u l d be no misunderstanding as to what those o b j e c t i v e s were, as they were a p u b l i s h e d document. The f i r s t l e v e l was t h e r e f o r e very s o l i d and 120 p r o v i d e d a good foundation upon which to b u i l d the balance of the e v a l u a t i o n . F i g u r e 2. Claims E v a l u a t i o n Task Force Committee. Weighting of E v a l u a t i o n Processes. The amount of weight p l a c e d upon each of the e v a l u a t i o n stages i s i l l u s t r a t e d - by the p r o p o r t i o n a l b l o c k s of the pyramid. There was a very heavy weighting upon the I n t e r p r e t a t i o n of R e s u l t s but the g r e a t e s t emphasis was p l a c e d upon Recommendations.- Recommendations formed the f o u n d a t i o n of t h i s e v a l u a t i o n and the i n s t a b i l i t y .of the e v a l u a t i o n ' s methodology i s r e p r e s e n t e d by the i n v e r t e d pyramid. The f a c t based stages are o b j e c t i v e whereas the i n t e r p r e t a t i v e stages are s u b j e c t i v e i n nature. The methodology used by the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee was very b a s i c as i t merely 121 had to e s t a b l i s h whether or not there was compliance with the o b j e c t i v e s . T h i s t r a n s l a t e d i n t o a rudimentary "yes" or "no" format. Again, the methodology c o n s i s t e d of as k i n g a sample of the people i n v o l v e d with B u l l e t i n CDB894 whether or not they complied with B u l l e t i n CDB894. T h i s . i s v ery d i f f i c u l t to c h a l l e n g e except on the grounds of sample s i z e . That i s , does the s i z e of the sample and the method of sampling a c c u r a t e l y r e f l e c t the l a r g e r p o p u l a t i o n ? The r e s u l t s t h a t flowed from the a p p l i c a t i o n of the e v a l u a t i o n method are t h e r e f o r e q u i t e s o l i d . As an e v a l u a t i o n proceeds from t h i s p o i n t to a p o s i t i o n t h a t i s hi g h e r i n the pyramid, confidence i n the outcomes becomes s o f t e r as they now r e l y upon i n t e r p r e t a t i o n . Challenges c o u l d a r i s e at t h i s stage of an e v a l u a t i o n with r e s p e c t to how the accumulated data were i n t e r p r e t e d as opposed to how the data were accumulated. T h i s r e l a t i o n s h i p c o u l d be the s u b j e c t of f u t u r e s t u d i e s . T h i s study r e v e a l e d that the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee was i n a s t r o n g e r p o s i t i o n than i t would have been otherwise, as i t had a s t r o n g base upon which to develop i t s i n t e r p r e t a t i o n . An accurate i n t e r p r e t a t i o n w i l l l a r g e l y depend upon the experience and s k i l l o f the committee members i n i n t e r p r e t i n g r e s u l t s w i t h i n the context of the e v a l u a t i o n . 122 E v a l u a t i o n Approaches Must be F l e x i b l e i n P r a c t i c e T h i s study suggests t h a t as a s i t u a t i o n unfolds,, more than one model may be at work at the same time i n the e v a l u a t i o n p r o c e s s . That i s , the L i t i g a t i o n Management E v a l u a t i o n Task Force committee e v a l u a t i o n f u n c t i o n e d as a combination of an e x p e r t i s e o r i e n t e d e v a l u a t i o n along w i t h an o b j e c t i v e s o r i e n t e d e v a l u a t i o n . A g e n e r a l i z a t i o n t h a t may be made from t h i s study i s t h a t one e v a l u a t i o n model operates as the main focus of an e v a l u a t i o n , but other e v a l u a t i o n models may operate around t h a t primary model. As the s i t u a t i o n changes or the need a r i s e s , the focus of an e v a l u a t i o n a l t e r s s u f f i c i e n t l y to o b t a i n whatever i n f o r m a t i o n i s necessary f o r the main e v a l u a t i o n . That i s , the e v a l u a t i o n process may s h i f t from one model to another and then back again. T h i s leads to a h y b r i d approach to e v a l u a t i o n , where there i s one main e v a l u a t i o n process and from t h a t process, as the s i t u a t i o n or needs change or the e v a l u a t i o n model i n use " s t a l l s " , other models are brought i n , d i s c a r d e d , or r e - i n t r o d u c e d . T h i s s h i f t i n g of models r e q u i r e s t h a t the e v a l u a t i o n process being used i s f l e x i b l e i n both d i r e c t i o n and approach i n the a c t u a l b u s i n e s s s e t t i n g . E v a l u a t i o n s t h a t are formula o r i e n t e d are very focussed i n a s p e c i f i c form of evaluation,- and can become s t a t i c . Formulas t h a t are f i x e d are not adaptable to process change. 123 S t a l l P o i n t s The r e s u l t s of t h i s study i n d i c a t e t h a t the two models under examination are e f f e c t i v e to a c e r t a i n p o i n t . They serve the need of e s t a b l i s h i n g a framework f o r e v a l u a t i o n and a s s i s t i n the gross data g a t h e r i n g p r o c e s s . The c u r r e n t study i l l u s t r a t e s t h a t the Provus and Stufflebeam et a l . e v a l u a t i o n models reach a stage i n the e v a l u a t i o n process where they become i n e f f e c t i v e . I c a l l t h i s stage the s t a l l p o i n t . This s t a l l i n g p o i n t would appear to be v a r i a b l e depending upon the circumstances of the e v a l u a t i o n . In other words, given a d i f f e r e n t e v a l u a t i o n under d i f f e r e n t circumstances, the s t a l l p o i n t may be reached e a r l i e r or l a t e r . The i n d i c a t e d c o n c l u s i o n , however, i s t h a t at some p o i n t these models w i l l s t a l l . The impact t h a t t h i s c o n c l u s i o n w i l l have on e d u c a t i o n a l e v a l u a t i o n thought i s t h a t e v a l u a t o r s , i n a p p l y i n g models, must be cognizant of p o t e n t i a l s t a l l p o i n t s . Rather than t r y i n g to f o r c e a s i t u a t i o n to f i t the model, one must be prepared to r e c o g n i z e s t a l l p o i n t s and s h i f t from one model to another. I c a l l t h i s process "phase s h i f t i n g " . Phase s h i f t i n g (or t r a n s i t i o n ) occurs i n order t h a t a b r i d g e may be developed between one model and the next. This concept was e v i d e n t i n the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee where the committee began i n a s t r i c t l y o b j e c t i v e s o r i e n t e d approach .and then s h i f t e d to a management o r i e n t e d approach 124 once the d e t e r m i n a t i o n had been made t h a t there was non- compliance with B u l l e t i n CDB894. The committee continued to e v a l u a t e the l i t i g a t i o n process a f t e r r e d e f i n i n g i t s mandate but with a d i f f e r e n t f o c u s . The s i g n i f i c a n t c h a l l e n g e f o r e v a l u a t o r s w i l l be to p r e d i c t the p o t e n t i a l s t a l l p o i n t s and be prepared with a l t e r n a t i v e models as e v a l u a t i o n s proceed through model phase s h i f t s . T h i s study suggests t h a t e v a l u a t o r s must widen t h e i r focus from a s i n g l e model u t i l i z a t i o n approach to a m u l t i p l e model approach. T h i s w i l l r e p r e s e n t a dramatic change i n the.thought process w i t h r e s p e c t to the development of t h e o r e t i c a l models. Rather than l o o k i n g f o r j u s t i f i c a t i o n f o r t h e o r e t i c a l models and p r o v i n g t h e i r e f f i c a c y through a t e s t i n g process, one w i l l now have to look at the p o t e n t i a l of e v a l u a t i o n models w i t h i n a spectrum of e v a l u a t i o n models i n order to determine an h o l i s t i c approach to e v a l u a t i o n . I t w i l l no longer be adequate to look at e v a l u a t i o n s w i t h i n the context of an. approach based model. Rather, the e v a l u a t o r must become a connoisseur of e v a l u a t i o n i n order to choose the best models a v a i l a b l e w i t h i n the context of the s i t u a t i o n and apply them a p p r o p r i a t e l y i n order to have the best flow of e v a l u a t i o n throughout what w i l l now be a continuum of e v a l u a t i o n . The three main areas upon which to focus are the s t a l l p o i n t s , the phase s h i f t or t r a n s i t i o n p e r i o d and the b r i d g i n g of one model to the next i n order to s e l e c t the 125 a p p r o p r i a t e model or models which may f i t p o t e n t i a l f u t u r e s . A f o u r t h area of i n t e r e s t i s to look at e v a l u a t i o n as b e i n g a continuum r a t h e r than a s t a t i c p r o c e s s . . T h i s study suggests t h a t s p e c i f i c e v a l u a t i o n s may have s t a r t p o i n t s and end p o i n t s , but these are p l a c e d i n a past and f u t u r e context. T h i s l i n e of thought e s t a b l i s h e s t h a t . e v a l u a t i o n i s an ongoing process t h a t c o n s i s t s of t r a n s i t i o n s or phase s h i f t s from one e v a l u a t i o n form to another w i t h no t r u e end p o i n t . I t i s more of an e v o l u t i o n of e v a l u a t i o n r a t h e r than the completion of an e v a l u a t i o n . By l o o k i n g at an e v a l u a t i o n as b e i n g an e v o l u t i o n a r y process w i t h s t a l l p o i n t s , phase s h i f t s / t r a n s i t i o n s and b r i d g e s , models w i l l no longer n e c e s s a r i l y f a l l out of use. Rather, t h e i r use may be changed and l i m i t e d i n order to serve s p e c i f i c purposes. New l i n e s of thought may be developed w i t h r e s p e c t to e v a l u a t i o n , by r e v i s i t i n g and reviewing those e v a l u a t i o n models which were e f f e c t i v e i n the past and h a v e . f a l l e n out of use i n order to determine whether or not aspects of those models may serve a purpose or a r o l e i n c u r r e n t e v a l u a t i o n thought. Recommendations The i n t e r p r e t a t i v e stage of an e d u c a t i o n a l e v a l u a t i o n a l process, as d i s c u s s e d i n the e v a l u a t i o n pyramids above, i s not c o n s i d e r e d by e i t h e r Provus or Stufflebeam et a l . The e f f e c t o f . t h e " e x p e r t i s e " of the e v a l u a t o r s in- 126 i n t e r p r e t a t i o n of r e s u l t s by the e v a l u a t i o n committee would be a r e v e a l i n g t o p i c f o r f u r t h e r study. For example, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee had a broad range of c r o s s - d i v i s i o n a l r e p r e s e n t a t i o n and experience due to the makeup of the committee. The i n t e r p r e t a t i o n o f the r e s u l t s was s i m i l a r l y broad, c r o s s - d i v i s i o n a l , and on a grand s c a l e . T h i s may have been due as much to the d i v e r s i t y o f the committee members as to the scope of the mandate. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee was made up of members from the same d i v i s i o n w i t h s i m i l a r t e c h n i c a l e x p e r t i s e and focus. The recommendations of t h i s committee were very s p e c i f i c and focussed towards ad d r e s s i n g the o b j e c t i v e s . The emphasis was upon e v a l u a t i n g B u l l e t i n CDB894 with the aim of c o n t r o l l i n g the expense of l i t i g a t i o n and improving the l i t i g a t i o n p r o c e s s . These r e s u l t s may have been d i r e c t l y r e l a t e d as much to the ^ s p e c i a l i n t e r e s t s of the committee members as to the o r i g i n a l mandate. A recommendation f o r f u t u r e s t u d i e s would be to. examine s e v e r a l i s s u e s . 1. Is there evidence of s t a l l p o i n t s o c c u r i n g i n the use of other models i'n other s i t u a t i o n s , or i s t h i s l i m i t e d to the p a r t i c u l a r / p e c u l i a r study at hand? 127 2. I f these s t a l l p o i n t s e x i s t , i s there evidence t h a t other models have the a b i l i t y to take over where the i n i t i a l model has s t a l l e d ? 3. I f there are s t a l l p o i n t s and i f the use of other models would appear to be a p p r o p r i a t e i n order to move the e v a l u a t i o n o f f the s t a l l p o i n t , an examination of the t r a n s i t i o n or phase s h i f t would be r e q u i r e d i n order to determine the dynamics of the phase s h i f t p r o c e s s . 4. An examination of the models t h a t are used to break o f f an e v a l u a t i o n s t a l l p o i n t would be r e q u i r e d to determine whether or not these subsequent models a l s o have s t a l l p o i n t s . 5. I f subsequent s t a l l p o i n t s a l s o e x i s t , i t would be necessary to examine whether or not t h i s concept of s t a l l p o i n t s and phase s h i f t i n g i s p a r t of the process of an e v a l u a t i o n continuum. 6. Does the. e x p e r t i s e l e v e l o f the members of the e v a l u a t i o n committee have an e f f e c t , d i r e c t or i n d i r e c t , upon an e v a l u a t i o n ? 7. Does the exposure of c a r e e r r i s k to the members of an e v a l u a t i o n committee have an e f f e c t upon the d e s i g n s - f u n c t i o n and i n t e r p r e t a t i o n o f the r e s u l t s of an e v a l u a t i o n ? 12 These recommendations may r e q u i r e a long term examination of s e v e r a l e v a l u a t i o n s i n a v a r i e t y of e v a l u a t i o n s e t t i n g s i n order to e s t a b l i s h a foun d a t i o n f o r t h i s theory becoming g e n e r a l i z a b l e . 129 EPILOGUE Impact of E x t e r n a l I n f l u e n c e s on E v a l u a t i o n Model E f f e c t i v e n e s s In d i s c u s s i n g the p e r s o n a l r i s k of the members of an e v a l u a t i o n committee i t i s i n s t r u c t i v e to examine the impact of e x t e r n a l i n f l u e n c e s upon the two cor p o r a t e e v a l u a t i o n s t h a t were used i n t h i s study. By r e f l e c t i n g upon t h i s background the importance of examining e v a l u a t i o n committee member r i s k i s h i g h l i g h t e d and emphasizes the need f o r f u r t h e r i n v e s t i g a t i o n i n t o t h i s unexplored f a c e t Of e v a l u a t i o n . E x t e r n a l I n f l u e n c e s A c t i n g Upon the Claims T r a i n i n g E v a l u a t i o n Task Force Committee. In Spring, 1994 the then- Manager of the Human Resources Development Department (H.R.D.) was under pr e s s u r e to look f o r ways to reduce head count ( s t a f f ) w i t h i n h i s Department i n support of an o v e r a l l Human Resources D i v i s i o n downsizing i n i t i a t i v e . A p p a r e n t l y without c o n s u l t i n g other Senior Managers or h i s s t a f f he de v i s e d a p l a n to reduce head count w i t h i n h i s Department. He saw the Claims T r a i n i n g Department, which r e p o r t e d to him, as an area where he c o u l d downsize. As p a r t of the development of t h i s p l a n he spent time i n a f i e l d o f f i c e to acquaint h i m s e l f with f i e l d o p e r a t i o n s . He had l i t t l e or no d i r e c t claims experience and so spent one day i n a claims 130 o f f i c e to f a m i l i a r i z e h i m s e l f q u i c k l y with the pr o c e s s . This was where we f i r s t met and d i s c u s s e d t r a i n i n g . He. was aware t h a t I was working on my Master's Degree i n Edu c a t i o n and wanted to d i s c u s s some ideas w i t h me. We spent approximately two hours d i s c u s s i n g my course work and claims t r a i n i n g i n g e n e r a l . He a d v i s e d me of h i s p l a n and asked t h a t I keep i t c o n f i d e n t i a l u n t i l i t was announced. At the time, I was unaware t h a t our c o n v e r s a t i o n was anything more than a c a s u a l conceptual d i s c u s s i o n . He l a t e r communicated t h i s p l a n to the a f f e c t e d s t a f f members of the Human Resources Development Department and Claims T r a i n i n g Department and i t was at t h i s p o i n t t h a t a c r i s i s developed. The p l a n i n v o l v e d r e d u c i n g the head count i n the Claims T r a i n i n g Department which at t h a t time f e l l w i t h i n the Human Resources D i v i s i o n . T h i s was to be achieved by d e c e n t r a l i z i n g claims t r a i n i n g , which was conducted at Head O f f i c e by Claims T r a i n i n g Managers, and having F i e l d Managers assume a g r e a t e r r o l e and r e s p o n s i b i l i t y f o r t r a i n i n g . By having l e s s formal t r a i n i n g conducted at Head O f f i c e by Head O f f i c e T r a i n e r s there would be l e s s need f o r t r a i n i n g s t a f f w i t h i n h i s Department. A core of t r a i n e r s would be maintained f o r those areas of t r a i n i n g which c o u l d not e f f e c t i v e l y be d e c e n t r a l i z e d and the•remaining t r a i n e r s would be r e a s s i g n e d to f i e l d o f f i c e s . The f i e l d o f f i c e s belong to the Claims D i v i s i o n . By r e l o c a t i n g s t a f f the 131 Human Resources D i v i s i o n would reduce i t s head count but the r e s u l t i n g impact would be an i n c r e a s e i n the head count of the Claims D i v i s i o n . A l s o , the placement of t r a i n e r s , who had spent the l a s t ten to f i f t e e n years i n Head O f f i c e , was an u n s e t t l i n g and very emotional i s s u e . The Claims D i v i s i o n was upset f o r v a r i o u s reasons i n c l u d i n g the l a c k of c o n s u l t a t i o n and the f a c t t h a t more work would be imposed upon an a l r e a d y h e a v i l y loaded f r o n t - l i n e management. Senior management•reviewed the s i t u a t i o n and d i r e c t e d that a task f o r c e be c r e a t e d to eva l u a t e the Claims T r a i n i n g Department and to make recommendations. The Claims T r a i n i n g E v a l u a t i o n Task Force Committee would operate at arm's l e n g t h from the Manager of the Human Resources Development Department. The Manager of the Human Resources Development Department s e l e c t e d me to be on the task f o r c e s p e c i f i c a l l y as a r e s u l t of our pr e v i o u s c o n v e r s a t i o n s . The whole s i t u a t i o n was very d i f f i c u l t f o r the members of the Claims T r a i n i n g E v a l u a t i o n Task Force Committee as they were at p e r s o n a l r i s k from a ca r e e r l i m i t i n g p e r s p e c t i v e and had, as w e l l , a v e s t e d i n t e r e s t i n the outcome of the e v a l u a t i o n . The Claims T r a i n i n g E v a l u a t i o n Task Force Committee members i n c l u d e d a m i n o r i t y who were n e u t r a l with r e s p e c t to the e v a l u a t i o n outcome and the m a j o r i t y were s e l e c t e d from f r o n t l i n e s t a f f who were the r e c i p i e n t s of t r a i n i n g , f r o n t l i n e managers who would have 132 been d i r e c t l y a f f e c t e d by the o r i g i n a l p l a n of the Manager of Human Resources Development Department ( d e c e n t r a l i z i n g t r a i n i n g and moving i t out i n t o the f i e l d ) , o f f i c e managers who would have been h e l d accountable f o r implementing the o r i g i n a l p l a n , and claims t r a i n e r s who "would have been d i r e c t l y a f f e c t e d by downsizing and r e l o c a t i o n . The members had a l l of these, concerns wi t h them when they entered the f i r s t meeting. These concerns would be a r a t i o n a l e x p l a n a t i o n f o r why most of the f i r s t meeting s k i r t e d d i s c u s s i n g o b j e c t i v e s and was more focussed on the impact any changes would have on i n d i v i d u a l s . The Claims T r a i n i n g E v a l u a t i o n Task Force Committee c o u l d have requested c l a r i f i c a t i o n of the mandate and then i t s o b j e c t i v e s would have been c l e a r l y d e f i n e d . However, i n t h i s i n s t a n c e , having an open-ended and unfocussed mandate allowed the e v a l u a t i o n team the l a t i t u d e to i n t e r p r e t what i n f o r m a t i o n would be r e q u i r e d to s a t i s f y the mandate reasonably and d e v i s e an e v a l u a t i o n process which would minimize r i s k . While i t was never e s t a b l i s h e d as an operating' p l a n , the movement d i r e c t l y i n t o s t a k e h o l d e r i n t e r v i e w s and focus group s e s s i o n s allowed the Claims T r a i n i n g E v a l u a t i o n Task Force Committee to d e f l e c t some of the r i s k by t a k i n g on a r e p o r t i n g r o l e r a t h e r than an e v a l u a t i o n r o l e . In other words, the Claims T r a i n i n g E v a l u a t i o n Task Force Committee would be r e p o r t i n g on the 133 s t a t e o f a f f a i r s of the Claims T r a i n i n g Department by r e c o r d i n g the general views h e l d by the C o r p o r a t i o n through i d e n t i f y i n g the o p i n i o n s o f a r e p r e s e n t a t i v e c r o s s s e c t i o n of the p o p u l a t i o n t h a t e i t h e r had d i r e c t c o n t a c t with, or r e c e i v e d t r a i n i n g from, the Claims T r a i n i n g Department. The s e c u r i t y o f the l a r g e group reduced the amount of p e r s o n a l r i s k h e l d by the Task Force members. The Claims T r a i n i n g E v a l u a t i o n Task Force Committee recommendations were made with a r e l a t i v e amount of s a f e t y knowing t h a t they were suggested s o l u t i o n s to problems t h a t were i d e n t i f i e d by a l a r g e r group r a t h e r than-the r e s u l t s of an e v a l u a t i o n made by a s e l e c t group of i n d i v i d u a l s . The i n f l u e n c e of e x t e r n a l f a c t o r s c o u l d d i r e c t l y i n f l u e n c e the a p p l i c a t i o n of the Provus and Stufflebeam et . a l . E v a l u a t i o n Models as w e l l as the u s e f u l n e s s of the r e s u l t s of the e v a l u a t i o n . N e i t h e r Provus nor Stufflebeam et a l . c o n s i d e r s the i n f l u e n c e of e x t e r n a l f a c t o r s i n the desi g n of t h e i r models. E x t e r n a l I n f l u e n c e s A c t i n g Upon The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee. The L i t i g a t i o n Management E v a l u a t i o n Task Force Committee, on the other hand, d i d not encounter the same l e v e l of p e r s o n a l r i s k , as the impact of the e v a l u a t i o n would not a f f e c t people who were i n a peer or s u p e r i o r r e l a t i o n s h i p to the task f o r c e members. 134 A c u r s o r y f i n a n c i a l examination of o p e r a t i n g c o s t s e s t a b l i s h e d t h a t a c o n s i d e r a b l e amount of money was b e i n g spent on defence c o s t s . Given the p u b l i c nature of the c o r p o r a t i o n , i t was expected t h a t o p p o r t u n i t i e s to act as defence counsel would be made a v a i l a b l e to the p r i v a t e s e c t o r as opposed to c r e a t i n g an in-house defence bar. T h i s s i t u a t i o n allowed abuses i n s e v e r a l forms to take p l a c e . Such abuses i n c l u d e d o v e r b i l l i n g on time spent on telephone c a l l s and f i l e reviews, d e l a y i n g settlement or the h a n d l i n g of f i l e s to i n c r e a s e the p e r i o d o f b i l l a b l e time, t r a n s f e r r i n g f i l e s to counsel w i t h i n the same f i r m and chargi n g f o r new reviews each time t h i s occurred, and a s s i g n i n g minor f i l e s to s e n i o r counsel who charge at a high e r b i l l i n g r a t e and then t r a n s f e r r i n g f i l e s to a j u n i o r counsel j u s t b e f o r e t r i a l . The c o r p o r a t i o n found i t s e l f i n a d i f f i c u l t p o s i t i o n with r e s p e c t to having to use counsel from the p r i v a t e s e c t o r f o r defence work and yet having v i r t u a l l y no formal c o n t r o l over t h e i r b i l l i n g p r a c t i c e s . " E r r o r s " t h a t were d i s c o v e r e d by those claims s t a f f who were v i g i l a n t were brought to the a t t e n t i o n of defence counsel who u s u a l l y would not r e s i s t the c o r r e c t i o n . Given the volume of defence work i t was a d i f f i c u l t t ask to c o n t r o l these " e r r o r s " . Claims D i v i s i o n B u l l e t i n 894 was c r e a t e d , i n c o n s u l t a t i o n with defence counsel, to b r i n g i n a measure • of c o n t r o l to defence counsel b i l l i n g p r a c t i c e s . I t simply 135 r e q u i r e d defence counsel to review the f i l e m a t e r i a l at the out s e t of the l e g a l process and to develop a f i l e budget f o r l e g a l c o s t s . I f there was going to be an overrun, then defence counsel was r e q u i r e d to d i s c u s s the s i t u a t i o n with the a d j u s t e r . The a d j u s t e r was r e q u i r e d to monitor the . l e g a l f i l e h a n d l i n g from a c o s t p e r s p e c t i v e i n a d d i t i o n to g i v i n g f i l e d i r e c t i o n . O b j e c t i o n s were made at the time t h a t B u l l e t i n CDB894, L i t i g a t i o n Budgeting Process, was i n s t i t u t e d . From an a d j u s t i n g p o i n t of view i t was too cumbersome a pr o c e s s . From a defence counsel p e r s p e c t i v e i t was unreasonable to expect defence counsel to be able to f o r e c a s t a budget and be accountable f o r the accuracy of the budget of l i t i g a t e d f i l e s , which may take two years or more to s e t t l e or reach the courtroom. From the ou t s e t there were warning s i g n s t h a t compliance by s t a f f and defence counsel was going to be qu e s t i o n a b l e . Without a s c e r t a i n i n g t h a t these compliance i s s u e s were d e a l t with p r i o r to implementation, the intended outcomes were i n jeopardy. B u l l e t i n CDB894, L i t i g a t i o n Budgeting Process, was implemented with the e x p e c t a t i o n t h a t i t would save a c o n s i d e r a b l e amount of time and defence c o s t expenses. When these savings f a i l e d to m a t e r i a l i z e , the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee was formed to 136 evaluate the e f f e c t i v e n e s s of B u l l e t i n CDB894, L i t i g a t i o n Budgeting Process. From a p e r s o n a l r i s k impact p e r s p e c t i v e , the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee members were e s s e n t i a l l y r i s k f r e e . They were e n t e r i n g the s i t u a t i o n a c t i n g i n the r o l e of a u d i t o r s . T h e i r mandate was simply to determine whether or not defence counsel and a d j u s t i n g s t a f f were complying w i t h B u l l e t i n CDB894, L i t i g a t i o n Budgeting Process. I f there was compliance, then no-one was at r i s k . I f there was p a r t i a l or no compliance, the o n l y people on the Task Force who. would have been at r i s k were the Centre Manager, who was l e a d i n g the Task Force and r e s p o n s i b l e f o r the four Claims Managers i n h i s o f f i c e and s t a f f , and the Claims Manager (the p a r t i c i p a n t o b s e r v e r ) , who was r e s p o n s i b l e f o r e n s u r i n g t h a t my s t a f f complied with procedures. As the greater'onus was on defence counsel to comply and non-compliance was p o t e n t i a l l y c o r p o r a t e wide, the amount of p e r s o n a l r i s k was minimal. A l s o , being on the task f o r c e would p o t e n t i a l l y l e a d , without n e c e s s a r i l y d i r e c t i n t e n t , to a s o f t e n i n g of the c r i t i c i s m of management and a p o l a r i z a t i o n of c r i t i c i s m towards s t a f f and defence counsel should non-compliance prove to be the case. I wrote the f i n a l r e p o r t which was s c r u t i n i z e d and e d i t e d by the Centre Manager and Regional Manager, and while t h i s was not the case, the p o t e n t i a l a l s o e x i s t e d f o r c r i t i c i s m i n the f i n a l r e p o r t to be s l a n t e d away from management. In a s i m i l a r nature but not as obvious as i n the Claims T r a i n i n g E v a l u a t i o n Task Force Committee, there was an element of b i a s present which c o u l d a f f e c t both the d e s i g n of the e v a l u a t i o n and the r e p o r t i n g . o f the r e s u l t s . The members of the L i t i g a t i o n Management E v a l u a t i o n Task Force Committee were s e l e c t e d because of t h e i r experience and commitment to l i t i g a t i o n and the Tort, (the r i g h t i n law to sue another p a r t y f o r a wrong committed) system. This p r e d i s p o s e d commitment c o u l d not h e l p shaping the e v a l u a t i o n p e r s p e c t i v e . The p e r s o n a l r i s k , though minimal, i s j u s t another of the p o t e n t i a l i n f l u e n c i n g f a c t o r s which have no p r o v i s i o n or c o n s i d e r a t i o n w i t h i n e i t h e r the Provus or Stufflebeam et a l . E v a l u a t i o n Models and i s worthy of f u r t h e r study. 138 References Barnes C r a i g & A s s o c i a t e s (May, 1993). The Request For P r o p o s a l s : A b b e r a t i o n or Opportunity, USA Best, J.W. (1977). Research i n Ed u c a t i o n (3rd ed.). Englewood C l i f f s , NJ: P r e n t i c e - H a l l , Inc. Cousins, J.B. (1996). Consequences of Researcher Involvement i n P a r t i c i p a t o r y E v a l u a t i o n . S t u d i e s i n E d u c a t i o n a l E v a l u a t i o n , 22, 20-23. Garaway, G.B. (1995). P a r t i c i p a t o r y E v a l u a t i o n . S t u d i e s i n E d u c a t i o n a l E v a l u a t i o n , 21,- 85-102. Guba, E.G. & L i n c o l n , Y.S. (1989). Fourth Generation E v a l u a t i o n . Newbury Park, C a l i f o r n i a : Sage P u b l i c a t i o n s . House, E.R. (1995). P r o f e s s i o n a l E v a l u a t i o n : S o c i a l Impact and P o l i t i c a l Consequences. Newbury Park, C a l i f o r n i a : Sage P y u b l i c a t i o n s Langenbach, M. (1993). C u r r i c u l u m Models i n A d u l t E d u c a t i o n . Malabar, F l o r i d a : K r i e g e r P u b l i s h i n g Company. McMillan, J . H., Schumacher, S. (1989). Research i n Education: A Conceptual I n t r o d u c t i o n (2nd ed.). USA: Harper C o l l i n s P u b l i s h e r s . Meigs, R. F., Meigs, W. B. (1989). F i n a n c i a l A c c o u n t i n g (6th ed.). Toronto, Ont: McGraw-Hill. 139 M i l k o v i c h , G. T., Glueck, W. F., Barth, R.T., & McShane, S.L.(1988). Canadian Personnel/Human Resource Management: A D i a g n o s t i c Approach. Piano, Texas: Business P u b l i c a t i o n s , Inc. M i l l e r , J . P. S e l l e r , W. (1990). Curriculum: p e r s p e c t i v e s and p r a c t i c e . . Toronto, O n t a r i o : Copp C l a r k Pitman L t d . Robbins,. S. P., Stuart-Kotze (1986). Management Concepts and P r a c t i c e s (Canadian E d i t i o n ) . Scarborough, O n t a r i o : P r e n t i c e - H a l l Canada, Inc.- Stanley, J . C. & Hopkins, K. D. (1972). E d u c a t i o n a l and P s y c h o l o g i c a l Measurement and E v a l u a t i o n . Englewood C l i f f s , NJ: P r e n t i c e - H a l l , Inc. T y l e r , R. (1986) . Changing Concepts of E d u c a t i o n a l E v a l u a t i o n . I n t e r n a t i o n a l J o u r n a l of E d u c a t i o n a l Research, 10, 53-55. U n i t e d States Automobile A s s o c i a t i o n (1988) . L i t i g a t i o n Management. USA Wolf, R.M. (1987). E d u c a t i o n E v a l u a t i o n : The State of the F i e l d . I n t e r n a t i o n a l J o u r n a l of E d u c a t i o n a l Research, 11, 3-6. Workplace T r a i n i n g Systems Open L e a r n i n g Agency. (September 1993). Corporate T r a i n i n g Consortium I n i t i a t i v e : Managing D i v e r s i t y Videoconference C o s t - B e n e f i t A n a l y s i s of D e c e n t r a l i z e d Vs C e n t r a l i z e d D e l i v e r y Approaches. Vancouver, BC Worthen, B. R., Sanders, J . R. (1987). Educational- E v a l u a t i o n : A l t e r n a t i v e Approaches and P r a c t i c a l G u i d e l i n e s . White P l a i n s , N. Y: Longman. i.

Cite

Citation Scheme:

    

Usage Statistics

Country Views Downloads
United States 13 2
China 9 1
France 1 0
Philippines 1 3
Indonesia 1 0
Japan 1 0
City Views Downloads
Beijing 9 0
Unknown 7 0
Mountain View 2 1
Pasadena 2 0
Navarre 1 0
Jakarta 1 0
Tokyo 1 0
Isabang 1 1
Sunnyvale 1 0
Ashburn 1 0

{[{ mDataHeader[type] }]} {[{ month[type] }]} {[{ tData[type] }]}

Share

Share to:

Comment

Related Items