"Education, Faculty of"@en . "Educational Studies (EDST), Department of"@en . "DSpace"@en . "UBCV"@en . "Hasman, Ruth Margaret Reiner"@en . "2010-04-19T22:21:30Z"@en . "1982"@en . "Master of Arts - MA"@en . "University of British Columbia"@en . "The purpose of this study was to test the suitability of illuminative evaluation as a methodology for determining the value of residential adult education programs. Illuminative evaluation methodology was selected for several reasons. First, the methodology functioned independently of the program. Second, it permitted the flexibility needed to evaluate a developing program. Third, it provided a means of studying spontaneous events. Fourth, it allowed for representation of multiple viewpoints, and lastly, few studies of this methodology had been undertaken (Miles, 1981; Parlett & King, 1971). For those reasons, it seemed important to investigate the suitability of illuminative evaluation.\r\nA residential program was determined to be particularly suitable for testing illuminative evaluation because it had some unique advantages that did not exist in other program formats. The chief advantage of the residential format over the more traditional types was that of removing the participant temporarily from his ongoing responsibilities. This made it possible for the investigator to have continuous contact with the participants which is important for a methodology that relies on fieldwork techniques.\r\nIn this study, illuminative evaluation methodology was applied to the evaluation of a residential program at the Justice Institute of British Columbia. In order to test the suitability of the methodology, three criteria appearing frequently in the literature were judged appropriate to this study\u00E2\u0080\u0094technical adequacy, utility and efficiency. The literature suggested that an evaluation should produce technically sound information that is useful to some audience and is worth more to the audience than it costs (Grotelueschen, 1980).\r\nEvidence of the degree to which illuminative evaluation met these criteria was collected during the program. Techniques such as interviews, questionnaires, and observations were used to collect the evidence. The evidence was analyzed using quantitative and qualitative techniques to determine whether the methodology met the standards set by the criteria.\r\nThe evidence collected showed that this methodology satisfied the criteria requirements of technical adequacy and utility. Although it was weak on the efficiency criterion, the methodology compensated with particular strengths in utility and technical adequacy.\r\nFor further research, there are a whole host of possible areas that illuminative evaluation opens up. Further work needs to be done to develop specific tasks, questions, and/or procedures which could guide implementation of each stage of the illuminative evaluation methodology. Further studies could be done to contribute to the understanding of the methodology and studies could be done to determine the suitability of the methodology for evaluating other adult education program formats."@en . "https://circle.library.ubc.ca/rest/handle/2429/23856?expand=metadata"@en . "C.J An Assessment of Illuminative Evaluation as an Approach to Evaluating Residential Adult Education Programs by Ruth Margaret Reiner Hasman B.A., San Jose State College, 1967 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS in THE FACULTY OF GRADUATE STUDIES (Department of Administrative, Adult and Higher Education) We accept t h i s thesis as conforming to the required standard THE UNIVERSITY OF BRITISH COLUMBIA December 1982 \u00C2\u00A9 Ruth Margaret Reiner Hasman, 1982 In presenting t h i s thesis i n p a r t i a l f u l f i l m e n t of the requirements for an advanced degree at the University of B r i t i s h Columbia, I agree that the Library s h a l l make i t f r e e l y available for reference and study. I further agree that permission for extensive copying of t h i s thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. I t i s understood that copying or publication of t h i s thesis for f i n a n c i a l gain s h a l l not be allowed without my written permission. Department of /AcLoJL\u00C2\u00A3 Izdx^al^j The University of B r i t i s h Columbia 1956 Main Mall Vancouver, Canada V6T 1Y3 D a t e o2<\u00C2\u00BB; I9?g. i i ABSTRACT The purpose of this study was to test the s u i t a b i l i t y of illuminative evaluation as a methodology for determining the value of r e s i d e n t i a l adult education programs. Illuminative evaluation methodology was selected for several reasons. F i r s t , the methodology functioned independently of the program. Second, i t permitted the f l e x i b i l i t y needed to evaluate a developing program. Third, i t provided a means of studying spontaneous events. Fourth, i t allowed for representation of multiple viewpoints, and l a s t l y , few studies of thi s methodology had been undertaken (Miles, 1981; Parlett & King, 1971). For those reasons, i t seemed important to investigate the s u i t a b i l i t y of illuminative evaluation. A r e s i d e n t i a l program was determined to be p a r t i c u l a r l y suitable for testing illuminative evaluation because i t had some unique advantages that did not exist in other program formats. The chief advantage of the r e s i d e n t i a l format over the more t r a d i t i o n a l types was that of removing the participant temporarily from his ongoing r e s p o n s i b i l i t i e s . This made i t possible for the investigator to have continuous contact with the participants which is important for a methodology that r e l i e s on fieldwork techniques. In t h i s study, illuminative evaluation methodology was applied to the evaluation of a r e s i d e n t i a l program at the Justice I n s t i t u t e of B r i t i s h Columbia. In order to test the s u i t a b i l i t y of the methodology, three c r i t e r i a appearing frequently in the l i t e r a t u r e were judged appropriate to t h i s s t u d y \u00E2\u0080\u0094 t e c h n i c a l adequacy, u t i l i t y and e f f i c i e n c y . The l i t e r a t u r e suggested that an evaluation should produce te c h n i c a l l y sound information that is useful to some audience and i s worth more to the audience than i t costs (Grotelueschen, 1980). Evidence of the degree to which illuminative evaluation met these c r i t e r i a was c o l l e c t e d during the program. Techniques such as interviews, questionnaires, and observations were used to c o l l e c t the evidence. The evidence was analyzed using quantitative and q u a l i t a t i v e techniques to determine whether the methodology met the standards set by the c r i t e r i a . The evidence c o l l e c t e d showed that t h i s methodology s a t i s f i e d the c r i t e r i a requirements of technical adequacy and u t i l i t y . Although i t was weak on the e f f i c i e n c y c r i t e r i o n , the methodology compensated with p a r t i c u l a r strengths in u t i l i t y and technical adequacy. For further research, there are a whole host of possible areas that illuminative evaluation opens up. Further work needs to be done to develop s p e c i f i c tasks, questions, and/or procedures which could guide implementation of each stage of the illuminative evaluation methodology. Further studies could be done to contribute to the understanding of the methodology and studies could be done to determine the s u i t a b i l i t y of the methodology for evaluating other adult education program formats. iv TABLE OF CONTENTS ABSTRACT i i LIST OF TABLES v ACKNOWLEDGEMENT v i i CHAPTER I: INTRODUCTION 1 The Problem 4 Research Approach 5 Summary 8 CHAPTER I I : LITERATURE REVIEW 9 H i s t o r i c a l Emergence Of Evaluation 9 Social Sciences 11 Education 12 C l a s s i c a l Versus N a t u r a l i s t i c Paradigm 13 C l a s s i c a l Paradigm 18 Na t u r a l i s t i c Paradigm 24 Illuminative Evaluation 26 Summary . 30 CHAPTER I I I : METHODOLOGY 32 C r i t e r i a 33 Technical Adequacy 33 U t i l i t y 37 Ef f i c i e n c y 40 Study Site 43 Summary 47 CHAPTER IV: ILLUMINATIVE EVALUATION STRATEGY 49 P i l o t Phase 50 Issues C l a r i f i e d 51 Evaluation Process 51 Questionnaire Design 53 Data Co l l e c t i o n 56 Operational Program 61 Summary 61 CHAPTER V: RESULTS 63 Technical Adequacy 63 U t i l i t y 69 E f f i c i e n c y 78 Summary 82 CHAPTER VI: SUMMARY, CONCLUSIONS, IMPLICATIONS AND RECOMMENDATIONS 85 Summary 85 Conclusions 87 Technical Adequacy 88 U t i l i t y 90 E f f i c i e n c y 92 Implications And Recommendations 95 REFERENCES 99 REFERENCE NOTES ...106 APPENDIX A: Expectations Questionnaire 107 APPENDIX B: Mini-Session Questionnaire 110 APPENDIX C: F i n a l Questionnaire 112 APPENDIX D: Follow-Up Questionnaire 116 APPENDIX E: Revised Questionnaires 119 v i LIST OF TABLES 1: Some Basic Differences Between C l a s s i c a l And N a t u r a l i s t i c Paradigms 18 2: Illuminative Evaluation Stages 29 3: L i s t Of Evaluation Costs 42 4: Data Co l l e c t i o n Schedule 57 5: E f f o r t And Time Spent On Illuminative Stages 79 6: C r i t e r i a And Standards Used For Determining S u i t a b i l i t y Of Illuminative Evaluation Methodology 84 v i i ACKNOWLEDGEMENT The support and dir e c t i o n provided by the members of my thesis committee served as the motivation to complete this research. I g r a t e f u l l y acknowledge the wisdom and encouragement of Dr. T. J. Sork and Dr. J. E. Thornton. In addition I thank Dr. J. G. Dickinson for his i n i t i a l guidance in getting this study underway. I would l i k e also to thank Mr. Henry Kennedy, Director of Land T i t l e s for allowing me to evaluate his Land T i t l e School programs and Mr. Paul Dampier, Program Director at the Justice Institute for his help and guidance in the i n i t i a l stages of this project. To Annette Buckmaster, a very special friend, I owe sincere thanks for her un f a i l i n g moral support and encouragement during my last year. Special thanks also go to MarDell Parrish, for without his help in word processing, t h i s thesis might s t i l l be on the computer. Throughout the la s t five years many of my fellow students in adult education provided encouragement, advice and help. To each of them I extend thanks and appreciation. F i n a l l y , I would l i k e to express my sincere thanks to David who empathized with my \"thesis phases\" and to Janet and Brent who kept saying \"Are you finished yet, Mommy?\" Without their cajolement, support and encouragement, th i s project would not have been completed. 1 CHAPTER I INTRODUCTION The purpose of thi s study was to test the s u i t a b i l i t y of illuminative evaluation as a methodology for determining the value of r e s i d e n t i a l adult education programs. The study s p e c i f i c a l l y focused on the advantages/disadvantages, and strengths/weaknesses of thi s methodology. As the numbers of people seeking adult education have grown, new methods and techniques have developed to meet the increased demand (Apps, 1979; Houle, 1971). One of the fastest growing educational developments has been use of short term r e s i d e n t i a l group learning programs for adults. There are few people in education and business today who have not attended a r e s i d e n t i a l course, conference, seminar, colloquium or workshop. These programs provide participants with a concentrated experience, a change in environment and an opportunity for close interaction and mutual problem-solving with peers (Garside, 1969; Houle, 1971; M i l l e r , 1964; Schacht, 1960). Because of their concentrated nature, the programs are capable of providing 2 an experience with a powerful impact. To enhance the r e s i d e n t i a l program's capacity for impact, planners or evaluators should evaluate the program systematically, c o l l e c t i n g information from many sources. This information can be used to improve effectiveness, modify i n e f f e c t i v e procedures, and a s s i s t in designing both follow-up a c t i v i t i e s and future programs (Beckhard, 1956). Since program evaluation in education i s in the early stage of theory development, i t has become an area of intense academic inte r e s t . As might be expected a plethora of divergent views and terms have been created by those trying to describe, analyze, explain, theorize, or otherwise capture the essence of evaluation (Rusnell, Note 1). Despite interest in the process by academics, p r a c t i t i o n e r s in the f i e l d have been less enthusiastic. \"Among theorists evaluation is one of the most hotly debated a c t i v i t i e s in the educational process; among pra c t i t i o n e r s i t i s one of the most ignored\" (Davis & McCallon, 1974, p. 271). The foremost reason for reduced enthusiasm regarding evaluation i s the lack of guidance provided to pr a c t i t i o n e r s by the l i t e r a t u r e . In any new f i e l d , guidance i s expected from the experts through th e i r l i t e r a t u r e , but the l i t e r a t u r e about program evaluation has served more to confuse than to guide. Worthen (1974) noted that evaluation l i t e r a t u r e i s badly fragmented into unrelated pieces and i s as d i f f i c u l t to synthesize as i t i s to make a meaningful picture from a random handful of pieces to a jigsaw puzzle. Looking at the individual pieces i s 3 l i t t l e more he l p f u l , for the l e v e l of discourse in individual writings is often aimed at fellow evaluation theorists more than at schoolmen, thereby communicating a great deal of d e t a i l about a topic which lacks a larger context within which i t could be useful. Working under th i s handicap, busy pr a c t i t i o n e r s can hardly be faulted for not expending the necessary time to try to develop a clear picture from the current evaluation l i t e r a t u r e (p. 2). A second reason i s that evaluators have the problem of which d e f i n i t i o n to use. During i t s development, program evaluation has come to have many di f f e r e n t d e f i n i t i o n s . These d e f i n i t i o n s are derived largely from the emphasis placed on quantitative versus q u a l i t a t i v e studies. Program evaluation, according to Blackwell and Bolman (1977), should give individuals and systems some control over their mutual growth and development so that they can function optimally. It should be a \"systematic\" c o l l e c t i o n of information from many sources in order to improve planning effectiveness, to modify procedures where necessary, and to serve as a guide in planning future programs (Beckhard, 1956). Bass & Vaughan (1966) suggested that evaluations should be planned at the same time as the program and should constitute an integral part of the t o t a l program from beginning to end. Evaluation must be purposeful and not done just for i t s own sake (Steele, 1970). Since no d e f i n i t i o n s u i t s every s i t u a t i o n , a d e f i n i t i o n that t r i e s i s l i k e l y to f a l l short in numerous ways. Against such odds evaluators usually withdraw to their own d e f i n i t i o n s of evaluation (Stake, 1979). From the numerous d e f i n i t i o n s mentioned in the l i t e r a t u r e , two seem appropriate to t h i s study: 4 By the term evaluation, we mean systematic examination of events occurring in and consequent on a contemporary program--an examination conducted to a s s i s t in improving t h i s program and other programs having the same general purpose (Cronbach, 1980, p. 14). Evaluation i s a c o l l e c t i o n of methods, s k i l l s and s e n s i t i v i t i e s necessary to determine whether a human service i s needed and l i k e l y to be used, whether i t is conducted as planned, and whether the human service actually does help people in need. While doing these tasks evaluators also seek ways to improve programs (Posavac & Carey, 1980, p. 6). The Problem Although more than a hundred evaluation models have been developed since Tyler's objectives-centered model, evaluators are s t i l l looking for alternative models (Cronbach, 1980). They are seeking new ways to evaluate programs as well as ways to improve u t i l i z a t i o n of r e s u l t s . House (1972) put i t thi s way: \"Producing data i s one thing! Getting i t used is quite another\" (p. 412). Thus, i t appears that id e n t i f y i n g an appropriate methodology i s quite s i g n i f i c a n t . Since programs evolve and change over time, a l t e r a t i o n s for improvement may need to be made during the program. Therefore, a c r i t e r i o n of an evaluation methodology i s that i t should not require the program to stand s t i l l or stay the same in order to be evaluated (Katz & Morgan, 1974; Stake, 1978). In other words, the methodology should be independent of the program 5 being evaluated. The methodology i d e n t i f i e d that met the above requirement i s \"illuminative evaluation\" (Parlett & Hamilton, 1977), for i t involves examining the program without i n t e r f e r i n g , manipulating or r e s t r i c t i n g the a c t i v i t i e s . Illuminative evaluation methodology was selected for several reasons. F i r s t , the methodology functioned independently of the program. Second, i t permitted the f l e x i b i l i t y needed to evaluate a developing program. Third, i t provided a means of studying spontaneous events. Fourth, i t allowed for representation of multiple viewpoints, and l a s t , few studies of th i s methodology had been undertaken (Miles, 1981; Parlett & King, 1971). For those reasons, i t seemed important to investigate the s u i t a b i l i t y of illuminative evaluation. Research Approach Within educational evaluation, two d i s t i n c t paradigms can be found: the c l a s s i c a l and n a t u r a l i s t i c . Each has i t s own strategies, f o c i and assumptions. Most formal educational evaluation studies use the c l a s s i c a l paradigm which derives i t s methodology from experimental psychology. These studies assess the effectiveness of a program by examining whether or not i t has reached required standards on pre-specified c r i t e r i a . Studies of t h i s kind are designed to y i e l d objective numerical data that can be s t a t i s t i c a l l y analyzed. 6 Recently, however, there has been increasing resistance to evaluations of thi s type (Parlett & Hamilton, 1976; Smith, 1976; Stake, 1978). There i s a movement to use a second paradigm related to so c i a l anthropology. This paradigm requires a fundamentally d i f f e r e n t evaluation methodology from that used with the c l a s s i c a l paradigm. These two paradigms are discussed thoroughly in Chapter I I . Frequently evaluations based on the n a t u r a l i s t i c paradigm involve a case study of a program or project. Case study methodology according to Stake (1978) has f a l l e n into disrepute among s o c i a l s c i e n t i s t s ; however, he suggests that case studies are s t i l l needed in certain types of evaluations. For example, when the evaluation i s aimed at improvement of a s p e c i f i c program, when the information c o l l e c t e d i s for participants and not just s c i e n t i s t s , when the concern i s for individuals rather than broad generalizations, then a case study approach that i d e n t i f i e s unique c h a r a c t e r i s t i c s and idiosyncracies can be invaluable (Patton, 1978). The methodology tested in this study, illuminative evaluation, i s r e l a t i v e l y new and i s based on the n a t u r a l i s t i c paradigm. It i s not a standard methodological package but a general research strategy (Parlett & Hamilton, 1976). It i s a dynamic evaluation process which i s not t i e d to a single treatment, predetermined goals or outcomes, but rather focuses on the actual operations of a program over a period of time (Patton, 1978) . This process requires s e n s i t i v i t y to both q u a l i t a t i v e and quantitative changes in a program throughout i t s 7 development, not just at some end-point in time. Since illu m i n a t i v e evaluation i s b u i l t on d i v e r s i t y and adaptability, the strategies used are adaptable and e c l e c t i c . This i s extremely important in program evaluation, for innovative programs are often changed as planners learn what works and what does not, and as planners experiment and change t h e i r p r i o r i t i e s . In t h i s study, illuminative evaluation methodology was applied to the evaluation of a r e s i d e n t i a l program at the Justice Institute of B r i t i s h Columbia. This program i s described in Chapter I I I . In order to determine i t s s u i t a b i l i t y , illuminative evaluation methodology should meet certain c r i t e r i a . The l i t e r a t u r e suggests that an evaluation methodology should produce information that i s : (1) t e c h n i c a l l y sound, (2) useful to some audience and (3) worth more to the audience than i t costs (Grotelueschen, 1980). It was decided to use the above as c r i t e r i a for testing illuminative evaluation. Complete descriptions of these c r i t e r i a are found in Chapter I I I . Evidence o f the degree to which illuminative evaluation met these c r i t e r i a was c o l l e c t e d during the program. Techniques such as interviews, questionnaires, and observations were used to c o l l e c t the evidence. The evidence was analyzed using q u a l i t a t i v e and quantitative techniques to determine whether the methodology met the standards set by the c r i t e r i a . 8 Summary Chapter I provided general background as well as a statement of purpose, a statement of the problem and a description of the research approach. The remainder of thi s thesis i s organized into five chapters and appendices. The review of selected l i t e r a t u r e appears in Chapter I I . Chapter III provides the research methodology while Chapter IV contains the operationalization of the illuminative strategy. The results appear in Chapter V. Chapter VI includes a summary of the previous chapters and conclusions based on the research f indings. 9 CHAPTER II LITERATURE REVIEW The review of the l i t e r a t u r e presented here contains a brief description of the h i s t o r i c a l emergence of evaluation followed by the development of evaluation in s o c i a l sciences and education. Then the review i s directed to c l a s s i c a l and n a t u r a l i s t i c paradigms used in evaluation studies. The f i n a l section contains a review of the illuminative evaluation model, the focus of thi s study. H i s t o r i c a l Emergence of Evaluation Evaluation emerged in the 1600's (Cronbach, 1980) when natural science established i t s e l f as a powerful instrument for overturning t r a d i t i o n a l b e l i e f s . Since i t s early beginnings i t has developed in a variety of ways in various f i e l d s . Cronbach's (1980) review of the h i s t o r i c a l emergence of 10 evaluation \"reminds us that applied s o c i a l research, l i k e other human endeavors, develops not in a steady expansion but in spurts and slumps and changes of d i r e c t i o n \" (p. 23). As one reviews the current evaluation l i t e r a t u r e in a number of substantive areas--education, t r a i n i n g , community action, health, psychotherapy--an interesting pattern occurs. Regardless of the f i e l d , the same issues or concerns reappear. For example, common concerns include the \"roles\" of evaluation, the evaluation design, measurement and c o l l e c t i o n techniques, the n e u t r a l i t y of the evaluator, the value of observation, the function of formative evaluation, the use of objectives, the value of long-term studies, and u t i l i z a t i o n of data. Over the years, evaluation studies have been strongly influenced by the methodologies of a l l the s o c i a l sciences. Evaluation studies have become a r e f l e c t i o n of the diverse academic and professional i d e n t i t i e s , ideological and p o l i t i c a l outlooks and past career commitments of evaluation researchers (Freeman & Solomon, 1981). Because of evaluation's i n t e r d i s c i p l i n a r y nature, the methods of each d i s c i p l i n e , and the assumptions which underlie them, have been subjected to c r i t i c a l scrutiny and have benefited from revisions resulting from these new perspectives (Guttentag & Saar, 1977). Because of i t s strong influence on educational evaluation, the history of the development of evaluation in the s o c i a l sciences is reviewed below. This i s followed by a discussion of the development of evaluation in the f i e l d of education. 11 Social Sciences In the 1930's, s o c i a l science research changed i t s emphasis. Psychologists were beginning to undertake studies of an experimental character in and out of the laboratory. For example: Newcomb's (1943) study of attitude change among g i r l s at Bennington College, L i p p i t t and White's study of the impact of authoritarian and democratic leadership styles on children's group relationships ( L i p p i t t , 1940), and Kurt Lewin and his associates' studies on s o c i a l influence undertaken during the 1930's and 1940's. Then, there was the monumental applied-research program car r i e d out by Stouffer and associates on American sol d i e r s during World War II and the famous Western E l e c t r i c Studies of the 1930's that contributed \"Hawthorne E f f e c t \" to the vocabulary of s o c i a l science (Bernstein & Freeman,1975). In the 1950's and 1960's many soc i a l action and intervention e f f o r t s were scrutinized and evaluated by s o c i a l scientists--deliquency-prevention programs, penal-rehabilitation e f f o r t s , psychotherapeutic and psychopharmacological treatments, public housing projects and community organization a c t i v i t i e s . However, i t was not u n t i l the massive U.S. federal expenditures during the \"Great Society\" programs during the 1960's and 1970's that accountability began to mean more than assessing staff s i n c e r i t y or p o l i t i c a l head counting of opponents and proponents. In the 1970's, evaluation emerged as a p o l i t i c a l t o o l . During that time, evaluations were regularly required of a l l 1 2 health, education, and welfare programs. The requirement for evaluation was a p o l i t i c a l response to the perceived demand for increased governmental accountability. Educat ion The f i e l d of education t r a d i t i o n a l l y has had an interest in evaluation of c u r r i c u l a , i n s t r u c t i o n , programs, participants, and materials. This f i e l d has tended to consider i t s problems unique and i t s methods special and d i f f e r e n t from evaluation of other kinds of programs. However, as educational evaluation has followed innovative programming beyond the classroom to involve the s o c i a l issues of the day, i t has become almost indistinguishable from evaluation of other planned s o c i a l interventions. Weiss (1972) said: \"Educational evaluators have much to learn from--and to teach--those in other f i e l d s , and they have much to loose by developing special perspectives and a special vocabulary that i n h i b i t s communication and interchange of experience\" (p. 13). Pooling information benefits those facing similar problems across the range of program areas. Following a r e l a t i v e l y inactive period in the 1950's, development of educational evaluation theory was r e v i t a l i z e d in the mid 1960's. This r e v i t a l i z a t i o n was influenced by Cronbach (1963), Scriven (1967), Stake (1967) and Stufflebeam (1967). The f i e l d ' s development was further stimulated by the evaluation requirements of U.S. federal education programs launched in 1965, and by the U.S. accountability movement that began in the early 1970's (Stufflebeam & Webster, 1981). 13 This brief sketch illuminates the growth of evaluation in two d i s t i n c t f i e l d s . Although developing independently, evaluative methods in these f i e l d s have moved in the same di r e c t i o n . Demands from funding agencies have helped the trend towards accountability. While there has been continuity in the development of the evaluation f i e l d , a q u a l i t a t i v e change has occurred. With the emergence of large scale national projects in the 1960's, i t was found that evaluation approaches based on the c l a s s i c a l paradigm were simply inadequate to deal with the evaluation questions and issues posed by these projects (Cronbach, 1963). C l a s s i c a l vs. N a t u r a l i s t i c Paradigm What, then, are the options available for evaluative studies? The l i t e r a t u r e reveals two paradigms that are used to guide evaluations; they are the c l a s s i c a l and n a t u r a l i s t i c paradigms. The c l a s s i c a l paradigm comes from the t r a d i t i o n of experimentation in agriculture, which gave us many of the basic experimental techniques most widely used in evaluation. This paradigm assumes quantitative measurement, experimental design and multivariate, parametric s t a t i s t i c a l analysis. By way of contrast, the n a t u r a l i s t i c paradigm has i t s roots in the f i e l d s of anthropology and ethnography. Using the techniques of interviewing and personal observation, t h i s 1 4 paradigm r e l i e s on q u a l i t a t i v e data, and detailed description derived from close contact with the target of study. The c l a s s i c a l paradigm aims at prediction of phenomena, while the n a t u r a l i s t i c paradigm aims at understanding phenomena (Patton, 1978) . Which of these p a r a d i g m s \u00E2\u0080\u0094 c l a s s i c a l or n a t u r a l i s t i c - -provides best guidance for an evaluation? \"There i s of course no d e f i n i t i v e answer to that question....The choice between paradigms in any- inquiry or evaluation ought to be made on the basis of the best f i t between the assumptions.... and the phenomenon being studied or evaluated\" (Guba & Lincoln, 1981, p. 56) . Although the l i t e r a t u r e has shown that neither the c l a s s i c a l nor the n a t u r a l i s t i c paradigm is i n t r i n s i c a l l y better than the other, the debate goes on. Kuhn (1970) has pointed out that the two sides \" . . . w i l l inevitably talk through each other when debating the r e l a t i v e merits of their respective paradigms....[E]ach paradigm w i l l be shown to s a t i s f y more or less the c r i t e r i a that i t dictates for i t s e l f and to f a l l short of a few of those dictated by i t s opponent\" (p. 109-110). Since neither paradigm solves a l l problems, they should be viewed as alternatives from which the evaluator can choose. The evaluator should select the paradigm and the methodology that suits the type of program being evaluated and the nature of the evaluation questions, for paradigms only t e l l researchers what to emphasize, what to look for, what questions to be concerned with, and what standards to apply. In order to make those 15 choices, i t i s necessary to be aware of the assumptions of each. Although these two paradigms d i f f e r on a number of assumptions, the discussion below w i l l be limited to nine major assumptions (Guba & Lincoln, 1981). Philosophical base. Bogdan and Taylor (1975) d i f f e r e n t i a t e between the two relevant philosophical perspectives. \"One, positivism...seeks the facts or causes of s o c i a l phenomena with l i t t l e regard for the subjective states of individ u a l s . \" The second, phenomenology, \" i s concerned with understanding human behavior from the actor's own frame of reference. Since the p o s i t i v i s t s and the phenomenologists approach d i f f e r e n t problems and seek d i f f e r e n t answers, their research w i l l t y p i c a l l y demand dif f e r e n t methodologies\" (p. 2). Thus, the n a t u r a l i s t i c investigator, a phenomenologist, i s concerned with description and understanding of s o c i a l phenomena, while the c l a s s i c a l investigator, a p o s i t i v i s t , i s concerned with \" s c i e n t i f i c \" facts and their relationship to one another. Inquiry paradigm. A second difference between the two approaches can be found in the guiding paradigm. The c l a s s i c a l investigator, with his p o s i t i v i s t leanings, tends to see the world as composed of variables. Certain variables can be manipulated to determine their e f f e c t s on other variables. The n a t u r a l i s t i c investigator, on the other hand, i s concerned with description and understanding, and i s guided by a paradigm based on ethnography. Purpose. A t h i r d difference between the two approaches i s purpose. The c l a s s i c a l approach tests some proposition about a 16 r e l a t i o n s h i p c a l l e d a hypothesis. The purpose is to v e r i f y the hypothesis by testing ideas empirically. The purpose of the n a t u r a l i s t i c approach, on the other hand, is the discovery of relationships that can be observed rather than arranging for i t to happen under controlled conditions. Framework/design. Pre-ordinate, fixed designs are one of the hallmarks of a c l a s s i c a l approach, while emergent, variable designs are among the hallmarks of a n a t u r a l i s t i c approach. Setting. It i s clear from the above statements that the c l a s s i c a l investigator leans toward the laboratory setting for investigations, while the n a t u r a l i s t i c investigator c a r r i e s out investigations in a natural, non-contrived, environment. Conditions. The c l a s s i c a l investigator seeks to control conditions; the n a t u r a l i s t i c investigator opens the investigation to uncontrolled conditions as much as possible. Treatment. The concept of treatment is extremely important in c l a s s i c a l experimental science. To the n a t u r a l i s t i c investigator the concept of treatment i s very foreign since i t implies some kind of manipulation or intervention. Scope. C l a s s i c a l investigators must focus on a limited range of variables in order to be able to deal with them in the controlled, systematic way that characterizes t h i s approach. Conversely, n a t u r a l i s t i c investigators are more ready to consider any variable that appears relevant. They approach the problem from a h o l i s t i c view. Methods. Lastly, both c l a s s i c a l and n a t u r a l i s t i c researchers wish to be objective in their methodology, but the 17 meaning which they ascribe to that term i s quite d i f f e r e n t . The c l a s s i c a l investigator s t r i v e s for o b j e c t i v i t y in the sense of inter-subjective agreement. The n a t u r a l i s t i c investigator, places l i t t l e store in that form of o b j e c t i v i t y and str i v e s instead for confi r m a b i l i t y , i . e . agreement among a variety of information sources. The nine points of difference noted above are summarized in Table 1 (Guba, Note 2). The dimensions of the table i l l u s t r a t e the fundamental differences in viewpoints between c l a s s i c a l and n a t u r a l i s t i c approaches. Nevertheless, i t would be naive to believe that every c l a s s i c a l investigator would always conform to the points of view mentioned, just as i t would be absurd to suppose that a n a t u r a l i s t i c investigator would never deviate. 18 Table 1 Some Basic Differences Between C l a s s i c a l and N a t u r a l i s t i c Paradigms COMPARISON ITEM CLASSICAL NATURALISTIC Philosophical base Logical positivism Phenomenology Inquiry paradigm Exper imental physics Anthropology Purpose V e r i f i c a t i o n Discovery Framework/design Fixed Var iable Setting Laboratory Nature Condit ions Controlled Invited interference Treatment Stable Variable Scope Limited variables H o l i s t i c Methods Object i v e - - i n sense of inter-subject agreement Objective--in sense of f a c t u a l / confirmable C l a s s i c a l Paradigm The l i t e r a t u r e reviewed confirmed the dominance of the c l a s s i c a l paradigm with i t s quantitative, experimental bias. Campbell and Stanley (1963) c a l l e d t h i s paradigm \"the only avai l a b l e route to cumulative progress\" (p. 3). It was th i s b e l i e f in and commitment to the natural science model on the part of most prominent academic researchers that made the 19 c l a s s i c a l paradigm dominant (Patton, 1978). As Kuhn (1970) explained, \"a paradigm governs, in the f i r s t instance, not a subject matter but rather a group\" (p. 80). Those groups most committed to the dominant paradigm are found in u n i v e r s i t i e s where they not only employ the s c i e n t i f i c method in their own evaluation research but where they also nurture students in a commitment to that same methodology (Kuhn, 1970). Like the majority of evaluative studies, evaluations of short-term r e s i d e n t i a l programs belong to the group dominated by the c l a s s i c a l paradigm. A survey of the l i t e r a t u r e yielded only evaluations relying heavily on the assumptions and c h a r a c t e r i s t i c s of the c l a s s i c a l paradigm described previously. No studies were i d e n t i f i e d that conformed to the n a t u r a l i s t i c paradigm. Based on the c l a s s i c a l paradigm, the researchers in the studies reviewed u t i l i z e d either pre-experimental, true experimental or quasi-experimental designs. The following discussion i s limited to short-term r e s i d e n t i a l programs, since th i s study concerned testing an evaluation methodology on programs in t h i s format. The discussion separates and c r i t i q u e s the studies on the basis of design. One-shot Case Study Much evaluation research in education conforms to a design in which a single group i s studied only once (one-shot case study) subsequent to some treatment (conference, workshop) presumed to cause change. Three studies reviewed used the one-20 shot approach (Havelock, 1971; Milozarek, 1976; Scruggs, 1976). Ba s i c a l l y , the planners in the above studies wanted to know how program participants f e l t at the conclusion of the program. This simple form of evaluation requires that a set of systematic observations be made of one group at some specified time. These studies i m p l i c i t l y compare the r e s i d e n t i a l experience with other observed and/or remembered events. The inferences are based on general expectations of what the data would have been had the experience not occurred. In addition, \"the many uncontrolled sources of differences between any one study and potential future ones are so numerous that j u s t i f i c a t i o n in terms of providing a bench mark for future studies is hopeless\" (Campbell & Stanley, 1963, p. 7). Where only one group is measured, interpretation of the results is d i f f i c u l t and often unconvincing. \"This workshop was rated successful by the pa r t i c i p a n t s . In general, the open-ended responses were divided into two categories: outright praise, and requests for more time and depth of topic\" (Havelock, 1971). Without a comparison group, i t i s hard to know whether the results would have been equally good with some other program, or whether the program was actually responsible for producing the results at a l l . Pretest-Post Design If the question an evaluator i s seeking to answer cannot be addressed through one set of observations made at the completion of the program, then the next more complex research design 21 should be used. The pretest-posttest design i s used when the evaluator wants to know i f participants improved or at least did not deteriorate while being served by a program. Nine a r t i c l e s were i d e n t i f i e d that used th i s design (Cox, 1974; Deantonio,1973; Densmore, 1965; Dickinson & Lamoureux, 1975; Halverson & Thiesse, 1979; Pattison, 1968; Roberts & Holmes, 1971; V a l l a , 1975; Wohllenben, 1965). Like the one-shot case study, one cannot conclude that the program caused the improvement. The program might have caused the improvement; however, t h i s design i s not rigorous enough to permit such a conclusion. A l l of the above studies used either the one-shot case study or pretest-posttest design. These studies were highly l o c a l i z e d . Their value was limited to the program studied and therefore not generalizable. Sutton (1966) suggested that l o c a l i z e d studies should be appraised only in terms of the i r operational values to the i n s t i t u t i o n making them. True Experimental When evaluators want to discover the cause of changes in program participants, evaluations of greater complexity must be designed. In order to show that something caused something else, i t i s necessary to demonstrate that: \"(1) the cause precedes the supposed effect in time; (2) the cause covaries with the e f f e c t ; and (3) no other alternative explanations of the e f f e c t exist except the assumed cause\" (Posavac & Carey, 1980, p. 196). Campbell and Stanley (1963) suggest that only 22 true experimental and quasi-experimental designs w i l l prove causal r e l a t i o n s . Several studies of r e s i d e n t i a l programs have used the true experimental design. They used a pretest-posttest control group with participants randomly selected for the two groups (Bale & Molitor, 1978; Bunch, 1976; Conrad, 1976; Devlin, 1966; Jenkins, 1976). Occasionally researchers have used three randomly selected groups (Blaney & McKie, 1969; Peterson, 1971). In practice, evaluators are often in a position in which i t ' s impossible to randomly select groups and manipulate conditions. In those cases, evaluators choose a quasi-experimental design which controls some but not a l l \"threats to v a l i d i t y \" (Campbell & Stanley, 1963). In quasi-experimental designs, an intact group is selected as a control because of i t s s i m i l a r i t y to the experimental group. Sometimes t h i s type of group i s c a l l e d a \"comparison group\" to di s t i n g u i s h i t from a true control group. Including the comparison group permits a d i s t i n c t i o n to be made between the ef f e c t s of the program and the several alternative plausible interpretations of change. Because the comparison group i s tested at the same time as the experimental group, both groups have the same, amount of time to mature. H i s t o r i c a l forces presumably aff e c t the groups equally. Because both groups are tested twice, testing effects should be equivalent. F i n a l l y , the rates of participant loss between pretest and posttest can be examined to be sure they are similar (Posavic & Carey, 1980). 23 Quasi-experimental Design The quasi-experimental design most frequently used in the studies reviewed was the nonequivalent control group. This design necessitates the use of a pretest to provide control for selection bias. Only through a pretest can intact group equivalence be demonstrated, enabling the evaluator to compare the results of the two groups. Five of the studies using this design investigated one or more aspects of a p a r t i c u l a r program (Bringle, 1967; George & Green, 1976; Stewart, 1965; Torrence, 1966; Touzel, 1975). Four studies (Edelbach, 1973; Lacognata, 1961; Smallegan, 1971; Wientge & Lahr, 1966) compared r e s i d e n t i a l to non-residential programs. The above classical/experimental/quasi-experimental studies focused on isolated psychological variables, i . e . s a t i s f a c t i o n , anxiety, self-esteem. Such studies have not allowed insight into the complex impact that programs have on p a r t i c i p a n t s . Moreover, they have not always given program planners information needed to make programmatic decisions. F i n a l l y , much evaluation has been highly c r i t i c a l . It has been composed of negative punitive statements which t y p i c a l l y discourage, anger, and disappoint the evaluation audience (Patton, 1978). The problem for evaluation i s that the very dominance of the c l a s s i c a l paradigm with i t s quantitative, experimental emphasis appears to have cut off the great majority of evaluators from serious consideration of any alternative evaluation paradigm (Patton, 1978). 24 Na t u r a l i s t i c Paradigm Recently, however, there has been increasing resistance to u t i l i z i n g the c l a s s i c a l paradigm in evaluation studies (Parlett & Hamilton, 1976; Patton, 1978; Smith, 1976; Stake, 1978). One alter n a t i v e to the c l a s s i c a l paradigm i s the n a t u r a l i s t i c paradigm. The n a t u r a l i s t i c paradigm is not new but has i t s roots in ethnography and anthropology. A n a t u r a l i s t i c inquiry i s a dynamic process which i s not t i e d to a single treatment or predetermined goals or outcomes, but rather focuses on the actual operations of a program over a period of time (Guba & Lincoln, 1981; Parlett & Hamilton, 1976; Patton, 1978). This process requires s e n s i t i v i t y to both q u a l i t a t i v e and quantitative changes in a program throughout i t s development, not just at some end-point in time. Hamilton (1977) has characterized those a l t e r n a t i v e models as \" p l u r a l i s t \" evaluation models. That i s , models that take account of the value positions of multiple audiences. In p r a c t i c a l terms, p l u r a l i s t evaluation models (Parlett & Hamilton, 1976; Patton, 1975; Stake, 1967) can be characterized in the following manner: Compared with the c l a s s i c models, they tend to be more extensive (not necessarily centered on numerical data), more n a t u r a l i s t i c (based on program a c t i v i t y rather than program intent), and more adaptable (not constrained by experimental or preordinate designs). In turn they are l i k e l y to be sensitive to the d i f f e r e n t values of program participants, to endorse empirical methods which are couched in the natural language of the recipients, and to s h i f t the locale of 25 formal judgment from the evaluator to the participants (p. 339). There are many methodological questions that can be raised about n a t u r a l i s t i c inquiry, ranging from basic epistomological issues to operational or procedural matters. Guba (1978) has attempted to define the d i f f i c u l t i e s that face a n a t u r a l i s t i c inquiry. The major problem, as he saw i t , i s that of a u t h e n t i c i t y \u00E2\u0080\u0094 t h e establishment of the basis for trust in the outcomes of the evaluation. Other methodological problems are setting l i m i t s to the inquiry and focusing on the categories within which the data can be assimilated and understood. Despite the d i f f i c u l t i e s just mentioned, n a t u r a l i s t i c inquiry has begun to gain c r e d i b i l i t y . Leading evaluation theorists have been strongly interested in moving away from more c l a s s i c a l paradigms (Cronbach, 1980; Guba & Lincoln, 1981; Patton, 1978) and p r a c t i t i o n e r s have begun to apply n a t u r a l i s t i c techniques to evaluative studies (Erickson, 1977; Fienberg, 1977; Lutz, 1974; Parlett & Hamilton, 1976; Rist, 1975). A number of evaluation models have emerged which seem espec i a l l y congenial to the use of the n a t u r a l i s t i c paradigm. Five emergent models especially compatible with n a t u r a l i s t i c inquiry are: the Responsive Model (Stake, 1975), the J u d i c i a l Model (Wolf, 1979), the Transactional Model (Rippey, 1973), the Connoisseurship Model (Eisner, 1975), and the Illuminative Model (Parlett & Hamilton, 1977). These f i v e models have close philosophic and operational t i e s with n a t u r a l i s t i c inquiry. Their emergence at this time argues strongly for the u t i l i t y of n a t u r a l i s t i c inquiry for the f i e l d of educational evaluation, 26 and helps make the case that n a t u r a l i s t i c inquiry should be investigated as an alternative methodology. Illuminative Evaluation The illuminative evaluation model (Parlett & Hamilton, (1977) was chosen for testing because i t matched the philosophy and value system of the investigator. This model: (1) permitted the study of changing and emerging problems, (2) encouraged multiple viewpoints and perspectives, (3) focused on program a c t i v i t i e s and issues rather than outcomes and (4) provided for a means of studying spontaneous events and sit u a t i o n s . [I]lluminative evaluation, takes account of the wider contexts in which education programs function. Its primary concern i s with description and interpretation rather than measurement and prediction. It stands unam-biguously within the alternative methodological paradigm. The aims of illuminative evaluation are to study the innovatory program: how i t operates; how i t i s influenced by the various school situations in which i t is applied; what those d i r e c t l y concerned regard as i t s advantages and disadvantages; and how students, i n t e l l e c t u a l tasks and academic experiences are most affected. In short, i t seeks to address and to illuminate a complex array of questions (Parlett & Hamilton, 1977, p. 144). C h a r a c t e r i s t i c a l l y illuminative evaluations have three p r i n c i p a l stages: observation, inquiry, and explanation. The f i r s t stage i s an exploratory phase during which the investigator becomes knowledgeable about the program and people involved and t r i e s to understand and document the day-to-day 27 r e a l i t y of the setting or settings under study. No attempt i s made to manipulate, control or eliminate situations or program developments. Faculty, participants, planners and any other persons involved in the project are observed and interviewed. Documents are reviewed to obtain an h i s t o r i c a l perspective as well as a perspective on how people regard the innovation. The second stage is a narrowing and focusing process. It is an interactive process between evaluators and relevant decision-makers or information users. Narrowing and focusing the study means dealing with several basic concerns. What i s the purpose of the evaluation? How w i l l the information be used? What w i l l we know after the evaluation that we do not know now? What can we do after the evaluation that we cannot do now for lack of information? What topics or concerns should be selected for intensive investigation? Narrowing and focusing are key elements because programs are so complex and have so many l e v e l s , goals, and functions. There are always more potential study topics than there are time and resources to examine. The alte r n a t i v e s , therefore, have to be narrowed, c l a r i f i e d and redefined. When the alternatives have been c l a r i f i e d and defined, the evaluator must determine evaluation procedures. Illuminative evaluation does not have simple, standardized procedures for that function, so the evaluator might incorporate other models that offer guidelines for operationalizing the model. For example, i f the study focuses on participant reactions, the extent to which the program content was assimilated and/or the 28 change in job behavior, the evaluator might incorporate Kirkpatrick's (1967) or Hamblin's (1974) model. Both these models offer guidelines for operationalizing the evaluation. The t h i r d stage consists of seeking general p r i n c i p l e s underlying the organization of the program, spotting patterns of cause and e f f e c t , and placing individual findings within a broader explanatory context (Parlett & Hamilton, 1976). (See Table 2 for summary of the three stages.) Within the three stage framework of illuminative evaluation, the investigation can combine four d i f f e r e n t data gathering techniques permitting the program to be examined from a number of angles. These are (1) observation of the participants and events; (2) interviews with participants, resource persons, and administrators; (3) questionnaires covering many aspects of the program; and (4) h i s t o r i c a l research with existing documents. The following paragraphs describe the data gathering techniques in more d e t a i l . Observat ions are an essential part of illuminative evaluation. They are intended primarily to build-up a continuous record of on-going events, to add interpretive comments on obvious and latent features of the program, and to uncover t a c i t assumptions and interpersonal relationships. Interviews are used primarily to determine the perceptions and views of individual participants. Discovering the view of participants i s c r u c i a l to assessing the impact of the program. Informal interviews often provide unique insights into program processes experienced by d i f f e r e n t people. T a b l e 2 I l l u m i n a t i v e E v a l u a t i o n S t a g e s STAGES A C T I V I T I E S STAGE O N E \u00E2\u0080\u0094 O B S E R V A T I O N - r e v i e w o r d i s c o v e r what i s e x p e c t e d a t t h e o u t s e t The i n v e s t i g a t o r becomes - c o n s i d e r t h e q u e s t i o n s , h y p o t h e s e s o r i s s u e s a l r e a d y r a i s e d k n o w l e d g e a b l e a b o u t t h e p r o g r a m a n d - l o o k f o r p o s s i b l e s t u d i e s t o u s e a s a m o d e l s p e o p l e i n v o l v e d . - r e v i e w h i s t o r i c a l d o c u m e n t s - f o r m i n i t i a l p l a n o f a c t i o n - a n t i c i p a t e k e y p r o b l e m s , e v e n t s - c o n s i d e r p o s s i b l e a u d i e n c e s f o r p r e l i m i n a r y a n d f i n a l r e p o r t s STAGE TWO--INOUIRY - a r r a n g e a c c e s s t o p r o g r a m , n e g o t i a t e pl-an o f a c t i o n TrT^ i n v e s t i g a t o r n a r r o w s a n d - d i s c u s s a r r a n g e m e n t f o r m a i n t a i n i n g c o n f i d e n t i a l i t y o f d a t a , s o u r c e a n d r e p o r t s f o c u s e s t h e s t u d y . - i d e n t i f y i n f o r m a n t s and s o u r c e s o f p a r t i c u l a r d a t a - s e l e c t o r d e v e l o p q u e s t i o n n a i r e s o r s t a n d a r d i z e d p r o c e d u r e s i f a n y -work o u t r e c o r d - k e e p i n g s y s t e m -make o b s e r v a t i o n s , i n t e r v i e w s , u s e q u e s t i o n n a i r e - k e e p r e c o r d s o f a c t i v i t i e s a n d c h a n g e s STAGE T H R E E \u00E2\u0080\u0094 E X P L A N A T I O N - c l a s s i f y raw d a t a ; b e g i n i n t e r p r e t a t i o n s T h i s i s t h e a n a l y s i s a n d - g a t h e r a d d i t i o n a l d a t a , t r i a n g u l a t e d a t a t o v a l i d a t e k e y o b s e r v a t i o n s i n t e r p r e t a t i o n p h a s e . - s e a r c h f o r p a t t e r n s o f d a t a - s e e k l i n k a g e s b e t w e e n p r o g r a m a r r a n g e m e n t s , a c t i v i t i e s a n d o u t c o m e s - s e l e c t i l l u s t r a t i o n s , s p e c i a l i n t r e t a t i o n s -draw t e n t a t i v e i s s u e s , o r g a n i z e a c c o r d i n g t o i s s u e s - d e s c r i b e t h e s e t t i n g w here t h e a c t i v i t y o c c u r r e d - d r a f t r e p o r t s - d e s c r i b e m e t h o d s o f i n v e s t i g a t i o n - r e v i s e and d i s s e m i n a t e r e p o r t s 30 Questionnaires and tests are included to obtain information that sustains or q u a l i f i e s e a r l i e r , tentative findings. H i s t o r i c a l research using documentary and background sources provides information about the development of events. The gathering of background information yields an h i s t o r i c a l perspective of the way the program was regarded by d i f f e r e n t people before the evaluation began. This information can be obtained from l e t t e r s , minutes of meetings, and reports. The data gathered often suggest topics that need investigation and expose aspects of the program that otherwise would be missed. The three stages of illuminative evaluation do not function separately; they overlap and are in t e r r e l a t e d . The t r a n s i t i o n from stage to stage occurs as problem areas become progressively c l a r i f i e d and redefined. Beginning with an extensive data base, using the data gathering techniques mentioned above, the investigator systematically reduces the scope of the inquiry to give more concentrated attention to the emerging issues. This \"progressive focusing\" permits unique and unpredicted phenomena to be given due weight. It reduces the problem of data overload and prevents the massive accumulation of unanalyzed material (Parlett & Hamilton, 1976). Summary This chapter illuminated the h i s t o r i c a l development of evaluation. Although evaluation processes have been developed 31 independently, most f i e l d s have developed in the same direction--toward accountability. This trend toward accountability was noted both in soc i a l science evaluation and in educational evaluation. While there has been continuity in the development of the evaluation f i e l d , a q u a l i t a t i v e change has occurred. With the emergence of large scale U.S. projects in the 1960's, i t was found that the c l a s s i c a l evaluation approaches were inadequate to deal with evaluation questions and issues posed by those projects (Cronbach, 1963). One alternative to c l a s s i c a l evaluation that arose was n a t u r a l i s t i c inquiry which had i t s roots in the f i e l d s of anthropology and ethnography. Using the techniques of in-depth, open-ended interviewing and personal observation, t h i s approach r e l i e d on q u a l i t a t i v e data, and detailed description derived from close contact with the target of study (Patton, 1978). The illuminative evaluation model that has emerged from the n a t u r a l i s t i c paradigm permits the studying of changing problems, encourages multiple viewpoints, focuses on program a c t i v i t i e s and provides for a means of studying spontaneous events and situat i o n s . 32 CHAPTER III METHODOLOGY Illuminative evaluation methodology was selected for th i s study for several reasons. F i r s t l y , illuminative evaluation i s based on the assumption that evaluation should \"respond\" to the needs, interests, and perceptions of the partic i p a n t s rather than on measurement c r i t e r i a established a p r i o r i . Secondly, i t acknowledges that there are multiple r e a l i t i e s and multiple truths. Thus, unlike the majority of investigative e f f o r t s , illuminative evaluation e l i c i t s , considers, and builds on the in-depth information that i s provided by participants, instructors and coordinators a l i k e . F i n a l l y , data interpretations portray s i m i l a r i t i e s and differences in perceptions while describing the origins and context for such agreements and discrepancies (Guba & Lincoln, 1981). In order to determine the s u i t a b i l i t y of t h i s methodology, for evaluating r e s i d e n t i a l adult education programs, i t was tested on a selected s i t e . Then the results of the testing were compared with the standards set by pre-specified c r i t e r i a . If 33 the evidence of s u i t a b i l i t y meets the standards, the methodology can then be deemed suitable. The remainder of thi s chapter contains descriptions of the c r i t e r i a , standards, and study s i t e . C r i t e r i a Three c r i t e r i a appearing frequently in the l i t e r a t u r e were judged appropriate to this study. The l i t e r a t u r e suggested that an evaluation methodology should produce information that i s (1) technically sound, (2) useful to some audience and (3) worth more to the audience than i t costs (Grotelueschen, 1980). These c r i t e r i a , used to judge the s u i t a b i l i t y of this methodology, w i l l be described in the following paragraphs. Technical Adequacy Two standards of a tech n i c a l l y adequate methodology are the ob j e c t i v i t y and v a l i d i t y . Of the two standards mentioned, o b j e c t i v i t y i s probably the most con t r o v e r s i a l . \"For how can an inquiry be objective i f i t simply 'emerges'; i f i t has no careful control l a i d down a p r i o r i ; i f the observations to be made or the data to be recorded are not sp e c i f i e d in advance....\" (Guba & Lincoln, 1981, p. 124). The d i f f i c u l t y seems to stem from the meaning given to the term o b j e c t i v i t y . Scriven (1972) has pointed out that the terms objective and subjective are opposites, but they are widely used to refer to 34 contrasts in two d i f f e r e n t senses: a quantitative and a q u a l i t a t i v e one. In the quantitative sense, 'subjective' refers to what \"occurs to the individual subject, while 'objective' refers to what a number of subjects experience.\" In the q u a l i t a t i v e sense, \"'subjective' means unreliable, biased or probably biased, a matter of opinion, and 'objective' means r e l i a b l e , f a c t u a l , confirmable or confirmed, and so forth\" (Scriven, 1972, pp. 95-96). B a s i c a l l y , Scriven suggested that what one individual experiences is not necessarily unreliable, biased, or a matter of opinion, just as what a number of individuals experience i s not necessarily r e l i a b l e , factual, and conf i rmable. Illuminative evaluation methodology based on the n a t u r a l i s t i c paradigm emphasizes the o b j e c t i v i t y of the data while evaluation methodologies based on the c l a s s i c a l paradigm emphasize the o b j e c t i v i t y of the investigator. In the illuminative model, the o b j e c t i v i t y of the data i s of c r i t i c a l concern; i t should be both factual and confirmable. The second standard of a technically adequate evaluation methodology i s v a l i d i t y . Illuminative methodology emphasizes v a l i d i t y . It is concerned with the meaning and meaningfulness of the data co l l e c t e d and instrumentation employed. Does the instrument measure what i t purports to measure? Do the data mean what we think they mean? (Patton, 1978). Illuminative methodology makes the issue of v a l i d i t y central by getting close to the data, being sensitive to 35 q u a l i t a t i v e d i s t i n c t i o n s , developing empathy with program participants, and attempting to est a b l i s h a h o l i s t i c perspective on the program. This closeness to the data suggested by Denzin (1971) and others (Campbell, 1975; Guba & Lincoln, 1981; Patton, 1978) i s not the only legitimate way to understand human behavior, but i t is an alternative to the distance prescribed by the dominant c l a s s i c a l paradigm. The focus in the illuminative methodology i s on a v a l i d representation of what i s happening, not at the expense of r e l i a b l e measurement, but without allowing r e l i a b i l i t y to determine the nature of the data (Guba & Lincoln, 1981; Parlett & Hamilton, 1976). House (1980) pointed out that V a l i d i t y is provided by cross-checking d i f f e r e n t data sources and by testing perceptions against those of participants. Issues and questions aris e from the people and situations being studied rather than from the investigator's percept ions .... In constructing explanations, the n a t u r a l i s t looks for convergence of his data sources and develops sequential, phase-like explanations that assume no event has single causes (p. 280). In order to determine the technical adequacy of illuminative methodology, the data produced must be objective and v a l i d . In other words, i t must be shown that the data are both factual and confirmable as well as give a v a l i d representation of events. Two procedures were used to determine the technical adequacy of illuminative evaluation; those procedures are triangulation and continuous observation. Triangulation i s a process of cross-checking findings. Cross-checking enables the investigator to determine i f the data 3 6 c o l l e c t e d from multiple data sources confirm each other. Besides f a c i l i t a t i n g cross-checking, triangulation also increases the c r e d i b i l i t y of data through v a l i d a t i o n . In t h i s study, data from informal interviews, questionnaires, and observations were cross-checked. The process of combining these data sources produced data that were objective and v a l i d . Continuous observation i s also important in determining technical adequacy, because continuous observation w i l l provide a p r o f i l e of the program. In t h i s study the investigator b u i l t a continuous record of on-going events, transactions and informal remarks. Much of the on-site observation involved recording discussions with and between participants. These provided additional information that might not otherwise be apparent from more formal interviews and questionnaire responses. The data from continuous observation was used to provide a v a l i d picture of the program. In addition, those data were used in the triangulation process. For example, oral responses were cross-checked with written responses. If by using triangulation and continuous observation the data c o l l e c t e d could not be confirmed or validated, the illuminative evaluation methodology could not be judged technically adequate. I f , on the other hand, the data were confirmed through both triangulation and continuous observation, the methodology could be judged technically adequate. Thus, a technically adequate methodology provides evidence that i s both objective and v a l i d as defined in t h i s section. 37 U t i l i t y An evaluation methodology should be f l e x i b l e yet produce res u l t s that are useful. The results should be relevant, important, and credible. Once the evaluation i s completed, the l o g i c a l expectation i s that decision-makers w i l l use the r e s u l t s to make rational decisions about future programming. However, a l l too often the results are ignored. With a l l the money, time, e f f o r t and s k i l l that went into the a c q u i s i t i o n of information, why does i t generally have so l i t t l e impact? Weiss (1972) suggests several reasons: evaluation results do not match the informational needs of decision-makers, results may not be relevant to the l e v e l of decision-maker who received them, or results lack clear d i r e c t i o n for future programming. As House (1977) says evaluations \"can be no more than acts of persuasion .... Expecting evaluation to provide compelling and necessary conclusions is to expect more than evaluation can d e l i v e r . But i f i t cannot produce the necessary, i t can p r o v i d e t h e credible, the plausible, and the probable'\" (pp. 5-6) . In t h i s study, c r e d i b i l i t y was an important standard for judging the u t i l i t y of illuminative evaluation methodology. Assurance of c r e d i b i l i t y in illuminative evaluation i s probably best obtained through frequent and thorough interaction with participants as the information develops. Thus, information with limited c r e d i b i l i t y can be i d e n t i f i e d e a s i l y and either eliminated or strengthened. Of course, such a process could expose the investigator to biases. While t h i s p o s s i b i l i t y is 38 undoubtedly r e a l , the investigator in t h i s study hedged against biases through such safeguards as triangulation and continuous observation described previously. Thus, triangulation and continuous observation are useful for determining both technical adequacy and u t i l i t y . Another approach to increase c r e d i b i l i t y of evaluations is \"participant evaluation\" (Campbell, 1979). It i s a move toward using participant judgments as part of the evaluation i t s e l f to provide c r e d i b i l i t y checks: P a r t i c i p a n t s . . . w i l l usually have a better observational position than... outside observers of a new program. They actually have experienced the preprogram conditions from the same viewing point as they have the special program. Their experience of the program w i l l have been more relevant, direct and v a l i d , less v i c a r i o u s . C o l l e c t i v e l y , their greater numerosity w i l l average out observer .idiosyncracies that might dominate the report of any one ethnographer. While participants are asked to generate a lot of data in program evaluation, rarely are they d i r e c t l y asked to evaluate the program, to judge the adequacy, to advise on i t s continuance, discontinuance, dissemination, or modification. Rather than evaluating programs, participants are usually asked about themselves and their own adequacy. We are thus wasting a l o t of well-founded opinions (Campbell, Note 3). This study used Campbell's approach to produce credible data. The participants were asked to evaluate the program, to judge i t s adequacy and to advise on i t s modification. Two other standards of u t i l i t y are the relevance and importance of the data. In t h i s study, relevance and importance of data were determined through interviews and informal discussions with the program planner, director and Advisory Counc i1 members. 39 F l e x i b i l i t y i s also an important standard for judging the u t i l i t y of illuminative evaluation. F l e x i b i l i t y i s extremely important in program evaluation, for innovative programs are often changed as planners learn what works and what does not, and as planners experiment and change their p r i o r i t i e s . One of the chief advantages of illuminative evaluation i s f l e x i b i l i t y , since i t does not have prescribed constraints. The f l e x i b i l i t y of illuminative evaluation methodology allows the investigator to match the evaluation to the program. In addition, f l e x i b i l i t y insures that the program is not required to stand s t i l l or stay the same in order to be evaluated (Edwards & Guttentag, 1975). A f l e x i b l e , personalized evaluation design b u i l t upon close observer-participant and observer-instructor interaction lends i t s e l f to the highly informal, personalized environment of adult education. The course of t h i s study was not be charted in advance, since the course was dependent on the actual operation of the program. Within the three stage framework of illuminative evaluation, an information p r o f i l e was assembled using data c o l l e c t e d from four areas: observations, interviews, questionnaires and h i s t o r i c a l documents. To make a judgment regarding u t i l i t y , illuminative evaluation methodology had to provide evidence that met the standards described above. To meet the standards set, the data co l l e c t e d by various means had to be credible, important and relevant. In addition, the methodology had to pose no constraints on the program. 40 Ef f ic iency The l a s t c r i t e r i o n used to determine the s u i t a b i l i t y of illuminative evaluation methodology i s e f f i c i e n c y . An e f f i c i e n t evaluation methodology should produce results that are worth more than their costs (Groteleuschen, 1980). This c r i t e r i o n i s the most d i f f i c u l t to precisely define in terms of standards and measurements. Unlike technical adequacy and u t i l i t y , which are frequently described in the l i t e r a t u r e , e f f i c i e n c y \"when treated at a l l , i s treated almost tangentially\" (Haller, 1974, p. 405). The following i s i l l u s t r a t i v e of the reason for tangential treatment by many evaluation s p e c i a l i s t s : \"It embarrasses me to admit that I do not know anything about the measurement of costs. I w i l l have to leave that to somebody else\" (Stake, 1973, p. 312). Decisions cannot be made e a s i l y in advance as to what percentage of program resources should be expended on evaluation. On the one hand, every d o l l a r and hour spent on evaluation is taken from other aspects of the program, and those costs become a very important factor when i t comes time to make decisions (Haller, 1974). On the other hand, evaluation can be regarded as an investment in the future of the program. The value of the investment w i l l vary with accountability demands against the program and the value of reporting program performance. Reasonable costs for an evaluation can be decided by estimating the significance of issues and the l i k e l y impact of the evaluation. In a sense, e f f i c i e n c y represents the r a t i o of 41 e f f o r t to e f f e c t . Although various evaluation needs w i l l e n t a i l d i f f e r e n t expenditures of resources, some form of evaluation i s possible within any budget. To determine e f f i c i e n c y of a methodology, cost estimates in terms of outlay (such as supplies, space), time expenditures (such as administrative e f f o r t , interviews), and expertise needed (such as program personnel or instructors) should be examined. Cost estimates of acquiring information also should take into account hidden costs. These include time lost to the program by evaluating, alternative use of funds, and human costs such as invasion of privacy, dangers of creating negative attitudes and reactions, or generating pressure on program personnel (Grotelueschen, 1980). These may be compared with the costs of not evaluating. There w i l l be situations in which i t i s possible to assess cost in d o l l a r s and others in which i t is not. When costs can be reasonably measured in d o l l a r s , i t i s usually desirable to do so, although i t sometimes requires a l i t t l e ingenuity. Dollars, as measuring devices, provide a convenient, generalizable and comparable estimate of the evaluation costs (Haller, 1974). Table 3 l i s t s the costs to be determined during the course of the evaluation. Costs in time and/or money are c o l l e c t e d in the following general categories: personnel, materials and equipment, participant time and evaluator time. 42 Table 3 L i s t of Evaluation Costs CATEGORY TIME(hrs.) COST(dollar) PERSONNEL Administrative Di rector Secretary C l e r i c a l Staff Professional Instructors Consultants MATERIALS & EQUIPMENT Supplies Space Equipment PARTICIPANT TIME During program After program EVALUATOR TIME Before program During program Waiting time After program CONTINGENCY COSTS 43 The question to be answered by the c r i t e r i o n of e f f i c i e n c y i s whether the same outcomes could have been achieved at less cost. As greater demands are placed on limited f i n a n c i a l resources, questions of evaluation e f f i c i e n c y w i l l demand and receive closer consideration. The c r i t e r i a of technical adequacy, u t i l i t y and e f f i c i e n c y described above were used to determine the s u i t a b i l i t y of illuminative evaluation methodology. Evidence was co l l e c t e d at the s i t e described in the next section and matched against the standards set out above. Study Site A r e s i d e n t i a l program was determined to be p a r t i c u l a r l y suitable for testing illuminative evaluation because i t has some unique advantages that do not exist in other program formats. (1) The advantage of detachment from the usual routine and the sense of freedom t h i s imparts. (2) The advantage of an environmental break which affords a challenge by the new environment to another pattern of behavior. (3) The advantage of concentration on one f i e l d of work without the usual d i s t r a c t i o n s . (4) The advantage of time for assimilation and integration. (5) The advantage of intimacy of students and tutors which reinforces new knowledge. (6) The advantage of a community s p i r i t which encourages tolerance and open-mindedness (Schacht, 1960, pp. 2-3). 44 The chief advantage of a r e s i d e n t i a l program over the more t r a d i t i o n a l types was that of removing the participant temporarily from his ongoing r e s p o n s i b i l i t i e s . This made i t possible for the evaluator to have continuous contact with the part i c i p a n t s . Continuous contact i s important for a methodology that r e l i e s on fieldwork techniques. The Justice Institute of B r i t i s h Columbia was selected as a si t e for thi s study since i t offered numerous r e s i d e n t i a l programs. The Justice Institute as a post-secondary educational i n s t i t u t i o n i s a member of B r i t i s h Columbia's post-secondary network of colleges and i n s t i t u t e s . It provides leadership and coordination to support, develop and deliver a wide range of training and education programs for people working within the f i e l d of j u s t i c e and public safety. These programs are designed to improve the quality of justice and public safety for the c i t i z e n s of B r i t i s h Columbia. The Land T i t l e School program of the Justice I n s t i t u t e was i d e n t i f i e d as an appropriate r e s i d e n t i a l program for testing illuminative evaluation methodology because i t had the unique r e s i d e n t i a l c h a r a c t e r i s t i c s described previously. Moreover, i t met Edwards & Guttentag's (1975) c r i t e r i a for formal evaluation. They s p e c i f i e d that formal evaluation was appropriate i f a program was new, newly changed, or about to change. The Land T i t l e School program met that c r i t e r i o n , because i t was new. In addition, the program planner was seeking answers to the following types of questions. Is thi s program a good idea? If so, what can we do to make i t work as well as possible? If not, 45 how can we devise something better, given exi s t i n g constraints? (Edwards & Guttentag, 1975, Garside, 1969; Parlett & Hamilton, 1976). For those reasons, the Land T i t l e School was determined to be a suitable program to test illuminative evaluation methodology. The Land T i t l e School program was developed by the Justice I n s t i t u t e of B r i t i s h Columbia for the Land T i t l e Branch, Ministry of Attorney General. The Director of Land T i t l e s f e l t personnel working in the o f f i c e s should have an opportunity to better understand the legal background of their work. The staff often had been requested to interpret various regulations and procedures which were part of the r e g i s t r a t i o n process. Although staff weren't obliged to offer such assistance, in r e a l i t y i t was often the best way to expedite individual cases. The Director f e l t the more knowledgeable the staff member, the easier i t would be to s a t i s f y requests for explanations. The goal of the Land T i t l e School program was to provide Land T i t l e personnel with job enrichment courses in three areas: land law theory, environmental awareness and supplemental t r a i n i n g . The most important area was land law theory. In land law courses, the legal context for the land registry process was presented. Topics included p r i n c i p l e s of B r i t i s h Columbia land law, law history, and Land T i t l e Act. The second area was environmental awareness. The work of a Land T i t l e O f f i c e r e f l e c t s and i s effected by a c t i v i t i e s of the wider community. These courses aimed to provide a broader understanding of the relationship between the Land T i t l e O ffice 46 and the environment in which i t operates. As an example, sessions were presented on urban land use, B r i t i s h Columbia land history, operation of a lawyer's o f f i c e and an anthropological view of land. The t h i r d area was supplemental t r a i n i n g . Supplemental tra i n i n g was not essential to prescribed job performance or successful completion of promotional exams. Supplemental tra i n i n g was designed to give the participants greater insight into how to perform their various tasks. Examples of supplemental training are legal descriptions, documentation and public r e l a t i o n s . Three courses made up the Land T i t l e School program. They were the Introductory, Intermediate and Advanced Courses. The three day Introductory Course for newly hired clerks provided an overview of B r i t i s h Columbia's Land T i t l e system and i t s legal heritage. The core legal knowledge course was presented in the two week Intermediate Course. The intensive program gave participants an understanding of the law re l a t i n g to land and the l e g a l , s o c i a l and economic implications of land use. The fiv e day Advanced Course concentrated on s p e c i f i c land registry i ssues. Each course was divided into a series of half-day mini-sessions. Each mini-session was taught by a d i f f e r e n t resource person drawn from university faculty, consultants, and lawyers. The mini-session content was developed from a needs assessment conducted by the Justice Institute and conformed to the three content areas i d e n t i f i e d as essential--land law theory, 47 environmental awareness and supplemental t r a i n i n g . The Land T i t l e School program as outlined above met the c r i t e r i a for an appropriate s i t e for testing illuminative evaluation methodology. F i r s t l y , the Land T i t l e School was a new program. Secondly, the program coordinator was anxious to have the program evaluated. Lastly, i t was offered on a r e s i d e n t i a l basis which meant that the investigator would have continuous contact with the participants. Summary This chapter was divided into two sections. The f i r s t section contained a description of the c r i t e r i a used for judging the s u i t a b i l i t y of illuminative evaluation methodology. Three c r i t e r i a appearing frequently in the l i t e r a t u r e were judged appropriate to t h i s study: technical adequacy, u t i l i t y , and e f f i c i e n c y . The l i t e r a t u r e suggested that an evaluation methodology should produce information that is (1) te c h n i c a l l y sound, (2) useful to some audience and (3) worth more to the audience than i t costs (Grotelueschen, 1980). In t h i s chapter, the c r i t e r i a were defined, standards set, and the method for c o l l e c t i n g evidence to judge each c r i t e r i o n was described. In order to meet the set standards, the methodology must f i r s t provide evidence that is both objective and v a l i d . Second, the evidence must be credible, important, and relevant while the methodology remains f l e x i b l e . Third, the 48 evidence must be worth more than the costs. The second section of t h i s chapter contained a description of the s i t e selected for testing illuminative evaluation methodology. A r e s i d e n t i a l program was determined to be p a r t i c u l a r l y suitable for testing t h i s methodology because i t had some unique advantages that did not exist in other program formats. The chief advantage of a r e s i d e n t i a l program over the more t r a d i t i o n a l types was that of removing the participant temporarily from his ongoing r e s p o n s i b i l i t i e s . This made i t possible for the evaluator to have continuous contact with the participants, because continuous contact i s important in a methodology that r e l i e s on fieldwork techniques. The Land T i t l e School program of the Justice Institute of B r i t i s h Columbia was i d e n t i f i e d as an appropriate r e s i d e n t i a l program for testing illuminative evaluation methodology because i t had the unique r e s i d e n t i a l c h a r a c t e r i s t i c s described in t h i s chapter. The Land T i t l e School was a r e s i d e n t i a l program designed for voluntary job enrichment. It was intended to appeal to those who wished a greater insight into their work than was required to competently perform assigned duties. The courses designed for the program were Introductory, Intermediate and Advanced. These courses would be offered yearly on a re s i d e n t i a l basis. The fact that the Land T i t l e School was a r e s i d e n t i a l program and new made i t an ideal s i t e for test i n g the s u i t a b i l i t y of illuminative evaluation methodology. 49 CHAPTER IV ILLUMINATIVE EVALUATION STRATEGY This chapter provides a discussion of how illuminative evaluation was employed in th i s study. B r i e f l y , the illuminative methodology was u t i l i z e d to evaluate each phase in the development of the Land T i t l e School program. It was used to describe and understand relationships that could be observed in a natural, non-contrived environment under controlled conditions. The f i r s t or p i l o t phase of the Land T i t l e School program represented a t r i a l and error period during which new approaches or procedures were t r i e d out on a rather f l e x i b l e and e a s i l y revised basis. During the development of the program, some modifications occurred. The f i r s t p i l o t course was the Intermediate Course (March, 1980). In order to evaluate t h i s course, the illuminative evaluation methodology was employed. The success of th i s course led to the development of two more p i l o t courses\u00E2\u0080\u0094Introductory Course (November, 1980) and the Advanced Course (December, 1980). For consistency, the three 50 stage illuminative methodology was u t i l i z e d again. The main objective of the evaluation of the p i l o t phase was to learn enough to further develop the program. The second or operational phase of the Land T i t l e School program consisted of modifications of a l l three courses. Based on what was learned in the p i l o t courses, these courses were modified so that they stood the greatest chance of success. This f i n a l phase was also evaluated using the illuminative methodology. In order to operationalize the illuminative methodology, a number of steps had to be taken at the observation, inquiry, and explanation stages (see Table 2). These three stages were repeated for each course. The result was six complete evaluation cycles since each of the three courses was run twice. In the remainder of th i s chapter, processes used in each evaluation cycle w i l l be described. P i l o t Phase The f i r s t evaluation cycle of the illuminative methodology started with the p i l o t Intermediate Course. In the observation stage, the investigator became familiar with the Intermediate Course through analysis of background documentation, discussions with the program coordinator and Land T i t l e Director. This f a m i l i a r i z a t i o n process enabled the investigator to proceed to the inquiry stage. 51 The inquiry stage was an interactive process between evaluator and program planner. This stage consumed the major portion of the investigator's time because a number of steps had to be taken. For example, issues had to be c l a r i f i e d , the evaluation process determined, questionnaires designed and data c o l l e c t e d . The following paragraphs present detailed descriptions of the steps taken during this inquiry stage. Issues C l a r i f i e d F i r s t , the following issues were discussed and c l a r i f i e d - -the purpose of the evaluation, what process should be used, how the information would be used, what topics should be selected for intensive investigation. When these issues had been c l a r i f i e d , the s p e c i f i c evaluation process was determined. Evaluation Process Evaluation processes can be divided into a number of lev e l s and evaluation can be carried out at any of these l e v e l s . In t h i s study, i t was decided to concentrate on the h i e r a r c h i c a l l e v e l s i d e n t i f i e d by Kirkpatrick (1967) and Hamblin (1974). These l e v e l s , s t a r t i n g from the lowest l e v e l , are: (1) participant reactions, or how well they l i k e d the program; (2) learning or the extent to which the program content was assimilated; (3) behavior change, or the change in job behavior; 52 (4) results, or the change in organizational variables. Kirkpatrick and Hamblin assumed there was a cause and effect chain l i n k i n g the four l e v e l s . This h i e r a r c h i c a l chain could break at any of i t s l i n k s . For example, a person could react c o r r e c t l y but f a i l to learn; or he could learn, but f a i l to apply his learning on the job; or he could change his job behavior, but t h i s could have no e f f e c t on the organization. The job of the evaluator i s to determine i f the l i n k s in the chain hold and i f they don't where they broke and why. Evidence can be c o l l e c t e d at any of these l e v e l s , however, the degree of d i f f i c u l t y in c o l l e c t i n g evidence at each l e v e l increases as one ascends the hierarchy. The participant reaction l e v e l i s the simplest and easiest l e v e l . As the hierarchy i s climbed, the d i f f i c u l t y and the resources required to measure actual program outcomes generally increase (Bennett, 1975). The d i f f i c u l t y often starts at the behavior change and results levels because the evaluator does not usually have adequate information about or control over the non-training a c t i v i t i e s of the organization. Furthermore, the techniques which are used to evaluate at those levels w i l l normally be those which the organization already has at i t s disposal and uses for other purposes. If the appropriate techniques such as productivity measurements or cost-benefit analysis don't already exist in the organization, evaluation at the higher levels w i l l be impossible because techniques of this kind cannot be introduced for education or t r a i n i n g purposes alone (Hamblin, 53 1974). Therefore, in many cases i t may be impractical to evaluate at every l e v e l . In the case of this study, i t was impractical to evaluate at the results l e v e l , for the Land T i t l e Branch did not have techniques set up for evaluation at the results l e v e l . Therefore, i t was decided to concentrate on the f i r s t three l e v e l s : participant reactions, learning, and behavior change. Eesides describing the lev e l s of evaluation for this study, Kirkpatrick and Hamblin gave detailed examples, and suggested procedures and techniques that could be used in most programs. This guidance was lacking in the l i t e r a t u r e on the illuminative evaluation methodology. Questionnaire Design In the process of designing questionnaires, the investigator discovered that some theorists claim a program should never be c a r r i e d out unless i t has clear objectives (Bennett, 1975; Patton, 1978; Stufflebeam, 1967). Others say that, although i t ' s permissible to carry out such a program, i t is impossible to evaluate i t . However, there are people who disagree with both philosophies (Hamblin, 1974; Warr, Bird & Rackham, 1970). This study was guided by the l a t t e r authors, for t h i s program did not have any measureable objectives. It was d i f f i c u l t to set measureable objectives for the Land T i t l e School program or s p e c i f i c sessions even at the participant reactions and learning l e v e l s , because so l i t t l e was known about the participants' previous state of learning. In 54 cases when objectives are not formulated in measureable terms, the best way of assessing reactions or learning changes may be simply to ask participants whether they find the course interesting, whether they think their knowledge has improved in s p e c i f i c areas and/or the most important or most job-relevant point they remember from a program. In the absence of behavioral objectives which specify precise evaluation c r i t e r i a , the evaluation must adopt open-ended techniques. Due to the absence of measureable objectives, the questionnaires developed for t h i s study contained mainly open-ended questions. The questionnaires developed are described below. Expectations Questionnaire (See Appendix A). This questionnaire was used to assess participant expectations of the Land T i t l e School prior to the start of the program and at i t s conclusion. The questionnaire, developed by Warr, Bird, and Rackham (1970, p. 65), was used to obtain feedback on participants' expectations regarding the usefulness, enjoyment, relevance, and importance of the course. Participants approach a course with a set of expectations which are important in determining their reactions to the program. Mini-session Questionnaires (See Appendix B). Since the course was divided into a series of mini-sessions taught by d i f f e r e n t resource persons, a questionnaire was designed that could be used at the end of each session. The mini-session questionnaire was used to assess participants' reactions and perceived learning. Participants were asked to rate each mini-session on a scale of 1 to 7 from \"not very...\" to 55 \"extremely...\" on interest, relevance to job, and new information gained. In other words, participants were asked to make decisions about the usefulness of s p e c i f i c content areas in terms of three important contexts: relevance to their own expectations, perceived application to their work situ a t i o n and their previous knowledge. In addition to rating scales, participants rated the mini-sessions using open-ended responses. The questionnaire, o r i g i n a l l y developed by Warr, Rackham and Bird (1970), was adapted for use in this study by the investigator and program coordinator. The questionnaire was administered immediately a f t e r each mini-session. Fi n a l Questionnaire (See Appendix C). This questionnaire, administered immediately following the program, was used to determine how participants f e l t about the program. The instrument consisted of rating scales to assess participant reactions to the program as a whole and participant b e l i e f s about the relevance of information gained to t h e i r work. A series of open-ended questions was also included. Follow-up Questionnaire (See Appendix D). Forty-five days after completion of the course, participants were sent a questionnaire consisting of several sections of the f i n a l questionnaire. The remainder of the instrument contained a series of open-ended questions about the relevance of information, the effect of course on participant job behavior, and participant s a t i s f a c t i o n with the general program. 56 Data C o l l e c t i o n Lastly, in the inquiry stage of the p i l o t Intermediate Course, data were co l l e c t e d through observation, informal interviews and questionnaires. The investigator used the following procedures to c o l l e c t data through questionnaires. After the program director and Director of Land T i t l e s o f f i c i a l l y started the p i l o t Intermediate Course, the investigator introduced the study, s o l i c i t e d cooperation and gave instructions. A l l participants were instructed to select a numerical identity code that was to be used on a l l questionnaires. Then the expectations questionnaire was given to a l l p a r t i c i p a n t s . A packet of mini-session questionnaires was provided to participants so they could rate each session that was taught by a d i f f e r e n t resource person. These questionnaires were to be completed at the end of each mini-session and returned to the investigator. The f i n a l questionnaire was administered on the f i n a l day of the course during the evaluation session. The follow-up questionnaire was sent to participants f o r t y - f i v e days after course completion. Table 4 summarizes the data c o l l e c t i o n schedule. The data c o l l e c t i o n step completed the inquiry stage of the illuminative methodology. The steps taken during the inquiry stage enabled the investigator to proceed to the t h i r d and f i n a l stage of the methodology. In the explanation stage, a l l the data c o l l e c t e d from the p i l o t Intermediate Course were analyzed using either q u a l i t a t i v e Table 4 Data C o l l e c t i o n Schedule TECHNIQUE SOURCE TIME H i s t o r i c a l Documents Program D i r e c t o r Two weeks p r i o r to c o u r s e Such as c o u r s e p r o p o s a l , minutes of meetings, l e t t e r s , c o u r s e b r o c h u r e . E x p e c t a t i o n s P a r t i c i p a n t s P r i o r to s t a r t of m i n i -T h i s q u e s t i o n n a i r e a s s e s s e d p a r t i c i p a n t e x p e c t a t i o n s p r i o r to s t a r t of s e s s i o n s on the f i r s t day c o u r s e . For example, would i t be usefu1-use 1 ess; h e l p f u l - u n h e l p f u l ; of c o u r s e and important-unimportant. M i n i - s e s s i o n a l P a r t i c i p a n t s End of m i n i - s e s s i o n T h i s q u e s t i o n n a i r e concerned i n t e r e s t of s e s s i o n i n f o r m a t i o n g a i n e d , r e l e v a n c e to job, l e n g t h of s e s s i o n , l e v e l of s e s s i o n . O b s e r v a t i o n I n v e s t i g a t o r D u r i n g c o u r s e s of p a r t i c i p a n t s d u r i n g c o u r s e , at breaks. I n t e r v i e w P a r t i c i p a n t s D u r i n g c o u r s e s d u r i n g breaks b e f o r e and a f t e r c l a s s . E x p e c t a t i o n s & F i n a l P a r t i c i p a n t s F i n a l day of c o u r s e These q u e s t i o n n a i r e s a s s e s s e d f u l f i l l m e n t of p a r t i c i p a n t s ' e x p e c t a t i o n s and t h e i r f e e l i n g s at the end of the course. F o l l o w - u p P a r t i c i p a n t s 45 days f o l l o w i n g l a s t T h i s q u e s t i o n n a i r e was concerned with r e l e v a n c e of i n f o r m a t i o n , e f f e c t day of c o u r s e of c o u r s e on job behavi o r and s a t i s f a c t i o n with general program. 58 or quantitative techniques. Since the class size was small (n=43), only simple a n a l y t i c a l procedures were used. A l l quantitative data, obtained through questionnaires only, were arranged numerically by participant identity code. This technique enabled the investigator to trace the ratings of individual participants i f required. Mean scores were calculated for each rating scale using a small table top computer. These scores were used to develop summary and trend charts. Qualitative data obtained from questionnaires, observations, and interviews were typed to aid analysis. Data from questionnaires were arranged numerically by participant identity code; thus, each participant's comments could be cross-checked with the quantitative data. A l l coded responses for each mini-session were combined and typed. This procedure was followed for each open-ended question on the f i n a l and follow-up questionnaires. The procedure of combining and typing responses f a c i l i t a t e d ease of analysis and eliminated biases created by an individual's handwriting. A l l the data co l l e c t e d were used to est a b l i s h an information p r o f i l e of the program. The data were used to answer the questions posed by the program coordinator mentioned previously. Is th i s program a good idea? If so, what can we do to make i t work as well as possible? If not, how can we devise something better, given existing constraints? When interpretation of the data was completed, a f i n a l report was written and sent to a l l participants, instructors, 59 Land T i t l e r e g i s t r a r s , and Advisory Committee members (Hasman, Note 4). With the d i s t r i b u t i o n of the report, the f i r s t cycle of the three stages of the illuminative methodology was complete. The f i r s t evaluation cycle required more investigator's time than any of the subsequent cycles because a l l procedures for each stage had to be established. The investigator had to quickly become familiar with Land T i t l e work, the personnel, and the program. Then, with the cooperation of the program coordinator, issues had to be c l a r i f i e d and defined and evaluation processes determined. Next operationalization procedures had to be developed, questionnaires designed, and data c o l l e c t e d and analyzed. F i n a l l y results had to be reported. The second and t h i r d cycles of the illuminative stages started after development of the p i l o t Introductory and Advanced Courses respectively. Since there were no changes made during these cycles, they w i l l be described together. The issues had been c l a r i f i e d , evaluation processes determined and questionnaires designed during the p i l o t Intermediate Course, so the amount of time and e f f o r t involved in the observation and inquiry stages of the second and t h i r d cycles was decreased s i g n i f i c a n t l y . A few modifications, however, were made during the observation stage. A decision was made to discontinue the i d e n t i f i c a t i o n code of each participant for two reasons. F i r s t , the group size had been reduced to a maximum of twenty and second, there was no benefit in tracing individual responses. 60 Besides eliminating the numerical codes, some questionnaires needed a l t e r a t i o n s . For example, the sessional, f i n a l and follow-up questionnaires were modified as a result of suggestions from the program coordinator, the investigator and Advisory Council members (see Appendix E). The expectations questionnaire was replaced by an \"expectations warm-up\" exercise. This questionnaire was changed, because the program coordinator wanted to eliminate some of the evaluation forms. The modification provided both expectations data and a group \"warm-up\" (See Appendix E). After making the above modifications for both p i l o t Introductory and Advanced Courses, the investigator co l l e c t e d data by interviews, observations, and questionnaires. The data c o l l e c t i o n methods were the same as those described in the f i r s t c ycle. The explanation stage followed data c o l l e c t i o n . This stage followed the same procedures set down during the f i r s t cycle except that the responses were not coded by identity number and a desk calculator was used instead of a computer. A l l open-ended responses, interviews and observation notes were typed so that the investigator could interpret the re s u l t s . Like the p i l o t Intermediate Course evaluation, a report was written and di s t r i b u t e d (Hasman, Note 4). The second and t h i r d cycles through the illuminative stages were then complete. As mentioned, the time and e f f o r t involved during these evaluation cycles was reduced s i g n i f i c a n t l y due to the procedures established during the f i r s t cycle. 61 Operational Program The last three evaluation cycles were completed during the operational phase of the Land T i t l e School program. In the f i n a l cycles through the illuminative stages, the only changes made were to the program i t s e l f . After a few minor program al t e r a t i o n s , the three courses were in f i n a l operational form. Evaluations of each course followed the procedures established during the f i r s t three cycles of the illuminative stages. Since a l l three cycles were the same, they w i l l be described together. The investigator reconfirmed the focus of the evaluation and proceeded to gather data through observations, interviews, and questionnaires. In due course, the data were organized, interpreted, and reported. Upon presentation of the report to the program coordinator, participants and Advisory Council members, the f i n a l three cycles of the illuminative stages were completed. Summary This chapter provided a discussion of how illuminative evaluation methodology was employed in th i s study. The illuminative evaluation methodology was u t i l i z e d to evaluate a l l three courses of the Land T i t l e School program. Since each course ran in both a p i l o t and an operational form, six 62 evaluation cycles of the three-stage illuminative methodology were used. In the f i r s t cycle of the illuminative evaluation methodology, time was equally divided between the three stages: observation, inquiry, and explanation. During the second and t h i r d cycles of the methodology, time was reduced in the observation and inquiry stages. This i s due to the fact that the procedures had been established during the f i r s t cycle. Since only minor alt e r a t i o n s were made, the investigator did not focus e f f o r t s on those stages. The explanation stage also required somewhat less time. This was due to smaller class size which reduced the amount of data. The last three cycles of the illuminative stages encompassed the three course operational program. In these cycles, no changes were made to the evaluation procedures, so the majority of the investigator's time was spent on the explanation stage. The analysis took about the same time for a l l six cycles. 63 CHAPTER V RESULTS The l i t e r a t u r e reviewed for thi s study suggested that an evaluation methodology should produce information that is (1) technically sound, (2) useful to some audience and (3) worth more to the audience than i t costs (Groteleuschen, 1980). These c r i t e r i a described in Chapter III were selected for thi s study. Evidence used to assess the s u i t a b i l i t y of illuminative evaluation in rel a t i o n to these c r i t e r i a w i l l be presented in this chapter. Technical Adequacy A t e c h n i c a l l y adequate evaluation methodology w i l l produce evidence which meets the standards of o b j e c t i v i t y and v a l i d i t y as defined in Chapter I I I . In order to be judged technically adequate, the methodology must produce data that can be 64 confirmed. In other words, the burden of proof moves from the investigator to the information i t s e l f . Illuminative evaluation methodology encouraged c o l l e c t i o n of data from multiple sources and perspectives in order to cross-check and confirm r e s u l t s . This would ensure objective and v a l i d information. Procedures used in t h i s study for confirming results were triangulation and continuous observation. The evidence presented in the paragraphs below support the technical adequacy of the illuminative evaluation methodology. Triangulation Triangulation was used extensively in t h i s study to provide evidence of o b j e c t i v i t y and v a l i d i t y . Triangulation forces the observer to combine multiple data sources, research methods, and theoret i c a l schemes in the inspection and analysis.... 11 forces him to s i t u a t i o n a l l y check the v a l i d i t y of his causal propositions....It d i r e c t s the observer to compare his subject's theories of behavior with his emerging theoreti c a l scheme.... (Denzin, 1971, p. 177). The f i r s t example of triangulation involved cross-checking q u a l i t a t i v e and quantitative data from each mini-session. The quantitative data consisted of mean scores from each mini-session while the q u a l i t a t i v e data consisted of typed responses from each mini-session. As a result of cross-checking these two types of data, the ratings of each mini-session were confirmed. Add i t i o n a l l y , comments from the f i n a l questionnaire confirmed and further validated the mini-session data. 65 Besides cross-checking and confirming, the q u a l i t a t i v e data were used for interpreting quantitative data, because s t a t i s t i c s cannot t e l l why someone rated a session high or low. For instance, several participants rated sessions low on the variable \"new information gained\" while the majority rated the same session high. The investigator wondered why the ratings were low. The following comments illuminated the problem. This material was just covered by (Mr. Registrar) before this course. Everybody 'jogs'. I am aware of stress and health hazards and I would think most of the class would be. In another instance, a session rated poorly on \" i n t e r e s t \" and \"information gained\" but high on \"relevance.\" Without the q u a l i t a t i v e information, the investigator would not know i f there were problems with content and/or instructor. The q u a l i t a t i v e information revealed the problem: I've waited two weeks es p e c i a l l y for t h i s lesson and was thoroughly disappointed. I think t h i s s p e c i f i c area could prove b e n e f i c i a l to us a l l and gone into some depth. The l e v e l of 'presentation' we received was far below that which could prove useful and relevant. In addition to cross-checking and interpretation, q u a l i t a t i v e information provided a richness of description d i f f i c u l t to capture in a quantitative summary. Campbell (1975) noted that recognizing these functions of q u a l i t a t i v e data \"immediately legitimizes the 'narrative history' portion of most evaluative reports\" (p. 9). He suggested the importance of 66 q u a l i t a t i v e data be given formal recognition in the planning and execution of evaluations. \"Evaluation studies are uninterpretable without t h i s , and most would be better interpreted with more\" (p. 9). A second example of triangulation was comparison of responses on pre- and post-expectations questionnaires. Three variations of the expectations questionnaire were used. The f i r s t v a r i a t i o n consisted of comparing pre-course expectations using a semantic d i f f e r e n t i a l type scale to open-ended comments on the f i n a l questionnaire. The second va r i a t i o n consisted of the following. Participants were asked to write down their expectations on the f i r s t day of c l a s s . Then, on the f i n a l day, they were asked to check their expectations from the f i r s t day and see i f they were f u l f i l l e d . If their expectations were not met, they were asked to explain. For example, one participant wrote \"I'm hoping not to be bored.\" Comments from the f i n a l questionnaire re f l e c t e d that the participant had not been bored: I came into t h i s course expecting to be bored and was surprised about the competence of the instructors and the time they took to explain the answers to a l l our questions. In some cases, participants had certain expectations as a result of talking to former participants. The following comment i l l u s t r a t e s how participants can come with erroneous expectations and have them changed by the course. Did not have expectations but was more i n f o r -mative and interesting than I was led to believe by previous students. 67 The t h i r d variation consisted of participants presenting their expectations o r a l l y on the f i r s t day of c l a s s . Each participant's expectations were noted and typed. On the f i n a l day, participants wrote whether their expectations had been f u l f i l l e d . F i n a l l y , after comparing the written responses to the typed ones, the participants were asked to note any differences. It was found that many participants f e l t the course would be job training as r e f l e c t e d in t h i s comment: Thought i t would be more in l i n e with my work duties, but because of the way the system (L.T.O.) i s structured, I can see why i t was more general. A t h i r d example of triangulation involved data from interviews. The information from interviews was used to cross-check quantitative data from the mini-sessions. The investigator checked to determine i f the participants were saying one thing o r a l l y and another thing in writing. A f i n a l example of triangulation involved u n s o l i c i t e d information. Two participants c a l l e d the investigator to say that the information gained from the p i l o t Intermediate Course plus their notes had helped them pass their promotional exam. In another instance, several participants t o l d one instructor that he should be hired by Land T i t l e to offer his mini-session to a l l employees. These u n s o l i c i t e d comments further confirmed responses from the courses. 68 Continuous observation If one way to establish the v a l i d i t y of data i s through the use of triangulation, another way i s through the use of repeated observations. Eisner (1975) makes the point that \"One of the reasons why i t i s important... to have extended contact with an educational situation i s to be able to recognize events or ch a r a c t e r i s t i c s that are a t y p i c a l . One needs s u f f i c i e n t time in a si t u a t i o n to know which q u a l i t i e s characterize i t and which do not\" (p. 218). Thus, v a l i d i t y i s , to some extent, a function of the amount of time and e f f o r t which the investigator invests in repeated and continuous observation. Not only w i l l the investigator be able to d i f f e r e n t i a t e t y p i c a l from atypical situations, or identif y pervasive q u a l i t i e s which characterize a si t u a t i o n , but he w i l l also know when to give c r e d i t to the occasional idiosyncratic observation which nevertheless ca r r i e s great insight and meaning (Guba, Note 2). Continuous observation and extensive contacts are hallmarks of illuminative evaluation methodology. Continuous observation provides a variety of information that could not have been co l l e c t e d by any other means. Through observation, the investigator discovered the mood of the group. The information gained through observation was passed on to upcoming speakers, for the mood of each group was d i f f e r e n t . For example, one group didn't ask any questions during presentations, although at coffee breaks they asked many questions. Upon questioning the partic i p a n t s , the investigator discovered that the participants were hesitant to interrupt the speaker because they f e l t i t was 69 either rude or disruptive. This information about the group was passed on to succeeding speakers. The investigator also was able to observe the development of the group. Some participants were shy or unaccustomed to a pa r t i c i p a t i v e group. When thi s was observed, the investigator alerted the program coordinator. These insights were passed on to ensuing speakers who used the information to help the process along; for example, the speaker might use an \"ice breaker\" or warm-up exercise. Through continuous observation and extensive contact, the investigator was able to observe group trends, a l l e v i a t e misconceptions about the program and check how the program was being received. The above evidence supports the technical adequacy of the illuminative evaluation methodology. Since the q u a l i t a t i v e data were confirmed and validated by the quantitative data, t h i s methodology met the standards of o b j e c t i v i t y and v a l i d i t y as defined in Chapter I I I . U t i l i t y In order to meet the c r i t e r i o n of u t i l i t y , the illuminative evaluation methodology must remain f l e x i b l e while producing useful, v a l i d information. This methodology should produce evidence that meets the standards of relevance, importance, c r e d i b i l i t y , and f l e x i b i l i t y as defined in Chapter I I I . The following evidence is presented in support of the c r i t e r i o n of 70 u t i l i t y . Relevance and Importance Evidence of data relevance to the program planner is shown in the types of programming changes made as a result of the p i l o t Intermediate Course findings. For example, in subsequent courses, descriptive brochures were di s t r i b u t e d ; class size was reduced; time a l l o c a t i o n s for mini-sessions were varied; more breaks were scheduled; group discussions and f i e l d t r i p s were used rather than straight lecture; more audio-visual aids were introduced; and resource persons with teaching experience were sought. These changes made as a result of the evaluations were used to further develop the program. Written and oral reports by the investigator were important to members of the Land T i t l e School Advisory Council. The members found the information provided was useful in determining future directions and policy for the Land T i t l e School program. For instance, the investigator reported the lack of class discussion in the p i l o t Intermediate Course. Based on that information, the Council recommended a maximum class size of twenty which they f e l t would f a c i l i t a t e class discussion. The q u a l i t a t i v e information was relevant and useful to the resource people. Participant comments were forwarded to speakers, so that they could use the information to improve their courses. As evidenced through comments, a number of participants did not understand the relationship between certain mini-sessions and the work that they did. The participants 71 found i t d i f f i c u l t to bridge the gap between theory and practice. When an instructor was made aware of that d i f f i c u l t y , he t r i e d to explain the relevance or relationship. Furthermore, as a result of participant comments, several instructors requested tours through the Land T i t l e O ffice in order that they might better understand the needs of the pa r t i c i p a n t s . C r e d i b i l i t y Assurance of c r e d i b i l i t y was obtained through involvement. In order to obtain participant and decision-maker involvement in th i s evaluation, i t was necessary to gain their confidence by demonstrating interest in their opinions and willingness to act on their advice. The participants were asked to evaluate the course through s p e c i f i c , detailed comments and suggestions regarding changes, additions/subtractions, and modifications of each mini-session. The participants were t o l d that the course was designed to suit their needs. If i t was not relevant, important and/or appropriate, i t was their r e s p o n s i b i l i t y to respond accordingly on the questionnaires. The following is an example of the way participants were encouraged to evaluate the program: Please r e f l e c t on your experiences of the past week when answering the following items. Be candid in expressing your feelings, whether they are positive or negative. Make your comments very s p e c i f i c for they w i l l help us tremendously when we plan the next course. In addition, participants were given copies of the evaluation report, so they would have tangible evidence that the 72 information they generated was being read and - used. For example, several courses were developed from participant ideas. When that occurred, i t was mentioned to subsequent groups. A l l this helped establish the c r e d i b i l i t y of the evaluator and the u t i l i t y of the evaluation. The c r e d i b i l i t y of the study for the program coordinator and Advisory Council members was enhanced by involving them in the decision-making process. For example, they were involved in decisions concerning the nature, purpose, and methods of evaluation. Involvement of those persons encouraged them to keep informed by reading reports and attending meetings. F l e x i b i l i t y F l e x i b i l i t y i s inherent in most methodologies based on the n a t u r a l i s t i c paradigm. However, the question to be answered in this study i s can the illuminative evaluation methodology be f l e x i b l e yet produce technically adequate and useful data. One of the chief advantages of illuminative evaluation methodology i s i t s f l e x i b i l i t y , for i t does not have prescribed constraints. The illuminative methodology allows f l e x i b i l i t y in data c o l l e c t i o n techniques, types of data used and programming changes to name a few. Evidence of f l e x i b i l i t y i s presented below. Data Collect ion Techniques. A number of data c o l l e c t i o n techniques were used\u00E2\u0080\u0094observation, interview, questionnaire and h i s t o r i c a l documents. The illuminative evaluation methodology suggested how these techniques could be used and encouraged the 73 use of a l l four. However, certain techniques were more suitable for a s p e c i f i c stage of the methodology than were others. For example, h i s t o r i c a l documents were used only during the observation stage, because the investigator needed to gain insight and understanding of the project's background and development. During the inquiry stage, interviews and meetings were used to determine the nature, purpose, and focus of the evaluation. Interviews as well as observations and questionnaires were used exclusively to c o l l e c t data during each course. The use of these varied techniques allowed the investigator f l e x i b i l i t y as well as a means of cross-checking and confirming r e s u l t s . Questionnaire Development. F l e x i b i l i t y was c r i t i c a l during o r i g i n a l questionnaire design and subsequent modifications. By experimenting with evaluation instruments, the investigator developed and modified questionnaires. Since neither the program nor the mini-sessions had clear cut objectives, the investigator had to get as much information as possible by u t i l i z i n g open-ended questions. Then a s i f t i n g , narrowing and focusing process was used to reject those questions that were useless and refine and improve those that were useful. The questionnaires were redesigned and modified at the end of the p i l o t Intermediate Course and again at the completion of the p i l o t Introductory and Advanced Courses. This r e f i n i n g process ensured that only the most relevant data would be c o l l e c t e d . For example, after the p i l o t Intermediate Course, the Land T i t l e Director wanted to delete the question which referred to 74 part i c i p a n t s ' perceived relevance of the course. Since the purpose of the course was not job training but job enrichment, he f e l t that those questions might mislead the participants into believing the course should be job relevant. Thus, the job relevance questions were eliminated in subsequent questionnaires. Another example of change concerned expectations. It was decided during review of the p i l o t Intermediate Course that information of prior expectations could be gained through a warm-up exercise prior to the start of the program thereby eliminating one form. Data Collect ion Methods. In some instances the method of data c o l l e c t i o n had to be modified. For example, in the p i l o t Intermediate Course, i t was important to know i f the \" l e v e l of presentation\" and \"length of session\" were appropriate to the p a r t i c i p a n t s . The investigator t r i e d to measure those variables s t a t i s t i c a l l y , but the information obtained was not useful. For example, the ratings of \"length of session\" were mid-range with no s i g n i f i c a n t variance; that i s , the mini-sessions were neither too long nor too short. The ratings of \" l e v e l of presentation\" also clustered around the mid-point although there was some variance. Moreover, the participants made no written comments to c l a r i f y those ratings. The investigator, however, heard comments that seemed to contradict the mid-range ratings: \"The hardest part was to stay seated because I'm always running around the o f f i c e . \" Therefore, the investigator changed the data c o l l e c t i o n method during the p i l o t Intermediate Course from rating scales to 75 observations and interviews in order to obtain v a l i d and useful data. Through observations and interviews, i t was discovered that mini-sessions were too long. More importantly, participants needed more breaks. In subsequent courses, rating scales for \"length of session\" and \" l e v e l of presentation\" were eliminated, since they provided no useful information. Information on \" l e v e l \" and \"length\" was obtained through informal interviews and observations in a l l succeeding courses. Types of data. The data c o l l e c t e d in th i s study were not limi t e d to one type; both q u a l i t a t i v e and quantitative data were co l l e c t e d . The quantitative format enabled the investigator to produce summaries of mini-sessions quickly and accurately. These data were used for comparing individual mini-sessions as well as for comparing the ratings of two di f f e r e n t groups on the same mini-session. Comparisons could also be made of entire courses. For example, the p i l o t Intermediate Course could be compared with the operational Intermediate Course. Although reading and summarizing numerous lengthy responses to open-ended questions was a very time-consuming procedure, the q u a l i t a t i v e data gave the investigator insight into participants' perceptions. The following is a composite of participant perceptions from the p i l o t Intermediate Course. These composite perceptions give far more depth, richness and feeling than numerical ratings. I r e a l l y enjoyed t h i s course and obtained much valuable knowledge. It w i l l give me more confidence in my day-to-day work. I r e a l i z e the work involved in planning this course for us and I think i t ' s t e r r i f i c . It never hurts to learn 76 more about your job. The more knowledgeable I am about my job, the more I ' l l enjoy i t . Result better work! I feel we -should have a course l i k e t h i s once a year. Let me know the data. Bravo!! Besides providing insight, open-ended questions have other advantages that are useful for t h i s study. F i r s t , open-ended questions permitted v e n t i l a t i o n of participant feelings. Participants were given the opportunity to express their exact opinions in an open-ended response. If they had been asked to simply check items, they might have f e l t forced into responses that did not exactly match their a t t i tudes. For instance, participants were asked how relevant the course was to their jobs. The following are some examples of their responses: Dealt too much on his own point of view; got the impression he was trying to f l o g his book. This lecture i s not relevant to our jobs as i t was neither a presentation of new material nor an in-depth treatment of known information. Although the theories involved were relevant and possibly interesting the relevance seemed too far removed from our own experiences. Second, open-ended questions produced responses which drew the evaluator's attention to a s i t u a t i o n or outcome that was unanticipated when the course was developed and/or when questionnaires were designed. I had no idea of the problems faced by my fellow clerks in other o f f i c e s . It highlighted my weaknesses so now I can improve them. The c a l i b r e of the lecturers was excellent. A l l knowledge that a person gains over his/her 77 l i f e w i l l e ffect what and who that person i s or becomes in that a l l persons continue through the i r l i f e to change. That l i f e i s not as simple or as boring as i t sometimes seems! Third, open-ended questions did not l i m i t the range of possible answers as would closed response questions. For example, i f you wanted to know about participants' salient impressions of the program, an open-ended question asking for impressions i s better than a checklist of possible responses: If you were to reorganize the course, what would you change, leave the same, etc.? Explain. Program Changes. Due to f l e x i b i l i t y of the evaluation methodology, the program i t s e l f could be modified without inv a l i d a t i n g the study. This f l e x i b i l i t y i s extremely important in a developing program. For example, as a result of the evaluation of the p i l o t Intermediate Course, several programming changes were made. More \"stretch breaks\", f i e l d t r i p s , and longer lunch breaks were a few of the changes. These changes were made because participants were not accustomed to being students and found i t quite d i f f i c u l t to s i t for long periods of time. In addition, the evaluation results of the p i l o t Intermediate Course showed that the group size (n=43) was too large to f a c i l i t a t e interaction among participants and between participants and instructors. Since the size seemed to i n h i b i t discussion, classes were reduced to a maximum of 20. Due to the f l e x i b i l i t y of the methodology, these changes improved the program but did not effect the v a l i d i t y of thi s study. The above evidence supports the u t i l i t y of the illuminative 78 evaluation methodology. Since the methodology remained f l e x i b l e while producing useful, v a l i d information, i t met the standards defined in Chapter I I I . Ef f ic iency An e f f i c i e n t evaluation methodology should be worth more to the recipients than i t costs. This is not an easy c r i t e r i o n to measure for a l l evaluations require time and money. In order for the illuminative evaluation methodology to be judged e f f i c i e n t , the question to be answered by t h i s study was whether the same outcomes could have been achieved at less cost, because the investigator is resposible to make the most out of the resources a v a i l a b l e . The following evidence i s presented in support of the c r i t e r i o n of e f f i c i e n c y . It was impossible to attach a d o l l a r value to the time invested by various people during this study, for the bookkeeping involved would have increased the costs unnecessarily. As an a l t e r n a t i v e , Table 5 was developed to graphically summarize the r e l a t i v e amount of time spent on each stage of the illuminative evaluation methodology during the development of the Land T i t l e School program. This table shows that the greatest amount of time and, therefore money, was invested in the p i l o t Intermediate Course. The cost of the p i l o t Intermediate Course was high due to high developmental costs. As the time spent was reduced so were the costs. By the Table 5 Effort and Time Spent on Illuminative Stages P I L O T P R O G R A M OBSERVATION INQUIRY EXPLANATION INTERMEDIATE COURSE INTRODUCTORY COURSE 5 WEEKS 3 WEEKS O P E R A T I O N A L P R O G R A M OBSERVATION INQUIRY EXPLANATION 2 1/2 WEEKS 2 1/2 WEEKS ADVANCED COURSE APPROXIMATE EFFORT: = MINIMAL \u00E2\u0080\u00A2 3 WEEKS 2 1/2 WEEKS APPROXIMATE TIME: =AVERAGE =MAXIMUM = 1/2 WEEK =1 WEEK =1 1/2 WEEKS =2 WEEKS 80 time of the operational program, the costs had s t a b i l i z e d . The material and supplies costs remained constant throughout the program. Based on the experience in this study, an investigator should expect to spend between four and five weeks on a one week course of t h i s kind. Investigator time i s divided between the a c t i v i t i e s of the observation, inquiry and explanation stages of the methodology (see Table 2). The following explanation of Table 5 i s divided according to the three stages of the illuminative methodology. The observation stage of the p i l o t Intermediate Course involved considerable investigator time, minimal program coordinator time and no participant time. In this stage, the investigator became familiar with the program mainly through h i s t o r i c a l documents and discussions with the program coordinator. Being thoroughly familiar with the Land T i t l e School program, after the p i l o t Intermediate Course, the investigator required less time for this stage. As noted on the table, in subsequent courses, time was reduced s i g n i f i c a n t l y u n t i l i t reached a s t a b i l i z e d l e v e l . Based on the experience in th i s context, a person should expect to spend about one week or less on t h i s phase. The inquiry stage of the p i l o t Intermediate Course also required considerable investigator, coordinator and Advisory Council member time (approximately two weeks). It was during th i s stage that the issues were c l a r i f i e d , evaluation process determined, questionnaires designed and data c o l l e c t e d . Participants were involved in the data c o l l e c t i o n phase. 81 Participant time involvement was approximately the same for each course since only minor a l t e r a t i o n s were made to the questionnaires. The time spent on th i s stage during the p i l o t Introductory and Advanced Courses was reduced due to the fact that the evaluation design and procedures had been established during the p i l o t Intermediate Course. Only minor alt e r a t i o n s were made to questionnaires, so the investigator as well as coordinator and Council members did not invest much time. Since no changes were made to the evaluation procedures or questionnaires during the operational program, the time invested remained stable. The explanation stage involved both investigator and s e c r e t a r i a l time (approximately two weeks). This stage of the p i l o t Intermediate Course required s i g n i f i c a n t l y more time than subsequent courses for two reasons: (1) the class size was large (n=43); and (2) a l l the data from questionnaires were coded by student identity number. In the following courses the class size was reduced to a maximum of twenty participants which meant a reduction in the amount of data c o l l e c t e d . In addition, student coding was dropped for i t did not provide useful information. These two changes resulted in both reduced s e c r e t a r i a l and investigator time to approximately one week. Both q u a l i t a t i v e and quantitative data were c o l l e c t e d during t h i s study. Coding, typing and analyzing q u a l i t a t i v e information involved considerable s e c r e t a r i a l and investigator time. In comparison, coding and analyzing the quantitative information was rapid since only mean scores were calculated. 82 The actual analysis time was the same for a l l the courses after the p i l o t Intermediate Course (approximately one week). Although the q u a l i t a t i v e information required more time for c o l l e c t i n g , coding and analyzing than quantitative information, i t was more relevant and important to the resource people, program coordinator and Advisory Council members. Evidence in support of thi s statement is presented in the U t i l i t y section (\"Relevance and Importance\") of t h i s chapter. In addition, evidence in \"Types of Data\" in the U t i l i t y section support the u t i l i t y of the more costly q u a l i t a t i v e data. The above evidence was presented in support of the c r i t e r i o n of e f f i c i e n c y . The evidence presented i s only one aspect of the e f f i c i e n c y c r i t e r i o n . It addresses the c r i t e r i o n from the data cost perspective. The value of cost was determined by the investigator based on comments from Advisory Council members, participants, and resource persons not by an impartial person. Summary In order to determine the s u i t a b i l i t y of the illuminative evaluation methodology for evaluating r e s i d e n t i a l adult education programs, i t was tested on the Land T i t l e School program. This chapter contained the results of testing illuminative evaluation methodology on the selected s i t e . Evidence from the testing was compared with the standards set by 83 the pre-specified c r i t e r i a contained in Chapter I I I . Table 6 summarizes the c r i t e r i a and standards that were used to determine the s u i t a b i l i t y of illuminative evaluation methodology. The results indicated that: (1) The data were confirmed through triangulation and continuous observation. (2) The data c o l l e c t e d by various means were credible, important and relevant. (3) The data collected could not have been obtained by less costly methods. Thus, the evidence of s u i t a b i l i t y s a t i s f i e d the standards set by t h i s study. T a b l e 6 C r i t e r i a and Standards Used For Determining S u i t a b i l i t y of I l l u m i n a t i v e E v a l u a t i o n Methodology CRITERIA STANDARD TECHNIQUE Tec h n i c a l - Adequacy O b j e c t i v i t y and T r i a n g u l a t i o n p r o v i d e s evidence of o b j e c t i v i t y and v a l i d i t y by c r o s s - c h e c k i n g V a l i d i t y q u a l i t a t i v e and q u a n t i t a t i v e data, by a i d i n g i n t e r p r e t a t i o n of q u a n t i t a t i v e d a t a , and by comparing pre-post q u e s t i o n n a i r e r e s u l t s . Continuous O b s e r v a t i o n p r o v i d e s e v i d e n c e of v a l i d i t y through r e p e a t e d o b s e r v a t i o n . I t a l s o p r o v i d e s i n f o r m a t i o n u n a t t a i n a b l e by any other means, i . e . mood of group or development of group. Ut i 1 i t y Relevance and Importance W r i t t e n and o r a l r e p o r t s . These were used to p r o v i d e i n f o r m a t i o n f o r d e c i s i o n -making, f o r r e v i s i o n of m i n i - s e s s i o n m a t e r i a l , and f o r p r o v i d i n g p a r t i c i p a n t s with evidence t h a t the i n f o r m a t i o n they g e n e r a t e d was used. Cred i b i 1 i ty T h i s was i n s u r e d by involvement. A d v i s o r y committee members were i n v o l v e d i n d e c i s i o n - m a k i n g ; p a r t i c i p a n t s e v a l u a t e d the c o u r s e g i v i n g d e t a i l e d comments and s u g g e s t i o n s r e g a r d i n g changes. F1 ex i b i1 i ty F l e x i b i l i t y of t h i s methodology must be m a i n t a i n e d w h i l e p r o d u c i n g t e c h n i c a l l y adequate and u s e f u l data. F l e x i b i l i t y was demonstrated by changes made i n d a t a c o l l e c t i o n t echniques, q u e s t i o n n a i r e development, types of data used and program changes. E f f i c i e n c y C o s t / b e n e f i t Time and e f f o r t i n v e s t e d by v a r i o u s people d u r i n g t h i s study. Comparison of c o s t s of a n a l y z i n g q u a l i t a t i v e v e r s u s q u a n t i t a t i v e d a t a . Cost of data c o l l e c t i o n compared to u t i l i t y of d a t a as determined by the d e c i s i o n maker. \u00E2\u0080\u0094 CO 85 CHAPTER VI SUMMARY, CONCLUSIONS, IMPLICATIONS AND RECOMMENDATIONS Summary For many years, adult educators have been interested in evaluating their programs. U n t i l recently, most formal educational evaluation studies have used the c l a s s i c a l paradigm which derives i t s methodology from experimental psychology. However, there has been increasing resistance to evaluations of t h i s type (Parlett & Hamilton, 1976; Smith, 1976; Stake, 1978) and a movement to use an alternative paradigm related to s o c i a l anthropology has emerged. This a l t e r n a t i v e , the n a t u r a l i s t i c paradigm, requires a fundamentally d i f f e r e n t evaluation methodology from that used with the c l a s s i c a l paradigm. A number of evaluation models have emerged from the movement. These models have close philosophic and operational t i e s with the n a t u r a l i s t i c paradigm. Their emergence at this time argues strongly for the u t i l i t y of n a t u r a l i s t i c inquiry for 86 the f i e l d of education, and helps make the case that n a t u r a l i s t i c inquiry should be investigated as an alternative methodology. Illuminative evaluation was selected for t h i s study from the emergent models because i t i s r e l a t i v e l y new and based on the n a t u r a l i s t i c paradigm. It i s a dynamic evaluation process which i s not ti e d to a single treatment, predetermined goals or outcomes, but rather focuses on the actual operations of a program over a period of time (Patton, 1978). This i s extremely important in program evaluation, for innovative programs are often changed as planners experiment and change their p r i o r i t i e s . Residential programs are p a r t i c u l a r l y suitable for testing the value of the illuminative evaluation methodology because this type of program has some unique c h a r a c t e r i s t i c s that do not exist in other program formats. For example, r e s i d e n t i a l programs d i f f e r from most t r a d i t i o n a l types of programs because the participants are temporarily removed from their ongoing r e s p o n s i b i l i t i e s . This makes i t possible for the investigator to have continuous contact with the participants. Continuous contact i s important for a methodology that r e l i e s on fieldwork techniques. The Land T i t l e School of the Justice Institute of B r i t i s h Columbia was i d e n t i f i e d as an appropriate r e s i d e n t i a l program for testing illuminative evaluation methodology because i t s underlying assumptions (see Table 1) f i t the evaluation needs of the Land T i t l e School program. The methodology: 87 (1) allows for the study of open, changing systems and emergent problems; (2) encourages the representation of multiple viewpoints and value perspectives; (3) focuses on program a c t i v i t i e s and issues rather than outcomes; (4) provides a means of studying spontaneous events, situations, and c r i s e s ; (5) is sensitive to the context and setting. To determine the s u i t a b i l i t y of thi s methodology, evidence co l l e c t e d during the study was compared with the standards set by the pre-specified c r i t e r i a : technical adequacy, u t i l i t y and e f f i c i e n c y . It was f e l t that for the purpose of t h i s study, the illuminative evaluation methodology should produce information that was technically sound, useful to some audience and worth more to some audience than i t costs (Groteleuschen, 1980). Conclusions Evidence of the degree to which illuminative evaluation met the technical adequacy, u t i l i t y and e f f i c i e n c y c r i t e r i a was col l e c t e d during the Land T i t l e School program. Techniques such as interviews, questionnaires, and observations were used to c o l l e c t the evidence. The evidence was analyzed using q u a l i t a t i v e techniques to determine whether the methodology met 88 the standards set by the c r i t e r i a . The remainder of thi s section provides descriptive interpretations of the evidence co l l e c t e d on each c r i t e r i o n . Technical Adequacy Two major c r i t i c i s m s of illuminative evaluation and other n a t u r a l i s t i c inquiries appear in the l i t e r a t u r e . (1) Personal interpretations cannot be objective. (2) Descriptive studies cannot be v a l i d . In response to the c r i t i c i s m of o b j e c t i v i t y : any evaluation study\u00E2\u0080\u0094whether i t conforms to the c l a s s i c a l or n a t u r a l i s t i c paradigm\u00E2\u0080\u0094requires s k i l l e d human judgment. Human judgment is necessary at every stage of any study whether i t be descriptive or experimental (Guba & Lincoln, 1981; Patton, 1978). For example, i t i s used in choosing samples, constructing questionnaires, administering questionnaires, choosing s t a t i s t i c a l treatment, interpreting s t a t i s t i c a l data and presenting findings. Responses to the second c r i t i c i s m of v a l i d i t y are presented in the following quotes: a methodology, whether descriptive or i n f e r -e n t i a l , experimental or non-experimental, can seldom obtain v a l i d results unless c l o s e l y associated with substantive knowledge of the process being studied. (Bennett & Lumsdaine, 1975, p. 20). Evaluation data are never clearcut and absolute: studies are always flawed in some way, and there are always questions of r e l i a b i l i t y and v a l i d i t y . Error-free instruments do not and cannot exist in the measurement of complex human, s o c i a l , behavioral, and psychological phenomena (Patton, 1978, p. 180). 89 Although the process of c o l l e c t i n g data through various techniques i s time consuming and expensive, i t i s worthwhile. This process helps ensure the technical adequacy of the findings. Once information has been confirmed by two or more techniques, the uncertainty of i t s interpretation i s greatly reduced. Each technique contains i t s b i t of error, perhaps s u f f i c i e n t to cause rejection i f that were a l l that was ava i l a b l e . But when a series of b i t s of evidence are triangulated and a l l evidence tends in the same d i r e c t i o n , that d i r e c t i o n assumes greater b e l i e v a b i l i t y (Guba, 1968; Webb et a l . , 1966). In t h i s study a certain l e v e l of o b j e c t i v i t y and v a l i d i t y was attained by cross-checking and confirming data collected from h i s t o r i c a l documents, observations, questionnaires and interviews. The data col l e c t e d using the above techniques were validated and confirmed through triangulation and continuous observation procedures. No study can be completely objective and v a l i d as pointed out in the discussion above. However, by using a variety of techniques to cross-check, confirm and validate findings, c r i t i c i s m s of \"lack of o b j e c t i v i t y and v a l i d i t y \" are reduced. From the evidence presented in Chapter V, t h i s study met the standards of o b j e c t i v i t y and v a l i d i t y . 90 U t i l i t y The evidence presented in Chapter V demonstrated that the illuminative evaluation methodology could be f l e x i b l e yet produce useful results. Since programs are not s t a t i c , evaluation methodologies to be useful should be f l e x i b l e . Therefore, a model such as illuminative evaluation that unfolds through successive phases and strategies is more useful than a model based on uniformity and r i g i d i t y . In t h i s study, the evaluation design was not formulated in advance but continuously evolved and was modified as the evaluator interacted with participants and decision-makers. Data c o l l e c t i o n instruments and techniques also evolved and were refined as the program progressed. Due to the f l e x i b i l i t y of the model, these changes did not effect the v a l i d i t y or o b j e c t i v i t y of the data. The benefits of c o l l e c t i n g both q u a l i t a t i v e and quantitative data from a variety of perspectives are presented in Chapter V. Besides confirming and v a l i d a t i n g the quantitative data, the advantage of obtaining q u a l i t a t i v e data from many perspectives i s that the investigator can build on emergent insights by c o l l e c t i n g descriptive information that gives a useful, meaningful representation of what happened. A comprehensive description of what happened greatly aids judgment, decision-making, and u t i l i t y . In contrast, quantitative data are easy to code and analyze but have a number of weaknesses. It i s doubtful that much can be learned by asking participants to rate their perceptions. Scales l i k e 5-4-3-2-1 tend to obscure facts of f e e l i n g , not to 91 c l a r i f y them. Some participants use the scale backwards; others make a policy of \"never giving a 5 or a 1\". In addition, simply knowing that outcomes are high, low or di f f e r e n t does not reveal much about what to do about them. The results of the evaluation met the u t i l i t y standards of relevance, importance and c r e d i b i l i t y . The res u l t s obtained from the evaluations were credible to the investigator, program coordinator and resource people. Because the results were credible, they were u t i l i z e d . Meeting the pre-specified u t i l i t y standards required the involvement of a wide range of people--participants, program coordinator, Advisory Council members and the investigator. The investigator spent considerable time establishing rapport and coordinating the involvement and feedback process. Although those processes were time consuming and, therefore costly, they were important for this study for three reasons. F i r s t , the investigator established rapport and involved participants in the evaluation process. For example, the c r e d i b i l i t y and technical adequacy of the findings were s i g n i f i c a n t l y enhanced by checking out information with the part i c i p a n t s . In addition, with participant involvement in the evaluation, the investigator could be reasonably sure that the findings r e f l e c t e d the insights and judgments of the group. Second, coordinator and Council member involvement helped ensure the relevance and c r e d i b i l i t y of the data c o l l e c t e d for them. The investigator discussed the findings with the coordinator and Advisory Council members, helped them draw 92 implications and recommendations for action from the data and monitored the results of modifications made on the basis of the evaluat ion. Third, the involvement of the coordinator and Council members ensured the u t i l i z a t i o n of the data c o l l e c t e d . U t i l i z a t i o n of information c o l l e c t e d from evaluations is a c r u c i a l indicator of the value of evaluation. \"The basic rationale for evaluation i s that i t provides information for act ion... unless i t gains serious hearing when program decisions are made, i t f a i l s in i t s major purpose\" (Weiss, 1972, p. 318). Ef f ic iency Illuminative evaluation studies are c o s t l y , because they involve investigators for seemingly inordinate durations at considerable f i n a n c i a l expense to the program. This study was no d i f f e r e n t . Much time was spent in the developmental stages of t h i s study. The investigator needed to i n i t i a t e the sustaining relationships that made the evaluation possible and c r e d i b l e - -establishing rapport, and coordinating the involvement and feedback process. In addition, considerable time was spent developing a s p e c i f i c evaluation plan because the illuminative methodology lacked procedural guidelines to f a c i l i t a t e the evaluation a c t i v i t i e s . This i s a major weakness of the methodology and a reason for high developmental costs. Although the development and design costs were high during the i n i t i a l cycles of the methodology, they became less costly 93 in subsequent cycles for two reasons. One, the investigator did not have to spend as much time as in the i n i t i a l developmental stage. Two, the investigator became more p r o f i c i e n t in carrying out evaluation a c t i v i t i e s each time the program cycled through the illuminative stages. Thus, the illuminative evaluation methodology appears to be more e f f i c i e n t for on-going r e s i d e n t i a l programs than for \"one-shot\" programs. This study produced mainly q u a l i t a t i v e data. Qualitative data are both d i f f i c u l t to analyze and require more time for analysis than quantitative data. Qualitative data can also be confusing. But even under these circumstances, i f careful judgment i s exercised the data retains i t s value for decision-making. For example, occasionally a \"balancing phenomenon\" occurs in which comments contradict each other in almost equal numbers. Seventeen people w i l l say that the program moved too slowly; eighteen w i l l say that i t was too rapid. What does this r e a l l y t e l l an evaluator? Probably not that the program was either too slow or too fast, but that the design needs to provide more time for i n d i v i d u a l i z a t i o n . It might also t e l l the resource person that there i s too l i t t l e ongoing feedback during the session. The main weakness of q u a l i t a t i v e data i s that the resultant descriptions are often long and involved. A decision-maker or program planner does not always have time to read a long report in preparation for a decision. Thus, the investigator has to be selective in the information presented. The selection process is a potential source of bias which can harm the v a l i d i t y of 94 re s u l t s . The cost of the methodology must be compared with i t s benefits. The benefits of using q u a l i t a t i v e data presented in Chapter V showed the information was useful to the program coordinator, Advisory Council members, instructors and pa r t i c i p a n t s . Encouraging the participants to give their opinions and feelings made them feel part of the planning process. To enhance t h i s , they were given copies of the evaluation reports, so they could see both their contribution and the program in t o t a l i t y . The q u a l i t a t i v e information was relevant and useful to the resource people, for i t enabled them to improve their sessions by providing adequate information on which to act. The Advisory Council members and program coordinator found the data relevant, useful, and important for both decision-making and program planning. Thus, the benefits outweighed the costs of the evaluation methodology so i t s e f f i c i e n c y was adequate for t h i s program. Although the methodology i s not highly e f f i c i e n t , the loss in e f f i c i e n c y i s balanced by gains in the other two c r i t e r i a . In considering the importance of the c r i t e r i a , u t i l i t y i s a far more important factor in evaluation than e f f i c i e n c y , for far too many evaluation studies gather dust. If the evaluation i s not useful and u t i l i z e d , i t i s i n e f f i c i e n t regardless of the actual cost. Based on the interpretation of the findings of th i s study, the illuminative evaluation methodology is judged suitable for evaluating r e s i d e n t i a l adult education programs, for the 95 evidence c o l l e c t e d met the standards of technical adequacy, u t i l i t y and e f f i c i e n c y . An important question remains for investigators. Under what conditions does an evaluation methodology based on the n a t u r a l i s t i c paradigm provide the best guidance for evaluations? Neither the l i t e r a t u r e nor th i s study has shown that methodologies based on either p a r a d i g m \u00E2\u0080\u0094 c l a s s i c a l or n a t u r a l i s t i c \u00E2\u0080\u0094 i s i n t r i n s i c a l l y better than the other. The f i n a l choice between methodologies in \"any inquiry or evaluation ought to be made on the basis of the best f i t between assumptions...and the phenomenon being studied\" (Guba & Lincoln, 1981, p. 56). Implications and Recommendations Implications The findings of this study have the following implications for \u00E2\u0080\u00A2 researchers and pra c t i t i o n e r s desiring to use the illuminative evaluation methodology as a means of determining the value of a program. (1) The degree of f i t between the assumptions of the illuminative evaluation methodology and the program's evaluation needs i s an important consideration in choosing t h i s methodology. Illuminative evaluation methodology is p a r t i c u l a r l y suitable for evaluating adult education programs 96 which have complex goals that are d i f f i c u l t to define precisely and thus defy quantitative measurement. Because the i n i t i a l developmental costs are high, illuminative evaluation methodology i s more e f f i c i e n t when used with on-going programs. (2) The benefits of using t h i s type of evaluation increase i f decision-makers are involved, since the more decision-makers are involved with a project, the more they are apt to u t i l i z e the information. (3) The participants should also be a c t i v e l y involved to maximize the u t i l i t y of t h i s type of evaluation. By involving them, the investigator can be reasonably sure that the findings w i l l r e f l e c t the insights and judgments of the group. (4) Investigators desiring to use t h i s methodology need to develop good interview and observation s k i l l s , for these s k i l l s are c r i t i c a l to data c o l l e c t i o n . (5) Investigators need to become aware of their own biases. They should try to be understanding and open to d i f f e r i n g points of view, at the same time avoiding collusion or over-involvement which tend to create biases. Recommendations Based on the study completed and reported, the following recommendations are presented for those desiring to do further research or those desiring to employ this methodology. (1 ) Further work needs to be done to develop s p e c i f i c tasks, questions, a c t i t i v i e s and/or procedures which could guide implementation of each stage of the illuminative evaluation 97 methodology. Because the illuminative evaluation methodology i s weak on specifying evaluation a c t i v i t i e s , i t i s open for abuse and i t also diminishes the p o s s i b i l i t y of generalizing r e s u l t s . Procedural a c t i v i t i e s such as those presented in Table 2 could be used and/or further refined. (2) Educational programs which prepare individuals to become adult educators could be expanded to include the n a t u r a l i s t i c approach to evaluation. This approach could be assimilated into current adult education curriculum as an alte r n a t i v e to the c l a s s i c a l approach. (3) Further studies should be done to build up the understanding of the illuminative evaluation methodology. If evaluation studies using illuminative evaluation methodology were more accessible and/or published more frequently, investigators would be able to determine the s u i t a b i l i t y of t h i s methodology for other types of programs. (4) Further evaluations of r e s i d e n t i a l programs using illuminative evaluation methodology need to be done in order to contribute to the understanding of th i s methodology for r e s i d e n t i a l program formats. Further studies w i l l aid generalization of results, for there are obvious l i m i t a t i o n s to thi s study such as evaluator bias, limited g e n e r a l i z a b i l i t y , and small n's. In addition, the investigator played a dual role of program coordinator and evaluator. (5) Further studies could be done to determine the s u i t a b i l i t y of illuminative evaluation methodology for evaluating other types of adult education programs. Can 98 elements of thi s methodology be applied to other program formats? What modifications need to be made to the methodology? The findings in th i s study challenge adult educators to become more creative and resourceful in their approach to evaluation. They also present a model of one di r e c t i o n t h i s c r e a t i v i t y may take. 99 REFERENCES Apps, J . Problems in continuing education. New York: McGraw-H i l l , 1979. Bale, R.L. & Molitor, J.A. The Mountain-Plains Educat ion and Economic Development Program: Success or Failure?, 1978. (ERIC Document Reproduction Service No. ED 153 073). Bass, B.M. & Vaughan, J.A. Training in industry: The management of learning. Belmont, CA: Wadsworth Publishing Co., 1966. Beckhard, R. How to plan and conduct workshops and conferences. New York: Association Press, 1956. Bennett, C. Up the hierarchy. Journal of Extension, 1975, _1_3 (March, A p r i l ) , 7-12. Bennett, C.A. & Lumsdaine, A.A. Social program evaluation: Def i n i t i o n s and issues. In Bennett, C.A. & Lumsdaine, A.A. (eds.) Evaluat i on and exper iment: Some c r i t i c a l issues in assessing s o c i a l programs. New York: Academic Press, 1975. Bernstein, I.N. & Freeman, H.E. Academic and entrepreneur i a l research. New York: Russell Sage Foundation, 1975. Blackwell, B.L. & Bolman, W.M. The p r i n c i p l e s and problems of evaluation. Community Mental Health Journal, 1977, _1_3 (2), 175-186. Blaney, J.P. & McKie, D. Knowledge of conference objectives and effect on learning. Adult Education, 1969, J_9 (2), 98-1 05. Bogdan, R. & Taylor, S.J. Introduction to q u a l i t a t i v e research methods. New York: John Wiley & Sons, 1975. Bunch, M.B. Summat ive evaluat ion of Mountain-Plains Volume 11. 1976. (ERIC Reproduction Service No. ED 150 3987\"] Bringle, R.R. Eff e c t s of human relations laboratory training on f l e x i b i l i t y and attitudes toward supervision. (Doctoral d i s s e r t a t i o n , Oregon State University, 1967). Campbell, D.T. Assessing the impact of planned s o c i a l change. In Lyons, G.M. (ed.) Soc i a l research and public polic i e s . Hanover, N.H.: Public A f f a i r s Center, Dartmouth College, 100 1 975. Campbell, D.T. Degrees of freedom and the case study. Comparative P o l i t i c a l Studies, 1975 (2) 178-193. Campbell, D.T. & Stanley, J.C. Experimental and quasi - experimental designs. Chicago: Rand McNally, 1963. Conrad, R.W. Summative evaluation of Mountain-Plains Volume I I I . 1976\"! CERIC Document Reproduction Service NO. ED 150 399). Cox, W.F., J r . Development and implementation of a t r a i n i n g program for educat ional research and developmental personnel, Final Report. 1974. (ERIC Document Reproduction Service No. ED 095 157). Cronbach, L.J. Course improvement through evaluation. Teachers College Record, 1963, 64^ 672-683. Cronbach, L.J. (ed.) Toward reform of program evaluat ion. San Francisco: Josey-Bass Publishers, 1980. Davis, L. & McCallon, E. Planning, conducting, and evaluating workshops. Austin, Texas: Learning Concepts, 1974. Deantonio, E. Evaluation of the community school relations workshop, 1971 -1972, 1973. (ERIC Document Reproduction Service No. ED 089 147). Densmore, M.L. An evaluative analysis of selected university conference programs conducted at Kellogg Center for Continuing Education, Michigan State University. (Doctoral d i s s e r t a t i o n , Michigan State University, 1965). Denzin, N.K. The logic of n a t u r a l i s t i c inquiry. Social Forces, 1971, (50) 166-182. Devlin, L.E. The influence of directed instruction on learning in a conference setting. (Unpublished M.A. Thesis, University of Chicago, 1966). Dickinson, G. & Lamoureux, M.E. Evaluating educative temporary systems. Adult Educat ion, 1975, xxv, (2), 81-89. Edelbach, R.D. A study of the influence of r e s i d e n t i a l and non-r e s i d e n t i a l s k i l l training programs on trainee s e l f -esteem, occupational i d e n t i f i c a t i o n , and work motivation. (Doctoral d i s s e r t a t i o n , Rutgers University, The State University of New Jersey, 1973). Edwards W. & Guttentag, M. Experiments and evaluations: A re-examination. In Bennett, C.A. & Lumsdaine, A.A. (eds.) Evaluat ion and experiment: Some c r i t i c a l issues in 101 assessing soc i a l programs. New York: Academic Press, 1975. Eisner, E.W. The perceptive eye: Toward the reformaton of educational evaluation. Stanford: Stanford Evaluation Consortium, December 1975. Erickson, F. Some approaches to inquiry in school-community ethnography. Anthropology and Education, 1977(8). Fienberg, S.E. The c o l l e c t i o n and analysis of ethnographic data in educational research. Anthropology and Education, 1977(8), 50-57.. Freeman, H.E. & Solomon, M.A. Evaluat ion studies review annual, volume 6. Beverly H i l l s : Sage Publications, 1981. Garside, D. Short-term r e s i d e n t i a l colleges: Their origins and value. Studies in Adult Education, 1969, J_ (1), 2-30. George, C. & Green, D. Summary of outcome evaluat ion report for preparing educat ional t r a i n i n g consultants: s k i l l s t r a i n i n g (PETC-I), 1976. (ERIC Document Reproduction Service No. ED 134 627). Grotelueschen, A.D. Program evaluation. In Knox, A.B. & Associates. Developing, administering and evaluatinq adult educat ion. San Francisco: Jossey-Bass Publishers, 1 980') . Guba, E.G. Toward a methodology of n a t u r a l i s t i c inquiry in educational evaluat ion. CSE Monograph Series in Evaluation. Los Angeles: Center for the Study of Evaluation, 1978. Guba, E.G. & Lincoln, Y.S. E f f e c t i v e evaluation. San Francisco: Jossey-Bass Publishers, 1981. Guttentag, M. & Saar, S. Evaluat ion studies review annual, volume 2. Beverly H i l l s : Sage Publications, 1977. Haller, E.J. Cost analysis for educational program evaluation. In Popham, W.J. Educat ional evaluat ion: Current applications. Berkeley: McCutchan Publishers, 1974. Halverson, M.B. Facing the r e a l i t i e s : Some conference planning p r i n c i p l e s . Adult Leadership, 1974, 2 (2), 47-49. Halverson, M.B. & Thiesse, J. P r a c t i c a l evaluation and the regional conference approach. Lifelong Learning: The Adult Years, 1979, m (3), 4 - 6 . Hamblin, A.C. Evaluat ion and control of t r a i n i n g . London: McGraw-Hill, 1974. Hamilton, D. Making sense of curriculum evaluation. In 1 02 Shulman, L. (Ed.). Review of Education, V o l . 5 . Itasca, I I . : F.E. Peacock, 1 9 7 7 . Havelock, R.G. & Havelock, M.C. Preparing knowledge li n k i n g change agents in educat ion: A materials and training development project. F i n a l Report, 1 9 7 1 . (ERIC Document Reproduction Service No. ED 0 5 6 2 5 7 ) . Houle, CO. Residential cont inuing educat ion. Notes and Essays on Education for Adults # 7 0 . Publications in Continuing Education. New York: Syracuse University Publications, 1 9 7 1 . House, E. The conscience of educational evaluation. Teachers College Record, 1 9 7 2 , 73 ( 3 ) , 4 0 5 - 4 1 4 . House, E. The logic of evaluative agreement. Los Angeles: Center for the Study of Evaluation, UCLA, 1 9 7 7 . House, E. Evaluating with v a l i d i t y . Beverly H i l l s : Sage Publications, 1 9 8 0 . Jenkins, R.L., I I I . Summative evaluat ion of the Mountain- Plains community development component. An af feet ive evaluation report , 1 9 7 6 . [ERIC Document Reproduction Service No. ED 150 3 6 9 ) . Katz, D.S. & Morgan, R.L. A h o l i s t i c strategy for the formative evaluation of educational programs. In Borich, G.D. (ed.), Evaluating educational programs and products. E n g l e c l i f f s , N.J.: Educational Technology Publications, 1 9 7 4 . Kirkpatrick, D.L. Evaluation of t r a i n i n g . In Craig, R.L. & B i t t e l , L.R. (eds.) Training and development handbook. New York: McGraw-Hill, 1 9 6 7 . Kuhn, T. The structure of s c i e n t i f i c revolutions. Chicago: University of Chicago Press, 1 9 7 0 . Lacognata, A.A. A comparison of the effectiveness of adult resident i a l and non-resident i a l learning situations. Chicago: Center for the Study of L i b e r a l Education for Adults, 1 9 6 1 . L i p p i t t , R. Studies in experimentally created autocratic and democratic groups. University of Iowa Studies in Childwelfare, 1 9 4 0 , J_6 ( 3 ) , 4 5 - 1 9 8 . Lutz, T.W. & Ramsey, M.A. The use of anthropological f i e l d methods in education. Educat ional Researcher, 1 9 7 4 ( 3 ) , 5 -9 . Miles, W.R. Illuminative evaluation for formative decision-103 making. Evaluation Review, 1981, 5 (4), 479-499. M i l l e r , H.L. Teaching and learning in adult education. New York: Macmillan Co., 1964. Milozarek, G.J. F i e l d test and outcome milestone report for prepar ing educational t r a i n i n g consultants: Consulting (PETC-II), 1976. (ERIC Document Reproduction Service No. ED 126 157). Newcomb, T. Personality and soc i a l change. New York: Holt, Rinehart, & Winston, 1943. Parlett, M. & Hamilton, D. Evaluation as illumination: A new approach to the study of innovatory programs. In Glass, G.V. (ed.) Evaluat ion studies: Review Annual, Volume I. Beverly H i l l s , Ca.: Sage, 1976. Parle t t , M. & Hamilton, D. Evaluation as illumination: A new approach to the study of innovatory programmes. In Hamilton, D. et a l . (Eds.) Beyond the numbers game. London: Macmillan, 1977. Parle t t , M. & King, J.G. Concentrated study:A pedagogic innovation observed. London: Society for Research into Higher Education, 1971. Pattison, R.M. (ed.) Counselling educationally disadvantaqed adults. Indianapolis: Indiana State Department of Public Instruction, Division of Adult Education, 1968. (ERIC Document Reproduction Service No. ED 023 015). Patton, M.Q. Util i z a t i o n - f o c u s e d evaluat ion. Beverly H i l l s : Sage, 1978. Peterson, K.P. Influence of evaluative conditions and pre-conference contact on participants* evaluation of a conference. (Masters thesis, University of B r i t i s h Columbia, 1971). Posavac, E.J. & Carey, R.G. Program evaluation: Methods and case studies. Englewood C l i f f s : Prentice-Hall, 1980. Rippey, R.M. (ed.) Studies in transactional evaluation. Berkeley, Ca: McCutcheon, 1973. Rist, R.C. Ethnographic techniques and the study of an urban school. Urban Education, 1975(10), 86-108. Roberts, E.R. & Holmes, A. The professional development series for work experience educat ion. Evaluat ive report. 1971. (ERIC Reproduction Service No. ED 124 756). Schacht, R.H. Weekend learn ing in the U.S.A. Notes and Essays 104 on Education for Adults #29. Chicago: Center for the Study of Li b e r a l Education for Adults, 1960. Scriven, M. The methodology of evaluation. In Stake, R.E. (ed.) Curriculum evaluation. AERA Monograph Series on Curriculum Evaluation (Vol. O, Chicago: Rand McNalley, 1967. Scriven, M. Objectivity and s u b j e c t i v i t y in educational research. In Thomas, L.G. (Ed.) Philosophical redirection of educat ional research. 71st Yearbook, Part I, National Scoiety for the Study of Education. Chicago: The University of Chicago Press, 1972. Scruggs, J.A. Florida migratory chiId compensatory program state conference (September 1975). Evaluation report, 1976. (ERIC Reproduction Service No. ED 148 507). Smallegan, M. A comparison of two t r a i n i n g formats for persons with varying interpersonal needs. Adult Educat ion, 1971, 2J_ (3) , 1 66-1 76. Smith, M.L.; Gabriel, R.; Schott, J. & Podia, W.L. Evaluation of the e f f e c t s of outward bound. In Glass, G.V. (ed.) Evaluation Studies Review Annual, Volume I. Beverly H i l l s : Sage, 1976. Stake, R.E. The countenance of educational evaluation. Teachers College Record, 1967, 68, 523-540. Stake, R.E. Evaluation design, instrumentation, data c o l l e c t i o n , and analysis of data. In Worthen, B.R. & Sanders, J.R. Educat ional evaluat ion: Theory and practice. Worthington, Ohio: Charles A. Jones Publishing Co., 1973) . Stake, R.E. Evaluating the arts in education: A responsive approach. Columbus, OH: M e r r i l l , 1975. Stake, R.E. The case study method in s o c i a l inquiry. Educational Researcher, 1978, 7 (Feb.), 5-8. Stake, R.E.; Brown, C.; Hoke, G.; Maxwell, G.; & Friedman, J. Evaluat ing a regional envi ronmental learning system. Urbana: Center for Instructional Research and Curriculum Evaluation, 1979. Steele, S.M. Program evaluation: A broad d e f i n i t i o n . Journal of Extension, 1970 (summer), 5-17. Stewart, CW. A study of the results of a program of continuinq education for protestant clergy. Bloomfield H i l l s , Michigan: Institute for Pastoral Studies, 1965. (ERIC Document Reproduction Service No. ED 021 190). 105 Stufflebeam, D.L. The use of and abuse of evaluation in T i t l e I I I . Theory into Practice, 1967, 6 (June), 126-133. Stufflebeam, D.L. & Webster, W.J. An analysis of alternative approaches to evaluation. In Freeman, H.E. & Solomon,' M.A. (eds). Evaluat ion studies review annual, volume 6. Beverly H i l l s : Sage Publications, 1981. Sutton, E.W. Analysis of research on selected aspects of adult education. (Doctoral d i s s e r t a t i o n , Florida State University, 1969). Torrence, P.E. The Tuskegee experiment in adult t r a i n i n g . Adult Leadership, 1966, J_5 (3), 83-84; 96. Touzel, T.J. The s t a b i l i t y of attitudes in a longitudinal study resulting from a summer conference on i n d i v i d u a l i z e d inst r u c t i o n . (Doctoral d i s s e r t a t i o n , The University of Tennessee, 1975). V a l l a , D.C. An evaluative study of three workshops for the aging in nine planning d i s t r i c t s of V i r g i n i a . (Doctoral d i s s e r t a t i o n , V i r g i n i a Polytechnic I n s t i t u t e and State University, 1975). Warr, P.B.; Bird, M.W. & Rackham, N. Evaluat ion of management tra i n i n g . London: Gower Press, 1970. Webb, et a l . Unobtrusive Measures. Chicago: Rand McNally & Co., 1966. Weiss, C.H. Evaluating educational and s o c i a l action programs: A t r e e f u l l of owls. In Weiss, C. (ed.). Evaluating act ion programs. Boston: Allyn & Bacon, 1972. Wientge, K.M. & Lahr, J.K. The influence of soc i a l climate on adult achievement. The Impact of a resident i a l experience on learning and attitude change of adult students enrolled in an evening credit c l a s s . St. Louis: Washington University, 1966. [ERIC Document Reproduction Service ED 01 1 371). Wohllenben, A. The pattern of anxiety in r e s i d e n t i a l conferences. (Unpublished M.A. Thesis, University of Chicago, 1965). Wolf, R.L. The use of j u d i c i a l evaluation methods in the formulation of educational p o l i c y . Educat ional Evaluat ion and Policy Analysis, 1979, 57 (1), 19-28. Worthen, B.R. A look at the mosaic of educat ional evaluation accountabi1ity. Portland: Northwest Regional Educational Laboratory, 1974. 106 REFERENCE NOTES Note 1 Rusnell, D. Decisions in the design of evaluation. Unpublished manuscript, 1978. Note 2 Guba, E.G. Toward a methodology of n a t u r a l i s t i c inquiry in educational evaluation. Unpublished manuscript, February 1, 1978. Note 3 Campbell, D.T. Qualitative knowing in action research. Unpublished manuscript, 1979. Note 4 Hasman, R.M. Land T i t l e School Evaluation Report. Unpublished manuscript, 1980. 1 07 APPENDIX A Expectat ions Questionnaire LAND T I T L E SCHOOL P l e a s e g i v e us your f r a n k r e a c t i o n s and o p i n i o n s ; they w i l l h e l p us e v a l u a t e t h i s c o u r s e and improve f u t u r e programs. A l l i n f o r m a t i o n i s c o n f i d e n t i a l and w i l l be used o n l y to improve f u t u r e programs. We would be g r a t e f u l i f you would use an ID number of y o u r c h o i c e on t h i s form as i t w i l l h e l p us to a n a l y z e the r e s u l t s , P l e a s e use the same ID number on a l l f o r m s . ID t P l e a s e g i v e your o p i n i o n o f t r a i n i n g by c i r c l i n g the a p p r o p r i a t e number i n each of the o p i n i o n s c a l e s below. Example: , I 2 3 5 6 7 , r 1 1 1 \u00E2\u0080\u0094 \u00E2\u0080\u0094 i 1 1 I extremely very fairly 'n fairly very extremely between -V-COMPLICATED SIMPLE c o m p l i c a t e d 1 2 3 4 5 6 7 s i m p l e unprac t i c a l 1 2 3 4 5 6 7 p r a c t i c a l a c c u r a t e 1 2 3 4 5 6 7 i n a c c u r a te d u l l 1 2 3 4 5 6 7 i n t e r e s t i n g d i f f i c u l t 1 2 3 4 5 6 7 easy h e l p f u l 1 2 3 4 5 6 7 u n h e l p f u l f a s t 1 2 3 4 5 6 7 slow i m p o r t a n t 1 2 3 4 5 6 7 u n i m p o r t a n t How u s e f u l do you t h i n k t h i s t r a i n i n g w i l l be? 1 2 3 4 5 6 7 u s e l e s s u s e f u l How e n j o y a b l e do you t h i n k t h i s t r a i n i n g w i l l be? 1 2 3 4 5 6 7 d i d n ' t e n j o y e n j o y e d i t i t v e r y much v e r y much 109 LAND T ITLE SCHOOL P l e a s e g i v e us your f r a n k r e a c t i o n s and o p i n i o n s ; they w i l l help us e v a l u a t e t h i s c o u r s e and improve f u t u r e c o u r s e s . A l l i n f o r m a t i o n i s c o n f i dent i a 1 and w i l l be used o n l y t o improve f u t u r e c o u r s e s . ID # Based on your e x p e r i e n c e o f the pas t 2 weeks , p l e a s e g i v e your o p i n i o n of the Land T i t l e S c h o o l by c i r c l i n g the a p p r o p r i a t e number in each of the o p i n i o n s c a l e s be low. E xamp1e: C o m p l i c a t e d I m p r a c t i c a l D u l l D i f f i c u 1 t H e l p f u l F a s t I m p o r t a n t 1 2 3 CS 5 6 7 e x t r e m e l y v e r y f a i r l y T n f a i r l y v e r y e x t r e m e l y between to o s h o r t LAND T I T L E SCHOOL t o o l o n g 1 2 3 fr 5 6 7 S i m p l e 1 2 3 fr 5 6 7 P r a c t i c a 1 1 2 3 fr 5 6 7 I n t e r e s t i n g 1 2 3 fr 5 6 7 E a s y 1 2 3 fr 5 6 7 No t H e l p f u l 1 2 3 fr 5 6 7 S 1 ow 1 2 3 fr 5 6 7 Un i mpor t an t How u s e f u l was t h i s t r a i n i n g p r o g r a m ? 1 2 3 fr 5 6 U s e l e s s How e n j o y a b l e was t h i s t r a i n i n g p r o g r a m ? U s e f u 1 I 2 3 fr 5 6 7 E n j o y e d I t V e r y Much D i d n ' t E n j o y I t V e r y M u c h 1 10 APPENDIX B Mini -Session Questionnaire Name of Session: ID # the anumber e ^ f \u00C2\u00B0 l l 0 W i n g l t e m s ^ P l a c i n g a c i r c l e around I found thl\u00C2\u00AB (\u00E2\u0080\u00A2\u00E2\u0080\u00A2ilon to be: I-.,.. 9 , 5 , 6 7 i n l \u00C2\u00AB T r T y v e r y e x t r e m e l y between \u00E2\u0080\u00A2 h o r t long 1. Enjoyment of ses s i o n 1 2 3 ^ 5 6 7 Didn't enjoy i t very much Enjoyed i t very much 2 . Amount of new in f o r m a t i o n picked up during s e s s i o n 1 2 3 ij S 6 7 Taught me l i t t l e I Taught me a lot didn't know 3 . Relevance of ses s i o n to own job 1 2 3 h 5 6 7 Not very relevant Very relevant Length of session 1 2 3 if s 6 7 Too long Too short 5 . Level of p r e s e n t a t i o n 1 2 3 '4 5 6 7 Complicated Simple Comments & Suggestions: APPENDIX C F i n a l Questionnaire 1 13 ID # LAND TITLE SCHOOL EVALUATION OF TOTAL PROGRAM I have found t h i s program: chaotic unstimulating unimportant uninteresting I learned nothing not relevant to my job 1 2 3 k 5 6 7 1 2 3 it 5 6 7 1 2 3 it 5 6 7 1 2 3 5 6 7 1 2 3 it 5 6 7 1 2 3 it 5 6 7 well-ordered stimulating interesting I learned a lot relevant to my job 1. Which of your expectations of t h i s program were f u l f i l l e d ? 2. Which of your ex p e c t a t i o n s of t h i s program were u n f u l f i l l e d ? 3 . What i n f o r m a t i o n s k i l l gained through t h i s program i s most v a l u a b l e to you? Do you f e e l t h i s program w i l l e f f e c t your work? Yes No How much e f f e c t ? i 2 3 ' fr 5 6 7 l i t t l e = 0_o^ rouch effect s o _ s o effect Please e x p l a i n your answer: What were the major strengths of t h i s program? What were the major weaknesses of t h i s program? Did you r e c e i v e enough i n f o r m a t i o n on the content of t h i s program before coming? Yes ; No ,. I f no, would i t have been h e l p f u l to have t h i s i n f ormation? Yes ; No . What suggestions do you have f o r f u t u r e programs? a. Number of sessions b. Length of each s e s s i o n c. Subjects to be covered d. Changes you would make e. Other Based on your experiences of the past two weeks, would you come to the next l e v e l course. Yes ; No . Comment 1 1 6 APPENDIX D Follow-Up Questionnaire LAND TITLE SCHOOL FOLLOW-UP P l e a s e g i v e us y o u r f r a n k r e a c t i o n s and o p i n i o n s ; t h e y w i l l h e l p us e v a l u a t e t h i s p r o gram and i m p r o v e f u t u r e ones. A l l i n f o r m a t i o n i s c o n f i d e n t i a l and w i l l be u s e d o n l y t o i m p r o v e f u t u r e p r o g r a m s . '\u00E2\u0080\u00A2\u00E2\u0080\u00A2Ie w o u l d be g r a t e f u l i f y o u w o u l d use t h e sane ID number c h o s e n d u r i n g t h e c o u r s e . ID f P l e a s e g i v e y o u r o p i n i o n o f t r a i n i n g by c i r c l i n g t h e a p p r o p r i a t e number i n each o f t h e o p i n i o n s c a l e s b e l o w . Example: | 3 \u00C2\u00A9 5 6 i \u00E2\u0080\u0094 \u00E2\u0080\u0094 i 1 r extremely very fairly in fairly very extremely COMPLICATED between SIMPLE c o m l i c a t e d 7 s i m u l e u n p r a c t i c a l 1 2 3 7 p r a c t i c a l a c c u r a t e 7 i n a c c u r a t e d u l l _5 6 7 _ i n t e r e s t i n g d i f f i c u l t h e l p f u l f a s t 7 e a s y _5 6 7 u n h e l p f u l 7 s low i m o o r t a n t 7 u n i m p o r t a n t How u s e f u l was t h i s t r a i n i n g ? 1 2 3 4 u s e l e s s 6 7 u s e f u l How e n j o y a b l e was t h i s t r a i n i n g ? 1 2 3 4 d i d n ' t e n j o y i t v e r y much e n j o y e d i t v e r y much 1 18 Has t h i s p r o gram e f f e c t e d y o u r work? YES ; HO How much e f f e c t : 1 2 3 4 5 6 7 l i t t l e s o - s o much e f f e c t e f f e c t P l e a s e e x p l a i n y o u r answer: V.'hat i n f o r m a t i o n o r s k i l l g a i n e d t h r o u g h t h i s p r o g r a m i s most v a l u a b l e t o you? 'las y o u r o u t l o o k t o w a r d y o u r j o b changed? YES P l e a s e e x p l a i n : Do y o u have any s u g g e s t i o n s f o r f u t u r e c o u r s e s ? APPENDIX E Revised Questionnaires Mini-session F i n a l Follow-up Expectat ions JUSTICE INSTITUTE OF B.C. LAND TITLE SCHOOL Name o f s e s s i o n : P l e a s e r a t e t h i s s e s s i o n on t h e f o l l o w i n g i t e m s by p l a c i n g c i r c l e around t h e a p p r o p r i a t e number. P l e a s e be c a n d i d i n e x p r e s s i n g y o u r f e e l i n g s , w h ether t h e y are p o s i t i v e or n e g a t i v e . Note the d e f i n i t i o n s b e l o w . Example: I f o u n d t h i s s e s s i o n t o be: 1 2 3 ^ 5 6 7 . e x t r e m e l y veTy f a i r l y I R f a i r l y v e r y e x t r e m e l y b etween s h o r t l o n g D e f i n i t i o n s : INTEREST h o l d s y o u r a t t e n t i o n c a p t u r e s y o u r i m a g i n a t i o n s t i m u l a t e s HEW INFORMATION h a d n ' t h e a r d I t b e f o r e h e a r d I t b e f o r e but d i d n ' t u n d e r s t a n d RELEVANCE a p p l i c a b l e p e r t iner.t a p p r o p r i a t e INTEREST o f s e s s i o n t o me: 1 2 3 no i n t e r e s t much interest NEW INFORMATION g a i n e d d u r i n g s e s s i o n : 1 2 3 4 5 gained l i t t l e new information 6 7 gained a l o t o\u00C2\u00A3 new information 3. In terms of my job, I found the information from today's session to be: not useful 0 hiqhly useful u s e f u l b) previously NOT known both o l d and new previously known Comm en t s: FINAL EVALUATION LAND TITLE SCHOOL P l e a s e r e f l e c t on y o u r e x p e r i e n c e s o f t h e p a s t week when a n s w e r i n g t h e f o l l o w i n g i t e m s . Make y o u r comments v e r y s p e c i f i c . Y o u r comments w i l l h e l p us t r e m e n d o u s l y when we p l a n t h e n e x t c o u r s e . D e f i n i t i o n s : INTEREST RELEVANCE h o l d s y o u r a t t e n t i o n a p p l i c a b l e c a p t u r e s y o u r i m a g i n a t i o n p e r t i n e n t s t i m u l a t e s a p p r o p r i a t e 1. How r e l e v a n t do y o u now c o n s i d e r t h i s e n t i r e c o u r s e t o y o u r p o s i t i o n ? not very relevant relevant P e r s o n a l l y how i n t e r e s t i n g was t h i s c o u r s e ? 1 2 3 4 5 6 7 not very very in teres t ing in teres t ing D i d t h e c o n t e n t o f t h e c o u r s e a g r e e w i t h y o u r o r i g i n a l e x p e c t a t i o n s ? very l i t t l e moderately very much P l e a s e e x p l a i n y o u r answer: What i n f o r m a t i o n g a i n e d t h r o u g h t h i s c o u r s e do you f e e l w i l l be most v a l u a b l e t o you? How have you j u d g e d v a l u e ( i n question M)? (Please check one) Most p r a c t i c a l use Most remembered' Most r e v e a l i n g Most i n t e r e s t i n g O t h e r ( P l e a s e e x p l a i n ) What were t h e m a j o r s t r e n g t h s o f t h i s e n t i r e c o u r s e ? Be s p e c i f i c . What were t h e m a j o r weaknesses o f t h i s e n t i r e c o u r s e ? Be s p e c i f i c . P l e a s e g i v e any a d d i t i o n a l comments and/or s u g g e s t i o n s . 123 LAND TITLE SCHOOL FOLLOW-UP 1. What i s your present position? 2. How long have you held thi s position? 3. What course did you attend? 4. Have you shared with your fellow workers the handouts and information presented at the course? Yes No_ If yes under what setting ( i . e . s t a f f meetings, over coffee, etc.) 5. How often have you used the handouts from the course? 6. Did the information presented enable you to solve problems or meet situations on your job which previously you had not been able to do on your own? Yes No Explain: 7 . Did you find the information presented i n the course useful in your d a i l y work? Yes No 8. If you were to reorganize the course, what would you change, leave the same, etc.? Explain. Expectations Warm-Up Names are important to a l l of us. It f e e l s comforting to be addressed by name i n a strange environment. Having one-person introduce him/herself i s f i n e but there are few people who w i l l remember even h a l f the names mentioned. Using double-folded sheets of paper or old computer cards as desk-top name cards Is more useful than the stick-on type of name tag. Have each person p r i n t ( i n bold l e t t e r s ) the name they want to be c a l l e d on both sides of the card. Have each person put on the in s i d e \"Only f o r you to see\" the completion of these sentences: -What I'd r e a l l y l i k e to do r i g h t now i s . . . -I hope t h i s course won't be... -What I would l i k e to learn i n t h i s course includes... While t h i s information i s c o n f i d e n t i a l at t h i s stage, you may ask volunteers l a t e r on i n the session to share i t with the c l a s s . I t i s a technique to help members focus on t h e i r expectations, t h e i r present f e e l i n g s and t h e i r hopes. I t might also convey the notion that you care about these f e e l i n g s and are aware of t h e i r presence i n the room. "@en . "Thesis/Dissertation"@en . "10.14288/1.0055871"@en . "eng"@en . "Administrative, Adult and Higher Education"@en . "Vancouver : University of British Columbia Library"@en . "University of British Columbia"@en . "For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use."@en . "Graduate"@en . "An assessment of illuminative evaluation as an approach to evaluating residential adult education programs"@en . "Text"@en . "http://hdl.handle.net/2429/23856"@en .