A CRITICAL EXAMINATION OF THE FUNCTIONS AND METHODS OF EVALUATION RESEARCH By POYIN K.jAUYEUNG B.A. (Hons.) Siraon Fraser University, 1980 A THESIS SUBMITTED IN THE REQUIREMENTS MASTER PARTIAL Kn^ILLMENT OF FOR THE DEGREE OF OF ARTS in THE FACULTY OF GRADUATE STUDIES (The School of Coramunity and Regional Planning) We accept this thesis as conforming to the required standard THE UNIVERSITY OF BRITISH COLUMBIA October 1984 © Poyin K. Auyeung, 1984 In presenting t h i s thesis i n p a r t i a l f u l f i l m e n t of the requirements for an advanced degree at the University of B r i t i s h Columbia, I agree that the Library s h a l l make i t f r e e l y available for reference and study. I further agree that permission for extensive copying of t h i s thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. I t i s understood that copying or publication of t h i s thesis for f i n a n c i a l gain s h a l l not be allowed without my written permission. Department of Community and Regional Planning The University of B r i t i s h Columbia 1956 Main Mall Vancouver, Canada V6T 1Y3 Date October 1984 DE-6 (3/81) - i i -ABSTRACT This thesis examines the aims and methods of evaluation research, and, ultimately, considers what functions evaluation may serve in innovative programs. It addresses the controversy with respect to the inadequacy of the scientific experimental method in evaluating social action. Analysis of a case study indicates that applying the experimental method is problematic when programs have vague concepts and uncertain targets, which is almost certain in innovatory programs. Even i f that method were successful in generating reliable facts about program effects, i t would have overlooked some important subjective aspects of the program. Cur general conclusion suggests that i t is important to go beyond scientific experimental approaches by exploring alternative modes of evaluation. Critical theory, i t is suggested, is useful for studying the normative and latent aspects of programs and program effects. A main presupposition of this study is that an epistemological investigation, which examines the theories of knowledge underlying the aims and methods of evaluation, can make a worthwhile contribution to program evaluations. Inherent in evaluation are fundamental epistemological questions such as "What may be considered knowledge?", "What is the factual and value content of that knowledge?", and "For what purposes do we use enquiries?". Numerous figures in evaluation suggest that not enough attention is given to the conceptual analysis of evaluation purposes and methods. The literature on evaluation is the source of our analysis of the functions and methods proposed by various practitioners. We examine the - i i i -views of two prominent camps: those who support the scientific experimental methods of testing, measuring and generating facts; and those who use normative social theories and attend to value issues and subjective aspects of programs. Relying upon the literature related to social enquiry, the thesis examines diverse theories of knowledge: scientism, positivism, pragmatism, and finally, cr i t ical theory. The cr i t ical school advocates a flexible approach to enquiry which extends beyond scientific experimental methods. Following Kant, i t emphasizes the pervasive influence of a priori cognitive models in our generation of knowledge. It recognizes, too, the importance of the Hegelian view that the history of abstract thoughts and values is essential for understanding social affairs. As well, its critique incorporates the Marxian stance of focusing on the conflicts of interest among social classes. Accordingly, i t goes beyond the positivistic notion that bare facts devoid of value content can be generated. The cr i t ical approach, by considering multiple perspectives and by investigating latent aspects such as value systems, conflicting interests, perceived social needs and human consciousness, is appropriate for studying the complex nature of social action. Following upon these concerns, the following evaluation functions are proposed: to clarify theoretical assumptions of programs, to reveal and revise value assumptions and relate them to a history of ideas, to identify the social impacts of programs and the perceived needs of program - iv -participants, and ultimately, to foster the exercise of c r i t i c a l self-reflection among participants by encouraging decentralized decision-making and evaluation. These c r i t e r i a are applied to a case study, an evaluation of a community energy program. The conclusions drawn from the case study indicate that the controlled experiment should be supplemented with subjective evaluation approaches. The experiment may be useful at times, but at least with respect to social innovative projects, its application is problematic. With the presence of program features including vague program objectives, uncertain c r i t e r i a of program success, and unvalidated causal assumptions (attitude •* behaviour •*• actual energy savings) , i t is impossible to "test" (in the s c i e n t i f i c sense) hypotheses such as Program A -»- Causal Process B •> Objective C. Our c r i t i c a l examination suggests that, by concentrating on concrete program effects, the case study evaluation had overlooked important intrinsic factors affecting the program and plan. These factors are the program's theoretical and value assumptions, the perceived needs of recipients, and the conflicts of interest between the sponsors and the program designer. The c r i t i c a l school of thought reminds us that we have to persistently question the factual content, value basis and causal assumptions of our programs. C r i t i c a l theory, by identifying the limits of traditional enquiries and exploring alternative epistemologies, provides invaluable insights for developing guidelines for evaluation research. - V -TABLE OF CONTENTS Page ABSTRACT i i TABLE OF CONTENTS v LIST OF FIGURES v i i i ACKNOWLEDGEMENT i x CHAPTER 1 INTRODUCTION - A BRIEF HISTORICAL OVERVIEW 2 Thesis Purpose, The Context and Some Definit i o n s . . . . 6 The Scope, Objectives, Methods & Outline 11 CHAPTER 2 ALTERNATIVE FRAMEWORKS OF EVALUATION Introduction 19 An Analysis of Functions and Methods 19 The S c i e n t i f i c Experimental Framework 22 Normative Social Analysis 30 Soci a l Learning 31 I Hum inat ive/ Interpret ive 33 C r i t i c a l Consciousness 36 Systems Approach 40 De c i s i o n i s t i c 40 Systemic D i a l e c t i c a l 43 Pseudo-Evaluation 48 Conclusion 49 CHAPTER 3 EVALUATION AND EPISTEMOLOGY Introduc t ion 52 Identifying Connections Between Evaluation and Epistemology 52 - v i -Page The S c i e n t i s t i c Approach to Epistexoology 54 The Purpose of Critique as Held by Kant, Hegel and Marx 58 Positivism 64 Pragmatism 69 C r i t i c a l Theory 71 Conclus ions 78 CHAPTER 4 THE META-EVALUAT10N CRITERIA Introduc t ion . 81 The Obsolescence of Pure Factual Knowledge 81 As a Function of Disclosing Theoretical Assumptions 84 As a Means of Promoting C r i t i c a l Self-Reflection.... 85 As an Instrument of Identifying Social Impacts 87 As a Process of Attaining Emancipation from Dogmatism and Ideology 88 Surfacing and Revising Value Assumptions 90 Attending to the Perceived Needs of the Subjects 91 Encouraging Shared Decision-Making 91 Conclusions 92 CHAPTER 5 THE CASE STUDY - THE EVALUATION STUDY OF THE DEMONSTRATION PROGRAM ON COMMUNITY ENERGY CONSERVATION Introduction. 95 The Coranunity Energy Program 95 The Context 95 Program Objectives and Strategies 96 - v i i -Page The Program Evaluation 97 Evaluation Objectives 98 The Method 98 The Evaluation Results 100 A Critical Examination of the Case Study 105 Conclusions 121 CHAPTER 6 " CONCLUSIONS Introduction. 124 General Conclusions 124 Conclusions from the Case Study 129 Some Final Thoughts and Impressions 134 BIBLIOGRAPHY 137 APPENDIX 1 145 APPENDIX II 149 - v i i i -LIST OF FIGURES Page Figure 1 An Analysis of Alternative Frameworks of Evaluation 20 Figure 2 The Before and After Controlled Experimental Model 23 Figure 3 Types of Program Failure 26 Figure 4 Pre-test Post-test Control Group Experimental Design of the Community Energy Program 99 Figure 5 Variations on the Central Thrust of Process 104 Figure 6 Environmental Supports to Encourage Energy Conserving Behaviour 105 Figure 7 Theoretical Linkages in the Community Energy Program Model 109 o - i x -ACKNCWLEtta^ENT I w i s h t o e x p r e s s m y s i n c e r e a p p r e c i a t i o n t o m y a d v i s o r , P r o f e s s o r H e n r y H i g h t o w e r , w h o s e c o n t i n u o u s g u i d a n c e a n d p e r c e p t i v e c o m m e n t s w e r e i n v a l u a b l e t o t h e d e v e l o p m e n t o f t h e t h e s i s . M y t h a n k s a r e e x t e n d e d t o P r o f e s s o r P e t e r B o o t h r o y d f o r h i s h e l p f u l a d v i c e d u r i n g t h e l a s t s t a g e s o f p r e p a r a t i o n . T h e i r c o n s t r u c t i v e c r i t i c i s m s a n d e n c o u r a g e m e n t h a v e m a d e t h e w r i t i n g o f t h i s t h e s i s b o t h a p l e a s u r e a n d a g e n u i n e l e a r n i n g e x p e r i e n c e . S t a n S h a p i r o , L o u i s D ' A m o r e a n d G a r t h M a c G u i r e w e r e m o s t h e l p f u l i n p r o v i d i n g t h e b a c k g r o u n d m a t e r i a l f o r t h e c a s e s t u d y . L a s t , b u t n o t l e a s t , I w i s h t o e x p r e s s m y d e e p e s t g r a t i t u d e t o m y f a m i l y , a n d p a r t i c u l a r l y t o m y p a r e n t s , P u i - Y i n g a n d L a i A u y e u n g , f o r t h e u n f a i l i n g s u p p o r t a n d l o v i n g c a r e t h e y g a v e me t h r o u g h o u t m y a c a d e m i c p u r s u i t . - X -A profound thought i s i n a constant state of becoming; i t adopts the experience of a l i f e and assumes i t s shape. Likewise, a man's sole creation i s strengthened i n i t s successive and multiple aspects: h i s works. One after another, they complement one another, correct or overtake one another, contradict one another too. I f something brings creation to an end, i t i s not the victorious and i l l u s o r y cry of the blinded a r t i s t : " I have said everything," but the death of the creator which closes his experience and the book of his genius. ALBERT CAMUS, The Myth of Sisyphus: 84 - 1 -CHAPTER 1 INTRODUCTION: AN OVERVIEW OF EVALUATION & THESIS PURPOSE AND MfffflODS - 2 -INTRODUCTION - A BRIEF HISTORICAL OVERVIEW Evaluation research found i t s roots i n the s o c i a l sciences' p r i n c i p l e that the application of sophisticated research methods may f a c i l i t a t e our understanding and control of s o c i a l processes; thus, applying rigorous research i n public programs and plans may e l i c i t progressive s o c i a l change. This stream of i n t e l l e c t u a l development resulted i n the wide spread use of s c i e n t i f i c management techniques and systematic information management for designing and administering s o c i a l services i n the public sector (Rossi & Freeman: 1982). A concise h i s t o r i c a l review of some of the s o c i a l and p o l i t i c a l conditions of that era may shed some l i g h t on the expansion of evaluative research. Before World War I s o c i a l services for the poor, the old and the handicapped were largely handled by private organizations, volunteers, l o c a l c h a r i t i e s and family members. The Great Depression of the 1930's created a huge demand for s o c i a l services. Since that period, s o c i a l welfare plans and p o l i c i e s , correction programs and other government sponsored human services have grown extensively. The climax of th i s kind of intervention was U.S. President Franklin Roosevelt's New Deal s o c i a l programs, which aimed at solving problems of urban slums, malnutrition, crime, delinquency, and so forth. At that time, an Arkansas sociology professor demanded evaluation of those programs; and other concerned people from the public sector and the community also expressed interest i n ' t h e i r achievements and f a i l u r e s . Thereafter, - 3 -evaluation and i n broader terms, applied s o c i a l research began to f l o u r i s h (Rossi & Freeman: 1982). Further development of evaluation took place during W.W.II as the m i l i t a r y refined techniques i n monitoring soldier morale and i n assessing personnel and propaganda p o l i c i e s . The post war period was accompanied by rapid i n d u s t r i a l i z a t i o n , population growth and urban development. Large-scale programs were devised to supply community and recreational services, education, preventive health concerns and occupational t r a i n i n g . As w e l l , many developing countries i n Asia, L a t i n America and A f r i c a collaborated with i n d u s t r i a l i z e d nations to conduct programs for family planning, health, n u t r i t i o n and r u r a l community development (I b i d . ) . With the p r o l i f e r a t i o n of these large-scale government-sponsored s o c i a l a c t i v i t i e s , the academia, taxpayers and decision-makers expressed increasing interests i n th e i r r e s u l t s . In the late 1950's, grand evaluation studies became prevalent. Social Scientists applied knowledge of research methods—surveys, s t a t i s t i c a l analysis, experiments, quasi-experimental designs and so on—and r e l i e d on theories and concepts of psychology, sociology and public administration to help i d e n t i f y the results or effects'of s o c i a l projects. This trend continued through out the 1960's and 1970's. From the mid 70's to the early 80's, the austere economic climate experienced by many in d u s t r i a l i z e d nations called for a need to question - 4 -the continued growth of government programs. A common strategy adopted by many of these countries i s the reduction of government expenditures. Limited funding and resources result i n an increased demand for e f f i c i e n c y , effectiveness and thorough scrutiny of public projects. These trends suggest growing demand for evaluation of existing projects so that i n e f f e c t i v e ones may be c u r t a i l e d . Numerous recent studies on public administration and s o c i a l research suggest an upsurge of interest i n evaluation i n the public sector and the academia. In authorizing innovatory programs, the U.S. Congress frequently attaches a mandate for evaluation reports. In 1974, Congress permitted the General Accounting Office (GAO) to carry out i t s own evaluation. Versions of the Program Evaluation Act which passed Congress i n 1978 are l i k e l y going to boost evaluation a c t i v i t i e s (Cronbach: 1980). In Canada, evaluation i s gradually gaining support and formal recognition. During a 1982 conference on evaluation, the Comptroller General stressed the value of evaluation i n government planning during times of economic res t r a i n t (Rutman: 1982). Inherent i n these formal expressions of support are expectations and assumptions of what evaluation should and could contribute to s o c i a l p o l i c y and public administration. The h i s t o r i c a l overview has indicated that evaluation i s an integral part of so c i a l p o l i c y and program - 5 -development. I t i s essential, therefore, as evaluation becomes more central to po l i c y development, that i t be an object of scrutiny i t s e l f . The h i s t o r i c a l overview discloses s i g n i f i c a n t linkages between the development of evaluation and the concurrent "state of the a r t " of the.social sciences. Evaluation began with the efforts of s o c i a l s c i e n t i s t s to monitor and to assess s o c i a l programs; a perceived major function was to show evidence of the effects of these interventions. As Suchman (1967) claims, most evaluation studies are involved i n tracing the causes and effects of planned s o c i a l action. The common method, borrowed from the natural sciences, involves experiments, empirical data and observation. This approach gained prominence throughout the 1950's, 60's and early 70's. I n f l u e n t i a l works advocating this mode include Suchman's Handbook For Evaluative Research (1967), Campbell and Stanley's Experimental and Quasi-Experimental Designs for Research (1966), and Riecken and Boruch's Social Experimentation (1974). The l a t e 1970's saw a s h i f t away from the dominance of the experimental framework. Various writers, d i s s a t i s f i e d with that approach and viewing evaluation as going beyond the application of s c i e n t i f i c methods to the so c i a l world, have explored alternative approaches incorporating p o l i t i c a l and managerial aspects. Their common conviction i s that evaluation demands more than the simple application of rigorous research procedures for the purpose of generating f a c t u a l information. They hold that evaluation should be taken as a component - 6 -of a policy-making process characterised by the complex interplay of decision-making, planning and the accommodation of value and goal conflicts among social groups. The representative works of this approach include Cronbach's Toward Reform of Program Evaluation (1980), Parlett and Hamilton's Illuminative Evaluation (1972), Stufflebeam's Educational Evaluation and Decision-making (1973), and Dunn's The Obsolescence of Evaluation Research (1981). In considering these alternative views of evaluation, one is faced with a number of important questions. What are the perceived inadequacies of the scientific experimental approach? What do the alternative modes claim to offer that the dominant one cannot? What are the epistemological bases of a l l the respective approaches? As well, by what criteria can we ultimately judge program evaluation? This thesis attempts to provide insights and guidelines to assist us in answering these questions. THESIS PURPOSE The purpose of this thesis is to develop a framework of meta-evaluation criteria. This study is a meta-evaluation, that is, a critical examination of the assumptions, aims and methods of evaluation research. Major attention is devoted to epistemological principles of its functions and methods, particularly the presuppositions underlying the scientific experimental mode of research, and those embedded in - 7 -alte r n a t i v e approaches to enquiry. This conceptual investigation provides the foundation for us to decide what functions evaluation may serve i n s o c i a l action programs. THE CONTEXT This thesis addresses the concern that an examination of the theories of evaluation i s lacking. This concern has been expressed by such authoritative figures as Suchman (1967), Cronbach (1980), Weiss (1977), Ruddock (1981) and Dunn (1981). Suchman comments that evaluation issues have concentrated too much on the instrumentation of purpose and methods, while neglecting th e i r theoretical analysis. Dunn claims that technical refinement i s often achieved at the expense of conceptual sophistication. This concern presupposes that evaluation practices are inherently governed by theories of what should be achieved (assumptions of worth and interest) and what could be accomplished (assumptions of technical p o s s i b i l i t i e s ) i n s o c i a l research. Therefore, i f we want to understand the conduct of evaluation, we have also to examine i t s underlying p r i n c i p l e s . Fai l u r e to comprehend the conceptual basis of evaluation aims and methods may have two undesirable consequences. One i s the u n c r i t i c a l acceptance of a t r a d i t i o n a l aim and a standard method of evaluation. A case i n point i s the dominant view of evaluation as a means of producing factual information about s o c i a l causes and effects for the benefit of - 8 -policy-makers. Accordingly, the randomised controlled experiment i s commonly regarded as the standard evaluation method. However, there i s increasing discontent, as revealed by such writers as Croribach (1980) and Stake (1982), with respect to the i n a p p l i c a b i l i t y of the s c i e n t i f i c experiment in s o c i a l action settings. Another drawback i s the confusion with respect to the purpose of evaluation i n public policy-making. For instance, Weiss (1972) and Rutman (1982) ide n t i f y the discrepancy between the intended purpose of evaluation for contributing to decision-making, and the limited extent to which evaluation i s used for t h i s purpose i n practice. Both authors point to the irony that although many evaluation studies claim to be done for accountability and summative purposes i n decision-making (that i s , for assessing and deciding the continuation or abolishment of programs), empirical evidence demonstrates that program evaluation has not been important for determining resource a l l o c a t i o n i n s o c i a l program planning. Evaluation findings are very rarely used for deciding the "go/no go" of programs and plans, but are more often u t i l i z e d for management and program improvement. A l l these concerns point to a problem area of evaluation that needs to be examined from an epistemological perspective. Major e f f o r t s should be made to re-exaulne the c r i t e r i a of judging evaluation, which enta i l s a r e - d e f i n i t i o n of the role or function of evaluation i n s o c i a l programs. In th i s thesis, therefore, I attempt to examine c r i t i c a l l y - 9 -the conceptual foundation of evaluation i n the hope of constructing a framework of meta-evaluation c r i t e r i a . SOME EErTMTTONS I t i s d i f f i c u l t to provide a simple d e f i n i t i o n of evaluation as various writers hold different views regarding i t s nature, purpose and methods. However, there i s general agreement that evaluation connotes some form of description, analysis, and judgement of planned action. This consensus rests upon the planning aspects of evaluation, whose i m p l i c i t assumption i s that human a c t i v i t i e s can be planned, studied and improved. Despite the divergent opinions of what constitutes a successful evaluation, many authors agree that the purpose of the evaluation should be i d e n t i f i e d at the beginning of a study, since the purpose dictates the methods to be used, the questions to be asked, the issues to be addressed, and subsequently, the type of research findings to be collected. Different ends demand for different means. Therefore, a c r i t e r i o n of success i n evaluation i s the appropriateness of the method used to f u l f i l a given purpose. In addition, i t i s c r u c i a l that, as Ruddock (1981) emphasizes, sim i l a r weighting be given to assessing the worth of the stated purpose. Epistemology i s commonly understood as the study of the generation of knowledge, or the theory of knowledge. I t s fundamental - 10 -concern pertains to the questions, "What may be considered knowledge?", "How i s knowledge ascertained?", and "For what purpose i s knowledge to be used?". Epistemology i s sometimes used interchangeably with methodology, but there i s a d i s t i n c t i o n between them. Methodology refers s p e c i f i c a l l y to the study of methods, whose primary a c t i v i t y i s to inspect the procedure of generating knowledge but not the purpose of obtaining that knowledge. Habermas (1971) suggests that epistemology takes account of the knowing subject as he or she uses methods and thinks about ideas, but that methodology merely studies the l o g i s t i c s and rationale of methods and research procedures. As far as an objective of this thesis i s to investigate the purposes for which evaluators and the community use evaluation, epistemological considerations remain s i g n i f i c a n t . " Social action programs" refer to planned s o c i a l action that i s administered by a public or semi-public agency, and i s implemented with some degree of involvement from a designated population. They are often i n i t i a t e d with the aim of a l l e v i a t i n g some perceived s o c i a l problem. Suchman (1967) suggests two broad categories of s o c i a l action programs: demonstration and operational projects. Demonstration programs are designed to test the success or f a i l u r e of a strategy on a t r i a l basis. They are further c l a s s i f i e d into three types or stages. P i l o t programs, which are launched on the basis of t r i a l and error, stress the exploration of f l e x i b l e and innovative approaches, and - 11 -encourage v a r i a t i o n i n terms of how, where and by whom a project i s organised, whom i t reaches and so forth. Model programs, which are designed on the basis of findings obtained from the p i l o t stage, set out to demonstrate the success of a strategy i n achieving a desired objective under specified conditions. Prototype programs, which are conducted on the basis of knowledge and experience learned from the p i l o t and model stages, are applied i n a p r a c t i c a l setting under normal operating conditions. The other s o c i a l action program category, operational projects focuses on service improvement rather than assessing whether a service i s worthwhile or deliverable. This type of program concentrates on managing and aaministering the on-going operation of projects. THE SCOPE, OBJECTIVES AND METHODS Based on the l i t e r a t u r e review and past experience related to the subject matter, the following presuppositions have been formulated for guiding the development of the thesis: 1. Evaluation, when taken as a means of providing feedback and as a process by which we learn about our thoughts and actions, our p o l i c i e s and programs, i s a worthwhile endeavour. - 12 -2 . The Lack of conceptual analyses of the purpose and methods of evaluation exacerbates the confusion over i t s r o l e and i t s c r i t e r i a of success. Therefore, conducting an epistemological analysis i s desirable. 3. The t r a d i t i o n a l mode of evaluation of using the randomised controlled experiment to delineate s o c i a l causal processes may not be compatible with s o c i a l action settings; hence, i t should be c r i t i c a l l y examined. The following major questions must be dealt with in order for us to accomplish the purpose of th i s thesis. 1. What are the functions and methods of evaluation research as advocated by numerous proponents in the fi e l d ? What are the i r positions with respect to whether emphasis should be placed upon generating factual information or value issues? 2 . What are the basic principles of the schools of thought, namely positivism and pragmatism, which abide by the s c i e n t i f i c method of analysis i n adopting the procedures of testing, measurement and objective fact-finding? What are the theories, as expounded by Kant, Hegel, Marx and the c r i t i c a l theorists, which deal with the purpose of generating knowledge and the substance of that knowledge? How are these - 13 -issues connected to evaluation? What are th e i r implications for i t ? 3. What are the meta-evaluation c r i t e r i a — w h a t functions should evaluation f u l f i l i n s o c i a l action projects? 4. What are the stated functions and methods of the case study? Based on the c r i t e r i a developed, what functions have been performed and what others might evaluation serve i n community energy programs? I t i s not the objective of this thesis to explicate s t a t i s t i c a l methods and survey techniques of s o c i a l research. Rather, t h i s study attempts to examine the implications of various methodical frameworks i n the l i g h t of thei r commitment toward presenting facts and value issues. The thesis i s also limited with regard to the d e t a i l and e f f o r t that can be assigned to each of the various theories of knowledge and schools of thought. Instead, a more e c l e c t i c view i s presented. METHODS This study examines the functions and methods of evaluation research i n the form of a c r i t i q u e : 1 . Based on a review of the l i t e r a t u r e related to evaluation, I attempt to develop an interpretive understanding of the - 14 -divergent aims and methods of evaluation as they are recommended by a variety of concerned individuals within the f i e l d . 2. Relying upon the l i t e r a t u r e on s o c i a l enquiry, I intend to trace the h i s t o r i c a l development of those aims and methods by studying theories of knowledge and modes of s o c i a l enquiry. 3. Using a case study, I aim to study evaluation and to apply the meta-evaluation c r i t e r i a i n a concrete arena of program planning and evaluation. Information was gathered by reviewing the available documents and committee minutes, as w e l l as by interviewing a number of the people involved—the evaluator, the program designer and one of the sponsors of the community energy program. THE CASE STUDY The following c r i t e r i a , based on some of the main features of s o c i a l action programs, are used to select the case study. 1 . The program should belong to a demonstration type, characterised by i t s use of innovative approaches, i t s emphasis on t r i a l and error, and i t s f l e x i b l e or uncertain program objectives and goals. - 15 -2. There has to be a certain degree of community involvement i n the project. The involvement may take the form of having project recipients receive a s o c i a l service, or having community members participate i n project planning and implementat ion. 3. The problem i d e n t i f i e d by the project should be a r e l a t i v e l y new or speculative issue which necessitates further investigation. By referring to the preceding c r i t e r i a , a p i l o t demonstration community energy program has been chosen. I t was i n i t i a l l y funded by the federal government and was implemented i n September of 1979 i n three municipalities: Vernon, B.C.; Richmond H i l l , Ontario; and Fredericton, New Brunswick. A l l these municipalities adopted a similar community-based model i n delivering the program, although there were some variations i n the approach for accommodating differences i n the geographical, socio-economic and p o l i t i c a l attributes of each community. Their evaluation i s un i f i e d as a l l three c i t i e s were assessed by one evaluator using the same approach. "Community energy program" here refers to energy projects having char a c t e r i s t i c s of active involvement and support from comniunity organizations and l o c a l residents i n planning and implementation, as w e l l as strong education and awareness campaigns. - 16 -To date, Canadian energy conservation has r e l i e d mostly on technical research and development, f i n a n c i a l incentive strategies such as the Canadian Home Insulation Program (C.H.I.P.) and the Canadian O i l Substitution Program (C.O.S.P.), and awareness campaigns such as t e l e v i s i o n advertisements and the d i s t r i b u t i o n of pamphlets and booklets on conservation. However, community-based energy projects have not been given a high p r i o r i t y i n Canadian energy conservation p o l i c y , despite the interest expressed by some public o f f i c i a l s i n regard to the pot e n t i a l of community strategies i n energy conservation. Since energy conservation i s a r e l a t i v e l y recent public pol i c y concern, the planning, implementation and evaluation of l o c a l projects are s t i l l at an experimental stage. A l l these trends signal a need for investigating these projects. And since the lessons that could be learned from these projects largely depend on th e i r evaluation, i t i s sensible that efforts be directed towards understanding evaluation i t s e l f before we r e l y on i t s r e s u l t s . OUTLINE This thesis contains s i x chapters. The f i r s t chapter introduces the h i s t o r i c a l context of evaluation, i d e n t i f i e s the purpose of the thesis, and c l a r i f i e s the objectives and methods of the study. The second chapter contains an analysis of the divergent frameworks of the functions and methods. In the t h i r d chapter, theories of knowledge and concepts of s o c i a l enquiry are examined; together with the previous - 17 -chapter they b u i l d the theoretical foundation of the thesis. In the fourth chapter, based upon the prin c i p l e s and issues explored i n the preceding sections, c r i t e r i a to be used for judging evaluation are generated. The f i f t h chapter consists of the case study, including a description and an assessment of the project i n l i g h t of the meta-evaluation c r i t e r i a . The l a s t chapter presents some conclusions and recommendations. - 18 -CHAPTER 2 ALTERNATIVE FRAMEWORKS OF EVALUATION: AN ANALYSIS OF FUNCTIONS AND METHODS INTRODUCTION The literature on evaluation reveals numerous approaches to conceptualizing and conducting evaluation; their differences derive mainly from the emphasis placed on either scientific methods and factual arguments or normative social theories and value-based arguments. As Ralph Ruddock (1981) notes, "Arguments relating facts to value are the very substance of evaluation" (Ruddock: 1981:10). Not only are these two aspects (facts and value) central to evaluation, the tension between them provides a driving force for the branching out of different evaluation concepts and procedures. The divergence arises from the debate over the extent to which evaluation should either present facts or address value issues. This dissensus is reflected in the various perspectives on functions and methods. AN ANALYSIS OF FUNCTIONS AND METHODS The analytical framework constructed in this chapter is intended to further our understanding of the diverse aims and methods in light of their concentration on applying scientific experimental procedures or utilizing normative social theory as the basis of analysis. These two divisions are by no means mutually exclusive. In fact, some approaches attempt to assign similar weighting to both. For the sake of clarity, we shall generalize four different but related categories of evaluation. An illustration of this analytical framework presented on the following page will shed some light on our analysis. -20-SCIENTIFIC EXPERIMENTAL METHODS Decisi o n i s t i c (Stuff!ebean: 1973 a 1971) Systemic Di a l e c t i c a l (Dunn et a l : 1981) (Weilenmann: 1980) Scienti fic/Experimental (Suchman: 1967, 1972) (Rossi: 1972, 1982) (Campbell & Stanley: 1966) (Reiken a Boruch: 1974) Social Learning (Cronbach: 1980) II1uminative (Parlett & Hamilton: 1972) Cost Effectiveness Studies Information Management C r i t i c a l Consciousness (Ruddock: 1980) Fig. 1 An Analysis of Alternative Frameworks of Evaluation. - 21 -The arrows i n the diagram s i g n i f y orientation rather than absolute d i v i s i o n of the four camps. The upper quadrants represent studies that have a stronger commitment toward s c i e n t i f i c methods than the lower ones. Those on the l e f t represent studies containing a higher content of value issues than those on the r i g h t . The upper-right quadrant i s represented by the s c i e n t i f i c experimental framework, the lower-left by the learning model, the illuminative and c r i t i c a l consciousness frameworks which advocate the i d e n t i f i c a t i o n of value issues. The upper-left quadrant, consisting of the d e c i s i o n i s t i c and systemic-dialectical frameworks, stands for dual concentration. L a s t l y , the lower-right quadrant includes studies which are committed to neither aspect but are merely concerned with cost-effectiveness and information management. A paradigm may be understood as a model or a pattern which helps the researcher establish a context, a define a problem, i d e n t i f y the questions to be asked and choose the techniques to be employed. However, a paradigm does not necessarily provide a solution or prove a successful solution; instead i t i s largely a promise or an expectation of success which has to be further a r t i c u l a t e d , specified and tested (Kuhn: 1962). A conceptual framework i s a similar expression which e x p l i c i t l y discerns the subjective elements of b e l i e f s , values and assumptions constituting the paradigm or model. The notion of a conceptual order governing our perceptions and judgements may be traced back to the Kantian.sense of the "conditions of - 22 -knowing" or the "subjective conditions of knowledge"—that i s , the regulative ideas and cognitive forms which determine the kind of sensory data we perceive, the objects and events we pay attention to, and the kind of facts we select (Horkheimer: 1982). Therefore, i f we want to understand evaluation not merely i n terms of i t s contents and i t s data, but also the ideas and values governing those contents, we must trace the conceptual frameworks or the "subjective conditions" of evaluation. THE SCIENTIFIC EXPERIMENTAL FRAMEWORK In studying s o c i a l programs, a dominant trend i s the adoption of the experimental model borrowed from the natural sciences. While the proponents of th i s approach advocated rigorous research methods and the development of basic knowledge, they were also extremely concerned with applying that knowledge to policy-making and concrete s o c i a l action. "Social experimentation" was the concept and the model used by proponents such as Campbell (1966) and Reiken and Borugh (1974). Suchman1 s support of th i s approach can be detected i n his view on the functions of evaluation. To some extent evaluative research may offer a bridge between 'pure' and 'applied' research. Evaluation may be viewed as a f i e l d test of the v a l i d i t y of cause and effect hypotheses i n basic science whether these be i n the f i e l d of biology ( i . e . medicine) or sociology ( i . e . s o c i a l work). Action programs i n any professional f i e l d should be based upon the best available s c i e n t i f i c knowledge and theory of that f i e l d . As such, evaluation of the success or f a i l u r e of these programs are intimately t i e d into the - 23 -proof and disproof of such knowledge. Since such a knowledge base i s the foundation of any action program, the evaluative research worker who approaches his task i n the s p i r i t of testing some theoretical proposition rather than a set of administrative practices w i l l i n the long run make the most s i g n i f i c a n t contribution to program development. (Suchman: 1967:170) In short, under t h i s paradigm, the main functions of evaluation are to test the v a l i d i t y of cause-and-effect hypotheses, to contribute to basic knowledge and, ultimately to use th i s basic knowledge to assist the planning, development and operation of s o c i a l service programs. Im p l i c i t i n this model are some n a t u r a l i s t i c presumptions which treat the research procedures of the natural sciences as the standard to be applied to s o c i a l a c t i v i t i e s . The n a t u r a l i s t i c interest in delineating the cause and effect of the physical world i s translated into the evaluator's concern for the causal relationships of the s o c i a l world. These assumptions w i l l be c r i t i c a l l y examined i n the succeeding chapter. The most highly recommended method i s the Before and Aft e r Controlled Experimental Model: Before After Experiment XI X2 Control Y1 Y2 I f (X2 - X I ) > (Y2 - Y1) then program has the desired effect F i g . 2 The Before and After Controlled Experimental Model. - 24 -Before a program (stimulus) i s launched, measurements w i l l be taken from the experimental group which w i l l experience the program (X1), and from the control group which w i l l not experience i t (Y2) but which has very s i m i l a r characteristics to the experimental group. After the termination of the program, measurements w i l l again be obtained from both groups (X2 and Y2 respectively). I f the difference between the "before" and "after" measurements of the experimental group i s greater than that of the control group, then the program i s considered to have e l i c i t e d the desired effect, and i s regarded as successful. This model i s ideal and superior i n terms of minimizing bias and achieving s t a t i s t i c a l sophistication. However, i t demands a s p e c i f i c setting i n which to operate. And those controlled settings may not be e a s i l y found outside of the laboratory. Adaptations of the ideal model, known as quasi-experiments, e n t a i l less stringent conditions (Campbell & Stanley: 1966). When the experimental controlled method i s executed on s o c i a l programs, i t presumes and dictates a s p e c i f i c procedure of conducting those projects. Program A (input or stimulus) i s to be performed to achieve objective B (output or response) through a causal process C. For example, i n welfare, a t r a i n i n g course (A) i s to be established for reducing unemployment among the physically handicapped (Obj. B), under the assumption that such trai n i n g would increase the c a p a b i l i t i e s and confidence of the handicapped, thus help them i n finding jobs (C). The task of evaluation research i s to test the v a l i d i t y of t h i s hypothesis: Program A •+ Causal Process C •»• Objective B. The basic pre-requisites - 25 -of executing the test are well-defined objective(s), measurable c r i t e r i a by which the attainment of the objective may be judged, and knowledge of the causal process. Legitimate questions to be raised are, "What kinds of programs s a t i s f y these pre-requisites?;" "Does the program at hand f i t into t h i s category?;" " I f not, what are alternative evaluation approaches?;" and "What are the consequences i f the program being evaluated i n the experimental manner does not possess those pre-requisites?." I f the pre-requisites are not present i n the project, and i f the findings indicate l i t t l e or no ef f e c t , then i t i s very d i f f i c u l t to conclude whether the causal process did not materialise (theory f a i l u r e , meaning the theoretical assumptions with respect to the cause and effect relationships are i n v a l i d ) , or whether the program was not implemented s a t i s f a c t o r i l y enough to activate the causal process (program f a i l u r e ) . This confusion defeats the o r i g i n a l purpose of t h i s paradigm, which i s to test the v a l i d i t y of cause-and-effect hypotheses. Furthermore, t h i s impasse hinders program improvement and development because we do not know whether the program or the theoretical proposition i s at fa u l t (Weiss: 1972). - 26 -Successful Program Theory Fai l u r e Program Failur e Program Program Program set i n motion set i n motion didn't set i n motion Causal Process led to Desired Obj ective which Causal Process Desired Obj ective didn't lead to which Causal Process Desired Obj ective would have led to Fig. 3 Types of Program Fail u r e . (Weiss: 1972:38) Suchman (1972) recognises the impracticality of the ideal experiment and suggests some of i t s approximations. The panel/prospective design i s the second best choice, followed by survey design and case study. The panel design i s based upon (X2 - Y1). This means the program i s considered to have an effect i f measurements from the control group before the program are less than those from the experimental group after the program. The survey design (X2 - Y2) does not require "before" measurements since i t only quantifies the difference between the experimental and control group after a project has been carried out. The case study, commonly considered the weakest s t a t i s t i c a l l y among these three alternatives, only c o l l e c t s information from the experimental group after the program and does not need a control group. - 27 -Considerations on Some Problematic Issues This thesis i s concerned with innovatory or demonstration projects which are s t i l l at a developmental stage or are proceeding on a t r i a l and error basis, and therefore are not l i k e l y to have fixed or well-established program objectives and strategies. I t i s susceptible to change and improvement throughout the duration of i t s execution. The s t i p u l a t i o n of a fixed innovatory program i s not only unfeasible s o c i a l l y because of the prevalence of intervening factors but i s also counter-productive to program development, hence the o r i g i n a l exploratory purpose of demonstration projects i s also defeated (Weiss: 1972). Without a fixed or well-defined program by which the hypothesis of Prog •*• Cause ->• Obj can be tested, the basic purpose of the experimental approach for contributing to factual knowledge i s refuted as w e l l , as i s i t s function of evaluation research. The existence of uncontrolled intervening variables increases the l i k e l i h o o d of having disregarded findings on the basis of i n s u f f i c i e n t s c i e n t i f i c substantiation. By the same token, a program could be j u s t i f i e d or defended on the basis of the p o s s i b i l i t y of theory f a i l u r e instead of program implementation f a i l u r e . This kind of r a t i o n a l i z a t i o n might discourage program o f f i c e r s from learning about program f a u l t s . The requirement of producing factual knowledge about casual processes i n s o c i a l action settings appears rather over-optimistic, at least with respect to innovatory programs based largely upon new ideas, new insights on s o c i a l change, programs which are l i k e l y complicated by the - 28 -interruption of unforeseen circumstances. Another consideration i s the a v a i l a b i l i t y of measurable s o c i a l indicators. As mentioned e a r l i e r , the procedure of the experimental model permits only the c o l l e c t i o n of information that i s e a s i l y tangible, operational and quantitative. Therefore, the research findings are i n a way predetermined by the research method. In addition, how much i s considered the "desired effect" seems to be largely an arbitary decision. The amount of change regarded as desirable appears to depend mainly on the researcher 1s judgement or the preference of a decision-maker. These considerations cast doubts on the factual basis of evaluation findings. The preference for fixed and well-defined variables i n program design not only constricts the f l e x i b i l i t y and innovativeness and design, but also serves the end of summative rather than formative evaluation. These two terras were coined by Michael Scriven (1974). The former connotes evaluation that provides information at the end of a f u l l - c y c l e of a project i n order to i s o l a t e some specified program inputs, thus enabling the researcher to hold the inputs constant and to distinguish t h e i r effectiveness. By contrast, formative evaluation produces information that i s being fed back to a program throughout i t s duration so as to allow i t to develop. Formative studies appear more capable of serving the exploratory nature of innovative aproach than do summative ones. - 29 -Whatever thei r promises, summative studies deliver l i t t l e i n terras of revealing accurately the effectiveness of a specified input. Besides, as Weiss (1977) notes, evaluation findings are in practice often used for improving programs rather than deciding whether to continue or discontinue them. Suchman (1972) expresses preference for formative or process evaluation rather than summative or outcome studies under the reasoning that process studies have more potential for disclosing how and why a program succeeded or f a i l e d . The one-shot, p a s s / f a i l study i s considered u n r e a l i s t i c because evaluation results seldom d e f i n i t i v e l y "prove" a program to be a complete f a i l u r e ; hence evaluation should improve program flaws. So f a r , our discussion suggests that the pre-requisites of the experimental paradigm discourage formative studies. The problems mentioned above do not necessarily c a l l for abolishment of the experimental model. Some writers think rather about what improvements could be made, or what supplementary approaches might be applicable. Suchman, although a prime proponent of the experimental model, does not perceive i t to be the only legitimate method. In fact, he acknowledges the inherent normative and subjective nature of evaluation as an a c t i v i t y performed to test valued objectives and goals. According to him, the ideal t a c t i c i s to use multiple approaches which incorporate both subjective and objective appraisal. Subjective investigation may involve c o l l e c t i n g the subjective opinion of program recipients and s t a f f s as they evaluate the program. Objective 0 - 30 -measurements may be assembled by adopting standard experimental procedures i n testing hypotheses and effectiveness. To t h i s point, our discussion suggests doubts as to the adequacy of the experimental and summative methods to handle demonstration programs whose exploratory nature demands f l e x i b l e and adaptive program design and evaluation. The rigorous and r i g i d research rules of these methods tend to overlook subjective aspects and unintended consequences of s o c i a l actions. When the unintended consequences have any direct or indirect effects on the program recipients or other segments of a community, then recognizing these consequences may not only benefit program development, but may also influence our judgement of the worth of the program. Considerations of worth often depend upon value judgements and value issues, to which s c i e n t i f i c methods pay l i t t l e or no attention. NORMATIVE SOCIAL ANALYSIS As the shortcomings of the s c i e n t i f i c experimental mode became more apparent to some writers, alternative frameworks emerged. Several of these alternative approaches share some commonalities, such as th e i r recognition of divergent s o c i a l values influencing s o c i a l programming, their attention to the complexities and uncertainties of s o c i a l processes, and t h e i r "soft" methods or somewhat undeveloped methods of - 31 -research. As w e l l , each of these alternatives possesses a theme of i t s own. A Social Learning Framework According to some evaluators, the ultimate function of evaluation i s to educate society about human a c t i v i t i e s . The experimental model i s not necessarily taken as diametrically opposed to newer ones; rather i t i s perceived as the basis from which other options develop and expand. The learning process i n evaluation i s expressed succinctly by Lee Cronbach (1980), a well-respected figure i n the f i e l d of educational evaluation. Society i s constantly learning about i t s own procedures and the d i f f i c u l t i e s i t i s trying to overcome. And the process by which society learns i s evaluation, whether personal, impressionistic, or systematic or comparatively objective. (Cronbach:1980:12) Under this frame of reference, evaluation i s characterized by i t s comprehensiveness and educative function. I t i s supposed to serve society i n a broad spectrum rather than be used as an exclusive tool for the benefit of decision-makers, experts and planners. Under the learning framework of Cronbach (1980), evaluation should contribute to soci a l thought and perception. In other words, i t should be an e d i f i c a t i o n of s o c i a l problems and in j u s t i c e s ; i t should improve human services and the welfare of c i t i z e n s ; i t should encourage - 32 -more affected parties to participate i n the p o l i t i c a l process by-bringing the actors together and disclosing the decision-making machinery and processes; l a s t l y i t should reveal and revise the values, goals and assumptions of the actors. Above a l l , evaluation may also serve a symbolic function of checks and balances, i n that policy-makers and program s t a f f s , knowing that t h e i r actions are p u b l i c l y scrutinised, may be more responsible and responsive to the program recipients. These functions are constructed under the presumption of a p l u r a l i s t i c society, having d i f f e r i n g values, goals, interests and knowledge-levels among policy-makers, program o f f i c e r s and c i t i z e n s . Then, evaluation serves to surface the heterogeneous values and revise them i f necessary. The primary concern i s not merely to contribute objective knowledge to experts and the academia, but to disseminate that knowledge , s k i l l s of problem-solving and knowledge of the decision-making process, to the masses. Cronbach's view of the r e s p o n s i b i l i t y of evaluation to take part i n p o l i t i c a l processes i s shared by Robert Stake (1982), who affirms that i t should deal with fundamental issues or pol i c y directions of programs. However, Peter Rossi (1982), i n direct response to Stake's remark, argues against the p o l i t i c a l interference of evaluation. His reasoning rests upon the notion that p o l i c i e s and goals of public programs should be determined by the p o l i t i c a l process and not by evaluators whose job i s to c l a r i f y program goals, and to examine whether or not p o l i c i e s and programs reach the pre-determined objectives. - 33 -Hence, the duty of public o f f i c i a l s i s to provide technical information without questioning p o l i c y issues. Whereas Stake's viewpoint proposes that the p o l i t i c a l system has to be reformed or supplemented with participatory mechanisms through which the needs and interests of the recipients may be revealed and may revolve into p o l i c i e s and programs. And evaluation i s seen to be one of those mechanisms. Within the learning model, the methods of conducting evaluation have not been elaborated. This may suggest the model's indifference to setting up methodological rules. In general, methods ranging from subjective to more objective enquiries, and multiple approaches combining different research techniques are recommended. However, Cronbach (1980) i s extremely c r i t i c a l of the s c i e n t i f i c controlled experiment, which he believes requires impartial and impersonal enquiries and i s thus unsuitable for solving s o c i a l problems. An Illvminative/Interpretive Framework While Cronbach's learning model elaborates on functions but not methods, P a r l e t t and Hamilton (1972) provide the rationale, the structure by which a subjective and q u a l i t a t i v e manner of evaluation may be performed. Illuminative studies attempt to assess a program on the basis of i t s rationale, i t s evolution, i t s operations and i t s network of s o c i a l and i n s t i t u t i o n a l factors, a l l of which are believed to influence the program. This type of study consists primarily of descriptive accounts and interpretation rather than measurements and prediction. - 34 -Producing such information may take the form of observation, interviews and c o l l e c t i o n of h i s t o r i c a l records. This kind of qua l i t a t i v e investigation attempts to expose the subjective elements of the program, such as the remarks and response of parti c i p a n t s , issues r e l a t i n g to personalities and i n s t i t u t i o n a l p o l i t i c s . Adhering to qua l i t a t i v e and subjective findings does not necessarily mean that research procedures are unaccountable or ambiguous. In fact, this approach c a l l s for an evaluation that can be scrutinized by outside researchers or other interested parties. To make t h i s cross-checking procedure possible , research processes have to be documented, the c r i t e r i a used for selecting research material c l a r i f i e d , and the methodological and theoretical p r i n c i p l e s made e x p l i c i t . In short, the subjective aspects governing the frame of reference of the researcher have to be made apparent. Two major assumptions provide the impetus for the development of the illuminative framework. F i r s t , that s o c i a l processes, human a c t i v i t i e s , are very complex. There are many interwoven factors not read i l y observable or eas i l y detectable as d i s t i n c t e n t i t i e s , governing human thought and action. The second point proceeds from the f i r s t . Relying as i t does upon discernible variables, measurements and averages, the experimental model i s not s u f f i c i e n t for elucidating the complexities of s o c i a l processes. I t i s argued that by measuring and quantifying, one neglects - 35 -subjective data such as the concerns of the recipients and program s t a f f . By concentrating on r i g i d l y defined variables, one becomes insensitive to unintended effects resulting from the program. By using averages and s t a t i s t i c a l significance tests, one tends to overlook unique features and d i v e r s i t y among participants and s o c i a l processes. P a r l e t t and Hamilton describe the experimental method as a "technological s i m p l i f i c a t i o n of r e a l i t y . " They acknowledge that the illuminative framework probably increases uncertainty about human events through i t s attempt to expose complexity. They assert that the h i s t o r i c a l , subjective and q u a l i t a t i v e information generated by the l a t t e r approach would unveil the complexity. Peter Rossi's affirmation of the superiority of the experimental model i s p a r t l y based on the defence of i t s potential to produce generalizable data and c h a r a c t e r i s t i c s . While recognising the existence of both general and unique aspects i n human events, he describes the discovery of the general aspects as a more worthwhile pursuit of knowledge. In fact, the unique aspects are not regarded as a valuable source of knowledge. These opinions are again a direct reply to Stake's preference for unique features and subjective evaluation. Rossi and Stake's disagreement can be attributed to t h e i r c o n f l i c t i n g theories of knowledge. The a f f i n i t y towards creating generalizable data s i g n i f i e s an interest i n predicting and c o n t r o l l i n g s o c i a l processes; correspondingly, only objectively based information i s considered - 36 -knowledge. By contrast, Stake (1982) holds that the subjective expressions of needs and values, whose q u a l i t a t i v e overtones often do not match s t a t i s t i c a l formats, are worth knowing as a source of s o c i a l enquiry. Based upon our explication on the methodology of illuminative evaluation, i t Is evident that t h i s type of investigation aims to discover, to understand or to illuminate human a f f a i r s . I t s stated purpose i s to gather and present information about a program from various perspectives. The contribution made to decision-making i s only one of i t s functions but i s not the primary one. A Critical Consciousness Framework Recurring themes of c o n f l i c t i n g values and consciousness-raising mark the c r i t i c a l consciousness framework. Ruddock (1981) considers evaluation a value-laden exercise because programs and p o l i c i e s are always assessed against some valued objectives and goals. However, i r o n i c a l l y , most studies treat the valued objectives as given and merely investigate the relationships between program strategies and objectives, without considering the worth of those objectives. The central premise i n Ruddock1s framework establishes that there are c o n f l i c t i n g values and contradictions i n s o c i a l i n s t i t u t i o n s within the p l u r a l i s t i c society. By treating the valued objectives as given, evaluation overooks other values and interests which may deserve - 37 -attention. An example drawn from the f i e l d of education demonstrates the prevalence of c o n f l i c t i n g values i n s o c i a l i n s t i t u t i o n s . Evaluating the d e s i r a b i l i t y or effectiveness of a curriculum or a teaching method depends largely on whether or not the curriculum i s devised to f u l f i l the students, the i n s t i t u t i o n or society's need. Con f l i c t i n g needs that are not recognized and resolved may generate negative consequences. A teaching method i t s e l f may be very well thought out and l o g i c a l l y sound. However, i f i t does not r e f l e c t the need of the students, i t i s of l i t t l e use (Ruddock: 1981). S i m i l a r l y , i n s o c i a l service programs, where a program objective clashes with the need f e l t by the recipients, the program w i l l l i k e l y f a i l to generate any desirable change, however well-thought out the program strategies. Proceeding from that premise, the function of evaluation should then surface the c o n f l i c t i n g values and perceived needs of di f f e r e n t s o c i a l groups, and should also investigate t h e i r sources. The ultimate purpose i s to widen and deepen our awareness of the m u l t i p l i c i t y of perspectives and needs r e l a t i n g to a given problem (Ruddock: 1981). I t i s here recognized that the purpose of the evaluation i s paramount because i t determines the methods to be used. The use of multiple-purposes and multiple-methods i s considered the most admissible. I t i s essential that the underlying assumptions and the worth of a stated purpose be c r i t i c a l l y assessed. Given Ruddock's concern for evaluation being an instrument for system preservation, i t - 38 -i s advised that the perspectives of the recipients have to be included i n the studies (Ruddock: 1981). The premise elucidating the complex and subjective nature of s o c i a l enquiries constitutes the foundation of Ruddock's methodical framework. Under the postulate that s o c i a l processes stem from human intentions and consciousness beyond empirical research, s o c i a l enquiries that r e l y s olely on objective methods are inadequate. According to him, each method, whether comparatively objective or subjective, has i t s own special function. The key i s to use each technique appropriately to serve a designated purpose within a comprehensive framework. A three-phase evaluation, designed to answer- such questions as what i s happening, how i t happens and why i t happens, i s recommended. At the i n i t i a l stage, objective methods of measurements and quantification w i l l be applied to c o l l e c t factual information on p a r t i c u l a r social-economic trends and characteristics of the population. This kind of data may take the form of percentages, p r o b a b i l i t i e s , or other s t a t i s t i c a l formats. These s o c i a l s t a t i s t i c s are essential for indicating what i s happening i n the community being studied. The second stage requires the use of more subjective methods such as illuminative evaluation and description of how certain trends happen. At t h i s step, a longitudinal study of the happenings may - 39 -indicate some of the s o c i a l or i n s t i t u t i o n a l channels through which changes occur. Las t l y , the t h i r d stage attempts to ascertain why certain trends happen. The investigation i s no longer concerned with s o c i a l s t a t i s t i c s , but with s o c i a l dynamics, changing s o c i a l forces and the sectors of power. This part of the enquiry i s l i k e l y to be problematic because i t s analysis r e l i e s on issues and theories about human behaviour, s o c i a l forces and value-systems, a l l of which are normatively based. Information used i n this type of analysis can be derived from personal accounts of individuals, and accounts on history and society. Methods such as depth interview, l i f e h i s t o r i e s , interactionalism and psychodynamics can also be u t i l i z e d . Consistent with i t s o r i g i n a l theme on consciousness r a i s i n g of the m u l t i p l i c i t y of perspectives and issues, Ruddock summarizes his position very l u c i d l y : The argument of t h i s text, however, i s that we need to work with a f l e x i b l e or extended consciousness and a readiness to switch levels without losing focus. . . . In the s o c i a l sciences, and i n evaluation, i t i s not necessary to adopt Sartre's theory or any other global system, but we need to work with an awareness of the inter-relatedness of phenomena, and th i s should make us w i l l i n g to change the scale of our discourse at anytime, and without embarassment. (Ruddock: 1981:43) - 40 -SYSTEMS APPROACH: DUAL CONCENTRATION The evolution of evaluation i s marked by i t s increased complexity i n both form and content, i t s heightened recognition of the pervasiveness of value-based objectives and i t s use of multiple methods incorporating s c i e n t i f i c empirical research and normative analysis. Systems analysis i s used to study s o c i a l problems and evaluate s o c i a l programs. Two representative models are the d e c i s i o n i s t i c and systemic-dialectical frameworks. The Decisionistic Framework Within t h i s construct, evaluation i s defined as "the process of delineating, obtaining and providing useful information for judging decision alternatives" (Stufflebeam: 1973:129). I t s primary function i s to provide information for decision-makers. The information i s based on factual knowledge and c l a r i f i c a t i o n of i t s i m p l i c i t values. The evaluator i s not supposed to decide on the value choice, which presumably i s l e f t to the decison-makers to determine. Then, the purpose and the design of the evaluation conform to the needs and expectations of the decision-makers. This model presumes that decision-making i s a r a t i o n a l process which b a s i c a l l y r e l i e s cn factual information i n the selection of alternative p o l i c i e s . The ra t i o n a l decision-making model has been - 41 -challenged by Croriback (1980) , Lindbloom and Cohen (1979) and Perrin (1982). According to them, decision-making i s very diverse, amorphous and p l u r a l i s t i c , r e f l e c t i n g the interests of pressure group and parties, and often r e s u l t i n g i n a compromise between the competing interests. Based on the d e c i s i o n i s t i c model, the c o l l e c t i o n of information or monitoring i s paramount to evaluation. Its decision-making structure i s b u i l t with multiple feedback loops, through which information i s constantly transmitted to decision-makers and program managers during the program, i n order that the program can be improved and changed. The information being circulated i s b a s i c a l l y factual and s c i e n t i f i c . I t i s derived from monitoring the changes that occur during and after a program for the purpose of finding out whether or not the changes meet the program objectives. In other words, this i s an attempt to e s t a b l i s h causal linkages of program and effects, so that the evaluator can claim that given condition A and objective B, strategy C would achieve B (Stufflebeam: 1973). I t i s recommended that in early program development when objectives and strategies are vague, process evaluation be considered more appropriate than product evaluation because flaws could be i d e n t i f i e d and remedied at an early stage. Two strategies may be used for process evaluation. One aims to id e n t i f y or monitor continuously the potential sources of f a i l u r e i n a program be they due to personnel problems, communication obstacles or i n s u f f i c i e n t resources. Another strategy i s to maintain a record of program procedures. Both strategies - 42 -serve the purpose of informing managers about what i s happening to a program and what i s going wrong (Ib i d ) . Stufflebeam (1973) casts serious doubts on the u t i l i t y of the controlled experimental design i n evaluating educational programs and demonstration projects. Its r i g i d requirements of using treatment and controlled subjects, and of inducing uniform program output to a l l treatment subjects, do not f i t into educational programs. Instead, l e s s confined methods such as interaction analysis, interviews, r a t i n g scales, s t a f f meetings and Program Evaluation and Review Technique (P.E.R.T.), are suggested. Besides providing factual information, the d e c i s i o n i s t i c model also stresses the need to c l a r i f y the values which impinge on the formulation of evaluation c r i t e r i a . At least four kinds of values have been distinguised: i n s t i t u t i o n a l values (what the Board thinks) , community values, subsystems maintenance values (of co-workers), and the evaluator 1s values (Stufflebeam et a l : 1971). Although the model c l a r i f i e s these different values, i t makes no attempt to revise them; i n s t i t u t i o n a l values would probably override the others since the purpose of t h i s framework i s to serve decision-makers, presuming that the decision-makers are elected representatives heading the Board. One implication of t h i s type of evaluation i s that i t ultimately maintains the status quo of the dominant authority along with i t s prescribed value system. However, another approach which also i s based on systems theory and procedures, claims to be c r i t i c a l of the established system instead of preserving i t . - 43 -The Systemic-Dialectical framework The systemic d i a l e c t i c a l evaluation approach perceives erroneous problem d e f i n i t i o n s , c o n f l i c t i n g values and c o n f l i c t i n g performance c r i t e r i a as the basic characteristics of public p o l i c y and program. I t attempts to develop a procedure by which these c o n f l i c t s are surfaced, negotiated and resolved. I t does not impose any single set of evaluation c r i t e r i a , but u t i l i z e s two or more problem d e f i n i t i o n s , assumptions, methods and observers to assess a polic y or project (Dunn et a l : 1981). In broad terms, the function of evaluation i s to improve public p o l i c i e s and programs. In p a r t i c u l a r , i t should provide a systenic framework by which a l l relevant elements and their relationships are i d e n t i f i e d . The relevant elements to be i d e n t i f i e d and investigated are the normative issues determining evaluative c r i t e r i a and program objectives, the factors of personal motivation e l i c i t i n g action; the roles and viewpoints of diverse stakeholders influencing program performance, and l a s t l y the ways and means by which people u t i l i z e evaluation results (Ibid). The mistake (the Type I I I error) made by most programs i n defining the problem i s a major factor pinpointed by the systemic d i a l e c t i c a l framework. The following view renders the central merit of thi s framework: - 44 -In my f i r s t operation research problem, I f e l l into the trap of working on a wrong problem . . . In a f i r s t course i n s t a t i s t i c s , the student learns that he must constantly balance between making an error of the 1st kind ( i . e . rejecting the n u l l hypothesis when i t i s true) and an error of the 2nd kind ( i . e . accepting the n u l l hypothesis when i t i s f a l s e ) . . . . John Tukey suggested that practitioners a l l too often make errors of a 3rd kind: solving the wrong problem. (Ibid: 264) According to Dunn, this error i s made when an evaluator accepts u n c r i t i c a l l y the problem d e f i n i t i o n and objectives put forward by a single stakeholder. The remedy recommended i s to u t i l i z e a method that e x p l i c i t l y and systemically examines c o n f l i c t i n g perspectives of the same problem. Different stakeholders should be encouraged to suggest different problems as experienced or f e l t by them; the evaluation thus takes an active r o l e i n defining the problem through a synthesis of multiple perspectives. The manner i n which we design s o c i a l programs and p o l i c i e s ultimately depends upon our perception of the problem to be resolved. Most s o c i a l prograns are corrective. That i s , when an undesirable trend i s detected or predicted (high crime rate, i n s u f f i c i e n t housing for the poor, t r a f f i c congestion and so f o r t h ) , a program i s devised to improve the s i t u a t i o n . Before any corrective measures can be prescribed, one must diagnose the problem that i s thought to lead to the undesirable trend. Only then can any corrective measure tackle the designated problem. Therefore, the d e f i n i t i o n of the problem i s extremely c r u c i a l . I t sets o f f the whole series of events and shapes our formulation of the objectives and strategies geared toward solving our designated problem. - 45 -The methods that have been devised to carry out these functions involve the use of numerous approaches, ranging from questionnaires, archives, and participant observation to more objective research of s t a t i s t i c a l and experimental techniques, depending on the different stages of evaluation. 1Assumptional Analysis' i s the main thrust of t h i s methodical construct. One begins the task by specifying the issue areas and by selecting stakeholders on the basis of involving proponents of as many diffe r e n t viewpoints as possible. P r o b a b i l i t y sampling w i l l not be used because the purpose i s to have s o c i a l l y meaningful groups, not representative set of individuals. Instead, Theoretical and Sociometric Sampling, which selects groups and individuals because of the i r different s o c i a l positions, w i l l be applied. Then, assumptions, including causal (X -> Y), functional (Purpose X i s to do Y), and e t h i c a l (X ought to do Y), a l l have to be i d e n t i f i e d . This i s followed by stages of p r i o r i z i n g assumptions, reducing them, debating them, and f i n a l l y developing a synthesis of these assumptions. After the assumptions have been selected on the basis of t h e i r r e l a t i v e importance and r e l a t i v e certainty, they w i l l be selected depending on t h e i r potential contribution to u t i l i z a b l e p o l i c i e s and strategies. The f i n a l stage i s to design measures that correspond with the assumptions being chosen. Thereafter, conventional research methods may be used (Dunn: 1981). - 46 -This framework may contribute to the development of evaluation i n terms of i t s relentless e f f o r t to disclose the internal frames of reference governing our thoughts and actions. I t s implications for evaluation are that no aspect should be taken for granted, be i t evaluative c r i t e r i a , problem d e f i n i t i o n , objectives or purposes. In addition, the suggestion that merely surfacing c o n f l i c t s i s i n s u f f i c i e n t but that some form of negotiation of the c o n f l i c t s i s necessary, introduces a controversial r o l e for evaluation. Evaluation hereby becomes an agent of s o c i a l change, i n the sense of uprooting and revising the dominant value system. Considerations of Some Problematic Issues In our preceding discussion, we can detect some degree of concordance between the two systems approaches to evaluation. Both accept the premise that s o c i a l processes are characterized by the dynamic relations of inter-related elements operating within a defined system. Their divergence rests upon th e i r d i s t i n c t approaches i n dealing with t h i s "organized complexity". The d e c i s i o n i s t i c model tackles organized complexity by dissecting the decision-making process into discrete u n i t s . This allows information to be channelled from element to element through the superflous feedback loops. Dissenting values are surfaced but not necessarily resolved. The model's primary purpose i s to f a c i l i t a t e decision-making. - 47 -By contrast, the systemic-dialectical framework aims to reveal and reconcile c o n f l i c t i n g values and assumptions. I t centers on the latent aspects of our thoughts and actions, that i s , what assumptions make us think i n certain ways and what motivations guide our action. I t constructs a systemic manner of analysis by which the inter-connected issues and assumptions are made known. Weilemann (1980), affirming his support for the systems mode of evaluation, argues that the systems approach can achieve what the s o c i a l sciences have not been able to accomplish. He regards the basic s o c i a l science model of tracking down a simple cause-effect r e l a t i o n as impossible, and nonsensical. Instead, he r e l i e s on systems and cybernetic theories that expound on the c i r c u l a r i t y and inter-relatedness of s o c i a l phenomenon, and the mutual causal relationships whereby A affects B and B affects A (Ruddock: 1981). Yet, i f t r a d i t i o n a l s o c i a l science methods have not been able to demonstrate s a t i s f a c t o r i l y the simple "A causes B" relationship, how can systems theory master even more complex and m u l t i - l e v e l l e d relationships of causes and effects? Ruddock (1981) exposes the shortcomings of Weilenmann's stance very astutely. Systems theory may be useful i n administration, service, m i l i t a r y or welfare organizations where there i s r u l i n g consensus on goals and objectives. However, where value and interests are diverse and ununiformed, applying systems approach may be inappropriate, time and resource - 48 -consuming. And that the grandeur scheme of hierachy of feedback loops, sub-systems and sub sub-systems may be beyond the threshold of feasibility, given our current state of the art of our knowledge of social processes, and our resource constraints in the practical world. (Ruddock: 1981 :94) PSEUTX>-EVALUATION Included in our fourth quadrant are studies whose commitment to either the scientific method or social value analysis is minimal compared to the others mentioned above. Some writers, including Dunn (1981) and Stufflebeam (1980), refer to this type of evaluation as pseudo-evaluation. Let us briefly mention some of these approaches. Some studies are done for the purpose of continuously supplying information to managers and staff directing and controlling programs. The main difference between this approach and the decisionistic framework is that the former provides information for developing and defending the worth of program instead of losing information to judge its worth. Such studies try to find out whether or not the program activities are implemented according to schedule, on budget and with the expected results. This type of "management information system" utilizes systems analysis such as Program Planning and Budgeting System (P.P.B.S.), compute-based information systems, cost analysis and audits (Stufflebeam: 1980). Cost effectiveness studies aim to find out whether program expenditures are used effectively. This type of information is - 49 -important for administrators, whose al l o c a t i o n of resources i s based on costs and p r o f i t s . Cost considerations may be important for evaluation since they are commonly regarded as one of the major determinants of resource a l l o c a t i o n . However, they do not constitute evaluation by themselves i f they do not relate costs to some value issues (Ruddock: 1981). CONCLUSIONS This chapter has presented the diverse aims and methods of evaluation. This d i v e r s i t y r e f l e c t s the c o n f l i c t i n g opinion as to whether evaluation should apply s c i e n t i f i c experiments and supply f a c t s , or explore subjective methods of analysis and cope with value issues. The s c i e n t i f i c experimental framework assigns to evaluation the r e s p o n s i b i l i t y of generating factual knowledge about s o c i a l causes and effects; t h i s factual knowledge, i n turn, i s used to test hypotheses (eg. Program A -> Causal Process C -> Objective B). I t s methodical structure demands the presence of well-defined objectives, the a v a i l a b i l i t y of measurable variables, a knowledge of causal processes, and the control of intervening factors. However, when these features do not e x i s t , as i t i s common i n s o c i a l innovatory programs, the s c i e n t i f i c controlled experiment becomes problematic. Yet, there are alternative modes which acknowledge the complexity of s o c i a l action and which aim to investigate subj ective and value - 50 -issues. One such mode advocates s o c i a l learning so that evaluation may benefit society as a whole, and not j u s t a selected group of decision-makers. Another approach treats the interpretive understanding of multiple perspectives and needs as the major task of study. Another sees consciousness-raising as the ultimate purpose—the need to be aware of c o n f l i c t i n g values and objectives, and to be f l e x i b l e with our frames of reference. I f our aim i s to discover the subjective aspects and the c o n f l i c t i n g interests i n s o c i a l programs, and i f we want to explore f l e x i b l e approaches to enquiry, these alternative modes seem highly promising. - 5 1 -CHAPTER 3 EVALUATION AND EPISTEMOLOGY - 52 -INTRODUCTION The previous chapter has explicated the various frameworks of the aims and methods of evaluation research. In t h i s chapter, the epistemological theories embedded in these aims and methods w i l l be examined i n greater depth. I s h a l l scrutinise the p r i n c i p l e s of positivism and pragmatism which are i n t r i n s i c to the s c i e n t i f i c experimental framework. As w e l l , I s h a l l review c r i t i c a l theory—an approach extending beyond scientism and positivism—which proposes alternative modes and which acknowledges the s i g n i f i c a n t influence of subjective values i n s o c i a l a f f a i r s and i n the generation of knowledge. The theories of knowledge propounded by Kant, Hegel and Marx w i l l also be discussed. IDENTIFYING OONNECTIONS BETWEEN EVALUATION AND EPISTEMOLOGY The h i s t o r i c a l overview section demonstrated that evaluation depends to a large extent upon the development of the s o c i a l sciences, whose concern for delineating the causes and effects of so c i a l processes has translated into a primary concern of evaluation. This emphasis on causality derives largely from the natural science mode of observing the causal relationships of physical properties. The i n t r i c a t e linkages between the sciences and epistemology has been emphasized by Kar l Mannheim, who argued that epistemological - 53 -presuppositions are embedded in every scientific enquiry, and should be critically examined. Once we realize that although epistemology is the basis of a l l the empirical sciences, i t can only derive its principles from the data supplied by them, and once we realize, further, the extent to which epistemology has hitherto been profoundly influenced by the ideal of the exact sciences, then i t is clearly our duty to inquire how the problem will be affected when other sciences are taken into consideration. (Mannheim: 1936:292) Mannheim implied that the traditional epistemology of the sciences may not be perfect, that they too should be subject to scrutiny and revision, especially when a non-natural science issue is being dealt with. Mannheim's interest in developing a new epistemology appears to stem from his historical perspective on the growth of knowledge. According to him, what is considered knowledge or what is intelligible to human beings is always subject to change resulting from the flux of historical experience. Each historical period defines its problem of 'knowing' within its conceptual construction, which in turn arises from the experiences of the people in that era (Ibid.). Correspondingly, ideas cannot be understood autonomously in some 'inner dialectic,' but have to be related to their social and historical source. In other words, we do not think abstractly in a contemplating mind, but construct our world views accordingly to our social status and role (Coser-. 1971). Mannheim's social epistemology suggests that there is no single absolutely valid and relevant method in - 54 -s o c i a l enquiry, as any method i s subject to transformation i n accordance with the s o c i a l development of different eras. This c a l l s for the need to explicate the epistemological presuppositions of enquiry as i t i s applied i n various s o c i a l settings, and to examine c r i t i c a l l y t h e i r relevance to changing s o c i a l conditions. He opposed the natural science methods and p o s i t i v i s t i c views, claiming that they are both not suited for studying the dynamic and h o l i s t i c character of c u l t u r a l a f f a i r s . THE SCTENTISTIC APPROACH TO EPISTEMOLOGY The s c i e n t i s t i c perspective on knowledge affirms that only science produces knowledge rather than seeing science as one form of possible knowledge. Therefore, knowledge can only be ascertained through the application of s c i e n t i f i c procedures. Accordingly, epistemology, which t a c i t l y assumes that there are different forms of knowledge, i s undoubtedly disregarded by scientism and replaced by.the philosophy of science, whose task i s limited to a self-understanding of the sciences. Epistemology, however, investigates beyond s c i e n t i f i c methods and the philosophy of science (Habermas: 1971). K a r l Popper (1957) supports a s c i e n t i s t i c stance with respect to epistemology. According to him, epistemology should be i d e n t i f i e d with the theory or the choice of s c i e n t i f i c methods. The central problem of epistemology has always been and s t i l l i s the problem of the growth of knowledge . . . and the growth of knowledge can be studied best by studying the growth of s c i e n t i f i c knowledge. (Popper: 1957:15) - 55 -He further confirms the necessity of using empirical methods i n order to a t t a i n s c i e n t i f i c knowledge. "The theory of knowledge . . . may accordingly be described as a theory of the empirical method—a theory of what i s usually ca l l e d experience," (Ibid.: 39). Habermas (1971) i n s i s t s upon the broad perspective and the c r i t i c a l basis of epistemology which takes no type of enquiry for granted, even science, which can only be examined as one category of possible knowledge. Knowledge i s to be taken as neither absolute nor as the " s c i e n t i s t i c self-understanding of the actual business of research." The s c i e n t i f i c method was b r i e f l y introduced i n the previous chapter and w i l l now be discussed i n greater d e t a i l . Popper (1957) holds that a statement i s s c i e n t i f i c only i f i t i s testable, and that information that i s empirical, factual and quantifiable i s more testable than that which i s not. Therefore, testing, r e p l i c a b i l i t y , empirical evidence, facts and measurement are necessary features of the s c i e n t i f i c method. There are certain areas, such as value issues and the knowing subject, to which s c i e n t i f i c enquiry i s i n d i f f e r e n t . Although he acknowledges that there are value questions which may be s i g n i f i c a n t and should be addressed, Popper takes these to be outside of s c i e n t i f i c enquiry. Besides, the l o g i c a l analysis of s c i e n t i f i c knowledge does not investigate the process of conceiving a new idea, or how a new idea occurs to a person, but examines the idea i t s e l f , and the methods and results of examination. In other words, concepts such as consciousness, - 56 -ideology and emancipation, a l l of which relate to the states of mind of the subject, are then considered i n s i g n i f i c a n t . The processes of thinking, knowing and believing as they are experienced by the subject, are deliberately dismissed. I t i s c r u c i a l not to presume a un i f i e d s c i e n t i f i c method. On the contrary, there are various streams, each having i t s own aims and procedures upon which the general framework of testing i s constructed. Two representative streams are induction and deduction. Inductive reasoning proceeds from observations of p a r t i c u l a r incidents to general conclusions and universal laws. Repeated observations or experiments serve to j u s t i f y or v e r i f y the established theories or laws (McConnell: 1981). The t r a d i t i o n a l view of s c i e n t i f i c method had the following stages i n the following order, each giving r i s e to the next (1) observation and experiment (2) inductive generalization (3) hypothesis (4) attempted v e r i f i c a t i o n of hypothesis (5) proof or disproof (6) knowledge . . . . (Ibid.: 44) The deductive reasoning developed by Popper, the c r i t i c a l s c i e n t i f i c method, advocates an alternative procedure. I t progresses from a general stance to more s p e c i f i c inferences. I t rejects testing through v e r i f i c a t i o n by proving a theory; rather, i t pleads for f a l s i f i c a t i o n , that i s by disproving a theory. Popper believes that s c i e n t i f i c enquiries should st a r t by identifying a problem, then developing tentative solutions which should be c r i t i c i z e d and whose - 57 -errors should be detected. The res u l t i s generation of a new set of problems which w i l l provide impetus for the process of c r i t i c a l r e v i s i o n to repeat i t s e l f . The deductive method, commonly considered more acceptable than induction, proceeds along these stages: (1) problem (usually rebuff to existing theory or expectation) (2) proposed solution; i n other words a new theory (3) deduction of testable propositions from the new theory (4) test i.e. attempted refutations by, among other things (but only among other things), observation and experiment (5) preference established between competing theories. (McConnell: 1981:44). The s c i e n t i s t i c and c r i t i c a l approach suggest at least two general directions for the conduct of evaluation. The former stance considers legitimate only the use of s c i e n t i f i c - e m p i r i c a l methods i n evaluating s o c i a l actions. I t supports the view that experimental methods should be the standard evaluation mode. The methodological problems encountered i n evaluation are only to be solved by devising alternative s c i e n t i f i c methods of research. The c r i t i c a l perspective proposes a need to investigate broader and multiple approaches to evaluation, even i f they do not conform to the s c i e n t i f i c manner of research. This position does not necessarily affirm the superiority of no n - s c i e n t i f i c methods over s c i e n t i f i c ones. However, t h i s view i s instructive to us i n two aspects: we need to understand our procedure of enquiry i n a context that exceeds our own frame of reference; and we need to be tolerant and w i l l i n g to adopt - 58 -alternative modes of enquiry and theories of knowledge. I t should be noted that the c r i t i c a l approach to epistemology does not reject the empirical method; rather, i t disavows i t s claim to be the standard procedure of enquiry. Abraham Kaplan (1964) exposes the myth held by scientism i n r e l a t i o n to the arbitrary boundary drawn between value issues and s c i e n t i f i c undertakings. Scientism i s connoted as "the pernicious exaggeration of both the status and function of science i n r e l a t i o n to our values" (Kaplan: 1964:401). Scientism holds that pol i c y must be accompanied by charts, graphs, equations and computerized data, and that to acknowledge value issues i s u n s c i e n t i f i c . Kaplan recommends an approach which recognises the diversive range of cognitive styles and interests which constitute s o c i a l enquiry. THE PURPOSE OF CRITIQUE OF KNOWLEDGE AS HELD BY KANT, HEGEL & MARX Immanuel Kant (1724-1804), commonly perceived to be the f i r s t c r i t i c a l philosopher to establish the c r i t i q u e of knowledge, grounded the formal recognition of epistemology. To Kant, knowledge i s to be crit i q u e d by our cognitive f a c u l t i e s of reason and s e l f - r e f l e c t i o n . This process of reasoning w i l l f a c i l i t a t e reason to be aware of i t s own formal structure. This reasoning endeavour needs no j u s t i f i c a t i o n other than for the sake of reasoning i t s e l f (Habermas: 1971). By examining Kant's theory on human perception, we can detect some clues as to why - 59 -the reasoning endeavour has such a paramount standing i n his epistemology. In the Kantian perspective, we never know or see objects as they r e a l l y are; what we know of them depends on our own thinking models. Theories of causality are among these cognitive models. These models are the a p r i o r i conditions of knowing, that i s , the cognitive forms and causal assumptions which determine the kind of objects we see and the events we pay attention to. Therefore, we cannot a t t a i n absolute knowledge, but r e l y only on the "conditions of knowledge" or the cognitive forms to allow us to recognise certain perspectives of this "absolute knowledge." The bare object of knowledge per se cannot be recognised by us (Kant: 1781). I f we want to examine the basis of our knowledge, we must trace i t s cognitive models. I f these underlying models are products of reason, then reasoning i t s e l f i s the most appropriate means of tracing the model. Hence, the c r i t i q u e of knowledge. Although Kant stresses the importance of speculative a p r i o r i knowledge, he also argues for the empirical necessity of expanding knowledge. This i s supported by his exposition on the d i s t i n c t i o n between a n a l y t i c a l and synthetical judgements. A n a l y t i c a l judgements merely divide a given concept into elements, as well as relationships among the elements. Synthetical judgements on the other hand, expand a given concept by adding a predicate drawn from experience. He concludes that empirical judgements are a l l synthetical, and thus are necessary - 60 -f o r expanding knowledge. To him, both speculative reason and empirical knowledge are essential enquiries: . . . But although a l l our knowledge begins with experience, i t does not follow that i t arises from experience . . . a question which deserves at least closer investigation, and cannot be disposed at f i r s t sight, whether there exists a knowledge independent of experience . . . such knowledge i s called a p r i o r i , and distinguished from empirical knowledge, which~lias i t s sources a p o s t e r i o r i , that i s , i n experience. (Kant: 1781:2) The contribution of Kant's cognitive-empirical perspective to the development of epistemology i s two-fold. One contribution i s the elucidation of the constitution of knowledge and facts: that there i s no bare knowledge or bare fact as such. Knowledge and fact are guided by cognitive models forming the different conditions of knowledge; thus, c r i t i q u e of knowledge i s j u s t i f i e d . The second contribution i s s i g n i f i c a n t because of i t s controversial nature, and i t s consequent stimulation of new epistemological theories. This i s the notion of the seemingly unending process of reasoning. I f reasoning i t s e l f i s the ultimate purpose of c r i t i q u e , then reasoning i t s e l f has to be critiqued as w e l l . Accordingly, the c r i t i q u e of knowledge takes no forms of knowledge for granted, but r e l e n t l e s s l y questions the pre-suppositions that determine the frame of reference of a given investigation. Upon that basis, a controversial issue arises as to the presuppositions of c r i t i q u e i t s e l f . In other words, one has to question the way one questions, and - 61 -t h i s quest has to go on i n d e f i n i t e l y . The unconditional doubt required by Kant's c r i t i q u e i s subjected to a vicious c i r c l e . The c r i t i c i s m was taken up by Georg Hegel (1770-1831), whose epistemology resolves the c i r c l e of unconditional doubt to which Kant's c r i t i q u e i s subjected. Reflective experience replaces unconditional doubt as the purpose of c r i t i q u e . Hegel establishes the phenomenology of mind which asserts that the r e f l e c t i v e process of c r i t i q u e i s knowledge already. Then epistemology cannot be j u s t i f i e d i n i t s own terms (as a challenge of presuppositions), but only as a phenomenological experience—the process of r e f l e c t i o n as experienced by the subject who may then a t t a i n consciousness. Although Hegel's phenomenology resolves the c i r c l e of unconditional doubt i m p l i c i t i n Kant's c r i t i q u e , i t b a s i c a l l y makes reasoning superflous by l e g i t i m i z i n g reasoning as an experience instead of r a d i c a l l y transforming the i d e a l i s t epistemology (Habermas: 1971). Both of these i d e a l i s t t r a d i t i o n s served as an impetus for the construction of Marxian m a t e r i a l i s t epistemology. In opposition to both Kant and Hegel, K a r l Marx (1818-1883) believed that neither l o g i c a l reasoning nor phenomenological s e l f - r e f l e c t i o n can provide the j u s t i f i c a t i o n for c r i t i q u e of knowledge; rather c r i t i q u e has to be based upon the s o c i a l l i f e processes, the material production and labor conditions of the human species. This m a t e r i a l i s t perspective, which uses the political-economic structure as - 62 -the basis of c r i t i q u e — t h e c r i t i q u e of p o l i t i c a l economy-marks a milestone i n the development of epistemology. Based upon Marxian theories, the purpose of attaining knowledge i s to emancipate the human species. Emancipation requires two s p e c i f i c types of knowledge: technically exploitable and s e l f - r e f l e c t i v e . The former serves to f a c i l i t a t e the process of material production, and consequently to promote technological progress. In Marxist terms, humans are relieved from physical labor as machines are invented to substitute for manual work. This kind of technological progress i s possible only with the growth of s c i e n t i f i c knowledge, of the natural sciences (e.g. physics, which discovers natural laws of motion) (Habermas: 1971). The l a t t e r catagory, s e l f - r e f l e c t i v e knowledge, although t a c i t l y contained i n Marxist notions of class consciousness and ideology, was not formally acknowledged by his epistemology, and was consequently incorporated into the technical and instrumental knowledge of the former. Although Marx acknowledged the two types, the significance of s e l f - r e f l e c t i v e knowledge i s subsumed by technically exploitable knowledge. However, i n order for the human species to achieve f u l l emancipation, i t has to att a i n a l e v e l of c r i t i c a l consciousness s u f f i c i e n t to free i t s e l f from ideological delusion and norms of domination. Ideology serves to reinforce dogmatically some prescribed norms or t r a d i t i o n s . What Marx does not seem to elaborate i s that i n order bo at t a i n emancipation, the subjects must possess the faculty to - 63 -assess c r i t i c a l l y and recognise the ideological forms. This faculty can be r e a l i z e d only i n the Hegelian concept of c r i t i c a l s e l f - r e f l e c t i o n as carried out by the subject (Habermas: 1971). Hegel's notion of c r i t i c a l s e l f - r e f l e c t i o n may be understood as the recognition of the evolution of ideas that has accompanied human history; and sp e c i a l l y , the r e a l i z a t i o n of the abstract ideas and values that have shaped our thinking and attitudes (Taylor: 1975). As w e l l , Kant emphasizes also the significance of c r i t i c a l r e f l e c t i o n i n discovering the subjective conditions moding our concepts of the objective world. He elucidates: "Reflection (reflexio) i s not concerned with objects themselves, i n order to obtain d i r e c t l y concepts of them, but i s a state of the mind i n which we set ourselves to discover the subjective conditions under which we may ar r i v e at concepts. . . . Many a judgement i s accepted from mere habit, or made from i n c l i n a t i o n , and as no r e f l e c t i o n precedes or even follows i t c r i t i c a l l y . . ." (Kant: 1781:202) There i s l i t t l e doubt that Marx rad i c a l i z e d Kant and Hegel's epistemology by introducing the element of s o c i a l labor, an action based a c t i v i t y as opposed to the exclusive thought processes as held by h i s two predessors. While Marx disagreed with some of their theories, he also accorded with them i n other instances. He inherited the Hegelian notion of the d i a l e c t i c s of the self-formative process of the human species, that humans construct new standards of knowledge and new modes of consciousness after a process of c o n f l i c t . Marx applied this notion to - 64 -the c o n f l i c t of s o c i a l classes, resulting i n the emergence of a new class and socio-economic conditions, and also a new consciousness. In accordance with Kant, Marx acknowledged an objective nature or an external world existing apart from human conception. Concommitantly, an empirical basis of knowledge i s taken by them as legitimate means of r e a l i z i n g t h i s objective world. Kant merged empirical and l o g i c a l forms, while Marx proposed to expose the empirical r e a l i t y of socio-economic conditions to raise class consciousness (Habermas: 1971). Marxian notions of human history and society could have contributed to a s o c i a l theory of knowledge, namely, that a r a d i c a l c r i t i q u e of knowledge should be based on the s o c i a l history of humankind through the medium of s o c i a l labor, class struggle and s o c i a l interaction. However, t h i s leap did not happen. On the contrary, the Marxian theory of knowledge succumbed to the 'absoluteness of s c i e n t i f i c knowledge' as held by the p o s i t i v i s t school. On the other hand, inadequacies of Marxist p r i n c i p l e s spurred the b i r t h of the c r i t i c a l school of thought, through which Marxist theories are re-examined, re-constructed and expanded ( I b i d . ) . POSITIVISM Positivism, as indicated by Auguste Comte (1798-1857), relates to the s c i e n t i f i c methodological framework which concerns the 'actual' instead of the merely 'imaginary.' The 'positive s p i r i t ' i s assigned to research procedures that guarantee s c i e n t i f i c o b j e c t i v i t y . I t s three - 65 -fundamental doctrines are empirical and methodical certainty, and the technical basis of knowledge. I t affirms that a l l knowledge has to be achieved through sense-observation. Anything that i s neither observable nor detectable by sensory experience i s disregarded. Only empirically based information produces facts which can increase the o b j e c t i v i t y of our investigation. Another p o s i t i v i s t dogma stipulates that s c i e n t i f i c enquiry should place p r i o r i t y on the method over the substance of investigation (Habermas: 1971). The standard i s the positive method which treats observation, experimentation and theory formulation as necessary s c i e n t i f i c procedures (Coser: 1971). Furthermore, i t holds that technical, objective knowledge allows humans to exert control over nature and society, thus promoting technological progress. Comte i s sometimes regarded as the father of the s o c i a l sciences, as he aimed to create a science of society to explain s o c i a l occurrences and to predict the future. His science of society i s b a s i c a l l y borrowed from Francis Bacon's p r i n c i p l e of the natural science, 'To see i n order to foresee". The natural science approach to studying nature, to discovering natural laws, i s then transferred to the study of society i n i t s attempt to discover laws of s o c i a l action (Coser: 1971). We s h a l l f i n d that there i s no chance of order and agreement but i n subjecting s o c i a l phenomena, l i k e a l l others, to invariable natural laws, which s h a l l as a whole, prescribe for each period, with entire certainty, the l i m i t s and character of s o c i a l action. (Corser: 1971:4) - 66 -The natural science i s praised because of the l e v e l of technological progress made possible by i t s generation of technical ' knowledge. The s o c i a l science i s taken as the highest i n the hierarchy of the sciences, on which s o c i a l science i n turn depends. By looking to s c i e n t i f i c interests and methods as the only basis for knowledge, positivism replaced the theory of knowledge with the philosophy of science. The philosophy of science i s concerned only with procedures and rules of research and theory construction, but undermines the significance of the knowing subject and the purposes for which the subject uses the knowledge. I t s pre-occupation with the meaning of facts but not the meaning of knowledge i m p l i c i t l y equates fact with knowledge. By merely dealing with the v a l i d i t y of facts , i t leaves out the a p r i o r i constitution of facts as experienced by knowing subjects. Positivism regresses behind the l e v e l of s e l f - r e f l e c t i v e c r i t i q u e advocated by Kant and Hegel (Habermas: 1971). Facts as interpreted i n the p o s i t i v i s t i c mode contain two dimensions: the sensations of color, temperature, shape, oror, etc. (elements), as detected by us; and the physical bodies or things exerting those elements which are independent of us. Even though sensory evidence i s accounted for , the perceiving subject who experiences the sensation i s deliberately ignored i n favour of the elements of sensations. As Ernst Mach observes: - 67 -I can resolve a l l of my physical attributes into elements that cannot be further analysed: colors, pressures, temperature, odors, spaces, times, etc. These elements reveal themselves dependent on circumstances both outside and inside my bodily environment. I f and only i f the l a t t e r i s the case, we also c a l l these elanents sensations . . . What i s primary i s not the ego (the knowing subject) but the elements, sensations . . . (Habermas: 1971 : 82, 83) By neglecting the perceiving subject, without whose experience of sensation the elements have no way of being known, positivism reduces knowledge to bare sensations as they are measured by s c i e n t i f i c operations. In an attempt to maximize objective knowledge, positivism aligns i t s e l f with facts which are believed to exist uninterruptedly. Anything beyond sensory experience i s disregarded as 1pseudo-problems'; the perceiving subject with i t s 'inscrutable ego' i s also deleted from th e i r enquiry (Habermas: 1971). Kant, Hegel and Marx suggest that our sensations do not depend upon the physical world t o t a l l y but also depend on our own experiences. Kant i d e n t i f i e s a p r i o r i cognitive forms as the causes of our sensations. Hegel's abstract idealism implies that our sensations of the world depend primarily upon our own s e l f - r e f l e c t i o n of ideas. And a Marxian stance suggests that our sensations depend on our socio-economic posi t i o n . The p o s i t i v i s t i c notion of the p o s s i b i l i t y of bare facts and bare sensation i s refuted by Kaplan (1964). To him our observations of these sensory elements are always guided by some a p r i o r i conceptual - 68 -formulation and often interrupted by our interpretation of them. Perception i s then impossible without a conceptual and interpretive process by which these elements are understood as objects and events. Physical events and objects are always interpreted from s o c i a l and psychological perspectives. A l l these arguments cast doubts on the purely factual content of our sensory data, and hence also refute the idea that s c i e n t i f i c knowledge i s merely a record of sensory elements. Kaplan further emphasizes that interpretation i s guided not only by cognitive formulation but also by our values, our standards or pri n c i p l e s of worth. The boundary between facts and values i s ambiguous. Underlying values always determine the kind of facts we c o l l e c t and the di r e c t i o n of our research. He concludes that values cannot be excluded from science, but they can be made more objective and e x p l i c i t . Then the problem of s o c i a l enquiry i s not whether values e x i s t i n our research but rather, how they are to be empirically grounded and made known. Logical positivism, a modern science and philosophy developed i n the 1920's, b a s i c a l l y abides by a l l the three p o s i t i v i s t doctrines with the addition of the v e r i f i a b i l i t y p r i n c i p l e . I t asserts that a statement that i s neither formally (as i n lo g i c or mathematics) nor empirically v e r i f i a b l e by sense-observation i s meaningless (Magee: 1982). - 69 -PRAGMATISM Pragmatism replaces the epistemological concern for the o r i g i n of knowledge with the outcomes and effects of knowledge as i t i s applied i n action settings. I t aims to put knowledge and ideas to the use of instrumental action and technical control. Knowledge serves the function of a s s i s t i n g us to choose between alternative strategies of action. As d i s t i n c t from the Kantian approach to using l o g i c a l forms to study the constitution of knowledge, i t applies l o g i c to the connection between knowledge and action, and i m p l i c i t l y assumes a l o g i c a l connection between them. Charles S. Pierce (1839-1914), the founder of pragmatism, claimed that truth i s derived not merely from examining the l o g i c a l structure of statements but rather from the l o g i c of enquiry as i t i s applied i n the context of purposive-rational action. Enquiry i s seen as a means by which we attempt to understand and to exert control over the environment. I t i s operated under certain methodical and l o g i c a l rules which are decided by those who participate i n the process of enquiry. Under th i s condition the subjects are recognised as active participants i n deciding upon the structure of enquiry, and are not eliminated as i n positivism (Habermas: 1971). Within the pragmatist framework, a b e l i e f i s considered v a l i d only i f i t i s manifested i n the context of behaviour and action. B e l i e f - 70 -i s defined by Pierce as ". . . the establishment of a habit; and diffe r e n t b e l i e f s are distinguished by the different modes of action to which they give r i s e , " (Ibid: 120). Beliefs have to be validated i n the context of behaviour. Therefore, knowledge i s only meaningful i n the context of instrumental action. Accordingly, knowledge i s only taken as a means for achieving an end i n action and behavioural settings. The methodical rules of f u l f i l l i n g the lo g i c of enquiry are fi r m l y grounded i n controlled action settings. This i s the area where p o s i t i v i s t i c influence i s evident. The s c i e n t i f i c method i s regarded as most desirable for enquiry. I t serves the function of discovering laws through induction, discovering causes by hypothetical inference, and predicting effects by deduction. The experiment i s seen to be appropriate i n accomplishing these tasks, since i t allows for precise and controlled observation by which b e l i e f s can be validated. Therefore, as far as experiment i s concerned, only action and behaviour that are observable, controllable and measurable can be accounted for. Once we have established v a l i d b e l i e f s by executing the logi c of enquiry, we can predict that specified conditions w i l l e l i c i t p a r t i c u l a r kind of events or behaviour to occur. Predicting future actions and events i s a fundamental concern of pragmatism, but this concern may be problematic i n the s o c i a l context since i t i s very d i f f i c u l t to be certain of the cause and effect of s o c i a l actions. Another problem with the pragmatist mode of prediction i s i t s heavy reliance on l o g i c . I t s aim i n using knowledge to e l i c i t - 71 -c o l l e c t i v e purposive-rational action assumes that the enquirers and the community operate on a simi l a r l o g i c a l and ra t i o n a l basis. The behaviour of the community i s predicted according to the lo g i c held by the investigator. As Kaplan (1964) c r i t i c i z e s t h i s assumption, the community i s not j u s t affected by the logi c of statements and p o l i c i e s , and then led to act accordingly. Rather, there are s o c i a l and psychological factors which from the outset influence the community1s interpretation of lo g i c and pol i c y . Therefore, what seems from the viewpoint of the enquirer to be a l o g i c a l course of action may not be sensible from other viewpoints. Unless the pragmatist framework i s w i l l i n g to accept that s o c i a l action i s due not only to l o g i c , and to acknowledge other factors motivating human action, or else i t s l o g i c of enquiry w i l l f a i l to achieve i t s purpose of e l i c i t i n g c o l l e c t i v e r a t i o n a l action and of f a c i l i t a t i n g decision-making. This c a l l s for a mode of enquiry that i s not constrained within the p o s i t i v i s t and pragmatist framework of observation, objectivism, and instrumentation. CRITICAL THEORY The c r i t i c a l school was formed i n the 1920's i n Frankfurt, hence i t i s also known as the Frankfurt School. I t explores alternative principles and methods of enquiry. A major theme guiding the c r i t i c a l theorists i s the need or the d e s i r a b i l i t y of emancipation. According to - 72 -them, i n order to be t r u l y emancipated, one has to adopt a c r i t i c a l and s e l f - r e f l e c t i v e approach which views nothing, be i t a statement, an i idea, or an action, as natural, given or universally true. As Habermas (1971) notes, c r i t i c a l s e l f - r e f l e c t i v e knowledge presupposes the need for emancipation of the subjects, whose exercise of c r i t i c a l reason can free them from dogmatism and ideology. This appears to be the ultimate goal of c r i t i c a l theory. With regard to methods, the c r i t i c a l school uses h i s t o r i c a l reason as a form of c r i t i q u e . This i s a study of the hist o r y of individuals, society, events, and most important, a hist o r y of ideas and value systems. As Horkheimer claims There are connection between the forms of judgements and the h i s t o r i c a l periods . . . the c l a s s i f i c a t o r y judgenment i s t y p i c a l of pre-bourgeois society: This i s the way i t i s and humans can do nothing about i t . The hypothetical and disjunctive forms belong especially to the bourgeois world: under certain circumstances, this effect can take place; i t i s either t h i s or that way. C r i t i c a l theory maintains i t need not be so; humans can change r e a l i t y and the necessary conditions for such a change already e x i s t . (Arato & Gebhardt: 1982:384) The c r i t i c a l school's ideal form of critque attempts to incorporate the Kantian t r a d i t i o n of investigating the a p r i o r i constitution of knowledge, the c r i t i c a l s e l f - r e f l e c t i v e epistemology of Hegel, and the m a t e r i a l i s t basis of enquiry advocated by Marx. At the same time, i t rejects some elements of each of these t r a d i t i o n s . I t does not concur with the Kantian notion that cognitive reason i s the - 73 -ultimate producer of knowledge - that there may be other modes such as value and emotive elements determining the generation of knowledge. I t downplays the significance that Hegel places on the Idea i t s e l f as a moving force of history. I t believes that our knowledge of the world has to originate from r e a l l i f e conditions as experienced and f e l t by us, but that i t does not exist merely i n our minds. As w e l l , Habermas (1971) comments that c r i t i c a l s e l f - r e f l e c t i o n i s i n t u i t i v e at a subjective l e v e l , but i s emancipatory at a p r a c t i c a l l e v e l when the subject i s free from a dogmatic attitude and habits of l i f e . Then s e l f - r e f l e c t i v e knowledge i s not necessarily r e s t r i c t e d within the mind but may exert p r a c t i c a l consequences. Marxist theory l a i d the foundation for the c r i t i c a l school, whose sp e c i f i c mandate i s to re-construct Marxism. Herbert Marcuse remarks that c r i t i c a l theory sets out to explore the q u a l i t a t i v e aspects, the q u a l i t y of l i f e of society and not j u s t material change. Marxism increasingly concentrates on economic development, on production and on equal d i s t r i b u t i o n of products. However, qu a l i t a t i v e change such as self-determination, emancipation and freedom from alienation have not received comparable attention. The c r i t i c a l school also perceives a need to integrate psychology with Marxian theory which neglects the individual as a thinking, active and conscious being. I t studies domination of the r u l i n g class not only i n i t s material aspects but also on the l e v e l of the psychology of the less privileged class - the f a l s e consciousness imposed upon them i n the form of ideology (Magee: 1982). - 74 -The c r i t i c a l school sees a need to revive the importance of the knowing subject, arguing that the way i n which the subject perceives the world often r e f l e c t s an objective r e a l i t y about s o c i a l conditions. Therefore, the subjects have to be brought to l i g h t , t h e i r consciousness revealed, and t h e i r subjective r e a l i t y disclosed. C r i t i c a l theorists affirm that subjective values are the foundation of knowledge and facts. The generation of facts i s ultimately determined by the values that establish our terms of reference in the f i r s t place. Since the value component i s largely a latent aspect, or at least something that i s not e a s i l y measured and observed, using t r a d i t i o n a l experimental methods and stringent empirical research may not be s u f f i c i e n t for revealing value issues. Values operate at a conscious or sub-conscious l e v e l . Therefore, empirical knowledge, which acknowledges only what i s conceivable, may overlook the inconceivable processes of the mind, and correspondingly, values as w e l l . Horkheimer remarks: New forms of r e a l i t y , especially those emerging from the h i s t o r i c a l a c t i v i t y of human beings, l i e beyond empiricist theory . . . s c i e n t i f i c prognoses rarely refer to them as ' s i g n i f i c a n t changes,' for , understandably, there i s a lack of observational data. (Arato & Gebhardt: 1982:404) As far as the empirical method i s concerned, anything that i s not v i s i b l e through i t s methodical screen i s simply non-existent. Latent categories such as motivation and drive refer to "probable behaviour." - 75 -Another common assumption i s the correlation between quantity and significance, that the larger the quantity the more s i g n i f i c a n t i t i s . By t h i s means, anything that i s not e a s i l y quantifiable or not large i n number i s automatically considered s t a t i s t i c a l l y i n s i g n i f i c a n t and i s disregarded. Though c r i t i c a l theory recognizes the l i m i t a t i o n s and inadequacies of empirical methods, i t does not abandon these methods t o t a l l y but uses them as a legitimate mode of enquiry. Horkheimer advocates "a comprehensive empirical research apparatus into the service of problems of s o c i a l philosophy." Such an apparatus attempts to apply empirical research to a broad range of subjects that may include art, r e l i g i o n , public opinion, l i f e s t y les, entertainment, and so f o r t h (Arato & Gebhardt: 1982). I t i s held that the t r a d i t i o n a l s c i e n t i f i c method of measuring allows only a s p e c i f i c empirical r e a l i t y to be revealed, and that t h i s r e a l i t y may be j u s t one among many others. Therefore, by using the c r i t i q u e of h i s t o r i c a l reason to study a diverse range of subjects that are empirically based, the c r i t i c a l school attempts to expose various empirical r e a l i t i e s . For instance, aesthetic c r i t i c i s m i s one mode of enquiry explored by them. Through a r t , the f e l t d i s s a t i s f a c t i o n or new v i s i o n communicated by the a r t i s t expresses an empirical r e a l i t y that i s relevant and may be detected by c r i t i c a l examination of the art work. However, numerous c r i t i c a l theorists i n s i s t that c r i t i c a l theory i s neither anti-science nor a n t i - s c i e n t i f i c . What they attack, rather, - 76 -i s " s c i e n t i f i c method," the dominant method of enquiry according to which any enquiry or problem that does not f i t within the prescribed methodic procedure i s taken to be u n s c i e n t i f i c . I t i s emphasized that c r i t i c a l theory i s not an alternative to s c i e n t i f i c method or t r a d i t i o n a l theories; instead, they are i t s raw material or point of departure (Horkheimer: 1982). Marcuse (1982) elaborates that the c r i t i c a l school i s p r i m a r i l y against the lack of self-awareness of the prevalence of valued ends and objectives i n science. Reason i t s e l f i s often taken to be an objective t o o l for the purpose of devising e f f i c i e n t means and implementing valued ends which are considered beyond r a t i o n a l challenge. This relates to the o b j e c t i v i s t i c i l l u s i o n that facts can be disassociated from values. P o s i t i v i s t s undermine the contribution of subjective elements to knowledge and facts. However, a c r i t i c a l approach holds that one becomes more objective when one recognizes one's value-choice. Then o b j e c t i v i t y i s maximized, as one exposes the subjective elements of one's research. In order to make the subjective aspects objective, one has to reveal the s u b j e c t i v i t y i n some way. Adorno (1982) i n s i s t s that instead of trying to eliminate the inevitable s u b j e c t i v i t y operative i n any perspective, we should "bring this s u b j e c t i v i t y back to i t s o b j e c t i v i t y , " by being s e l f - r e f l e c t i v e i n our procedures. Habermas (1971) disagrees with the pragmatist neglect of the subjective use of knowledge, the view that knowledge i s to be used only - 77 -as a means for survival and for e l i c i t i n g action. The pragmatist, claims Habermas, overlook the knowledge which constantly determines the goal of our l i f e and actions. This category of knowledge does not operate at the l e v e l of technical control and instrumental action, instead i t i s reflected i n communicative action and interaction of s o c i a l groups. Pragmatism r e s t r i c t s the purpose of knowledge to the contribution toward instrumental action and technical control. But i t f a i l s to take seriously the subjective use of knowledge, which i s manifested i n s o c i a l interaction and ideological control, and which ultimately reinforces our value basis. C r i t i c a l theorists go beyond the pragmatist notion that b e l i e f s are only v a l i d when they are translated into some form of observable action. Instead, they a l i g n themselves closer to the Hegelian notion that the generation of a new consciousness transforms the way r e a l i t y i s perceived and at the same time becomes a new r e a l i t y . They challenge the presupposed gap between theory and praxis, often misunderstood as the application of theory. I t i s argued that applying a theory presumes a well-defined gap between theory and praxis which has to be bridged by application. However, every new theory offers new insights about "something," and thus changes the i d e n t i t y of that "something" through our new attitude or expectation or our action toward i t . Correspondingly, "theory application" r e s t r i c t s r e a l i t y to an autonomous entit y which i s independent of our attitude and our theories (Arato & Gebhardt: 1982). - 78 -By acknowledging attitude and consciousness as part of r e a l i t y , together with action and v i s i b l e objects, theory becomes part of r e a l i t y . Theory i s the latent aspect while v i s i b l e action and objects are the manifestation of r e a l i t y . Along t h i s l i n e of thinking, enquiry does not have to be translated d i r e c t l y into instrumental action, plans, p o l i c i e s and programs before we can consider the enquiry useful. There may be a less tangible use in the form of new insights, new theories and problem r e d e f i n i t i o n . This d i a l e c t i c a l learning process, which encourages us to ponder, question and re-evaluate our established perspective, may be an essential but overlooked purpose of s o c i a l enquiry. CONCLUSIONS This chapter has discussed the divergent purposes for which s o c i a l enquiry i s conducted, as well as the methods used. When so c i a l enquiry i s used to generate objective knowledge for the use of technical control, the value basis of that knowledge i s often undermined; the category of knowledge that determines our subjective values i n the f i r s t place i s ignored. I f we abide by the p o s i t i v i s t method, which disregards anything that i s not readily observable through i t s methodical framework, our enquiry w i l l be indifferent to those intangible and latent aspects of knowledge. When the method i s developed at the expense of the user, the knowing, thinking and acting subject has to remain passive. When instrumental action i s taken to be paramount, the c r i t i c a l s e l f - r e f l e c t i v e experience of the subject, whose exercise of th i s c r i t i c a l faculty may bring emancipation, i s seriously depreciated. A l l these inadequacies, as recognised by the c r i t i c a l school, serve to confirm the d e s i r a b i l i t y of adopting the c r i t i c a l approach to s o c i a l enquiry. In addition, as opposed to scientism and positivism, c r i t i c a l theory advocates a f l e x i b l e approach which explores alternative p r i n c i p l e s and methods, and which also investigates the subjective and less tangible aspects of s o c i a l a f f a i r s . In accordance with Kant, the c r i t i c a l school affirms the pervasiveness of subjective cognitive models in our generation of knowledge. From Hegel, i t has inherited the view that systems of abstract thoughts and values, ideologies guide human history. In the t r a d i t i o n of Marx, i t pays scrupulous attention to the c o n f l i c t of interests among s o c i a l groups i n i t s study of s o c i a l processes. In conclusion, the c r i t i c a l approach provides an ideal framework for the study of s o c i a l action, which i s often accompanied by a complex intermingle of subjective values, c o n f l i c t of interests and human consciousness. - 80 -CHAPTER 4 T H E JVEJ^-EVAII IATION C R I T E R I A : T H E C R I T E R I A SELECTED FOR JUDGING EVALUATION STUDIES INTRODUCTION In the previous chapter, we examined numerous theories of the functions and methods of s o c i a l enquiry. In th i s chapter, we s h a l l attempt to id e n t i f y some l i n k s between those theories and evaluation. Our aim i s twofold: we s h a l l disclose some of the implications that those theories have on the purpose and methods of evaluation and we s h a l l determine the evaluation functions. THE OBSOLESCENCE OF PURE FACTUAL KNOWLEDGE Positivism considers that the generation of facts i s necessary and s u f f i c i e n t for enquiries, and that the generation of facts i s best achieved by observation, experimentation and theory formulation. The p o s i t i v i s t i c influence on evaluation i s evident: the primary aim of t r a d i t i o n a l evaluation i s to supply basic and factual knowledge about s o c i a l causes and effects. The recommended method i s the controlled randomized experiment which allows the evaluator to observe and measure s o c i a l occurrences. Not only i s t h i s aim unfeasible i n s o c i a l settings (as i t has been indicated i n chapter two), i t also has undesirable s o c i a l implications (as implied by the c r i t i c a l t h e o r i s t s ) . Past evaluation of soc i a l demonstration programs showed that the s o c i a l environment i s not as simple and manageable as the p o s i t i v i s t s assume. Where s o c i a l and innovative projects are involved, there i s a great deal of uncertainty - 82 -with respect to program objectives, the amount of change desired, and the a v a i l a b i l i t y of measurable c r i t e r i a . Above a l l , we do not have sound knowledge of s o c i a l causal processes. And yet, most programs are designed with certain assumptions about s o c i a l causal processes: Program A -*- Causal Process B •*• Objective C. I t i s only by being able to conclude confidently that A has led to B, B has created C, that we can claim to have generated factual information about s o c i a l causes. However, when both B and C are dubious variables by themselves, attempting to produce factual knowledge about s o c i a l programs i s rather u n r e a l i s t i c . Therefore, i f we view evaluation only as a supplier of factual knowledge about s o c i a l processes to be used for decision-making, we over-simplify s o c i a l a f f a i r s , and may well over-emphasize the role that facts play i n decision-making. To accept the p o s i t i v i s t i c notion that bare facts and bare sensations can be observed presupposes that a stringent research method can generate hard facts for the p o l i c y makers. These facts are considered value-free, and i t i s up to the p o l i t i c i a n s to select facts according to t h e i r personal value orientation. These presumptions disregard the p o s s i b i l i t y that value issues are intermingled with facts, and are not simply superimposed on facts. As Habermas (1971) suggests, every research method and mode of enquiry already expresses an interest or a value preference. The s c i e n t i f i c -empirical method has a technical i n t e r e s t — t h e aim to control and - 83 -instrumentalize action. Critical theory has an emancipatory interest—the goal of liberating the subjects from dogmatism and ideology of the status quo. Therefore, the assumption that pure facts can be created is illusionary. Furthermore, the scientistic mode does not treat the experimental method as only one of many legitimate methods, but considers i t to be the standard research device. As Kaplan (1964) and Myrdal (1969) claim, values exist in a l l kinds of enquiries: the task is to make them explicit and empirical. Lynd (1967) further suggests that when value issues are not revealed, they are blurred in commonly accepted terms and traditional habits. As the critical theorists contend, these dogmatic views and habits become ideology wherein values are uncritically accepted. In addition, when values are submerged, the less priviledged and less powerful social groups often have to conform to the beliefs of the more privileged ones. Therefore, i f evaluation does not debate the conflicting interests of its stakeholders, those who affect decisions (the politicians and bureaucrats) will have vast opportunities to dominate those who are affected (the program recipients and the community). The program subjects are commonly treated as passive recipients whose faculty to judge and decide is discouraged. In such a way, programs are evaluated from an established point of view, reflecting the interests of the status quo, thus reinforcing a form of social inequality. - 84 -AS A FUNCTION OF DISCLOSING THEORETICAL ASSUMPTIONS/COGNITIVE MODELS The generation of bare facts i s refuted by Kant's notion of the "constitution of knowledge and facts," which accentuates the subjective influences affecting the production of knowledge and facts. Kant assigns the subjective factors as the possible cognitive forms (Kant: 1781). Therefore, a Kantian stance holds that the design of our projects and plans i s dictated by a cognitive model out of other possible models. And evaluation may serve the function of investigating the theoretical models of our programs and plans. Since c r i t i c a l reason i t s e l f i s the ultimate purpose of enquiry, and since unconditional doubt should be applied to a l l investigations, then the presuppositions of every enquiry have to be assessed c r i t i c a l l y . Accordingly, every evaluation approach, including the pos i t i v e method and the experimental procedure, has to be scrutinized i n the l i g h t of other evaluation models. This continuous scrutiny of the evaluation i t s e l f is. taken to be necessary. Apart from Hegel's c r i t i c i s m of the vicious c i r c l e that Kant's unconditional doubt i s subjected to, there i s another problematic issue. When reasoning i t s e l f i s taken to be the ultimate purpose of attaining knowledge, the diverse purposes for which the subjects use enquiries may be overlooked. I f i t i s assumed that every one who evaluates or who uses evaluation does so for a r a t i o n a l purpose, that - 85 -i s , to exercise c r i t i c a l reason, then the i r r a t i o n a l elements operating i n the conduct of enquiry are dismissed. However, examining the s o c i a l p o l i c y scene indicates that policy-making and resource a l l o c a t i o n are not e n t i r e l y based upon factual and s c i e n t i f i c information. Why evaluation i s carried out and whether or not i t s results are adopted are not necessarily determined by the exercise of c r i t i c a l reason. Rather, c o n f l i c t s of personal interest often play an important part (Cronbach: 1980). Nevertheless, Kant's over-reliance on the r a t i o n a l model does not depreciate his merit i n underscoring the significance of subjective cognitive models and causal assumptions affecting our course of enquiry. AS A MEANS OF PROMOTING CRITICAL SELF-REFLECTION Hegel envisioned a more subjective j u s t i f i c a t i o n for c r i t i c a l reason, namely that the ultimate purpose of c r i t i q u e i s to allow the subjects to experience the r e f l e c t i v e process of c r i t i c a l reason so that they may at t a i n c r i t i c a l consciousness. His idealism i s marked by his v i s i o n that the evolution of ideas guides human history. I t i s held that the self-generation of the human species i s brought about by co n f l i c t s of ideas, whereby the formulation of new ideas overcome old ones. This self-generation i s possible when the subjects recognize that di f f e r e n t ideas and value systems mode human history, and thus devise new ones for further progress (Haberraas: 1971). - 86 -I f promoting c r i t i c a l s e l f - r e f l e c t i o n i s taken to be an evaluation function, then i t has to c l a r i f y the abstract ideas and values which subconsciously guide our plans and programs. I t should encourage us to investigate c o n f l i c t i n g systems of thought and diverse s o c i o - p o l i t i c a l world views, and to r e f l e c t upon alternatives for further progress. We do not merely assess the internal relationships of a program model, but scrutinize the model as i t represents a way of thinking and an attitude that are pervasive to a people at a certain period i n history. More s i g n i f i c a n t , these abstract ideas can be transformed through c r i t i c a l s e l f - r e f l e c t i o n . Evaluation then serves the purpose of promoting the exercise of c r i t i c a l s e l f - r e f l e c t i o n toward our programs and p o l i c i e s . This r e f l e c t i v e experience, as practised by those who participate i n the enquiry, appears to be the ultimate goal. Under the Hegelian t r a d i t i o n , evaluation becomes a change agent, although the changes are confined at the l e v e l of the mind: that i s , changes i n attitude, i n values and consciousness. I t i s assumed that changes i n ideas are s u f f f i c i e n t for bringing about changes i n human his t o r y . However, i f we presuppose that j u s t changing our ideas of the world would change history, then we might dismiss the s o c i a l and material aspects of history. And i f we only assess the abstract ideas embedded i n our action, we might disregard the s o c i a l and material impacts of our programs. Hegel's thesis i s instructive i n the sense that the inner states or the consciousness of the subjects are revived, and the significance of abstract ideas defended; however, rel y i n g on the c r i t e r i a of the i d e a l i s t t r a d i t i o n too stringently may d i s t r a c t us from - 8 7 -the s o c i a l and material consequences of our plans and programs. AS AN INSTRUMENT OF IDENTIFYING SOCIAL IMPACTS The pragmatists tackle the i d e a l i s t problem by attempting to apply knowledge i n action settings. To them, the purpose of enquiry i s to u t i l i z e knowledge for the benefit of instrumental action and technical control. Therefore, only b e l i e f and knowledge that can be understood i n the context of action are meaningful to them. This pragmatist notion concedes with the t r a d i t i o n a l mandate of evaluation, which aimed to bridge the gap between knowledge and s o c i a l action. I t i s commonly accepted i n the f i e l d that we have to understand the effects of our programs i n the hope that we can exert technical control over future actions. The interest i n prediction i s shared by both pragmatism and t r a d i t i o n a l evaluation. In order to predict, one has to know the causes and effects. And thi s mandate, as discussed i n the previous section, i s rather d i f f i c u l t to f u l f i l . While the pragmatists consideration of concrete action settings i s well taken, t h e i r obsession with the technical and instrumental aspects of action may not be s u f f i c i e n t . The pragmatic approach of Marx which stresses the importance of using s o c i a l and material conditions as points of departure i n studying human history may be more tenable. According to him, s o c i a l labour and class c o n f l i c t are the action settings on which enquiries should be - 88 -based. This Marxian stance s i g n i f i e s the necessity of paying attention to the c o n f l i c t i n g interests of different s o c i a l groups, which are reflec t e d i n t h e i r varying economic and s o c i a l conditions. From Marx's analysis, we can in f e r that c o n f l i c t of interests exists i n public programs and p o l i c i e s i n the sense that there are dif f e r e n t stakeholders, each having i t s own interests and i t s own de f i n i t i o n of need. Social p o l i c i e s and programs which presumably affe c t the d i s t r i b u t i o n of material resources and s o c i a l services w i l l inevitably create s o c i a l and material impacts on different s o c i a l groups. Then, c r i t i q u e serves the function of c l a r i f y i n g the d i s t r i b u t i o n of s o c i a l services as they are received by the program participants, simultaneously, disclosing t h e i r c o n f l i c t i n g interests and needs. AS A PROCESS OF ATTAINING EMANCIPATION FROM DOGMATISM AND IDEOLOGY To the c r i t i c a l theorists, the desire of humanity to achieve emancipation from dogmatic and ideological t r a d i t i o n i s a fundamental working assumption. This value disposition i s consistently held and expressed by them. Although a related concept had been implied i n Hegel's c r i t i c a l s e l f - r e f l e c t i v e approach, and was advocated i n the Marxian notion of consciousness r a i s i n g , the c r i t i c a l theorists are o r i g i n a l i n pronouncing t h i s form of emancipation a human need and a human ri g h t . - 89 -From a c r i t i c a l point of view, public enquiries should not merely serve p o l i t i c i a n s or planners i n devising new plans and p o l i c i e s , which only benefit instrumental action for technical ends, but should encourage the exercise of c r i t i c a l r e f l e c t i o n so that the community i s liberated from dogmatic thinking and t r a d i t i o n a l habits. This r e f l e c t i v e process i s realized at the individual l e v e l when the subject attains consciousness through c r i t i c a l thinking. Furthermore, when we study the patterns of s o c i a l interaction within a community, we may discover whether the nature of i t s s o c i a l r e l a t i o n s are based upon emancipation or domination. Through s o c i a l interaction, s o c i a l groups express norms and interpersonal habits when they relate to each other. These norms and habits are the manifestations of s o c i a l relations that are either based on self-determination and equality, or are based on exploitation and domination. When the nature of s o c i a l relations i s characterised by emancipation and self-determination, the subjects are encouraged to practice c r i t i c a l thinking and to c u l t i v a t e a r e f l e c t i v e attitude. By exercising c r i t i c a l r e f l e c t i o n , they are then i n a position to assess the ideological forms and dogmatic norms which dominate and constrain t h e i r values, thinking and l i f e s t y l e s . Hence, they may a t t a i n emancipation when they use c r i t i c a l consciousness to assess and to change those ideological traditions (Habermas: 1971). A primary concern of the c r i t i c a l theorists i s to change ideological patterns and norms of domination. However, i n order for the - 90 -subjects to change these patterns and norms, they have to be aware of them. This c r i t i c a l consciousness can only be realised at a subjective l e v e l when the subjects assess the patterns of s o c i a l interaction and study the ideological forms. Both studies c a l l for a subjective method of enquiry. The c r i t i c a l approach advocates narrowing the gap between the r u l e r and the ruled, the planners and the community, as well as the evaluator and the program recipients. Therefore, i t i s not j u s t the evaluation findings that are paramount, but the experience of c r i t i c a l examination and learning being shared by the evaluator and the community. I t i s through the evaluation process that the faculty to exercise c r i t i c a l thought upon our programs and p o l i c i e s may be continuously reinforced. Surfacing and Revising Value Assumptions The c r i t i c a l theorists think that i n order for us to be free from dogmatism, we have to recognise our subjective values before we can change our frame of reference. Since subjective values are the foundation of our knowledge and facts, i t i s necessary to make our value choice e x p l i c i t i f we want to be more objective about an issue. Value assumptions dictate our program objectives, problem d e f i n i t i o n and program strategies. We w i l l be i n a better position to c r i t i c a l l y assess our program i f we can step beyond our immediate subjective frames of reference. Therefore, revealing the value assumptions of programs and p o l i c i e s i s a v i t a l function of evaluation. - 91 -With respect to revisin g value assumptions, Hegel's thesis offers valuable insights for a theory of s o c i a l change. That i s , value change may be f a c i l i t a t e d by studying the history of abstract ideas and value systems. Being exposed to the h i s t o r i c a l values moding human history, we become aware of h i s t o r i c a l changes. Accordingly by juxtaposing our current value position i n the context of these h i s t o r i c a l value systems, we may become conscious of the transformation of ideas through time, and also, may be more prepared to anticipate and formulate changes for the future. Attending to the Perceived Needs of the Subjects According to c r i t i c a l theory, the subjects who are involved i n the enquiry have to be given the opportunity to express t h e i r perception of needs. Since there are diverse value choices, broadening our knowledge of other d e f i n i t i o n s of a problem allows us to step outside of our immediate frames of reference. Concurrently, the subjects may become more conscious of thei r value orientation and subjective references. This type of investigation i s primarily a subjective study. The c r i t i c a l theorists believe that the s c i e n t i f i c experimental method only reveals one empirical r e a l i t y among others, and that sometimes more subjective and interpretive methods are more rewarding i n revealing the value preferences of different s o c i a l groups. Encouraging Shared Dec is ion-making The p o s i t i v i s t i c mode of investigation implies that the subjects have to remain as passive appendages of the established status quo, that - 92 -t h e i r decision-making faculty i s often stripped from them. But the c r i t i c a l approach maintains that the thinking, acting and knowing subjects have to be encouraged to participate i n evaluation and decision-making. I t i s held that there i s always a c o n f l i c t of interests i n society whereby the more privileged groups have more opportunities to dominate over the less advantaged ones. Their ways of thinking and l i v i n g are often imposed upon those who do not possess wealth and decision-making authority. I t i s by attending to the views of the subjects and allowing them to determine courses of actions that this form of domination may be minimised. There are s o c i a l benefits i n permitting the community and program recipients to evaluate programs and plans, to suggest pol i c y directions and to take part i n project planning. I f programs are imposed upon the community without f i r s t consulting them and understanding their needs, the community w i l l be affected by decisions and plans which may be irrelevant to th e i r f e l t needs. When the recipients participate i n decision-making, the i r a b i l i t y to exercise c r i t i c a l assessment may also be improved. CONCLUSIONS Despite our presentation of empirical facts and s t a t i s t i c a l graphs, our programs and p o l i c i e s , and th e i r evaluation, are loaded with value assumptions with respect to perceived s o c i a l need and goals. The d i f f i c u l t y of generating accurate facts about s o c i a l causes and events - 93 -i s another signal t e l l i n g us not to r e l y heavily on facts when we evaluate our plans and actions. Correspondingly, the purposes for which we use evaluation are primarily determined by our value disposition. C r i t i c a l theory presents a very relevant and enlightening perspective upon which the selection of our functions i s based: i t elucidates i t s value assumption behind the exercise of c r i t i c a l thinking and c r i t i c a l s t u d i e s — t h e need of the subjects to a t t a i n emancipation from dogmatic views and habits. Expanding from this fundamental assumption, t h i s chapter has i d e n t i f i e d numerous functions of evaluation: to disclose our theoretical assumptions i n programs and plans; to bring to l i g h t the value assumptions, and t h e i r r e l a t i o n to the history of ideas; to i d e n t i f y the s o c i a l impacts of our programs; to attend to the perceived needs of those who are involved and affected by the program; and to encourge c o l l e c t i v e decision-making through which the faculty to exercise c r i t i c a l s e l f - r e f l e c t i o n may be nurtured. Above a l l , i t i s desirable to accept diverse aims and methods i n our s o c i a l enquiries, so that more subjective methods, methods outside of the t r a d i t i o n a l stringent research, may be considered and explored. - 94 -CHAPTER 5 THE CASE STUDY: THE EVALUATION STUDY OF THE DEMONSTRATION PROGRAM CN COMMUNITY ENERGY CONSERVATION - 95 -INTRODUCTION This chapter has three objectives. I t w i l l b r i e f l y describe the program that has been evaluated. I t w i l l i d e n t i f y the objectives, methods, process and results of i t s evaluation. F i n a l l y , based upon the c r i t e r i a developed i n the previous chapter, i t w i l l consider what functions evaluation has served and may serve i n the community energy program. THE COMMUNITY ENERGY PROGRAM The Context Research related to energy consumption and conservation has shown that general promotion programs for encouraging energy conservation are no longer eff e c t i v e for inducing conserving behaviour; however, programs that offer personalized service, active community involvement and education may be more promising (Anderson & Cullen: 1979). As well the American experience with energy conservation has indicated that many community-based programs have been successful i n reducing energy usage. The c i t i e s of Portland, Oregon; Davis, C a l i f o r n i a ; and Seattle, Washington are among the numerous examples. A key element of these programs i s active community p a r t i c i p a t i o n . A c r i t i c a l question, pondered by interested public o f f i c i a l s and concerned individuals, i s whether the success of these programs i s mainly due to active community involvement. In any event, evidence from both the research and applied - 96 -f i e l d on conservation has provided an incentive for the t r i a l of community based conservation programs i n Canada. The Case Study A Federal p i l o t demonstration project on community energy conservation was implemented i n 1979 i n three Canadian c i t i e s : Vernon, B r i t i s h Columbia; Richmond H i l l , Ontario; and Fredericton, New Brunswick. The project was i n i t i a l l y funded by the federal m i n i s t r i e s of Energy, Mines and Resources, Consumer and Corporate A f f a i r s , and Supply and Services. The central purpose of th i s program was "to develop energy conservation practices at the community l e v e l through c i t i z e n p a r t i c i p a t i o n and i n i t i a t i v e s " (D'Amore: 1981). I t s s p e c i f i c objectives were to achieve a higher l e v e l of awareness and knowledge of the energy sit u a t i o n and conservation practices; to promote conserving attitudes and behaviour; to accomplish actual energy savings; and to find out whether active c i t i z e n involvement i s an effec t i v e means of achieving conservation so that t h i s program could be used by other Canadian municipalities as a model for designing conservation p o l i c i e s and proj e c t s . 1 The program strategy adopted was community animation—that i s , Based on personal inteviews conducted with the program designer, Louis D'Amore; the program coordinator, Garth Macguire; and the evaluator, Stan Shapiro, between 1982-84. - 97 -developing connection and "networks" with community leaders, key organizations and i n s t i t u t i o n s i n the hope of gaining the i r support i n planning and i n p a r t i c i p a t i n g in conservation projects. This was the r e s p o n s i b i l i t y of a program coordinator appointed i n each of the three c i t i e s . Their jobs were to heighten energy concerns and to disseminate relevant information on energy saving a c t i v i t i e s and techniques to various s o c i a l groups. Under th i s project, individuals, groups and municipal resources were mobilized to bring about conservation. In September 1979, a series of s p e c i f i c energy projects was started i n the business and i n d u s t r i a l sector, the public schools, the municipal government, the media and so forth. A central premise of t h i s program i s that individuals, groups and i n s t i t u t i o n s are more l i k e l y to adopt conserving behaviour when the entire community shares that e f f o r t , when they are given the opportunity to plan the goals and devise the measures, and when a l l segments of the community support those goals and measures (D'Amore: 1981). Hence the c i t i z e n involvement approach was taken to be a desirable way of encouraging conserving behaviour. In th i s program, p a r t i c i p a t i o n was not an end i n i t s e l f , rather, i t was a means by which conservation may be successfully achieved. THE PROGRAM EVALUATION The program was evaluated by someone who also held a major r o l e i n designing the program and who came up with the o r i g i n a l idea for the - 98 -project. Being a part of the organization responsible for designing and managing the program, he i s considered an "inside evaluator." As indicated by him, two main evaluation objectives were: 1. To determine program effects, i n p a r t i c u l a r , the amount of energy awareness, conservation behaviour and actual energy savings that resulted from the program. In other words, "how successful was the program i n achieving i t s objectives?" 2. To find out whether "community involvement" i s an ef f e c t i v e means of promoting conservation practices. In other words, the evaluation was to id e n t i f y the causal relationship, i f any existed, between community p a r t i c i p a t i o n and energy conservation. 2 The Method The Pre-test Post-test Controlled Experiment i l l u s t r a t e d i n Fi g . 4 was the method employed. For each of the three m u n i c i p a l i t i e s , a control community which did not experience the project and which was believed to resemble the test community was used for comparison. Penticton was selected as the control community f o r Vernon, New Market for Richmond H i l l , and Moncton for Fredericton. Based on an interview conducted i n May 1984, with the evaluator, Stan Shapiro. - 99 -Data on the s i x municipalities were collected before and after the program had been launched. The program would be considered successful i f the difference i n energy usage between the pre-test and post-test results of the demonstration c i t y were more than the difference of the pre-test and post-test results of the control community. This method i s an outcome evaluation measuring what happens after a program has been operated. Two types of variables were to be measured. DEMONSTRATION COMMUNITY CONTROL COMMUNITY Pre-Test (Sept. 1979) i Attitude & Behaviour Survey Actual Energy Saving ( Attitude & Behaviour Survey Actual Energy Saving Post-Test (Sept. 1980) I I I I I I I I I I f (B - A) i s more than (D - C), then the program i s considered successful. F i g . 4 Pre-Test Post-Test Control Group Experimental Design of the Community Energy Program. 1. Aggregate energy consumption data for the c i t y were collected to measure the actual energy saved. The energy data were c l a s s i f i e d by t y p e — e l e c t r i c i t y , natural gas, home heating o i l and gasoline. - 100 -2. Conservation attitudes, knowledge and reported conserving behaviour were measured by a telephone survey of 250 households i n the demonstration and control communities. According to the i n i t i a l evaluation plan, a process evaluation was to be conducted as w e l l . The community groups who participated i n the program were encouraged to provide the i r subjective evaluation of the project through personal interviews (D'Amore: 1980). The Evaluation Results Outcome from the household survey. Results from the household survey recording changes i n attitudes and behaviour show that, o v e r a l l , the three test communities did not produce any signficant changes when the i r results are compared to the control c i t i e s . The basic c r i t e r i o n upon which the evaluator judged whether a change was s i g n i f i c a n t or not was based on the methodology of the pre-test and post-test experimental design mentioned e a r l i e r . That i s , i f the difference between the f i r s t wave score (1979 survey) and the second wave score (1980 survey) i n the test c i t y i s greater than the difference i n the other c i t y , then the net difference indicates that the test community has done better than the other. Two tests were employed for interpreting the res u l t s , the index test and the t-test (refer to Appendix I for s t a t i s t i c a l d e t a i l s ) . For the index test, i t was decided that the difference i n score between the two c i t i e s had to be greater than an index of 2 i n order for the change to be c l a s s i f i e d as s i g n i f i c a n t . In other words, a register - 101 -of +4 in one city but +2 in another was interpreted as a comparable change; while +6 in one but +3 in the other was classified as a significant change (Shapiro: 1980). However, upon what rationale this "greater than two rule" was established is obscure. This rule may alter the overall result or conclusion of the survey. Altogether 45 questions were presented. According to the index test, there were 14 cases of comparable change in both Vernon and Richmond H i l l , while Fredericton had 16. For significant change, Vernon yielded 18, Richmond H i l l 22 and Fredericton 12 (Refer to Appendix I for statistical details). A t-test, which is commonly considered a more reliable statistical testing, was executed to double-check the index test. However, the overall results or changes were found out to be even less significant. That may mean the "greater-than-two rule" was too low to be used as a standard of measuring the difference. The t-test almost doubled the number of insignificant changes. Out of the 45 questions, there were 30 comparable changes in both Vernon and Richmond H i l l , and Fredericton had 33. Both Vernon and Fredericton produced 6 significant changes, while Richmond H i l l had 12. These data lead us to conclude that at least in the short run, community animation had not produced any measurable and favorable effects in increasing energy awareness or inducing conserving behaviour. However, we have to treat these data and evaluation with - 102 -caution as there are major inadequacies embedded i n the evaluation. Outcome from the community a c t i v i t i e s . Apart from the r e s u l t s of the pre-test and post-test experiment, there was an additional source of project outcome assessment. This was the compilation of a l i s t of conservation a c t i v i t i e s and measures that materialized i n the test municipalities after the program had been launched (refer to Appendix II for d e t a i l s ) . This information can give us a general idea of the o v e r a l l impact that was created i n each community. Although the information does not contain any rigorous s t a t i s t i c a l testing, the a c t i v i t i e s were largely either d i r e c t l y or i n d i r e c t l y a r e s u l t of the animator's e f f o r t i n trying to gain support from community groups and i n s t i t u t i o n s . An impression gained from t h i s l i s t i s that the animation process has created noticeable impact i n terms of guiding and encouraging the adoption of energy conservation plans, p o l i c i e s and technology among various community organizations and l o c a l i n s t i t u t i o n s within the test communities. Process evaluation. Although the i n i t i a l evaluation plan related to process was abandoned, the animation process was b r i e f l y analysed and assessed. The three cximmunities had different animation t a c t i c s , t a i l o r e d to the s p e c i f i c characteristics of each community. The Steering Committee was the vehicle through which concerned c i t i z e n s and organizations could take part i n formulating goals and planning projects for conservation. - 103 -Each community applied a d i s t i n c t animation process, as i l l u s t r a t e d i n Figure 5 on the next page. In Richmond, the process can be described as a " s o c i a l marketing" approach, i n which the animator attempted to establish contacts with the general public through the u t i l i z a t i o n of various forms of caiimunications media and workshops. However, the process did not receive much support from the l o c a l i n s t i t u t i o n s and i t s progress was hindered by the Lack of commitment shown by the Town Council. Vernon's program was marked by "community development out-reach," i n which the animator expended a great deal of e f f o r t i n contacting as many community groups as possible. Since City Council provided only mediocre support, the Steering Committee and the Task Groups l o s t momentum gradually. Vernon had the largest number of community organizations involved d i r e c t l y i n conservation projects. The model undertaken by Fredericton can be call e d "community development-institutional," because i t received immediate and f u l l support from the City. Strong support from the Province, the University and the School D i s t r i c t followed. The Steering and Working Committee gained momentum, and the network of support expanded to c i t i z e n groups and the general public (D'Amore: 1981). - 104 -F I G . 5 V A R I A T I O N S ON T H E C E N T R A L T H R U S T OF P R O C E S S ( S o u r c e : D ' A m o r e : 1 9 8 1 : 49 ) Public / / R i c h m o n d H i l l V E R N O N F R E D E R I C T O N - 105 -A CRITICAL EXAMINATION OF THE CASE STUDY Based upon the evaluation objectives of and methods employed by th i s program, i t appears that great emphasis was placed upon presenting facts about the program effects. The program was concerned with " s o c i a l experimentation," through which the community energy program model, as i l l u s t r a t e d below, i s to be tested i n s o c i a l settings. Providing Incentives F i g . 6 Environmental Supports to Encourage Energy Conserving Behaviour. Source: D'Amore, Louis, 1980, Urban Forum, V. 4, No. 6. The evaluation of th i s program was obstructed by complications and unforeseen circumstances, and consequently, did not f u l f i l many of the terms stated i n i t s i n i t i a l plan. I n s u f f i c i e n t and inappropriate energy data, funding shortages, c o n f l i c t s of interest and an overly ambitious program model marked the sources of the complications. - 106 -U t i l i z i n g the functions enumerated i n the previous chapter, we s h a l l consider the functions that evaluation has served or might serve i n t h i s community energy program. 1. Disclosing Theoretical Assumptions The Kanatian notion of the "subjective conditions of knowledge" reminds us that the design of our plans, programs, and research are guided by theoretical assumptions about causal processes. Therefore, the effectiveness of plans and programs i s p a r t l y dependent upon the v a l i d i t y of those theoretical models. When we evaluate our programs, i t i s necessary to c l a r i f y those underlying cognitive models and, i f possible, to examine t h e i r v a l i d i t y . As Weiss (1972) points out, i f the program f a i l s , and i f we do not know whether the theory i s v a l i d or not, we w i l l not be able to distinguish between theory and program f a i l u r e . This community energy program was designed with several theoretical assumptions. However, as far as we can determine from the evaluation plan and documents, t h i s evaluation neither c r i t i c a l l y examined nor formally acknowledged the significance of the program's th e o r e t i c a l assumptions. Consequently, i t did not d i f f e r e n t i a t e between theory and program f a i l u r e . The evaluation results have shown that the program had l i t t l e measurable effect. The program had two theoretical assumptions which necessitate c r i t i c a l enquiry. One theorises that i f people are aware and knowledgeable of energy conservation p r i n c i p l e s , issues and measures, - 107 -they are more l i k e l y to adopt conserving behaviour. However, t h i s presumption was not tested by the evaluation, nor was i t supported by any empirical studies. Therefore, the success of the program depended upon a causal assumption, awareness + behaviour, which i s untested and uncertain. Then, without knowledge of the causal process, and without any means by which the causal assumption i s to be tested by the evaluation, i f the program f a i l s , i t i s very d i f f i c u l t to conclude whether the program i t s e l f was not w e l l implemented, or whether the theory was not v a l i d . The other untested theory i s the connection between energy conserving behaviour and actual energy savings. This program model presupposes that a l l energy conservation measures w i l l lead to actual energy reduction. However, t h i s causal relationship i s again not necessarily v a l i d when uncontrolled intervening factors e x i s t . When people adopt a conservation technology, the technology i t s e l f may be i n e f f i c i e n t , that i s , i t may not produce net energy savings. Or, people may not apply the measure i n a proper manner for i t to e l i c i t energy reduction. Therefore, even i f people adopt more conservation measures, there may not be a net energy gain. Net energy analysis, which attempts to measure the amount of energy consumed by a technology or an a c t i v i t y , i s an extremely complicated and problematic task ( G i l l i l a n d : 1978). I t should be noted that this program only f a c i l i t a t e d conservation through community animation and education. However, we have to evaluate the results i n energy savings and behaviour, which were - 108 -not direct program outputs. An i l l u s t r a t i o n i n Figure 7 c l a r i f i e s t his point. Between program and effects, certain intervening factors that may interrupt the causal process were not accounted for. Consequently, the conclusions drawn pertaining to the causal relationship between program and i t s effects are neither r e l i a b l e nor purely f a c t u a l . I f the significance of these cognitive models were acknowledged by the evaluation, i t might have been more attentive to the subjective models governing our plans and programs. 2 . Revealing and Revising Value Assumptions and Relating Them to a History of Ideas Value assumptions, manifested as program goals and objectives, establish our d e f i n i t i o n of a problem and decide what i s worthwhile for us to achieve i n our programs. A Hegelian stance affirms that the formulation of objectives i s subconsciously guided by the abstract ideas we hold. These abstract ideas represent a way of thinking and an attitude that are pervasive to a people at a certain h i s t o r i c a l moment. Therefore, besides s c r u t i n i z i n g the theoretical models, evaluation also has to expose these abstract ideas and relate them to a h i s t o r y of ideas. The point of viewing them i n a h i s t o r i c a l perspective i s that these attitudes and ideologies can be transformed through time, and so can our value assumptions. PROGRAM DESIGN & OBJECTIVES V Energy Program Implanentation Animation & Information MEASUREMENT CONTROL FAILURE ( F a i l e d t o c o n t r o l f o r i n t e r v e n i n g f a c t o r s l i k e i n e f f i c i e n t energy technology, economic growth and r e c e s s i o n , and na t i o n a l energy p o l i c i e s ) I Awareness & Knowledge of Conservation Community Groups Adopt Energy Conservation Measures I Energy Savings PROGRAM FAILURE (Program was not wel 1 implemented to c a r r y out the o b j e c t i v e s of in c r e a s i n g energy awareness and knowledge) \ / THEORY FAILURE (Energy awareness does not n e c e s s a r i l y lead to conserving behaviour; People d i d not apply the conservation technology c o r r e c t l y ) I NET ENERGY SAVINGS o OUTCOME EVALUATION F i g . 7 Thev Theoretical Linkages in the Community Energy Program Model. - 110 -By surveying the attitudes of the community, th i s evaluation has, to a certain extent, exposed the value orientation of the community. By attempting to change prevalent values, that i s to increase t h e i r conservation awareness, th i s program i m p l i c i t l y admitted that there was a value difference between program objectives and the community. Also, a value change toward increased conservation i s needed. The evaluation dealt with t h i s value issue by aiming to measure the change that took place. This program recognized a value problem embedded i n the conservation controversy when i t aimed to tackle a few value-related variables such as attitude change and community perception. Therefore, value issues were recognised. However, c o n f l i c t i n g values were not c r i t i a l l y examined and revised. I t appears the evaluation did not take the program objectives as objects of scrutiny; thus, i t regarded the program goals as given and determined the success of the program according to them. The process evaluation as outlined i n the o r i g i n a l plan was not conducted. This was due mainly to inadequate funding and a c o n f l i c t of interests. The program sponsors were primarily interested i n finding out about the program outcome i n terms of energy and d o l l a r savings. The program designer also considered i t worthwhile to assess the animation process as i t was applied i n each community and to include the subjective evaluation as expressed by the p a r t i c i p a t i n g individuals and groups. F i n a l l y , the subjective evaluation was abandoned and a b r i e f assessment of the animation process was done. - 1 1 1 -The c o n f l i c t of interest among different stakeholders i n t h i s case study occurred not only i n the evaluation objectives, but also existed i n the program objectives. The sponsors perceived the purpose of the program as a means of reducing energy consumption and developing a conservation model to which other communities could refer. However, the program designer was also interested i n exploring how c i t i z e n s respond to the program when they were given the opportunity to formulate goals and to plan a c t i v i t i e s through a community involvement process. The c o n f l i c t may be traced to two opposing viewpoints. One party wanted to obtain short-term and tangible results i n energy savings for the use of technical control. The sponsors were interested i n concrete program outcome, which often serves the purpose of accountability. The other party, the designer and the evaluator, was more interested i n long-terra developments of energy programs and the lessons learned from the animation process. This may contribute to the management of program and the production of knowledge. The evaluation dealt with the c o n f l i c t i n g objectives by incorporating both, and as a r e s u l t , demanded a large amount of resources and funding i n order for the tasks to be accomplished. Our review i n chapter two has suggested that i n the p r a c t i c a l world of policy-making and program implementation, the amount of funding and the l e v e l of knowledge required by grand and ambitious projects are often not r e a l i s t i c . In t h i s case study, both of these constraints were relevant. I t appears that the c o n f l i c t of interest, which was - 112 -manifested i n i t i a l l y i n the program objectives and subsequently i n the evaluation objectives, had to be resolved before the program and i t s evaluation could be executed with a reasonable degree of s a t i s f a c t i o n . This i s the area where evaluation may c o n t r i b u t e — i n assessing the c o n f l i c t of interests, the program objectives and i t s resource requirements i n the hope of devising a program that i s implementable and an evaluation that i s conductable. Under t h i s condition, evaluation has to supply inputs to the selection of program objectives. I t should be considered a s i g n i f i c a n t part of the entire program i n the beginning, rather than be treated merely as a follow-up procedure or an ad-hoc a c t i v i t y . However, i n th i s case, the program seemed to have dictated the evaluation rather than vice versa. The program objectives to a large extent pre-determined what functions and objectives the evaluation should serve. The fact that the evaluator was not from an outside organization independent of program design and management may have discouraged evaluation from performing a cross-checking function and a c r i t i c a l assessment. As chapter three and four have suggested, i t i s easier to step outside of our immediate frame of reference when we review other perspectives and other subjective evaluation toward the problem. When the evaluator i s deeply involved with program design and management, i t may be d i f f i c u l t for him to c r i t i c i z e the program objectively. When the evaluation i s content with the frame of reference - 113 -defined by the program, i t s potential to c r i t i c i z e the program's ' assumption i s reduced. There i s another value issue, the c r i t e r i a of program success, which deserves our attention. How much change or how much increase i n conservation should be considered desirable? How much difference i n attitude, behaviour and energy reduction between the test and control c i t y should constitute success? Ultimately, who sets t h i s standard? There seemed to be no agreement, pr i o r to the program, as to how much change should be achieved. This problem i s not unusual i n demonstration program, which have few previous lessons from which to learn i n order to set operational objectives and d e f i n i t i v e targets. Decisions are often a r b i t r a r i l y made according to the discretion of the evaluator. Due to the experimental nature of innovative programs, project designers are frequently confused about what to accomplish and anticipate. Therefore, the issue of how much change should be achieved, and ultimately, the standard of program success, i s not so much a factual issue but more a value decision. Whose standard of success i s being used? Given that the amount of change desired has been decided, the use of more standardized methods and s t a t i s t i c a l testing may be useful for indicating i n a summarized and quantitative form the chances of error occurring i n our observation of change. Moreover, s t a t i s t i c a l testing and graphs cannot replace the value-based standard of the degree of s o c i a l change desired. - 114 -3. Identifying the Social Impacts This evaluation shares the pragmatist concern with applying knowledge i n instrumental action and exerting technical control. This orientation i s apparent i n i t s two objectives. I t was preoccupied with discovering the effects i n action settings after the program had been implemented, rather than s c r u t i n i z i n g the i m p l i c i t thoughts and values governing the program design. The former pertains to the pragmatist framework, the l a t t e r , to the i d e a l i s t canp. By studying program effects, i t was hoped that this knowledge would aid the planning and development of future community energy programs. Overall, t h i s evaluation placed more weight on the generation of factual knowledge than on the investigation into subjective value issues. This i s obvious from i t s objective: to discover whether the community involvement element i s an effective means of achieving conservation. In order to f u l f i l t h i s aim, "community involvement" has to be treated as a d i s t i n c t program input that could be isolated and observed. I t was assumed that t h i s variable could be factored out and observed. How far has the evaluation achieved i t s pragmatist functions? Is o l a t i n g the "community involvement" variable i s far from being simple and tenable. What kind of a c t i v i t i e s should be considered involvement—attending events, adopting conservation measures, planning and managing the program? Who are to be included as the community? And how are these variables measured? I t was decided i n the end that i t was - 115 -v i r t u a l l y impossible to factor out and operationalize the community involvement element i n the program. On the other hand, the task of revealing the program effects was not completed, either. This evaluation was over-optimistic with respect to the types of energy data that i t could c o l l e c t ; t h i s may also be a sign of i t s over-reliance on generating factual information and i t s neglect of the d i f f i c u l t i e s involved. Aggregate energy consumption data were not rea d i l y available in the desired form required by the controlled experiment. Data on gasoline and home heating o i l were not r e l i a b l e because the gasoline stations and o i l suppliers kept t h e i r sales record i n a format that did not serve the purpose of the evaluation. In the case of Vernon, another drawback was that B.C. Union Gas f a i l e d to supply information on natural gas. Without knowing the consumption of natural gas, program s t a f f could not get accurate account of consumption s h i f t . The decline i n home heating o i l may very w e l l be due to the fact that households shifted to natural gas usage. Ever since the o i l embargo of 1973 and subsequent o i l price escalation, there has been great impetus for consumers to replace heating o i l with other, f u e l types. With regard to e l e c t r i c i t y , the data obtained from B.C. Hydro were stored on the basis of a one year cycle, which did not coincide with the program's cycle s t a r t i n g i n September. On the other hand, acquiring timely energy consumption data on individual households would have been most suitable, but was considered unfeasible. - 116 -There i s l i t t l e doubt that the accuracy of the assessment on actual energy savings was not satisfactory. This part of the outcome evaluation did not produce any r e l i a b l e facts about program effects. At t h i s point, two questions come to mind. Under what conditions w i l l our energy data be more securely available? To what extent can and should our evaluation depend on such hard data to generate factual information on program effects? There are two options available for dealing with the f i r s t question. One involves narrowing the program scale so that only a s p e c i f i c area of concern has to be monitored, such as aiming at a s p e c i f i c sector and s o c i a l group. The other option, which i s less p r a c t i c a l , i s to make the required data available at immense cost i n term of funding and resources. The second question, which i s more s i g n i f i c a n t since i t enquires about the f e a s i b i l i t y and worth of c o l l e c t i n g t h i s type of hard data, w i l l be discussed i n greater d e t a i l i n subsequent sections. Furthermore, the complicated program model discouraged generalization about program and effects. As i s evident from our e a r l i e r discussion, the linkages between attitude and behaviour, behaviour and actual energy savings, were not soundly established. Due to the presence of these untested and i n v a l i d linkages, i n f e r r i n g a causal relationship between program and effects i s very dubious. One questions the f e a s i b i l i t y of predicting s o c i a l events and exerting technical control i n s o c i a l action settings, where human knowledge, behaviour, intentions and motivations a l l operate i n a complex interplay of mutual causes and e f f e c t s . - 117 -I t seems that, r e l a t i v e l y speaking, the east problematic investigation i s the survey on the attitude and awareness variable, which was a more direct output of the program. This may also be the area where the pre-test post-test control experiment i s useful for revealing change. Besides, i t s problem with data c o l l e c t i o n might have been reduced i f the program concentrated on a smaller scale, for instance j u s t i n the r e s i d e n t i a l or i n d u s t r i a l sector, instead of intervening every segment of the community. Therefore, i t appears that the pragmatist framework may be easier to apply i n smaller s o c i a l scale, and at the lower end of our theoretical model where assumptions pertaining to human behaviour are minimal. The pragmatist mode i s also useful for summarising the concrete empirical situations about socio-economic trends, energy consumption patterns, and so forth. Accordingly, instead of presupposing the f e a s i b i l i t y of obtaining objective knowledge about s o c i a l causes and program effects, i t may be a more tenable aim to attempt to study the program impacts as they are interpreted by different stakeholders. This type of interpretive survey may expose the subjective perspectives, the c o n f l i c t of interest among different program participants. As a Marxian stance i n s i s t s , these c o n f l i c t s of interest among different s o c i a l groups are the s o c i a l action settings upon which enquiries should be based. The subjective evaluation, which was supposed to be done by program participants and which might resemble t h i s type of interpretive enquiry, was not carried out as stated i n the i n i t i a l plan. Nevertheless, the program designer assessed the animation process i n the three communities. The d i s t i n c t - 118 -i n s t i t u t i o n a l and s o c i a l contraints i n each community were i d e n t i f i e d and analysed; t h i s revealed the diverse interest and commitment of dif f e r e n t s o c i a l groups. 4. Attending to the Perceived Needs and Problems of the Program Participants The c r i t i c a l theorists i n s i s t upon the necessity of recognising and considering the frames of reference of other parties, especially those who usually do not possess decision-making authority. This may be done by enquiring how the participants perceive a problem and define t h e i r needs. Were the p a r t i c i p a t i n g individuals and groups given the opportunity to interpret the energy conservation problem? Were they encouraged to express t h e i r perception of the usefulness of the program? According to the o r i g i n a l plan, there should have been a process evaluation i n which the participants had the opportunity to assess the program. However, t h i s task was abolished, p a r t l y due to i n s u f f i c i e n t funding, but largely, I believe, due to a c o n f l i c t of evaluation objectives as held by the sponsors and the designer. Nevertheless, the c i t i z e n s and community organizations who took part i n program planning ( i . e . the Steering Committee) were invited to review the development of the program. I r o n i c a l l y , t h i s element was not fonnally recognized as a component of the evaluation. - 119 -5. Encouraging Shared Decisionmaking Accepting the c r i t i c a l perspective maintaining that the subjects have to be treated as thinking, acting and knowing individuals, program participants then have to participate i n the planning and evaluation of the program. The central theme of t h i s energy project was to involve the c i t i z e n s i n deciding the goals and strategies for carrying out conservation schemes. I t was believed that i f a community takes part i n deciding program directions, i t would be more supportive of the entire project, and conservation could be achieved more e f f e c t i v e l y . In each community, the Steering Committee, comprised of various community leaders and concerned individuals, was prompted to assess constantly program progress and decide upon i t s future courses. Again, i r o n i c a l l y , t h i s type of appraisal was not acknowledged as a component of the formal evaluation plan. 6 . Cultivating the Exercise of Critical Self-Reflection for the Purpose of Attaining Emancipation from Dogmatism and Ideology This function c l e a r l y represents a value disposition with regard to carrying out enquiries: that i t i s worthwhile for the subjects to be emancipated from dogmatic and ideological thinking and habits through the practice of c r i t i c a l r e f l e c t i o n . In t h i s project, community involvement was primarily treated as a means for implementing energy conservation. By the same token, the - 120 -participants were encouraged to assume an active role i n planning and decision-making for the purpose of gaining the i r support of the program, rather than attempting to foster the f a c i l i t y of c r i t i c a l thinking among them. Hence, although t h i s evaluation has, to a certain extent, s a t i s f i e d the preceding two functions of encouraging shared decision-making and attending perceived needs, i t does not necessarily follow that i t also aimed at c u l t i v a t i n g the exercise of c r i t i c a l r e f l e c t i o n among the participants. This i s because the functions were carried out to f u l f i l a purpose other than emancipation and consciousness r a i s i n g . Shared decision-making and community involvement can also be used as a means for program management and public rel a t i o n s . This discrepency between means and ends further demonstrates the need for us to examine the underlying value disposition embedded i n the c r i t e r i a and functions. Therefore, i t i s not j u s t the evaluation findings that are important for future planning, but the process of c r i t i c a l examination and learning that may be shared between the evaluator and the community. However, when this process i s not shared by the community, the exercise of c r i t i c a l r e f l e c t i o n i s also suppressed. This suppression i s not uncommon i n the evaluation f i e l d . As Weiss (1972) remarks, program managers and planners are often reluctant to expose mistakes, f a i l u r e s and complications of programs. Negative findings are often concealed. This may be due to t h e i r lack of the concern for c r i t i c a l reason and c r i t i c a l learning. - 121 -CONCLUSIONS In t h i s case study, the evaluation objectives and the method outlined i n i t s i n i t i a l plan indicate that great emphsis was placed on generating factual knowledge about the effects of the energy program. However, the evaluation results show that t h i s evaluation did not succeed i n producing r e l i a b l e facts about program effects. This f a i l u r e i s largely due to the existence of a vague program variable, problems with energy data c o l l e c t i o n , and the design of a complicated program model containing dubious assumptions regarding human behaviour and human intentions. Above a l l , the generation of facts i s ultimately determined by value-based standards. This evaluation did not pay s u f f i c i e n t attention to the subjective factors and value issues implanted i n the program. I t did not c r i t i c a l l y examine the theoretical assumptions; accordingly, i t overlooked the i n v a l i d linkages affecting the r e l i a b i l i t y of program re s u l t s . I t underestimated the significance of c o n f l i c t i n g systems of thought held by the different stakeholders and f a i l e d to scrutinize and resolve the c o n f l i c t i n g program objectives held by the sponsors and the designer. Correspondingly, the process evaluation, which could be a valuable subjective assessment, was abandoned. I r o n i c a l l y , the subjective appraisal done by the Steering Committee was not considered a component of the formal evaluation. - 122 -Above a l l , although i t served a function of encouraging shared decision-making among program s t a f f and participants, the strategy was carried out as a means of attaining c i t i z e n support rather than as an end i n i t s e l f — a s an end, that i s , of shared decision-making as a worthwhile and necessary s o c i a l pursuit. This evaluation did not f u l f i l the function of fostering the exercise of c r i t i c a l r e f l e c t i o n among the participants. Hence, the fundamental value dis p o s i t i o n of the c r i t i c a l school—the d e s i r a b i l i t y of attaining emancipation—was not realized by the study. This convinces us of the necessity of examining the underlying value orientation of our program strategy and evaluation c r i t e r i a i f we want to investigate whether they have adopted a c r i t i c a l approach. - 123 -CHAPTER 6 CONCLUSIONS - 124 -mTRODUCTION This l a s t chapter draws some general conclusions frcm the analysis of the functions and methods of evaluation and from the divergent theories of s o c i a l enquiry. These w i l l be followed by a discussion of the s p e c i f i c conclusions gathered from the case study. GENERAL CONCLUSIONS In retrospect, this thesis commenced with the presupposition that an epistemological analysis of evaluation i s a necessary and worthwhile pursuit, given that our purpose i s to develop a framework of meta-evaluation c r i t e r i a . Epistemology examines some fundamental issues related to s o c i a l enquiry: What kind of functions do so c i a l enquiries serve? Which methods are employed to f u l f i l those functions? What kind of knowledge may be generated from s o c i a l enquiries? These questions have s i g n i f i c a n t bearing upon the c r i t e r i a that we select for judging evaluation. Besides, one of the conclusions drawn from this investigation i s that the c r i t i c a l framework of conducting s o c i a l enquiry i s desirable. One of the princ i p l e s of the c r i t i c a l approach, which c a l l s for the need to explore and consider multiple approaches to and methods of generating knowledge, confirms the significance of an epistemological study of so c i a l research. We s h a l l summarize the conclusions of our investigation i n the following pages: - 125 -1. Numerous figures i n the evaluation f i e l d , such as Ruddock (1981), Durm (1981), Cronback (1980) and P a r l e t t and Hamilton (1972), hold that s o c i a l processes are characterized by a complex interplay of human intentions, motivation and values; that these subjective issues are often not e a s i l y observed and defined by the t r a d i t i o n a l controlled experimental framework but may deserve investigation. In accordance with t h i s viewpoint, various renowned theories of s o c i a l enquiry also affirm the influence of subjective factors on s o c i a l a f f a i r s . Kant highlights the influence of a p r i o r i cognitive models on our generation of knowledge. Hegel accentuates the force of systems of abstract thoughts and values affecting human h i s t o r y . Marx emphasizes the c o n f l i c t of interests among di f f e r e n t s o c i a l classes that accompany s o c i a l a f f a i r s . Therefore, the s c i e n t i f i c research method, which treats testing, r e p l i c a b i l i t y , measurement, facts and o b j e c t i v i t y as i t s necessary ingredients, may not be able to investigate these subjective issues. I f we accept the premise that s o c i a l action i s complex and f u l l of subj ective influences which may not be discerned by the controlled experimental framework, we w i l l then have to expand our frame of reference to explore diverse approaches and subjective methods, i n the hope of revealing those subjective issues which also deserve attention. 2. The c r i t i c a l approach to conducting enquiry i s very appropriate for s o c i a l action settings. This i s because i t adopts multiple perspectives and methods, i t allows f o r a f l e x i b l e framework, - 126 -and i t investigates such latent aspects of s o c i a l action as value systems, c o n f l i c t s of interest and perceived needs, and human consciousness. I t incorporates the princ i p l e s held by Kant, Hegel and Marx mentioned above. In addition, i t accentuates the value d i s p o s i t i o n of i t s approach: that the ultimate purpose of conducting enquiry i s to allow the subjects to be emancipated from ideology and domination through the practice of c r i t i c a l s e l f - r e f l e c t i o n and the r a i s i n g of consciousness. Proceeding from the basic value orientation of the c r i t i c a l school, we have suggested s i x functions to be served i n evaluation: To examine theoretical assumptions of our programs and plans. To reveal and revise value assumptions or objectives, and relate them to a histor y of abstract thoughts and value systems. To i d e n t i f y the s o c i a l impacts of our programs as they are experienced by the participants. To expose the perceived needs of the program recipients and the i r problem d e f i n i t i o n s . To foster shared decision-making among participants so that norms of domination are minimised between the decision-makers and the c i t i z e n s , program and recipients, and the evaluator and the community. To c u l t i v a t e the exercise of c r i t i c a l r e f l e c t i o n for the purpose of r a i s i n g the consciousness of the subjects so that - 127 -they can be emancipated from dogmatic and ideological t r a d i t i o n s . 3. The s c i e n t i s t i c approach, which affirms that only s c i e n t i f i c methods of research are legitimate standards of enquiry and which disregards value issues, i s inadequate for i n s o c i a l action settings. Correspondingly, the p o s i t i v i s t mode may not be very relevant to many s o c i a l action projects. This i s because the primary aim of positivism i s to produce factual knowledge about s o c i a l causes and effects. Positivism considers only the e a s i l y observable and definable variables meaningful. However, s o c i a l demonstration programs usually contain vague objectives, program variables which are not e a s i l y measured and operationalized, uncertain c r i t e r i a of success and uncontrolled intervening factors. In addition, i f we evaluate s o c i a l programs from a p o s i t i v i s t standpoint, then certain subjective and value related concerns, which are beyond i t s frame of reference but which deserve attention, w i l l be neglected. Therefore, i t i s both unfeasible and undesirable to r e l y heavily on the p o s i t i v i s t mode of enquiry. 4. When pragmatist p r i n c i p l e s are exercised with d i s c r e t i o n and applied with adaptations, they may be useful for studying the impacts of s o c i a l action. I t s primary concern for applying knowledge i n action settings, and i t s interest i n discovering the consequences of our actions and plans i n concrete situations, are important supplements to Kant and Hegel's i d e a l i s t emphasis on the role that cognitive models and ideas play i n shaping human a f f a i r s . The significance of th i s synthesis - 128 -can be deduced from the use of the socio-economic base as the source of enquiry as advocated by Marx, as wel l as from the sociology of knowledge and the s o c i o - h i s t o r i c a l perspectives adopted by Mannheim. However, the pragmatist concern for prediction, for technical control of future actions, and for discovering program effects, may not be very applicable to s o c i a l action. A l l these demands require a knowledge of s o c i a l causal processes. Very often, programs are instrumentalized so that they conform to the methodical constraint of the controlled experiment designed for finding out program effects. Under this kind of r e s t r i c t i o n , other legitimate concerns which cannot be detected by i t s methodical structure are screened out. Therefore, i t seems the pragmatists' basic interest i n concrete s o c i a l settings i s acceptable but not t h e i r prescribed methods. Hence, we have to come up with some adaptations. One i s the Marxian notion of using the concrete socio-economic base as the action settings from which class c o n f l i c t s , s o c i a l roles and authority, and norms of domination and subserviency may be detected. Apart from re l y i n g on s t a t i s t i c s on socio-economic patterns, we may also observe patterns of s o c i a l interaction and communication through which s o c i a l c o n f l i c t s may be revealed. Another i s to discover the s o c i a l impacts as they are experienced by a community or individuals. This i s a way of learning the consequences of our programs and plans, not so much by using objective and generalised information, but more by relying on subjective information gathered from divergent perspectives and various s o c i a l groups. - 129 -Furthermore, one more r a d i c a l adaptation of the pragmatist framework i s to use Mannheim's notion of "decentralised planning" as the testing ground for our plans and programs. In thi s sense, we do not test the v a l i d i t y of a program model by experimenting with i t i n a s o c i a l setting, and by monitoring i t s effects and the recipients' reaction. 'Decentralised planning' proceeds with the aim of allowing program recipients and the community to review, c r i t i q u e , and debate on the model. This i s to be followed by considering and incorporating t h e i r suggestions i n program model revision, when they are appropriate. Therefore, program models and ideas are not exclusively tested by technicians and planners; they are also examined by the people who w i l l be affected. In t h i s manner, we may be able to bridge the gap between knowledge and concrete action settings, given that shared planning and the expressions of the community r e f l e c t some concrete l i f e s ituations. A l l of these adaptions are merely b r i e f introductions. Further s p e c i f i c a t i o n and research to develop and refine these alternative subjective evaluation are recommended to be carried out. CONCLUSIONS FROM THE CASE STUDY 1. The case study indicates that many of the problems associated with using the controlled experiment i n the evaluation of s o c i a l action projects are relevant. This demonstration energy program had vague objectives. I t i s d i f f i c u l t to determine from the program s t a f f s whether attaining energy awareness, actual energy savings or developing the community involvement process was the ultimate purpose. This - 130 -program also had an immeasurable variable (community involvement) , and uncertain c r i t e r i a of success (unknown amount of energy reduction desired). Above a l l , i t contained unvalidated linkages about causal processes (attitude -> behaviour -> actual energy savings). The presence of these program features, made controlled experimental evaluation problematic. Correspondingly, this evaluation did not produce factual information about s o c i a l causal processes for the benefit of po l i c y making; nor did i t come up with results that are objective and s t a t i s t i c a l l y sound for this purpose. Therefore, using the p o s i t i v i s t approach and the pragmatist method are problematic for t h i s evaluation. 2. This evaluation did not accomplish the pragmatist function of discl o s i n g program effects. One then questions the u t i l i t y of the pre-test post-test controlled experiment i n t h i s program. The least dubious program effect i n th i s evaluation seems to be the attitude variable because i t i s a more direct program output which evades the unvalidated linkages. This leads us to conclude that the pragmatist mode may be more suitable to simpler program models containing minimal assumptions about human intentions and human behaviour. 3. With respect to the d i f f i c u l t y of revealing program eff e c t s , such as behaviour and energy savings, we are not concluding that they should be disregarded e n t i r e l y . When they are involved, two precautions should be taken. The case study shows that the pre-test post-test controlled experiment i s best suited for programs involving a small - 131 -s o c i a l scale and a simple theoretical model. Hence, we have to use discretion when we apply this experiment. On the other hand, even when t h i s experiment seems to be suitable for some conditions, the causal inferences drawn regarding program and effects may s t i l l be imperfected by unforseen circumstances and uncontrolled factors. Hence, we have to question p e r s i s t e n t l y the r e l i a b i l i t y of those inferences. Both these precautions imply the importance of not treating the s c i e n t i f i c experimental framework as the standard method, but of being able to consider and develop diverse frameworks and perspectives. 4. By using the c r i t i c a l approach to judge t h i s evaluation, we found that i t did not assess several s i g n i f i c a n t subjective aspects of the program, such as the theo r e t i c a l and value assumptions, and the c o n f l i c t of interests between the sponsors and the designer. I t abandoned i t s i n i t i a l plan of gathering subjective evaluation from the recipients. Had i t carried out i t s o r i g i n a l plan of conducting subjective evaluation, some lessons might have been learned regarding how di f f e r e n t participants perceived the usefulness of the program, how various s o c i a l groups were affected by the program and what program improvements may be relevant to the community. Answers to these questions may be very i n s t r u c t i v e for planning and implementing future community energy programs. The experiment did not generate any of t h i s kind of subjective information; thus, i t has to be supplemented with more subjective evaluation approaches, l i k e the adaptations suggested i n (4) i n the previous section, and the functions proposed i n (2). - 132 -5. I f evaluation i s to contribute to on-going program development, i t should be treated as an integral part of the entire program planning and management process from the beginning. On-going program evaluation i s c r u c i a l to demonstration projects since they are characterized by t r i a l and error, by vague and c o n f l i c t i n g objectives and by arbitary outcome c r i t e r i a . A l l these features necessitate continuous program revision. The controlled experiment which assesses a program after a f u l l program cycle and requires holding the program variables constant, may not be appropriate for evaluating s o c i a l innovative programs. The case study confirms that i f the c o n f l i c t i n g progran objectives were resolved at an early stage, those c o n f l i c t s might not surface i n the evaluation objectives. And i f the unvalidated theoretical linkages were i d e n t i f i e d at the beginning, the program model and i t s implaTientation may be improved. I f the problems with data c o l l e c t i o n were realized at the outset, the program scale might have been reduced or the outcome assessment abandoned outright. The value of the process subjective evaluation might have gained more support. A l l these d e t a i l s affirm the d e s i r a b i l i t y of on-going evaluation of demonstration programs. 6. The characteristics of demonstration projects mentioned i n (5) above also serve to confirm the merit of a c r i t i c a l evaluation. I f we accept that arbitrary standards and vague objectives are inevitable i n innovative projects, then we have to seek a reasonable manner of coping with them. I t seems that we can be less dogmatic about our - 133 -arbitrariness by being aware of i t and by being w i l l i n g to a l t e r our valued-based standards. This awareness can be achieved by exercising c r i t i c a l s e l f - r e f l e c t i o n upon our theoretical and value presumptions, and more important, by r e l a t i n g them to a histor y of systems of abstract thoughts and values. I t i s by juxtaposing our recent standards with a history of ideas that we can r e a l i z e the transformation of standards through time; i t i s inevitable and "natural" to change our standards and perspectives. Therefore, we have to be liberated from dogmatic and ideological t r a d i t i o n s , whether such traditions take the form of the s c i e n t i f i c experimental framework or of centralised decision-making and evaluation. Traditions reinforce a habitual way of thinking and doing things without requiring us to r e f l e c t c r i t i c a l l y upon the functions, worth and h i s t o r i c a l context of these habits. 7. I t i s int r u c t i v e to refer back to the Hegelian concept of the d i a l e c t i c s of the self-generation of the human species, which maintains that i t i s the c o n f l i c t of ideas, the varying perspectives and their subsequent synthesis that permit the generation of new ideas and knowledge; and th i s synthesis w i l l ultimately lead to further human progress. I f we apply t h i s concept to evaluation, then i n order for enquiries to formulate a synthesis for on-going progress, f i r s t they have to i d e n t i f y the c o n f l i c t i n g and diverse perspectives of the participants. In order for the evaluator to consider the diverse perspectives and be c r i t i c a l of the program, i t seems that an outside evaluator who did not take part i n designing the program model and therefore i s not bound by i t s frame of reference, but who i s involved - 134 -with and aware of the on-going process of program operation, i s i n a better p o s i t i o n to exercise t h i s c r i t i c a l function than an inside evaluator. In the case study, an inside evaluator who i n i t i a t e d the o r i g i n a l concept of the program was appointed. This might have discouraged the evaluation from serving a c r i t i c a l function i n assessing the program from multiple angles and perspectives. SOME FINAL THOUGHTS AND IMPRESSIONS The primary task of evaluation i s to review our programs and plans. However, i n order to perform a c r i t i c a l and progressive function, evaluation should examine constantly i t s own stated functions and methods, and the value systems that i t i n t r i n s i c a l l y supports. This introspective style of c r i t i c a l r e f l e c t i o n i s v i t a l for the further progress of evaluation. The case study has demonstrated how subjective thoughts /and values influence the operation of the program and i t s evaluation. In i t s study of the h i s t o r i c a l development of various theories of knowledge, epistemology confirms that different types of knowledge are generated under different concepts of "knowing" held by the proponents i n each era. C r i t i c a l theory, which scrutinizes the l i m i t s of diverse modes of enquiry, prevents us from being contented with obsolescent tra d i t i o n s - 135 -and prescribed formulas. Above a l l , the emancipatory experience rendered by c r i t i c a l s e l f - r e f l e c t i o n and c r i t i c a l consciousness may provide an indispensible impetus for s o c i a l progress. - 136 -. . . human w i l l had no other purpose than to maintain awareness. But that could not do without d i s c i p l i n e . . . the dogged revolt against his condition, perseverance i n an e f f o r t considered s t e r i l e . I leave Sisyphus at the foot of the mountain! One always finds one's burden again. But Sisyphus teaches the higher f i d e l i t y that negates the gods and raises rocks. He too concludes that a l l i s we l l . This universe henceforth without a master seems to him neither s t e r i l e nor f u t i l e . Each atom of that stone, each mineral flake of that n i g h t - f i l l e d mountain, i n i t s e l f forms a world. The struggle i t s e l f toward the heights i s enough to f i l l a man's heart. One must imagine Sisyphus happy. ALBERT CAMUS, The Myth of Sisyphus: 85,91 - 137 -BIBLIOGRAPHY Adorno, Theodor W. 1982. "Subject and Object," in The Essential Frankfurt School Reader, Andrew Arato and Eike Gebhardt (eds.), the Continuum Publishing Company, New York. Allen, Edward H. 1975. Handbook of Energy Policy for Local Government, D.C. Heath & Co., U.S. Allen, Harris M. & David 0. Sears. 1979. "Against Them Or For Me: Community Impact Evaluations," in Improving Evaluations, Lois-Ellen Datta & Robert Perloff (eds.), Sage Publications, U.S. Anderson, Dennis & Carmen Cullen. 1977. Energy Research From A Consumer Perspective, for the Federal Dept. of Consumer and Corporate Affairs, Canada. Arato, Andrew & Eike Gebhardt (eds.). 1982. The Essential Frankfurt School Reader, Continuum: New York. Archer, Louise J. 1982. "Getting Past the Evaluation Assessment of C.H.I.P.," for the Ministry of Energy, Mines & Resources Canada, in Proceedings of the 3rd Annual Conference of the Canadian Evaluation Society, The University of Toronto. Berkeley, George. 1710. r'A Treatise Concerning the Principles of Human Knowledge," in The Empiricists, Anchor Books, New York. Beveridge, F.N. 1977. "The Economics & Accounting of Energy," in Energy Use Management, Rocco A. Fassolare (ed.), U.S. Dept. of Energy. Boruch, Robert F. & Hernando Gomez. 1979. "Measuring Impact: Power Theory in Social Program Evaluation," in Improving Evaluations, L.E. Datta & R. Perloff (eds.), Sage Publications: U.S. British Columbia Ministry of Energy, Mines & Petroleum Resources. 1981. Fuel for Thought, Canada. British Columbia Ministry of E.M.P.R. 1977. Municipal Energy Management, Canada. Burchell, Robert W. & George Sternlieb (eds.). 1978. Planning Theory in the 1980's. Center for Urban Policy Research, Rutgers University, N.J. Bredo, Eric, fie Walter Feinberg. 1982. Knowledge and Values in Social and Educational research, Temple U. Press, U.S. Bureau of Municipal Research. 1977. What Can Municipalities Do About Energy, Toronto, Canada. - 138 -Campbell, Donald T. & J.C. Stanley. 1966. Experimental and Quasi-Experimental Designs for Research, Rand McNally: III. Camus, Albert. 1955. The Myth of Sisyphus, trans, by Justin O'Brien, Random House, N.Y. Carnap, Rudolf. 1967. The Logical Structure of the World, trans, by Rolf George, University of California Press, U.S.A. Carter, Nova & Brian Wharf. 1973. Evaluating Social Development Programs, Canada. Central Mortgage & Housing Corporation. 1979. Local Energy Initiatives in Canadian Urban Settlements, by Judith Gibson, Canada. Chaudron, Douglas C. & Alan Ogborne. 1982. "Enhancing the Relevance of Evaluation," in Proceedings of the 3rd Annual Canadian Evaluation Society Conference, University of Toronto. Chelimsky, E. 1978. "Differing Perspectives of Evaluation," in Evaluating Federally Sponsored Programs: New Directions for Program Evaluation, C.C. Rentz & R.TR. Rentz (eds.), Jossey-Bass: San Francisco. City of Toronto. 1980. Cities Energy Conference Source Book, Toronto, Canada. Cohen, Alan M. 1981. "An Evaluator's Perspectives on Standards for Evaluation: Who Needs Them?" in Proceedings of the 2nd Annual Conference of the Canadian Evaluation Society, University of Toronto. Cohen, Alan M. 1982. "Program Evaluation: Why i t Seldom is Perceived as a Success," in the Proceedings of the 3rd Annual Conference of the Canadian Evaluation Society, University of Toronto, Canada. Comstock, Donald E. 1982. "A Method for Critical Research," in Knowledge and Values in Social and Educational Research, edited by Eric Bredo and Walter Feinberg, Temple U. Press, U.S. Coser, Lewis A. 1971. Masters of Sociological Thought, Harcourt Brace Jovanovich: U.S.A. Cronbach, Lee J. 1980. Toward Reform of Program Evaluation, Jossey-Bass Publishing Co.: U.S.A. Davidoff, Paul. 1978. "The Redistributive Function in Planning: Creating Greater Equity Among Citizens of Communities," in Planning Theory in the 1980's, T.W. Burchell & G. Sternlieb (eds.), Rutgers University: N.Y. D'Amore, Louis. 1981. Community Energy Conservation Project F i n a l Report on Phase I and I I . Canada, Unpublished. D'Amore, Louis J . 1980. "Community Energy Conservation: The Canadian Experience," i n Urban Forum, Feb. - Mar. 1980, Vol. 4, No. 6, Canada. Dunn, William, Ian M i t r o f f & Stuart Deutsch. 1981. "The Obsolescence of Evaluation Research," i n Evaluation and Program Planning, Vol. 4, pp. 207-218. Ehrenfeld, David. 1978. The Arrogance of Humanism, Oxford University Press Inc.: N.Y. Esmonde, P h i l . 1980. Community Energy I n i t i a t i v e s A Guidebook, for the Ministry of Energy, Mines & Petroleum Resources, B.C., Canada. Evans, John, Brent Ritchie & Gordon McDougall. 1979. "Energy Use and Consumer Behaviour: A Framework for Analysis & Po l i c y Formulation," i n Energy Po l i c y : The Global Challenge, by Peter Nemetz. Fischer, Frank. 1982. "Policy Evaluation: Integrating Empirical and Normative Judgements," i n Evaluation Studies Review Annual, V o l . 7, Sage Publication. Forester, John. 1982. "Know Your Organizations: Planning and the Reproduction of Social and P o l i t i c a l Relations," i n Plan Canada, March 1982, Canada. F r e i r e , Paulo. 1970. Pedagogy of the Oppressed. Seabury Press: N.Y. Friedmann, John. 1978. "Innovation, F l e x i b l e Response and Social Learning: A Problem i n the Theory of Meta-Planning," i n Planning Theory i n the 1980's, by Burchell and Sternlieb (eds.), Rutgers University, N.J. Friedmann, John. 1973. Retracking Ameria: A Theory of Transactive Planning, Anchor Press/Doubleday: N.Y. " Giddens, Anthony. 1977. Studies i n Social and P o l i t i c a l Theory, Hutchinson: London. Giddens, Anthony. 1979. Central Problems i n So c i a l Theory, Macmillan Press Ltd.: London. G i l l i l a n d , Martha N. 1977. "Energy Analysis: A Tool for Evaluating the Impact of End-use Management Strategies on Economic Growth," i n Energy Use Management, by Rocco Fazzolare, Pergamon Press: U.S.A. G i l l i l a n d , Martha N. 1978. Energy Analysis: A New Public P o l i c y Tool, American Association for Advancement of Science: U.S.A. - 140 -Gilmore, Alan. 1981. "Evaluation Standards: Some Lessons from the U.S. Experience," i n Proceedings of the 2nd Annual Conference of the Canadian Evaluation Society, University of Toronto, Canada. Habermas, Jurgen. 1971. Knowledge and Human Interest, Translated by Jeremy Shapiro, Beacon Press: Boston. Habermas, Jurgen. 1982. "On Systemically Distorted &OTitunication," i n Knowledge and Values i n So c i a l and Educational Research, edited by E r i c Bredo and Walter Feinberg, Temple U. Press, U.S. Heilbroner, Robert L. 1953. The Worldly Philosophers, Simon & Schuster, N.Y. Horkheimer, Max. 1982. "On the Problem of Truth," i n The Esse n t i a l Frankfurt School Reader, by Andrew Arato & Eike Gebhardt (eds.), Continuum: N.Y. Hume, David. 1748. An Enquiry Concerning Human Understanding, i n The Empiricists , Anchor Books: 1974, U.S. James, William. 1910. Pragmatism and Other Essays, Simon & Schuster: 1963, U.S.A. Kant, Immanuel. 1781. Critique of Pure Reason, trans, by Max Muller, Anchor Books: 1966, U.S. Kaplan, Abraham. 1964. The Conduct of Enquiry, Chandler Pub., Co., U.S.A. Kuhn, Thomas S. 1962. The Structure of S c i e n t i f i c Revolution, University of Chicago Press: Chicago. Lamb, Curtis W. 1970. National Evaluation of Community Action Programs, U.S.A. Lindbloom, Charles & R. Cohen. 1979. Usable Knowledge, Yale University Press: U.S.A. Lindbloom, Charles. 1959. "The Science of Muddling Through," i n Public Administration Review, Spring (19), U.S.A. Locke, John. 1690. "An Essay Concerning Human Understanding," i n The Empiricists , Anchor Books, 1974. Love, Arnold J . 1981. "Impact Evaluation: Program Evaluation for Human Services Organization," i n Proceedings of the 2nd Annual Conference of the Canadian Evaluation Society, University of Toronto, Canada. Love, Arnold J . 1982. "Joining Meaning & Methodology: N a t u r a l i s t i c Approaches to the Design of Responsive, Responsible, and Respectful Evaluations," i n Proceedings of the 3rd Annual Conference of the C.E.S., University of Toronto, Canada. -141 -Lynd, Robert S. 1967. Knowledge for What, Princeton University Press: N.J. Magee, Bryan. 1982. Men of Ideas. Oxford University Press: Great Britain. Mannheim, Karl. 1936. Ideology and Utopia, trans, by Wirth and Shils, Harcourt Brace & Co., New York. Marcuse, Herbert. 1982. "On Science and Phenomenology," in The Essential Frankfurt School Reader, Continuum Books: U.S.A. Marx, Karl & Frederick Engels. 1968. Marx and Engels: Selected Works, Progress Publishers: Moscow. Marx, Karl & Frederick Engels. 1965. The German Ideology. Lawrence and Wishart: London. McConnell, Shean. 1981. Theories for Planning. Heinemann: London. Ministry of Community & Social Services. 1974. Analysis and Design of Public Participation Program Evaluation, in Ontario, Canada. Myrdal, Gunnar. 1958. Value in Social Theory, Routledge & Kegan Paul Ltd.: London. Myrdal, Gunnar. 1969. Objectivity in Social Research, Pantheon Books, Random House: N.Y. National Research Council. 1977. Energy Consumption Measurement Data Needs for Public Policy, U.S.A. Nemetz, Peter. 1979. Energy Policy: The Global Challenge, Butterworth & Co., Canada. Parlett, Malcolm & David Hamilton. 1972. "Evaluation as Illumination: A New Approach to the Study of Innovatory Program," in Evaluation Studies Review Annual. Perrin, Burt. 1981. "Use of Evaluation: Decision-making Or . . .," in Proceedings of the 2nd Annual Conf. of the C.E.S., University of Toronto, Canada. Perryman, Gavin. 1975. Functions of Evaluation Research in Citizen Participation Programs, M.A. Thesis for the School of (Community & Regional Planning7 U.B.C, Vancouver, Canada. Pirsig, Robert M. 1974. Zen and The Art of Motorcycle Maintenance, Bantam Books: N.Y. Popper, Karl. 1959. The Logic of Scientific Discovery, Harper Torchbook, Harper & Row Publishers Inc., N.Y., 1968. - 142 -Rempel, warren F. 1965. The Role of Value in Karl Mannheim's Sociology of Knowledge, Mouton St Co., The Hague, Netherlands. Rossi, Peter H. 1972. "Boobytraps and Pitfalls in the Evaluation of Social Action Programs," in Evaluating Action Programs, by Carol Weiss (ed.) Allyn & Bacon Co., U.S.A. Rossi, Peter H. 1982. "Some Dissenting Comments on Stake's Review," in Evaluation Studies Review Annual, Vol. 7, pp. 62-64. Rossi, Peter H. & Howard E. Freeman. 1982. Evaluation: A Systematic Approach, Sage Pub., U.S.A. Ruddock, Ralph. 1981. Evaluation: A Consideration of Principles and Methods, Manchester Monographs 18, Dorset, Great Britain. Russell, Bertrand. 1946. A History of Western Philosophy, Counterpoint Books: London. Rutman, Leonard. 1982. "Program Evaluation in Canada: Issues and Prospects," in Proceedings of the 3rd Ann. Conf. of the C.E.S., University of Toronto, Canada. Salem urban Area Planning Committee. 1978. Salem urban Area Energy Study, U.S. Scriven, M. 1974. "Evaluation Perspectives and Procedures," Evaluation in Education, Popham, W.J. (ed.), McCutchan: Berkelay. Sewell, Derrick & Harold Foster. 1978. Images of Canadian Futures: The Role of Conservation and Renewable.Energy, for Environment Canada. Shapiro, Stan. 1980. Three Reports Prepared for the Steering Committee of NRG, Unpublished, Canada. Shapiro, Stan. 1979. Community Energy Conservation: Report on the Evaluation Programme, Report No. 10, Unpub., Canada. Sharp, Phillip & Ronald Brunner. 1980. "Local Energy Policies," in Energy Policy and Public Administration, by Gregory Daneke, U.S. Stake, Robert E. 1982. "A Peer Response: A Review of Program Evaluation in Education: When? How? To What Ends?" in Evaluation Studies Review Annual, vol. 7, Ed. by E.R. House, S. Mathison, J.A. Pearsol, H. Preskill, Sage Pub., U.S. pp. 53-60. Stufflebeam, Daniel. 1971. Educational Evaluation & Decision-Making, by the Phi Delta Kappa National Study Committee, F.E. Peacork Pub., U.S. - 1.43 -Stufflebeam, Daniel L. 1973. "Education Evaluation & Decision-Making," i n Educational Evaluation Theory and Practice, Worthen & Sander (eds.), Charles A.J., Pub. Co. U.S. Stufflebeam, Daniel L. 1973. "Evaluation as Enlightenment for Decision-Making," i n Educational Evaluation Theory and Pra c t i c e , Worthen and Sander (eds.). Stufflebeam, Daniel & William Webster. 1980. "An Analysis of Alternative Approaches to Evaluation," i n Education Evaluation and P o l i c y Analysis, May - June 1980, pp. 5-20. Suchman, Edward. 1967. Evaluative Research, Russell Sage Foundation, N.Y. Suchman, Edward. 1972. "Action for What? A Critique for Evaluation Research," i n Evaluating Action Programs, C.H. Weiss (ed.) , A l l y n & Bacon Inc., U.S.A. Taylor, Charles. 1975. Hegel, Cambridge University Press, Great B r i t a i n . Taylor, Charles. 1982. "Interpretation and the Sciences of Man," i n Knowledge and Values i n Social and Educational Research, ed. by E r i c Bredo and Walter Feinberg, Temple University Press, U.S. Treasury Board of Canada. 1981. Guide on the Program Evaluation Function, Program Evaluation Branch, Minister of Supply and Services, Canada. UNESCO. 1980. Evaluating Social Action Projects, France. U.S. Dept. of Energy, Bonneville Power Administration. 1982. Evaluation Plan f o r the Bonneville Power Administration Residential Energy Conservation Program^ University of Toronto. 1981. The Conduct of Evaluation: Planning, Implementation and U t i l i z a t i o n of Evaluation, Proceedings of the 2nd Annual Conference of the Canadian Evaluation Society, Toronto, Canada. University of Toronto. 1982. The Bottom Line: U t i l i z a t i o n of What, By Whom?, Proceedings of the 3rd Annual Conference of the Canadian Evaluation Society, Toronto, Canada. Webber, Melvin M. 1978. "A Difference Paradigm for Planning," i n Planning Theory i n the 1980's, Robert Burchell (ed.), Rutgers University, N.J. Weilenmann, A. 1980. Evaluation Research & Social Change, UNESCO. Weiss, Carol H. 1972. Evaluation Research, Prentice H a l l Inc., N.J. - 144 -Weiss, Carol H. 1972. (ed.) Evaluating Action Programs, Allyn and Bacon Inc., U.S.A. Weiss, Carol H. 1977. Research for Policy's Sake: The Enlightenment Function of Social Research,' in Policy Analysis, 3(4), pp. 531-545. Wilson, Steve. 1979. "Explorations of the Usefulness of Case Study Evaluations," in Evaluation Quarterly, Vol. 3, No. 3, Aug., pp. 446-459. Worthen, Blaine R. & James Sander. 1973. (eds.) Educational Evaluation Theory and Practice, Charles A.J. Pub. Co., U.S. Yin, Robert K. 1982. "The Case Study Crisis: Some Answers," in Evaluation Studies Review Annual, Vol. 7, 1982, Sage Pub., U.S., pp. 163-174. -145 -APPENDIX I THE COMMUNITY ENERGY PROGRAM HOUSEHOLD SURVEY Source: Shapiro, Stan. 1980. Three Evaluation Reports Prepared f o r the Steering Committee. Unpub. Canada. A t L . L uU c Cliait|jc R e l e v a n t D i m e n s i o n j r c e i v a d S e r i o u s n e s s 2b E n g e r g y C o s t 2c E n e r g y S h o r t a g e V e r n o a P c n t i c t o n I n d ex t - t e s t Richmond H i l l Newmarket Index KH Kll t - t e s t RH F r e d e r i c t o n Monc t o n Index t - t e s t X' • -M M O p i n i o n S t a t e m e n t s 6a "Hoax" V 6b T e c h n o l o g i c a l f i x V 6c 10% r e d u c t i o n 1' 6d Not i n f l a t i o n c a u s e V 6e Too d e p e n d e n t - B o t h N RH B o t h RH F F F F M F F F P o l i c y S u p p o r t -9a L o c a l a d v e r t i s i n g V 9b I n c r e a s i n g c o s t -J)c S e p a r a t e g a r a g e yd D i s c o u r a g e c a r use V A t t e n d e d s c h o o l y e a r P RH N RII RH F M M F M K Both-' . n p o r t a n c e I I n d i v i d u a l E f f o r t s Community E f f o r t s V 6V IV 5KH 3RH 2P OP 4 N O N 61ns 131ns 51ns 111ns 7F 5M 2 I ns 4F 3M 71ns - 1 + 7 -R e p o r t e d II u Ii a v i o u r 35 R e l e v a n t D i r a e u s i o n 4a L e s s Heat F u e l 4b L e s s lilo,c 11 i e i uy 4c L e s s G a s o l i n e ,0a T h e r m o s t a t Lower "10b L i g h t s o f f 0c Hoc w a t e r l o w e r -iOd L e s s h o t w a t e r \0e F u r n a c e s e r v i c e — O f I n s u l a t i o n I0g D r i v i n g l e s s _0U More f r e q u e n t s e r v i c i n g O i C a r p o o l i n g T O j More p u b l i c t r a n s -p o r t j _ K e t u r n a b l e b o t t l e s _ . x H a s t e S e p a r a t i o n V e r n o n P e n t i c t o n Index t - t e s t : l a M b --5 a L5e -5J ! 5c V P V V V p p V p Lower day t h e r m o s t a t P Lower n i y h t t h e r m o s t a t home s a v i n g s gas ' s a v i n g s D i s c u s s e d f r i e n d s . A t t e n d e d m e e t i n g s P a r t i c i p a t e d i n p r o j e c t , B o t h * 1' P V. P P V V Richmond H i l l Newmarket 1 ndex t - t e s t 1J o t h N RH RH RH RH RH RH N RH RH RH RH RH RH RH RH RH RH RH RH RH RH RH RH KH F r e d e r i c t o n Moncton Index. t - t e s t M F M H M F 9V AV IS RH 8 RH 7P 5P I N I N 61ns 131ns 6 Ins 13 Inr 6M 121; 2F IN 191ns L y Co sir, u n i t y P u c c c p t i o n R e l e v a n t D i m e n s i o n N e i g h b o u r Usage 5a E l e c t r i c i t y 5b H e a t i n g f u e l 5c G a s o l i n e V e r n o n P e n t i c L o n I n d ex t - t e s t Richmond H i l l Newmarket Index RII. N t - t c s t RH N B o t h F r e d e r i c t o n Honet on Index t - t e s t F M I n v o l v e d Community A c t i o n 16 O v e r a l l 17a S h o p p i n g C e n t r e s V 17b Ac work v 17c D r i v e l e s s / d o u b l e up p 17d P u b l i c b u i l d i n g s 17e At f r i e n d s V V P B o t h * N N RH N M M M Both*. 3V 2Uns IV' 4P AIns 2RH 1RH AK 2 N 31ns 61ns I F 6M 21ns OF ' i 7I3S. A TOTAL, 4 - 5 QtVAeSTiolAt l2 v 6V - 1:4? -APPENDIX II A COMPILATION OF THE CONSERVATION ACTIVITIES THAT MATERIALISED IN THE THREE TEST CCWMUNITIES Source: D'Amore & Associates Ltd. 1981. Community Energy Conservation Project F i n a l Report on Phase I & I L i Unpub. Canada. COMPARATIVE PROJECT OUTCOMES MFOERICTON RICHMOND HILL VERNON PROVINCIAL PROVINCIAL PROVINCIAL Dept. of Ministry of Ministry of Environment - Provided literature on Energy - Provided funding of $500. Energy, Mines environmental issues and re- and Resources Contributed $3,750 to cycling. GO Transit - To set up new locker units project. for bicycles at Richmond - To publish re-cycling hand- Hill station. REGION book. Ministry of North Okanagan tnergy Industry and Regional District- Contributed $3,750 to Secretariat - Consultant to Energy Secre- Tourism - Energy Bus brought to two projecI. tariat Is Dieiifcer of Steering local industries. Committee and technical advisor to project. - Consult dot to Energy Secre-tariat designed and Imple-mented a "Cunmiril ty Energy - Collaborating In examining -• feasibility of wood-chip burning furnace at Ode 11 Park Lodge. Standing Comm-ittee on Energy - Received brief on project. MUNICIPAL MUNICIPAL MUNICjPAL City - Provided of!Ice space. Town - Provided funding of $4,350. City Contributed $3,750 to project. - Equivalent of one full time - Provided office space. person seconded to project. - "Plan Vernon", city's new - Provided secretarial master plan Is to incorp-- Secretarial support. support. orate energy conservation. Ideas being considered - General support as required. Mayor - Chairman of Steering Comm- Include: solar reserves. ittee and fully involved tree shading, insulation Mayor - Chairman of Steering Conn- In project. standards, narrower streets. it tee and fully involved tn home Siting policies, energy project. - Proclamation of "Save - 10 efficient public transit. Days". recycling by-laws. - Proclamation of ftbrubry as Energy Conservation Month. Energy Manage- - ln-house energy conservation Director Admini-ment Comntttee - Consultant assisted in comntttee set up. Initiating traffic study strative Services - tn-house energy conservation of central business district - Central filing system to programs. for energy efficient move- eliminate paper waste. ment. rmiuRicTON Administrative ' .••I vll I" . I'li'.i'iil.il Ion ul pin.lri I In ftilmliil-.li.illvi' Dl l l i i r , ul New Hiuii.wlik. Ir.ll III IN|iji l i i » n l l i . i l l l i light', swllilii'.l In "lil Inl lll'l" I" l<>w ti.il I Ii: linui:,. I redericton 1raiis 11 Cily Imjlncer Id 1 ri'iil Inn Department Used project logo and dev-eloped a "Ride With Us" campaign - "Save Energy". Collaborator in feasibility ol pilot program to separate garbage at source. Collaborator In examining feabibillty of wood-chip burning furnace at Odell Park I oilqe. Public library - Sponsored two film nights. EDUCATION District No. 26 School Board - Has Committee examining curricula at al l levels to develop projects and pro-grams relating to Energy Conservation. - Print a monthly newsletter sent to al l teachers. - Guest lectures by consultant New Brunswick Community College - Organized an evening course in Energy Conservation. -2-i i ic i iMrmn 11111 I tWI l|V KIIIFH)!' • Mfllt l.iHliill I I r r NIl l lMMIfl III I I Hydro Publ ic l i b r a r y l i l ' l l l ' I ' i l I I i l l I . l l 111 .1 < 11 III I l l l III l l l M I M - I ' l i r t ' l V l l l . l i l . | i | t ' I I H - | l i I III l.llmi . l l tun III i l r v i I n p l l i ' l • i l l rungy '" n l r . r l v.i I i n i l p i n gram M I Ih torn l i i i n l •. ii I I I I Id n M Ki l l - Hydro b i l l s u'.ril In ini lmli Inserts promoting project ,11 11villrv. - Sponsored series ol five evening workshops nn Inergy Conservation. Information p.iifc<ti|r |iti|>.i 1 nl Im p.n I Ii Ipiinl'. ill i'.11I1. - Increased collection of energy publications to B K ' P t Illl I I'.IM ' .I l l l ' l l i . l l l . l , - Sponsored energy film night - Assisted In di-velnping tlis play at Shnpplng Hi 11. EDUCATION York County Board of Education Seneca College Specific Schools - Cooperation by consultant in developing Energy Con-servation progranming. - Developed three. 1 day courses on Energy Con-servation. - Assisted In developing display at Shopping Hall . A series of f ive, one page information sheets on Energy Conservation pre-pared by consultant for one school to be brought home by children. i l l y Hayor Lighting Engineer Hiinlenanre Hinager. EDUCATION Vernon Schools VI M N I I N H'Ui.llii' i | . r . .il '.I'M.HII- |il.inl tiring I1.11111".'.fit In li.-.tl lull lil|IM|-.. II11 yi I lll'l In ( 11 v Im • I ' I I I K I . Inlitrnwil inn l>tiii turn- In 1.111 /I ' l l ' , nn 1 i'i yi I IIII| . Bicycle path ptnpi r„ i l by Rotary Cluti 111 - i 111| I ml i •• it. (Mil In Im.I "I II ir . in Energy Ciinlrii'in i ' " Im I'llll. Proclamation nl 19lin ,r, Ini'igy f.nnsei v.il inn Vc.n . I nl I y 111 vi 11 vi • 11 In pi 11 | i i I. Street IIghtIng being changed from 11 urn I ' M nil In high pressnri- sodium. Also, wattage being lowenil. Ten year cnsl/liriipl i I analysis ol city's resource utiIization. Talks held with all teachers and principals. Energy curriculum booklets circulated. " Energy coordinator idenl-I f led In each si 'linn I .mil each a I tended one day work-shop. District teacher's resource centre developed an energy bibliography. Schools using information resources provided by project. -5-University of New Brunswick Lxteusion llept. iM-pl • I in e\li y FRIDIRIC10H Consultant acted as coor-dinator for a newspaper course In Inergy Conser-vation tn lie given through local newspaper; uml still -seguenlly across Canada. (,ol lalmratioii In examining teas Hi 1111y nt wood-chip burning lurnace at (dlell Park lodge. Specific Schools RICHMOND lllll Home audits done hy science and geography student', at. one secondary school. VI RNON Vernon Schools School Buildings Okanagan Col lege Classes involved in home energy audits, energy re-lated science projects, energy conserving cooking methods, charting aulixuilii le e I f 11:1 em I es, mm 11 or I m| M III H 11 energy ti'.e, ieiyi l iui| at M liool , f»tt . . . Journal Km < ta'.'. developed ,i •.Ilili' tape |o i".in 1.11 Inn nn "Ineigy Uinlies". - Hi anu i lass Incorporated energy conservation Into their major production nl the year. Inergy audits of al l school buildings completed. Conservation plans to be developed for next year based on energy audits. l ighting workshop held. fnvironmental control syslon to be Installed in Vernon Secondary. formation of school hoard conservation committee. Contributed J3.750 to project Inergy Conservation is a major design feature of college building plans. - Inergy technicians program being developed. I HI III UK. KIN COMMUNITY ORGANI/AI l o i r , Conservation Council of New Brunswick Active participation on Steering and Working Conmlttees. Designed approach for pi lot project to separate garbage al source. Conducted several workshops on conservation. He*I I' Board U l e Labour CounciI Hade available tn project slide-tape presentation on Inergy ( on.rrv.il Ion. General cooperation and support to project. Developed policy to support Energy Conservation pro-grams in work place and conservcr practices among members. New Brunswick Power Women's Group Various Volunteer Organ 12a lions and Church Groups Consultant and Resource person assisted in deslgnim a 1? week Summer Youth Employment Program on Energy Conservation. Preparation of a recycling handbook for the hone. Consultant gave talks on Energy Conservation as It relates to l i fe -sty les and Home Economics. HIUIHilMli l l l l l ( i m i N H V OHCANI/AI Kill', Richmond Hi l l Recycling Comni ttee Chamber of Commerce - Participation on Steering Conmtttee. - General support for pro JIM I and by prnjei t. Collaborated in developing Energy Conservation sem-inars for local business and Industry. Solar Inergy Soc lely dua l chapter to be formed as a resell I nt Inteie.l generated by prnjei I . Various Local Groups and Organizations - Talks by consultant on Inergy Conservation. "Helpmate" - Community information service acted as channel of communication lor pro-ject. v i tmoN IIIHMIIIIIV (IKI.AN1/A1 ION'. (.Iiarolirr ol roimieice - Accepted res|Miusibl Illy ol meeting Vernon's committ-ments to project. - Contributed olf ice spai •• to project. - Doing energy audit as example for business community. '.oi lai I" I an-il log r.oimi 11 B.C. Hydro - President ai leil as (.halt person ol Steering (mini Ittee and lully Involved III piiijei I . - Provided secretarial set vice' to project. - Provided energy curriculum kits for primary classes In use throughout d ist r ic t . - General support and 1 0 I I -aboration. Inland Natural Gas - General support and co l l -aboration. Hotels/Hotels - The two main lodges inst-ituted conservation program. Hospital - Regional confercnie held on Energy in Vernon. Iludai - Working toward building a demonstration energy mn serving home. Real (state Board - Contributed $500 towards the development of Energy Conservation packet. - Use NRG Action logo on front of weekly Journal. - Slide-sound presentation ot Canadian Real Tstate Asso-clatlon being used lota J lit. - 5 -RICHMOND KILL VEJUION Real ist.itc lli>.n <l Attempting tn be a model Board for Canada In Energy Conservation. New Ventures - Businessman examining feas-ib i l i t y of an energy eff ic -ient building to house energy businesses and a resource centre. - Design and marketing nl energy and environmental control systems by electrician. - Energy consulting business started. Genera I One business blnr.k nrganl/ml to recycle; recycling bin built lor their use. Huhawk Oil Co. Recycling Society technical Inlnimatlon pro-vided to a nunfcer ol Industries, each examining Energy Conservation. Has proposed a regional oi l recycling depot with all proceeds going tn Recycling Society. Has received additional support from several se i lms. Newsprint now sold to Kelowna Instead of China. Various Local Groups and Organizations Markets for aluminum and I In Identified and now being accepted. Amount of paper being re-ceived has Increased by ?). Talks given by consultant and resource person on Energy Conservation. fREDCRICTON MEDIA Channel 10 Community Television - Regular coverage of project. - Open 1Ine talk show. - Prepared one hour video-tape documentary of project. Dally Gleaner - Project coverage. - "Energy Alert" column. - I'ulillshed Energy Index. CBC r. iNii «: I H I Kegular coverage. RICHMOND HHL MEDIA Classlcomm Cable Liberal CFGM - Produced a weekly television program on Energy Conser-vation activities in the community. - Weekly column of "Energy Savers". - Excellent coverage of pro-ject. - Five page energy supplement for "Save 10 Days". • Local radio coverage. "INFORMATION CIIANNII S" Richmond Hill Hydro Chamber of Commerce Citizen's Advisory Group - Information conni-Block Parents Association unicated through Field Naturalists Recycling Committee Schools (newsletters) Public library School Libraries Family "Y" Seneca College Town Offices Hillcrest Mall mailings of these organizations. "Energy Action" pamphlets on display. Also placed emphasis on pro-moting fner|ave, "llcalllne" and Canadian Home In-sulation Program (CHIP). VERNON Volunteer Action Groups - Solar greenhouse approved for construction by u private school. - Farmers market approved I one year. - NRG float built lor winle carnival. MEDIA Vernon Dally News - Weekly energy column. - General coverage of the project. - Two special energy supple ments. Common i ty Cable Television - Weekly program on Energy Conservation.
UBC Theses and Dissertations
A critical examination of the functions and methods of evaluation research Auyeung, Poyin 1984
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.
- 831-UBC_1984_A8 A99.pdf [ 7.07MB ]
- JSON: 831-1.0096200.json
- JSON-LD: 831-1.0096200-ld.json
- RDF/XML (Pretty): 831-1.0096200-rdf.xml
- RDF/JSON: 831-1.0096200-rdf.json
- Turtle: 831-1.0096200-turtle.txt
- N-Triples: 831-1.0096200-rdf-ntriples.txt
- Original Record: 831-1.0096200-source.json
- Full Text
Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url: