Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Development and application of a methodology for evaluating adult basic education projects Harvey, Sheldon Robert 1981

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


831-UBC_1981_A8 H37.pdf [ 3.58MB ]
JSON: 831-1.0055882.json
JSON-LD: 831-1.0055882-ld.json
RDF/XML (Pretty): 831-1.0055882-rdf.xml
RDF/JSON: 831-1.0055882-rdf.json
Turtle: 831-1.0055882-turtle.txt
N-Triples: 831-1.0055882-rdf-ntriples.txt
Original Record: 831-1.0055882-source.json
Full Text

Full Text

DEVELOPMENT AND APPLICATION OF A METHODOLOGY FOR EVALUATING ADULT BASIC EDUCATION PROJECTS by SHELDON ROBERT HARVEY B.A., University of Winnipeg, 1970 B.Ed., University of Manitoba, 1975 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS in THE FACULTY OF GRADUATE STUDIES (Adult Education Division, Department of Administrative, Adult and Higher Education Faculty of Education University of B r i t i s h  Columbia)  We accept this thesis as conforming to the required standard.  THE UNIVERSITY OF BRITISH COLUMBIA July, 1981 ©  Sheldon Robert Harvey, 1981  DE-6  In p r e s e n t i n g  this thesis in partial  f u l f i l m e n t of  requirements f o r an advanced degree at the  the  University  of B r i t i s h Columbia, I agree t h a t  the L i b r a r y s h a l l make  it  and  f r e e l y a v a i l a b l e for reference  study.  I  further  agree t h a t p e r m i s s i o n f o r e x t e n s i v e copying of t h i s t h e s i s f o r s c h o l a r l y purposes may  be  department o r by h i s o r her understood t h a t  granted by  representatives.  my  It is  copying or p u b l i c a t i o n of t h i s t h e s i s  f o r f i n a n c i a l gain  s h a l l not  be  allowed without my  permission.  The U n i v e r s i t y of B r i t i s h 2075 Wesbrook P l a c e Vancouver, Canada V6T 1W5  (2/79)  the head o f  Columbia  written  ABSTRACT In British Columbia, adult basic education  (ABE) has  evolved  into a significant  verge of  gaining  program area on  the  acceptance as a legitimate and important part of the public education  system.  If the resources  currently committed to  these programs are to be s o l i d i f i e d imperative  the  present  program  and  impact  increased, be  i t is  measured,  the  needs, resources, processes and outcomes be articulated  and  the benefits of increased programming be predicted.  This is  best  program  accomplished  through  effective  use  of  evaluation. The f i e l d of program evaluation is characterized by a lack of well developed theory, absence of methodology  a series of complex models, an  and  an  abundance of  designs  and  checklists which are not tied to a sound theory or model. The  purpose  reviewed  of  the  evaluation  this  extant  of  adult  study was models  and  education  evaluating  threefold.  methodologies programs.  methodology  for  innovative,  developed.  Finally, the methodology was  evaluation of an adult basic education  i i  Firstly, it  ABE  for  Secondly, programs  the a was  used to guide an  project conducted by  a  British  Columbia  evaluated accomplished methodology  provided  college,  While  few of its a useful  and  the  stated  project  being  objectives, the  flexible  structure to  guide the evaluation process. It is hoped that this methodology will be f i e l d tested on a variety of ABE programs and that additional research will result  in an even more sophisticated methodology  designed  to strengthen the ties between the best theories and models and the f i e l d of practice.  iii  TABLE OF CONTENTS Page ABSTRACT  i i  TABLE OF CONTENTS  iv  LIST OF FIGURES  vi  ACKNOWLEDGEMENTS  vii  CHAPTER ONE INTRODUCTION  1  PURPOSE OF THE STUDY  3  NEED FOR EVALUATION  .3  PLAN OF THE STUDY  8  CHAPTER TWO LITERATURE REVIEW  9  Definitions of Evaluation  10  The Tylerian Model  11  Scriven's Model of Evaluation  12  Stake's Concept of Evaluation  15  Decision Management Strategies  17  COMPARISON OF EVALUATION MODELS  22  CHAPTER THREE AN OUTLINE OF A METHODOLOGY  26  EVALUATION PROCESS ACTIVITIES  29  Defining the Boundaries of the Evaluation  30  Preparing the Program Statement  35  iv  Page Creating a Specififc Evaluation Plan  37  Performing the Evaluation  40  Reporting the Evaluation  41  CHAPTER FOUR INTRODUCTION  44  SUMMARY OF PRE-E V AL UATI ON ACTIVITIES THE EVALUATION  44 46  Defining the Boundaries of the Evaluation.... 46 Preparing the Program Statement Creating a Specific Eval uat i on . Pl an  50 51  Performing the Evaluation  56  Reporting the Evaluation Results  58  SUMMARY AND OVERVIEW  59  CHAPTER FIVE SUMMARY  61  CONCLUSIONS  65  BIBLIOGRAPHY  70  APPENDIX A  74  v  LIST OF FIGURES Page 1.  Decision-Making Settings  5  2.  The Pathway Comparison Model  14  3.  Data Collection and Analysis  16  4.  Types of Decisions  20  5.  Outline of an Evaluation Methodology  28  6.  Defining the Boundaries of the Evaluation  34  7.  Preparing a Program Statement  36  8.  Creating a Specific Evaluation Plan  39  9.  Performing the Evaluation  40  10. Reporting the Evaluation Results  41  11. Sample Impact Model  55  12. Conceptualizing the Project  98  vi  ACKNOWLEDGEMENTS I would like to thank several members of the Department of Administrative, Education.  Adult  First,  and Higher  Education,  Dr. G. Dickinson,  Faculty of  who offered me the  opportunity to participate in an exciting series of research and development projects, one of which formed the basis of this study.  He also offered support and expert, consistent  advice throughout.  Dr. P. Cookson spent long hours reading  various drafts of this paper, wrote voluminous critiques and regularly drew my attention back to the forest whenever I insisted  on staring at the trees.  Dr. D. Rusnell  was  helpful in the development of an earlier paper on evaluation which i n i t i a l l y drew me to the area of program evaluation. Robert and Mae Harvey, my parents, consistently and lovingly encouraged and supported me in this as well as many other projects.  I would like to express appreciation to my wife,  Betty Harvey, who persevered, cajoled, edited and encouraged when each was required. Carol  McGillivray and Sherry  Estes  typed  this  document,  often at odd hours and from a semi-legible draft.  I express  my sincere appreciation to a l l of those mentioned above.  vii  CHAPTER I  INTRODUCTION It is widely accepted in Canada that at least nine years  of  schooling  literacy. aged  required  Nearly five million  fifteen  schooling. people  are  and  over,  to  people  have  less  attain  functional  in the population,  than  nine  years  of  British Columbia alone has in excess of 378,000  or twenty-four  percent  of the adult population in  this category (Dickinson, 1979, p. 1). If the definition of basic education usually applied to the pre-adult population, up to and including twelve years of schooling, is extended to adults, the number lacking basic education is even more dramat i c.  Many correlational  studies have i l l u s t r a t e d  the  relationship between a lack of education and such variables as low income, high unemployment, high welfare dependence and other social established  problems.  in adult  In addition, i t has been well  education  that  the  best  educated  participate in continuing education while the undereducated participate rarely or not at a l l . institutions  of  British  The public educational  Columbia  opportunities for the undereducated  have  provided  few  of this Province (Faris,  2  1979), but the need for adult basic education  (ABE) programs  is well documented (Faris, 1979;  1978;  1976). as  Dickinson,  Thomas,  It appears therefore, that many of these individuals  well  as  systematic  society  in  general  would  benefit  from  a  approach to providing appropriate, high quality  educational opportunities for the undereducated adult. The British  Ministry of  Columbia has  numerous reports Committee  on  and  Province  of  not articulated a policy on ABE,  but  briefs,  the  Continuing  Education  and  of  the  including the Community  described this as an area of high p r i o r i t y .  Report of  Education, The  have  Report of  the Committee on Adult Basic Education (Faris, 1979,  p.  17)  stated that: ...the main prerequisite for closing the gap between the need for and the provision of Adult Basic Education was an integrated and systematic approach. Adult Basic Education as i t presently exists in British Columbia is characterized by an ad hoc approach which has resulted in uncertainty, insecurity, and inefficiency in the deployment and use of resources. One  of the strategies adopted .by the Ministry to  ameliorate  this situation has been the sponsorship of pilot projects at various  educational  improve ABE  institutions  to  initiate,  programs or to develop innovative  expand  or  approaches.  3  Given  the nature  and purpose  of such  projects  i t is  imperative that systematic formative evaluation be b u i l t - i n , both to maximize the likelihood  of success and to provide  data for summative evaluation and dissemination  of findings  to other institutions and regions. PURPOSE OF THE STUDY The purpose of this study was threefold. it  reviewed the extant  evaluation  of adult  methodology education  models and methodologies education  for evaluating  programs.  innovative,  projects was developed.  Firstly, for the  Secondly, a  short  term  adult  Finally, the methodology  was used to guide an evaluation of the Adult Basic Education project conducted by a British Columbia college. NEED FOR EVALUATION Those charged with the responsibility for program planning benefit  and evaluation from  an evaluation  commit to such a study. potential  benefit programs  for  which  study and what  programs resources  will to  Most educators would agree that the  of an evaluation  programs than for others. those  must decide  is greater  for some  The d i f f i c u l t y is in identifying which  an  evaluation  will  be  4  cost-effective.  Stufflebeam  has  developed  a  conceptualization which addresses this situation. Stufflebeam  (1973)  presents  four  different  decision settings which result from the intersection of two continua.  The f i r s t continuum, degree of change, refers to  the expected outcome of the a c t i v i t y ranging change to a large change. grasp, high.  The second continuum, information  refers to the quantity and  available  to  the  from a small  program  quality of  planners,  Figure 1 i l l u s t r a t e s the  ranging  resulting  information from  low  to  decision-making  settings. The produce  a  first  completely  understanding outcomes. reality.  decision setting, metamorphism, would  of Such  The  new  need, a  system  required  situation  is  based resources  The  small changes intended  and  a  total desired  rarely encountered  homeostasis setting is typical  on-going program.  upon  in  of a stable,  decisions that are made result in to maintain  or restore the program.  5  FIGURE 1 Decision Making Settings 1. METAMORPHISM  2. HOMEOSTASIS Acti vity: Restorat i ve Purpose: Mai ntenance Bas i s : Technical standards and quality cont rol 3. INCREMENTAL ISM Act i vity: Developmental Purpose : Cont i nuous i mprovement Bas i s : Ex pert judgment plus structured  SMALL  Activity: Ut o p i a n Pu rpose: Complete change Bas i s : Overarchi ng theory  4. NEOMOBILISM Act i vity: Innovat i ve Pu rpose : Invent i ng , testing, and di ffusi ng solut i ons to significant prob1 ems Bas i s : Conceptuali zation, heuristic investigation, and structured inquiry  DEGREE OF CHANGE  LARGE  (From "Educational Evaluation and Decision Making" by D.L. Stufflebeam. In B.R. Worthen & J.R. Sanders, Educat i onal Evaluation: Theory and Practice, Worthington: Charles A. Jones Publishing, 1973 ).  6  The risks solid  are small  grasp  of  since these decisions  the  relevant  are based upon a  information.  Incremental  decision-making produces small changes based upon incomplete information.  This is characteristic of small pilot  operated within larger on-going programs. intended to test expense.  some new concept  While homeostatic  program,  incremental  neomobilism, without  results  a solid  pressing  need  for  attempt  base.  action  for  or  to  maintain a  to  move  a program  large  Such situations or the  risk  tend  The fourth quadrant of  from a desire  data  Such projects are  very l i t t l e  decisions  decisions  slowly to a new position.  with  projects  perceived  Figure 1,  changes result  but  from a  opportunity  to  make a quantum leap.  The many  project  large  scale  neomobi1istic  described projects,  setting.  new i n s t i t u t i o n  in this can  be  The project  attempting  to  study,  similar  described  involved  a  as  to a  relatively  serve a different  clientele,  u t i l i z i n g new methods and materials without the benefit  of a  supportive theoretical  this  or empirical data base.  situation,  the  assist  program planning and implementation  to  the  document  need for evaluation  all  aspects  of  the  Given  was especially  project  so  great  process that  to and  future  projects would have more information with which to work.  7 Evaluation  is a  much  discussed  but  practised a c t i v i t y in ABE. A review of published studies shows that they tend to f a l l  little  evaluation  into two categories -  system-wide surveys and end-of-program reports. The collects  former serves a valuable function in that i t  data  on a system-wide  basis.  Knox, Mezirow,  Darkenwald and Beder (1972) have developed called  perspective  discrepancy  assessment  suited for this type of evaluation.  a methodology which  is well  This strategy, however,  relies heavily upon comparing percentile distributions among groups  and therefore  is not d i r e c t l y  evaluation of a smaller program. areas  of  decision  making  applicable  to the  In addition, the six major -  recruitment,  staffing,  instruction, staff development, collaboration, goal setting - are not the most important  issue  in a l l ABE programs,  especially innovative pilot projects. The second category of evaluation studies, end of project  reports, seldom meet the c r i t e r i a  Typically, these reports the  of evaluation.  are prepared in order to satisfy  requirements of a funding  organization  and do l i t t l e  more than describe the events which transpired during the 1i fe of the project.  8  What is required by many ABE projects and programs is a methodology which makes evaluation an integral part of the program planning process.  The data, both descriptive  and judgmental,  the planners  should  assist  in modifying  various aspects of the program to maximize the likelihood of achieving better  the stated goals  match  methodology  or in revising  the r e a l i t i e s should  also  of a new  facilitate  the goals to  situation.  The  the completion  of  summative reports since good formative evaluation simplifies the summative evaluation process. PLAN OF THE STUDY The following  chapter  presents a review of the  literature pertaining to the leading definitions and models of educational evaluation as they  relate  to this  study.  Chapter Three presents a methodology based upon one of these models.  Chapter  Four describes the implementation of this  methodology in conducting project.  The final chapter draws conclusions based upon the  information study.  an evaluation of an ABE pilot  gathered  and experiences  resulting  from this  9  CHAPTER I I  LITERATURE REVIEW The evaluation of adult basic education programs has  been  seriously  theoretical  base.  limited Most  by  the  evaluation  lack plans  of  a  sound  have  been  implemented without the assistance of a conceptual framework to guide the process.  Traditionally, evaluation has been  defined in terms of participant s a t i s f a c t i o n , participation patterns, and dropout rates, and characterized by the ad hoc collection of data which often answers questions no one is asking.  Further, most evaluation reports are submitted too  late to affect the project or program under study.  ABE  programs tend to be in a state of constant change because of t h e i r dynamic nature and fact  and  methodologies  the  precarious funding.  inadequacies  of  this  evaluation  significant benefit could be derived from the  development of a methodology which r e a l i t i e s of ABE programming. than an  current  Given  inconvenience  takes into account  the  Evaluation must become more  imposed by funding authorities  and  begin to serve a dual purpose of program improvement and demonstration of accountability.  10 Definitions of Evaluation A conceptualization of the evaluation process will be s i g n i f i c a n t l y influenced by i t s definition. (1972) dictionary defines judge".  evaluation  The Webster  as "to examine and  While there is nothing technically incorrect with  this d e f i n i t i o n , i t offers no direction as to how i t can be operationalized.  Several  educational  evaluators  have  modified this basic definition in an attempt to remedy that problem.  Tyler (1950) states that evaluation is essentially  the process  of determining  to what extent the educational  objectives are actually being realized by the curriculum and instruction. describe  and  Stake (1967) suggests that evaluation must judge,  and  Scriven  (1972)  argues  that  evaluation should appraise not only the achievement of goals but also judge the merits of the goals themselves. accepted  and  Stufflebeam.  useful  definition  has  been  A widely  offered  by  It states:  "Evaluation is the process of delineating, obtaining and supplying descriptive and judgment information, concerning some object's merit, as revealed by i t s goals, plans, process and product and for some useful purpose such as decision making and accountability" (1976, p. 14). While there each  seems  to  are s i m i l a r i t i e s  place  the  emphasis  among definitions, on  a  different  11 aspect of the process.  A brief examination of the models  associated with these definitions will  serve to i l l u s t r a t e  their different emphases. The Tylerian Model The Tylerian Model emphasizes the degree to which programs are achieving their intended objectives.  There are  five steps in the process. 1.  Analysing objectives to two basic dimensions. the behaviour expected the key content that is  identify and c l a r i f y their These dimensions are: (a) from the student, and (b) to be mastered.  2.  Identifying situations that will give the student a chance to express the behaviour related to the cont ent.  3.  Selecting or developing record the behaviours.  4.  Analysing the amount of change that's taking place in students.  5.  Analysing the strengths and weaknesses of the program in helping students achieve the objectives (Steele, 1973, p. 154).  instruments  that  will  While the attainment of objectives is an important aspect of a program to evaluate, i t is not the only one. Programs may, which  are  objectives.  for example produce important totally This  ignored  by  is especially  the  side effects  official  apparent  in ABE.  program For  12  example, i t is common for ABE return  to  school  potential  is  with  present  students to approach their  understandable to  either  structure the  situation such that they experience may  restore confidence  and  trepidation.  The  learning  frequent success which  instill  a desire to continue  learning or to teach the material in a manner which reinforce their past f a i l u r e s .  Thus, while the stated goals  refer to cognitive areas there may be important side effects.  The  may  Tylerian Model  attitudinal  is thus too narrow an  approach for evaluating ABE programs. Scriven's Model of Evaluation Scriven whether  argues that  objectives have  been  in addition to achieved,  assess the merit of the objectives. allows the program.  evaluator to judge the  determining  evaluation must  This dual assessment overall  merit  of the  Scriven differentiates between the goals and the  roles of evaluation.  The goal, he suggests, is always to  arrive at an overall judgment of merit.  While the roles are  many and varied, they are of two types, formative evaluation which  is  a  developmental  process  aimed  at  program  improvement, and summative evaluation which is an objective, independent  assessment  of  merit.  For  both  types  the  evaluation process is seen as essentially one of collecting  13 relevant information, condensing i t , and judging the merit of a program.  In addition to the concepts and  definitions  referred to above, Scriven has developed what he calls  the  Pathway Comparison Model. The Pathway Comparison Model consists of a series of ten steps  intended  to guide an evaluation study.  The  f i r s t five steps represent  the conceptualizing phase, that  is from data to c r i t e r i a .  This phase can be viewed as a  process of understanding information  and  what needs to be done, gathering  condensing  the  information.  Steps  6-10  represent the credentialing phase, that is from c r i t e r i a  to  evaluative  conclusion.  an  examination  of  determination p.  10).  costs  Credentialing and  of a final  Figure  involves  alternative programs  and  the  judgment of merit (Scriven,  1973,  2 summarizes the  steps  of  the  Pathway  Compari son Model . This  is not  a linear model  nor  is i t intended  proceed through the steps only once. it  is an  interactive process  through the steps.  often  that  one  Properly implemented, requiring recycling  14  FIGURE 2 The Pathway Comparison Model 1. Characterization - what is being evaluated and at what level of generality of specif i city? 2. C l a r i f i c a t i o n of Conclusion with Client what form of evaluative conclusions does the client require. e.g. system x is superior to existing system or system x is superior to a l l available systems. 3. Causation - is i t an issue? it to be dealt with?  If so, how is  4. Comprehensive Check of Consequences includes looking for unintended outcomes. 5. Conceptualization (Compression) - t y p i c a l l y using preceeding data but may use some from steps 6-8. 6. Costs - since this is a comparison model one must examine the cost of a program in relation to the cost of similar or alternate programs. The opportunity cost should also be considered. 7. Consumer Characteristics - Market and need analysis examines the consumers of the product and the need for the evaluation. 8. C r i t i c a l Competitors - alternate pathways to the same goal. (1-7 for each of them). 9. Credentialing - combining. 10. Conclusions and communications - data processing, design, writing., dissemination. (Scriven, 1973, p. 28).  15  In addition to the  Pathway Comparison Model  formative - summative c l a s s i f i c a t i o n , Scriven has  and  described  an approach to evaluation called goal-free evaluation.  The  underlying  the  premise  is that  an  evaluator  unaware of  intended outcomes of a program will have greater objectivity and  a l l effects will  be assessed equally, not merely the  intended effects. Goal-free evaluation is intended not as a substitute  for  supplement.  goal-based  evaluation,  but  as  a  useful  Thus Scriven has taken a broader approach to  evaluation than the one suggested by the Tylerian Model. Stake's Concept of Evaluation Stake's thinking on evaluation has been influenced by the work of Scriven.  While Scriven has occupied himself  with  issues,  the  philosophical  Stake  has  attempted  to  develop a more s p e c i f i c , systematic procedure for performing evaluations. formal  Stake  differentiates between  informal  and  evaluations, and supports Scriven's position that an  evaluation is not complete until a judgment has been made. The  framework suggested by Stake is intended  collection and  data analysis.  to guide data  Three types of  information  are sought - antecedent, transaction and outcome - on four different judgments.  levels This  Countenance Model.  -  intents,  observations  approach to evaluation  standards  and  is known as  the  16 The descriptive data, intents and observations are analysed  by "finding the contingencies  transactions,  and  outcomes  and  between intents and observations"  among  finding  antecedents,  the congruence  (Stake, 1973, p. 117).  The judgment process follows a different model.  Each of the  types of data may be judged according to either a set of absolute standards or relative standards. FIGURE 3 Data Collection and Analysis  INTENTS  OBSERVATIONS  STANDARDS  JUDGMENTS  RATIONALE  DESCRIPTION MATRIX  JUDGMENT MATRIX  (From "The Countenance of Educational Evaluation" by R.E. Stake. In B.R. Worthern & J.R. Saunders, Educational Evaluation: Theory and Practice, Worthington: Charles AT Jones Publishing, 1973).  17 In addition to the Countenance Model, Stake has contributed dimension  the concept of evaluation  of responsive  evaluation.  is contrasted  with  This  preordinate  evaluation. "Preordinate studies are more oriented to objectives, hypotheses and prior expectations... Preordinate evaluators know what they are looking for and design the study so as to find i t . Responsive studies are organized around phenomena encountered often unexpectedly - as the program goes along" (Stake, 1976, p. 20). Responsive evaluation emphasizes the need to work with  those  who  may  be affected  by the evaluation, to  understand and be sensitive to their needs, to be willing to s a c r i f i c e some measure of precision to gain useful results, and  to  recognize  perspectives. theory  This  the  existence  process  of  differing  can be likened  value  to grounded  in that many of the directions and decisions are  determined as the program unfolds.  In practice  evaluation is often more concerned with  responsive  program a c t i v i t i e s  than with intents and outcomes. Decision Management St rat egi es A fourth major model of educational evaluation may be characterized  as decision management strategies and is  18 best  represented  by  the  work  of  Stufflebeam.  The  conceptualization of evaluation developed by Stufflebeam is called the CIPP Evaluation Model (Context, Input, Process, Product).  As Sufflebeam  noted, "servicing decision making  in change efforts was and is the most unique characteristic of the model. provide  It has been extended  information  both  for  recently, however, to decision  making  accountability in change efforts" (1974, p. 117).  and  Knowledge  of the decision situation is the central issue in the CIPP model.  Stufflebeam o r i g i n a l l y stated that, at a minimum, an  evaluator must know: 1.  who the decision makers are,  2.  what decision questions they must answer,  3.  what decision alternatives are to be considered,  4.  what c r i t e r i a are to be used in judging the alternatives, and  5.  the projected timing of the steps in the decision process.  In his 1969 a r t i c l e , he refined the categories of knowledge required about the decision situation and the following seven types of information:  cited  19 1.  the locus of decision making,  2.  the focus of the decisions,  3.  the substance of the decisions,  4.  the function of the decisions,  5.  the objects of the decisions,  6.  the timing of the decisions, and  7. . the relative c r i t i c a l i t y of decisions. The combination of these seven variables could be used  to  produce  a  multitude  of  decision  situations.  Stufflebeam felt the next stage of development should be a taxonomy of decision mutually exclusive.  situations which was  exhaustive  and  Such a c l a s s i f i c a t i o n scheme would then  be generalizable to a l l situations.  This was  accomplished  by classifying decisions as relating to either ends or means and  to  either  intentions  or  actualities  as  depicted  in  Figure 4. According to Stufflebeam four types of decisions suggest the  need for four types of evaluation,  terms context, Context  input,  evaluation  process  facilitates  and  product planning  were  hence the created.  decisions  by  examining such variables as the environment, the needs, and  20  FIGURE 4 Types of Decisions INTENDED  ENDS  MEANS  ACTUAL  PLANNING DECISIONS  RECYCLING DECISIONS  to determine  to judge and react to  obj ect i ves  attai nments  STRUCTURING DECISIONS  IMPLEMENTING DECISIONS  to design procedures  to ut i 1 i ze , cont rol , and refine procedures  (From "Educational Evaluation and Decision Making" by D.L. Stufflebeam. In B.R. Worthen & J.R. Saunders Educat i onal Eval uation: Theory and Practice, Worthington: Char! es A". Jones Publishing, 1973.  the opportunities.  This corresponds  closely to the needs  assessment component common to most adult education program planning models.  The result is information to assist in the  determination of objectives.  21 Input  evaluation  facilitates  structuring  decisions.  Given the program objectives, there are numerous  strategies  which  objectives. resources,  could  be  employed  These strategies are time,  to  achieve  appraised  budget, relevance  and  those  in terms  potential.  of The  resulting information guides the decision of which plan to select. Process  evaluation  implementing decisions. objectives. the  The  procedural  implementation  first design  provides  Process  of  is to detect or predict defects in or  i t s implementation  both  during  the  stages, the second is to provide information  the  (Stufflebeam, useful  for  evaluation has three main  for programming decisions, and the third record  information  procedures  1969).  The  and  events  information  is to maintain a as thus  for making adjustments while the  they  occur  provided  is  program is  running and for interpreting the results of the program.  Product  evaluation  This part of the Stufflebeam  serves  recycling decisions.  Model corresponds closely to  Tyler's approach to evaluation in that the emphasis is on attainment of objectives only to the exclusion of how or why this  occurs.  Essentially  it  is  a  process  of  operationalizing the objectives, assessing the attainment in  22  terms of the specified c r i t e r i a and comparing the results with some standard.  These a c t i v i t i e s  should  the program as well as at the conclusion.  occur during The results of  the context, input and process information provide the basis on which to interpret the product evaluation. COMPARISON OF EVALUATION MODELS Stufflebeam conceptual  bases  importance. evaluations  (1 969, for  If these  p. 45) has stated  evaluation  are  conceptions  of  are  that "the fundamental  faulty,  then  which are based on them must also be faulty".  This chapter has reviewed several definitions and four of the  leading  similarities  models  of  evaluation.  and differences  as well  The  models  as strengths  have and  weaknesses.  The Tyler model was reviewed not as a practical  alternative  for today's program evaluation  show the changes which have occurred years.  Michael  Scriven  has provided  needs, but to  in the past twenty the basis  for the  development of a theory of evaluation but much work needs to be done on his Pathway Comparison Model before used to guide evaluation studies.  i t can be  As Worthern and Saunders  (1973, p. 106) state, "the methods required  to r e l i a b l y  arrive at an overall appraisal have by no means been f u l l y specified.  Thus the practical  applications of Scriven's  23 suggestions, although appealing in the abstract, have not been f u l l y realized at this point in the development of the evaluation process". The models proposed by both Stake and Stufflebeam have been used as the conceptual  frameworks for developing  methodologies.  has  Steele  (1973)  listed  the  problem  situations in which each of these models is applicable. She groups these situations  into three types:  problems with  programming, problems in program management and problems in evaluation,  and  describes  their  applicability  as  shown  below. STUFFLEBEAM MODEL IS APPLICABLE IF I. You're having problems with programming with: 1. 2. 3. 4. 5. 6. 7.  setting p r i o r i t i e s , choosing among program p o s s i b i l i t i e s , developing objectives, selecting content and focus, examining factors affecting participation and leaving determining how you can use scarce resources most effect i vely, determining the results of programs. :  II. You're having problems with management with: 1. 2. 3. 4. 5. 6.  understanding the kinds of decisions involved in programmi ng, getting the most out of limited resources, deciding whether ideas for programs are good, understanding the relationship of evaluation to decision making, setting disputes when two or more methods or plans are advocated for doing the same thing, designing accountability strategies,  24 7.  developing a multicourse curricula or a multiactivity program, 8. assigning resources to staff units, 9. analysing weaknesses and problems in the operating of the unit, 10. working with an advisory committee, 11. improving use of time and other resources, 12. developing budgets, 13.  helping staff use evaluation as a management tool.  I I I . If you're having problems in evaluation: 1. improving staff attitudes toward evaluation, 2. forming judgments about programs, 3. organizing a comprehensive program review, accreditation team, etc. 4. determining what parts of the program to involve in your evaluation, 5. determining whether an extensive research effort is appropri ate, 6. determining the kind of data needed. STAKE MODEL IS APPLICABLE IF I.  You're having problems with programming with: 1.  understanding how program components interact to produce results, 2. examining the results of programs, 3. identifying key elements contributing to the success or failure of a program. II.  You're having problems with program management with: 1. using a systems approach to organizing programming, 2. considering input, process and outcome relationships of various parts of your program.  I I I . If you're having problems in evaluation: 1.  distinguishing between describing and evaluating program results, 2. determining what parts of the program to include in your evaluation, 3. handling a good deal of unrelated data and making sense out of how i t f i t s into patterns, 4. setting performance standards, 5. determining the kind of data needs. (From Contemporary Approaches to Program Evaluation: Implications for Evaluating Programs for Disabled Adults, by Sara Steele. Washington: Capital Pub., 1973).  25  Steele's  analysis indicates that Stufflebeam  has  produced a more comprehensive model, that is one which has application  in  a  greater  number  of  situations.  Stuff1ebeam s model also has the advantage of parsimony over 1  that of Stake.  This is p a r t i c u l a r l y true for the f i e l d of  adult education since Stuff1ebeam's Context, Input,  Process,  Product resembles the basic adult education program planning model of needs analysis, goal setting, design and management of  instruction  and  Stuf f l ebeam's model has  evaluation. been selected  For as  these the  reasons  basis  which to develop a methodology for the evaluation of projects.  upon ABE  The following chapter describes this methodology  which, while based on the work of Stufflebeam, an eclectic flavour.  c l e a r l y has  26 CHAPTER I I I  AN OUTLINE OF A METHODOLOGY Chapter Two outlined a number of leading models of program  evaluation.  Each  offers  a slightly  different  conceptualization of the evaluation process but a l l tend to address themselves to the questions  of "why" and "what"  while ignoring an equally important? "how".  The importance  of  of models is  theory  building  and the refinement  indisputable but i f evaluation role  in the program planning  is to assume i t s rightful process,  a methodology or a  taxonomy of methodologies must be developed.  Stufflebeam  (1969, p. 68) has stated: Once an evaluator has selected an evaluation strategy, e.g., context, input, process, product, he must next select or develop a design to implement his evaluation. This is a d i f f i c u l t task since few generalized evaluation designs exist which are adequate to meet emergent needs for evaluation. Grotelueschen et a l . have produced a design which can be used to guide the evaluation of programs which f i t the model of  a typical  ABE program.  This  typical  program i s  27  r e l a t i v e l y t r a d i t i o n a l , classroom-based, instructional and ongoing.  The  important areas of such a program are those  identified by the authors, namely, program emphases, program resources,  program  outcomes,  continuing  education and teacher evaluation.  professional  Thus whether the context  be classroom, l o c a l , state or federal this work has achieved its stated goal of examining "how activities  can  and why program evaluation  be related to improving instruction within  the local program context"  (p.v).  is on classroom instruction. design "does not  deal  The  As the  emphasis throughout authors state, this  extensively with  functions  counselling and budgeting that are vital related to instruction" (p.v). developed  by  Grotelueschen  but less d i r e c t l y  Thus, while et  al.  is  the  approach  valuable  evaluating the types of programs for which i t was it  such as  when  intended,  is not applicable to a l l the a c t i v i t i e s subsumed under  the term A.B.E.  Non-traditional a c t i v i t i e s such as many of  those funded by Training Improvement Project grants (Canada Employment Education  and Project  Immigration system  Commission)  (Ministry of  and  Continuing  Education) do  not  conform to the five administrative areas which are central to the work of Grotel ueschen et a l .  Such non-traditional  programs require a broader, more generalizable methodology which is anchored in a sound theoretical base.  28  The concepts  following  from several  evaluation  methodology  innovative  and  accomplished  section  of this  paper  models in an attempt flexible  non-traditional  enough ABE  has drawn  to design an  to  accommodate  programs.  by combining Stuff1ebeam's  This  was  CIPP model with a  series of five evaluation process a c t i v i t i e s (Figure 5). FIGURE 5 Outline of an Evaluation Methodology Evaluation Process Acti v i t i e s  Focus of the Evaluation Context  Input  Process  Product  Define boundaries Develop program Create a specific plan Perform evaluat i on Report findings  The five a c t i v i t i e s l i s t e d on the vertical axis result from a review of the literature regarding the steps, phases or  29 activities 1964;  which make up the evaluation  Fitz-Gibbon,  theorists  have identified  activities. in  1978; Grotelueschen,  process  (Verner,  1974).  Most  between four and eight discrete  The differences, however, result from variances  the definition  of what is included  in each  activity  rather than a substantive disagreement about what comprises the evaluation process.  As Verner and Booth (1964, p. 96)  state, "The evaluation  process is the same regardless of  whether i t is a total  program in a community or a single  institutional  session that  is being  evaluated.  The only  differences that appear are in terms of the variables to be considered and the scope of the task". EVALUATION PROCESS ACTIVITIES A fully  developed  evaluation  methodology would  require that each of the cells in Figure 5 contain a l i s t of all a c t i v i t i e s and decisions required at that stage of the evaluation and peculiar to that focus.  One advantage of  this approach is that i t allows the evaluator to define^ or limit the boundaries of the evaluation.  The four types of  evaluation (CIPP) may be used separately, in combination, or in t o t a l .  The specifics of each of the a c t i v i t i e s  vary  30  depending upon whether a context, input, process or produce evaluation  is  being  conducted.  The  remainder  of this section provides a general description of the issues which must be addressed at each stage. Defining the Boundaries of the Evaluation The f i r s t a c t i v i t y in the evaluation process is to define the boundaries of the evaluation.  In many situations  this will be the negotiations between the evaluator and sponsoring to  agency which lead up to an agreement or contract  evaluate.  should  the  be  Before  a mutual  either side  commits  itself,  there  understanding of what the  program is  about, what is expected of the evaluation, the  evaluator's  philosophy of or approach to evaluation, who the evaluation is  for, what the  relationship between sponsoring  agency,  program personnel and evaluator will be and the content and form of the evaluation reports. The collection  above process is comprised of three stages -  of data,  negotiation  and  achieving  consensus.  The negotiation and consensus steps consist of five steps identifying  audiences,  defining  focus,  defining  roles,  31 specifying  content and form and checking  interpretations.  Upon receiving a request for an evaluation proposal or an offer to conduct an evaluation, the evaluator should collect available  data  and become  program.  These  proposals,  annual  minutes  data  familiar  may  be  reports,  of meetings, second  with  studies.  A  source  associated  with the program.  (proposed)  in the form  budgets,  previous  the  needs  of program assessments,  evaluations and research  of data  is the  personnel  This includes the funding  agency personnel, the program personnel and present or past participants.  Once these data have been gathered and the  evaluator has at least a general overview of the (proposed) program, the negotiation  phase  can begin (Fitz-Gibbon,  1978). The identification of the audience is an important first  step of the negotiation  evaluation reports be addressed copies?  phase.  To whom will the  and who else will receive  This is more than a c l e r i c a l  question.  If the  report is addressed to the funding agency, there will be at least  the implication  function. director, function.  of a monitoring  If the report the implication  is addressed is of a  or accountability to the program  program  improvement  Thus, while any number of legitimate  audiences  32  may receive  the results of the evaluation  study, i t  is  imperative that the primary audience be i d e n t i f i e d . Once  the issue  of audience  identification i s  resolved,  the focus  and role of the evaluation  must be  defined.  Focus refers to the types of decisions which need  to be made - i f planning decisions, then context evaluation; if  structuring  implementing recycling 4).  decisions,  then  decisions, then  decisions, then  input  process  product  evaluation; i f  evaluation,  evaluation  and i f  (see Figure  The role of evaluation may be either formative for  program  improvement  or summative  for demonstration of  accountability and assessment of merit. These independent.  issues  are  interactive  Audience identification will  rather  than  influence and be  influenced by the role of the evaluation and both in turn will  affect the next issue - the relationship between the  evaluator and the project personnel.  The evaluator may be  an external, objective observer or a member of the program team participating in the program planning The final the  form  issue to be resolved  and content  process. at this stage is  of the evaluation  output.  The  33  evaluator may produce weekly, monthly, quarterly and/or end of project reports. formal  or  informal.  The reports may be written or o r a l , The  content may  be descriptive,  judgmental and/or recommendation making. Once a l l of the above issues have been resolved, the  evaluation  can proceed from a solid  understanding and common expectations.  base of mutual  Figure 6 summarizes  the process of defining the boundaries of the evaluation.  34 FIGURE 6 Defining the Boundaries of the Evaluation  1.  Collect program: a. b. c. d.  2.  audiences:  the primary audience the secondary audience those with a right to know  context input, process and/or product formative and/or summative  Define the  role of  the evaluator  and relationship  with  personnel.  Specify form and content of the evaluation output: a. b. c.  6.  (proposed)  documentation agency personnel personnel present participants  other project 5.  the  Define evaluation focus and roles: a. b.  4.  written funding program past or  and become familiar with  Identify the evaluation a. b. c.  3.  data  weekly, monthly, quarterly and/or end of project reports written or o r a l , formal or informal descriptive, judgmental and/or recommendation making  Check final  interpretation with program personnel.  35  Preparing the Program Statement The  program statement  is the working document on  which the evaluation study is based.  It contains basically  the program rationale, goals, intended and timeline.  a c t i v i t i e s , budget  If there exists a proposal for funding, then  some or a l l of this task may  be complete.  This is seldom  the case, however, since funding proposals are often based more on what the funding agency wants to hear than on what the  project  personnel  responsibility statement  of the  intend  to  do.  It  evaluator to validate  is  the  the program  i f one exists or to coordinate the preparation of  the statement  i f none exists.  This is a corroborative task  between the evaluator and the program personnel. The core of the program statement  is made up of  the program rationale, goals and intended a c t i v i t i e s . rationale states the need for the  program.  goals must be c l e a r l y stated and r e a l i s t i c . begin with Utopian goals. statement and  program  Programs often  preparation of the program  provides an opportunity to re-examine the goals  assure  activities  The  The  The  that should  they  are  attainable.  The  intended  be stated as s p e c i f i c a l l y as possible.  36 One test of the adequacy of a program's rationale intended  activities  is logical  contingency  goals and  (Stake, 1967).  Using Stake's terminology, one asks the question, "Can the intended outcomes be expected to occur as a result of the intended inputs and transactions?" A timeline and a budget are essential components of the program statement. valuable  planning  implementing  tools  If well and  constructed they are  provide  guidelines  for  the program and a c r i t e r i a against which to  judge the program as i t progresses. program statement  As stated e a r l i e r , the  is a working document and can be expected  to change as the program develops.  Figure 7 summarizes the  process of preparing a program statement. FIGURE 7 Preparing a Program Statement 1. 2. 3. 4. 5.  Clearly state the program rationale, goals and intended act i v i t i es Check the rationale, goals and intended a c t i v i t i e s for logical contingency Construct a program timeline Construct a program budget Check final program statement with program personnel  37  C reat i ng a Specific Evaluation Plan Given an understanding  of what is expected from  the evaluation and what the program is about, the evaluator can now construct a specific plan for the evaluation study. The f i r s t step is to identify the key issues.  If the focus  is on a product evaluation, the- issues will take the form of specific goals to  terminal  are  objectives.  long-term  measure  them  and  I f , however, the  complex, i t may  objectively.  evaluator must work from an  In  program  not be possible  this  situation,  impact model and  the  attempt to  measure the process goals and interim objectives.  An impact  model allows one to move backward' from an unmeasurable goal to  measurable a c t i v i t i e s and  the  goal  of  a  program  objectives.  is to  reduce  For example, i f the  incidence  of  i l l i t e r a c y by twenty percent, an interim objective might be to  produce a. specified, incremental  level  of a certain number of persons.  describe  the  planned  activities  interim objectives are to be met. goal would  increase in the reading  is to schedule attract  attendance.  ten  and  The  process  which must occur  goals i f the  An example of a process  advertise a l i t e r a c y course which  students  and  maintain  eighty  percent  The u t i l i t y of an impact model depends upon the  v a l i d i t y of the underlying assumption, that is that i f the  38  process goals are achieved then the interim objectives will be  achieved  and  ultimately  the  program  goal  will  be  achi eved. Not addressed where  a l l of the goals or a c t i v i t i e s  by an evaluation.  the  resources  productive.  The  of  need to be  Decisions must be made as to  the  allocation  evaluation of  the  will  be  resources  most  of  the  evaluation study should reflect the importance of the goal or a c t i v i t y to the overall program, the relative likelihood of  success, and  the  cost.  For example, one  would  not  allocate many resources to evaluating a r e l a t i v e l y minor component which has a high probability of success. other  hand, one  portion  of the  would  be  willing  to  use  a  On the  significant  resources to evaluate a crucial  activity  which had a lower probability of succeeding. When the key issues have been i d e n t i f i e d , the next task is to select the type of data which are indicative of success  or  failure the  and  audiences  of  question,  "What type  timeline  for  the  which  will  be  evaluation study. of data evaluation  should  credible  to  the  answers  the  collected?".  A  This be  answers  "when",  and  the  39  selection or construction of the instruments answers "how". The remaining component of the plan is the c r i t e r i a to be used for interpreting the data. can  In some cases, the c r i t e r i a  be stated very s p e c i f i c a l l y ,  least  one hundred  students will achievement determined  participants  for example, enroll at  or ninety percent  of the  gain at least two grade levels on a reading test.  In other  as s p e c i f i c a l l y  subjective standards.  cases,  and may  criteria  cannot be  require the use of  Figure 8 summarizes the process of  creating a specific evaluation plan. FIGURE 8 Creating a Specific Evaluation Plan 1.  Identify key issues  2.  Select credible data to be collected (the "what")  3.  Construct a timeline for the evaluation (the "when")  4.  Select and/or construct the instruments (the "how")  5.  Determine the c r i t e r i a for judging the data  6.  Check the plan with the program personnel  40  Performing the Evaluation  When accomplished,  the the  preced i ng  actual  implementation  of  a technical  task.  plan should be essentially collected  as per the timeline,  to the c r i t e r i a . encountered timelines altered,  will  need  be  goals redefined,  evaluation  The data are  is the ideal  More l i k e l y it  to  the  been  analysed and judged according  This, however,  in r e a l i t y .  have  act i vit ies  revised,  and budget  will  and is rarely be found that  intended altered.  activities This  is  one  of the values of evaluation as it provides the data base and mechanism reflected are as  for  rational  in the  not  made  part  of  change.  changes  should  program statement to assure that  arbitrarily the  All  overall,  and  independently,  integrated  summarizes the process of performing the  plan.  decisions  but  Performing the Evaluation  1.  Collect the data  2.  Analyse data  3.  Revise the evaluation plan as required  4.  Check analysis with the Program personnel  rather  Figure  evaluation.  FIGURE 9  be  9  41 Reporting the Evaluation Results All of the preceeding  a c t i v i t i e s w i l l be valueless  i f the results of the evaluation are those  in decision  credible  to the  making  evaluation  reports must be timely.  roles.  not  The  audience.  communicated to findings must  be  Additionally, the  Timeliness is especially important  for formative evaluations i f decisions are to be made while there is an opportunity to alter the program. should  have the  form  and  content  that  during the definition of boundaries phase.  The  reports  were agreed upon The role of the  evaluator in the decision making process should conform to agreement reached then.  Figure 10 summarizes the process of  reporting the evaluation results. FIGURE 10 Reporting the Evaluation Results 1.  Ensure c r e d i b i l i t y  2.  Ensure timeliness  3.  Form and content should conform to the agreement  4.  Assist decision makers in assessing required changes and examining alternatives  5.  Check total evaluation process with program personnel  42  For evaluation  ease  of  process  separately.  activities  They  independent.  discussions, are  each  have  not,  of  been  however,  the  five  dealt  with  discrete  The entire process is interactive.  Decisions  or changes made at any phase of the evaluation may any other phase.  or  affect  For example i f the focus of an evaluation  is process, the role is formative, and the audience is the project personnel then these decisions have an impact upon several other evaluation process a c t i v i t i e s .  The  evaluation  reports must be submitted in time to permit changes to the program, data collection cannot emphasize follow-up evaluation  procedures and  project personnel.  and  the  reports must be credible to the  Further i f there are significant changes  to the program statement in terms of the budget, objectives or  timeline, then  evaluation  this  timeline,  may  necessitate budget,  changes  data  to  the  collection,  instrumentation, and form and content of evluation reports. If educational by  providing  this  evaluation  is  to  have  an  impact  upon  practice, i t must begin to prove i t s usefulness information  goal, formative  for program improvement.  evaluation  should  be  given  Given priority  43  over summative evaluation. ABE  programs  which  are  This is especially important in evolving  and  developing.  If  evaluation results are to be u t i l i z e d , i t is essential that program personnel be aware and supportive the evaluation process.  of each phase of  The evaluator's role is to provide  the technical assistance and coordination necessary to make this  possible.  evaluation  of an  The ABE  use  of  this  methodology  Project conducted during  described in the following Chapter.  in  the  1979-80 is  44  CHAPTER  IV  INTRODUCTION The methodology outlined  in Chapter 3 of this  paper was designed for use in guiding the evaluation of ABE programs,  especially  programs.  innovative  or  non-traditional  The opportunity arose to test the u t i l i t y of the  methodology when the Continuing Education Division of the Ministry  of Education  evaluation  of a major  chose to l e t a contract for ABE project  implemented at a B.C. college.  being  the  designed and  This chapter w i l l  summarize  the a c t i v i t i e s which occurred prior to the commencement of the evaluation, describe the evaluation process a c t i v i t i e s and assess the u t i l i t y and efficacy of the methodology of guiding the evaluation study. SUMMARY OF PRE-EVALUATION ACTIVITIES In  1977 the Continuing Education Division of the  Ministry of Education s o l i c i t e d a proposal, from one of the province's colleges, for the development and implementation of an adult basic l i t e r a c y delivery system.  45  Previous only those  to  this  date,  programs sponsored  Immigration Commission.  the  by the  In the  and  IV l e v e l s .  Level  area of  through 8 was offered the  huge geographic  II,  area  of  at the  equates  to  community.  (one-third  incidence  offered  academic upgrading  Development  which  in only one  had  Canada Employment and  this meant Basic Training for S k i l l s III,  college  of  illiteracy  Grade 5  Thus,  the  II,  given  province), (14%),  the  relatively  high  and  the  traditional  lack of programs, the need for a non-traditional  delivery system seemed apparent.  Preliminary early  1977.  Official  project  was  when  motion  a  project  in  discussions  given  adult  recognition  by the  was  and  College  passed  basic  to  of  research the  need  Council  in  establish  literacy.  a  began for  such a  August,  1977  special  An early  in  draft  pilot of  the  project proposal was approved in principle by the council in September, state  of  Ministry  1978. limbo"  the  1979.  project  1979-80 f i s c a l budget.  in  officials.  January 10, to  The  proposal  November This  was 1978  year  March, was  pending  meeting  The College in  reported  did  Council  1979  included  and  to  in the  "in a  a meeting  not  occur  gave final  the  be  funding college  with until  approval for  the  operating  46 Since  the  project  and  fell  into  expensive,  was  ambitious  Stufflebeam's  in  decision  intent, making  setting of neomobilism (see p. 5), a large degree of change combined  with  low  Education  decided  information to contract  grasp, for an  the  Ministry  of  external, formative  evaluat i on. THE EVALUATION The following section, which describes the actual implementation  of the evaluation, is organized upon the five  evaluation process p. 28).  activities  described  in Chapter 3  Each of the process a c t i v i t i e s are referenced to  the appropriate figure in Chapter 3 so that the reader compare  (see  the  evaluation  methodology outlined there study  being  described  with  in this  the  may  actual  chapter.  The  numbers and letters in parentheses in the following sections refer to the items and sub-items found on each table. Defining the Boundaries of the Evaluation The  purpose of this  activity  was  to  focus  and  c l a r i f y what the evaluation was intended to do and for whom (see Figure 6).  The  first  step for the evaluator was  to  47  collect data and become familiar with the Project (1). To this  end,  interviews  were  arranged  with  Ministry of  Education (1-b) and College o f f i c i a l s (1-c), the most recent project proposal minutes  was reviewed,  (1-a) was obtained  a summary of College Board  and reviewed  as well  report (1-a) of the tour of a stage play designed  as the to raise  public awareness. Following  this  information gathering  exercise a  series of meetings with Ministry personnel resolved the next four  issues  (2-5) which  were  audience  identification,  definition of focus and roles, definition of relationships, and specification of the form and content of the evaluation out put. The primary audience of the evaluation was to be the College o f f i c i a l s of the proposal. Ministry  of  (2-a) charged with the implementation  The secondary audience was to be the  Education  officials  (2-b)  charged  with  monitoring the project and both the Ministry and the College had the right to share the evaluation findings with others (2-c) who where judged to have a legitimate interest.  48  Recalling Stuff1ebeam s four f o c i , context, input, 1  process and product, the decision was made that the study would be primarily a process evaluation (3-a) with enough emphasis on the product findings  in proper  focus  to be able  perspective.  to place the  The Context  and Input  components had been completed prior to the commencement of this  study.  A l l parties  agreed  on the need  for  the  development of an adult basic l i t e r a c y program (context) and the necessary resources resources  (inputs) appeared available. These  included the commitment of the College  i l l u s t r a t e d by the College Educational of  the College  Administration  charged with developing  Board as  Plan, the enthusiasm  and the faculty member  the proposal, and the a b i l i t y and  willingness of the Ministry of Education to fund the project over a two year period. The The  role of the evaluation was formative  evaluator's  role  was  (3-b).  to provide  consultative and  technical services to the project staff.  The intent was to  free the staff of the time-consuming and specialized duties of designing and implementing the evaluation and to provide a mechanism for accessing the distant resources of the lower mainland in general  and the University of British Columbia  49  in  particular.  The  accountability  remained  although some of the were  based  evaluation.  monitoring  with  data  Thus of  of  the  impressions  upon  between that  role  the  Ministry  created  collected role  independent  of  the of  Education,  the  made  formative  evaluators  external  for  and decisions  for  the  project  observers  (4) and  fell actual  members of the project team.  Since  the  evaluation  study  copies  the  to  Ministry of  the  College  administrator and the that  during  written describe  the  reports the  and  make  these  were  principal, project  to  be  the  year  (5-a)  (5-b)  would  be  during  funding  sent  of  It  the  recommendations  the (5-c)  intended  which  would  in  participants  -  the  College  the evaluator - the decisions  the  distances Ministry  and  maximize  likelihood of the project achieving the intended  Due to the geographic  question,  relationship  goals to  decided three  period  project  were discussed  plan the  goals.  between the of  with  project  and judgments regarding the  activities  the  college  was  submitted the  there  responsible  coordinator.  first  activities  making observations between  reports  Education was  three  Education and in a series of  50  bilateral  meetings  ( 6 ) but  The evaluator was able to occasions  for  establishment  a  total  nevertheless travel  of  time  behind  schedule  period  of  the  first  the  seven  visit  on three  days.  the  evaluation  cooperation this of  colleges  to the  This  this  study  made  the  was  seemed  seriously  always  to  be  While there was no overt  situation  and did  project  study  perceived as somewhat of a threat. of  by a l l .  of a trusting relationship d i f f i c u l t and since  by the  lack  agreed to  existed  have  an  throughout  impact  the  on how the  evaluation reports were received and acted upon.  Preparing the Program Statement  The program which  an  contains  statement  evaluation  study  is  is  the  based  working document on (see  program rationale goals,  the  Figure  intended  7).  It  activities,  budget and timeline.  The  project  proposal  requirements of a program statement  (1)  met  contingency  activities  (2),  of  that  the is  of  the  and constituted the plan  which the project personnel expected to follow. logical  all  rationale, "Can the  goals  intended  In checking and  intended  outcomes  be  51 expected to occur as a result of the intended transactions?"  inputs and  (Stake, 1973), i t was apparent, that despite  the resources, the stated goals were overly ambitious.  This  perception was communicated to the project coordinator but the decision was made to attempt to achieve a l l the goals. The timeline (3) and the budget (4) appeared well designed although  project a c t i v i t i e s were t i g h t l y scheduled  and would therefore require precise planning and effective management of time. It was agreed (5) that the project proposal would provide  the base line against  measure progress committed stated  itself  toward the goals. by contract  in the project  understanding  which the evaluation would While the college had  to perform those  proposal  there  was  activities  an  informal  that changes could be made by mutual agreement  as the project developed.  This  in fact  was one of the  responsibilities of the evaluator, to recommend revisions to the plan in order to maximize the attainment  of goals.  Creating a Specific Evaluation PI an The f i r s t step in creating a specific  evaluation  plan is to identify the key issues (see Figure 8).  In the  52 project in question there were three general goals (1), each with a series of related a c t i v i t i e s . Goal 1:  To increase the educational opportunities available to undereducated adults especially those who live and work in small, rural settlements, native Indian communities or who l i v e in a major centre but could not attend full-time classes.  Intended a c t i v i t i e s : 1. 2. 3. 4. 5.  to develop a curriculum for a tutor training course to train volunteer tutors to recruit 400 students to match tutors and students for purposes of instruction to provide support services and in-service training for the volunteer tutors 6. to establish and operate a "Laundromat Learning Centre" 7. to establish and operate an Oil Patch Camp Class Goal 2:  To develop public awareness of the need for and a v a i l a b i l i t y of adult basic education programs.  Intended a c t i v i t i e s : 1.  to develop a questionnaire for assessing educational needs 2. to recruit and train volunteers from the Women's Institutes to administer the questionnaire 3. to compile and assess the data 4. to plan and implement a publicity/awareness campaign including a newsletter, paper-graphics advertising, media advertising, stage play, tutor signs, and tutor merit awards. Goal 3:  To develop curriculum models and materials which will improve the quality of adult basic education programs within the college region and also have u t i l i t y for other regions of the province.  53 Intended a c t i v i t i e s : 1. 2. 3. 4.  to review existing curricula and learning materials and develop or adopt a rational, integrated curricula which w i l l be consistent throughout the college region to pilot test the above-mentioned curricula to publish the revised form of above-mentioned curricula to develop and publish other learning materials as required, in the areas of social orientation s k i l l s , foundation l i t e r a c y and work exploration and readiness. The  overall  project  goal  was  to  initiate  a  delivery system which would s i g n i f i c a n t l y reduce the level of i l l i t e r a c y in the college region. is  long-term  and  complex  within the  scope and  fact,  the  plus  evaluation,  Given that this  goal  i t s direct measurement was  not  resources of the  decision  determined  evaluation.  to conduct a formative,  that  this  study would  This process  concentrate  primarily upon the achievement (2) of interim objectives and the  actual  implementation  of  the  project  proposal  as  planned. The constructed  timeline to  (3)  parallel  for  the  development of the project.  the  proposed  Evaluation  evaluation  was  activities  and  reports were to be  submitted as soon as possible following September 30,  1979,  January 1,  plan  called  for  1980 the  and  April  first  1,  report  1980. to  The  evaluation  emphasize  curriculum  54 development, public  awareness and  project.  The  second  report  recruitment  and  training  and  the  management of  would  emphasize  the  final  the  tutor  report  would  summarize the entire f i r s t year and assess the impact of the project on the adults registered as learners. Given evaluation  the  and  the  geographic project  conducting an unobtrusive  distance  locale and  between  the  the  necessity  of  evaluation, i t was planned to use  the data (4) collected as an ongoing part of the project as the basis for the evaluation. and  post  tests  administering  administered  tests  For example, using the by  solely  for  the  tutors  the  pre  instead  purpose  of  of  the  The final task in the creation of a specific  plan  evaluat i on.  is  the  determination  Given that the objectives  and  of  focus was activities  criteria  for judging  on process and were  clearly  the  that the  data. interim  specified,  primary criterion was the performance of the task (5). example, the program statement called for the recruitment 400  adult  recruited.  students  and  the  criterion  was  the  the For of  number  Thus at this level of evaluation, the c r i t e r i o n  55 was the performance of the planned a c t i v i t i e s within the parameters of the budget  and the timeline.  At another  l e v e l , the c r i t e r i o n became the quality of the performance and an assessment of the impact these a c t i v i t i e s would have on the long term goal  had or  of reducing the level of  i l l i t e r a c y (5). This secondary level is more subjective and requires  the development  b r i e f l y describes the  of an impact model.  Figure 11  the model which was designed to reflect  relationship between  the project  and i t s  activities,  interim objectives and goals. FIGURE 11 Sample Impact Model Act i v it i es  Interim  Objectives  Example 1. raise the reading 1. recruit 400 tutors level of 400 adults by two 2. recruit 400 grade levels adult learners 3. train tutors 4. match tutors and learners 5. support tutor/ learner pairs  Goal Reduce level of i l l iteracy within the college region  56  The model  predicts  that  i f the a c t i v i t i e s are  performed as planned then the interim objectives  will  be  achieved and those will contribute to the achievement of the goal. to  The c r i t e r i a then were the measurement of the degree  which  the a c t i v i t i e s  were performed, the objectives  achieved and subjective assessment of the v a l i d i t y of the underlying impact model. Performing the E va1uat i on In actual  Chapter 3 (p. 30) i t was stated that, "the  implementation  of the evaluation  essentially a technical task. the timeline, analysed (see Figure 9).  plan  be  The data are collected as per  and judged according  to the c r i t e r i a  This, however, is the ideal and is rarely  encountered in the real world of education" 30).  should  (Chapter 3, p.  True to this prediction neither the project nor the  evaluation proceeded as planned.  Two a c t i v i t i e s which were performed prior to the beginning  of the project were successful.  creation of a stage play which depicted adult i l l i t e r a t e  These were the the plight of an  in today's society and the operation of a  57  volunteer tutor training program. were new to the college  Both of these a c t i v i t i e s  and the college  retrospect, the success of these high  personnel.  profile  In  activities  appear to have consumed the energy and enthusiasm of those charged with the management of the project.  This combined  with several developments which are described  in the f i r s t  evaluation  report  (see Appendix A) resulted in the project  getting off to a slow start. The evaluation plan called for the f i r s t report to examine  curriculum  development,  the  public  campaign and the management of the project. report was to emphasize tutor recruitment  awareness The second  and training while  the third report was to summarize and assess the learning of the students.  Due to the planned a c t i v i t i e s which did not  occur or occurred only in part, the plan of the evaluation study was altered considerably.  The data c o l l e c t i o n (1)  process became one of chronicling the a c t i v i t i e s which had occurred.  The analysis  (2)  was a comparison  of these  a c t i v i t i e s with those of the program statement and examining underlying  reasons for the lack of progress.  This analysis  produced data which were useful for identifying alternative actions for the consideration of the project personnel and for the revision of p r i o r i t i e s .  58  The  type of data u t i l i z e d  and the source of the  data differed greatly from the evaluation  plan.  The plan  called for the evaluation to u t i l i z e data generated by the various project a c t i v i t i e s , such as pre/post tests, student p r o f i l e s , dropout rates, but the fact that the project was chronically behind schedule and that many of the a c t i v i t i e s never occurred  shifted the emphasis of the evaluation.  The  main issue became the management of the project, and the main source of data collection became interviews with the project  staff  and  college  and  Ministry  of  Education  o f f i c i a l s (3 ). The  development described  above greatly affected  all aspects of the evaluation study including the content of the evaluation reports. Reporting the Evaluation Results The  evaluation  reports  (see Figure  10) were  submitted in the agreed format (3) and close to the planned timeline  (2).  In each report  recommendations (1,4) were  made which sought to assist the project personnel in setting both short term and long term p r i o r i t i e s . contained  in Appendix A.  The reports are  59  SUMMARY AND OVERVIEW Neither the project nor the evaluation proceeded as planned. did  As the evaluation reports describe, the project  not accomplish many of the stated goals.  cause of this  situation  was the failure  The immediate  to i n i t i a t e and  implement many of the planned a c t i v i t i e s . not  hired, curriculum  Personnel were  was not developed and the intended  numbers of tutor and learners were not recruited. The flexibility  evaluation  study  in part  of the methodology described  responded to the changing conditions  due  to the  in Chapter 3,  and produced results  which did have a positive influence on the project.  Some of  the recommendations were acted upon and, as in the famous Hawthorne studies, the mere presence of an outside observer may have resulted in greater productivity. As facilitates provide,  stated  summative evaluation.  for those  information  earlier, who  have  on the development  formative  evaluation  The evaluation  access  to them,  reports valuable  and implementation  of a  volunteer-tutor l i t e r a c y program and i l l u s t r a t e some of the p i t f a l l s of decentralized curriculum development.  60  In  conclusion,  the  methodology  conceptualization  of the evaluation  description  the  of  components of that combined  the  (Stufflebeam's)  activities  process.  strength with  the  process as well  as a  which  were  essential  The methodology successfully  of  the  provided  a  sound  practicality  evaluation of  a  model  series  of  evaluation process a c t i v i t i e s . The following chapter summarizes this makes recommendations  study, and  for the further development of the  evaluation of ABE programs.  61  CHAPTER  V  SUMMARY AND CONCLUSIONS This concluding chapter is divided into two major sections.  The f i r s t section summarizes the purpose of the  study, the development of the evaluation methodology and the pilot test. the  The second section draws and b r i e f l y discusses  conclusions  resulting  from  this  study  and makes  recommendations for further research. SUMMARY In  British  Columbia, ABE has evolved  into a  significant program area on the verge of gaining acceptance as a legitimate and important part of the public educational system. hoc  This development, however, has occurred  manner  uncoordinated, unstated committed  and resulted  in a diverse  ill-defined  or fuzzy to these  goals. programs  programs  in an ad  collection  seeking  to  If the resources  of  attain  currently  are to be s o l i d i f i e d and  increased, i t is imperative that the present program impact be measured, that needs, resources, processes  and outcomes  62  be  articulated  and  that  programming be predicted.  the  benefits  of  increased  This is best accomplished through  the effective use of program evaluation.  The state of development of evaluation for ABE has inhibited  practitioners from  technology  taking  for decision-making  advantage  of this  and accountability.  f i e l d of evaluation may be viewed as an hierarchy. base are checklists and as one moves  The At the  up the hierarchy  through designs, methodologies, models, and theories  there  is an increased a b i l i t y to apply the technology to a range of educational  situations.  S p e c i f i c i t y of guidelines and  evaluation a c t i v i t i e s , however, decrease as one moves up the hi erarchy.  Simplistic design and checklists are seldom linked to any of the leading models and tend to encourage the compilation  and  necessarily  addressing  evaluation  have  including Scriven,  analysis been  of  important developed  existing  data  issues. by  Models  several  Stake and Stufflebeam.  without of  researchers  These models,  while of interest to academics and students of evaluation,  63  provide  l i t t l e guidance to educational  methodologies  exist  which  are  practitioners.  useful  innovative, non-traditional ABE programs. theory of evaluation has yet to evolve.  for  Few  evaluating  A well developed Scriven's  work on  some of the more philosophical issues comes as close as any in beginning  the  theory  development  process.  Thus, the  f i e l d of program evaluation is characterized by a lack of sound theory,  a series of complex models, an  absence of  methodology and an abundance of designs and checklists which are not tied to a sound theory or model.  This  study  reviewed  and  models of evaluation, identified  analysed  the  the  leading  CIPP model as  being  superior, and developed an evaluation methodology based upon this model.  Five evaluation process a c t i v i t i e s were defined  which are general enough to be applicable to most evaluation a c t i v i t i e s while providing sufficient specific guidance to be  of  assistance  activities matrix  the  were crossed  which  outline  to  outlines  assists  in  informed  with the an  generalist.  CIPP model to produce a  evaluation  defining  the  These  methodology. focus  of  a  This proposed  evaluation study, either context, input, process, product or some  combination  of  the  four.  Each  of  the  process  64 a c t i v i t i e s was expanded to a l i s t of issues which should  be  addressed by an evaluation study. While each of the process a c t i v i t i e s was  described  separately, they are not discrete or independent.  They are  interactive and decisions made at any phase may other  phase.  approach  represented  by  the  methodology attempts to maximize the  likelihood  that  an  evaluation  The  overall  affect any  study is timely, credible, supported  involved in the program being evaluated,  by those  and useful to the  decision makers identified as the primary audience. Chapter 4 b r i e f l y described  a pilot test of the  methodology when i t was used to evaluate an ABE project that included  curriculum  non-traditional organized The  development,  delivery methods.  according  public  awareness,  and  This  description  was  to the outline presented in Chapter 3.  focus of the evaluation was  primarily on process with  sufficient emphasis on the product to allow the findings to be placed  in proper perspective.  evaluated  accomplished  methodology provided guide the evaluation  few  of  a useful process.  While the project  i t s stated and  being  objectives, the  flexible  structure  to  65  CONCLUSIONS The conclusions development Chapter  discussed  of the evaluation  3 and the evaluation  below result from the  methodology study  described  which  in  piloted the  methodology and is b r i e f l y described in Chapter 4 . On the positive side, i t is concluded that the methodology i s a useful tool for developing the evaluation process. model  an overview of  The methodology is grounded in a  (Stuff1ebeam's) which is as well developed as any  currently  available.  It further describes  a series of  a c t i v i t i e s which interact with various parts of the model. This combination p a r t i a l l y f i l l s the void which has existed to  date  between  the developing  theories  and models of  evaluation and simplistic checklists which are not based on any theory or model.  Thus the methodology goes part of the  way toward linking theory and practice. The  methodology  is especially useful  at the  beginning and at the end of an evaluation study, that i s , in focusing the evaluation plan and in reporting the results. Too often evaluations consist of collecting and summarizing  66  available data.  Using the matrix  produced by crossing the  evaluation foci and process a c t i v i t i e s , evaluators  will be  assisted in more c l e a r l y defining the nature and purpose of the study and will be better able to present their findings to the audiences of the evaluation. Evaluation, especially formative evaluation, loses much of i t s p r a c t i c a l i t y i f i t is not responsive needs of those who will  to the  use the results and i f i t is not  sensitive to the changes which occur in most programs as they develop.  Further, i f those involved in a program are  not supportive  or at least tolerant of an evaluation, the  task is complicated.  Properly implemented, the methodology  does maximize the likelihood that a study will be responsive to  needs, sensitive to changes and receive  program personnel. and the still  This approach is built into a structure  is i l l u s t r a t e d by the concluding  process stages.  support from  items in each of the  While every effort was made to communicate  purpose and plan  of the evaluation, the process was  perceived as a threat by the personnel of the project  on which an evaluation was conducted. described  Evaluation has been  as the "whore of the establishment"  1972 , p. 1) and this  (Scriven,  attitude persists among many adult  67  educators.  It is safe to conclude that evaluators need to  go to extraordinary lengths in their attempts to secure the support of a l l those involved in a program being  evaluated.  Despite the resulting d i f f i c u l t i e s , the methodology provided a f l e x i b l e structure for guiding the evaluation of a program which  strayed  significantly  from  the  original  plan  and  t i metable. On  the  negative  side, the methodology is not a  f u l l y developed package for use by those uninitiated  in the  l i t e r a t u r e and jargon of program evaluation.  If this work  is to have an impact on adult basic education  programs, i t  could be achieved through a workshop or series of workshops involving  those most  evaluation.  likely  to  be  charged  with  program  From a British Columbia perspective, this group  would consist of the directors of adult basic education from f i f t e e n community colleges and some twelve to fifteen school boards.  The explanation and discussion of the methodology  by  a group over a two-day period  such  significant  improvement of  both  quality  evaluation studies in this province.  could and  result  in  quantity  of  68 The studies  result of a search  in the area  systematic  for exemplary evaluation  of ABE was a realization  that few  evaluations are conducted and even fewer result  in enhancing the degree to which informed decisions can be made or accountability documented. marginal  nature  Given the t r a d i t i o n a l l y  of adult basic education  programs and the  p o s s i b i l i t y of a period of prolonged economic restraint, i t is imperative that evaluation move away from the case study and "happiness index" approach and toward a more systematic and sophisticated evaluation plan. Further  development  of the methodology  should  await several f i e l d tests which could f a c i l i t a t e creation of a comprehensive l i s t  of questions  or decision points for  each of the cells of Figure 5 (see page 28).  This  list  would, in effect, be an expansion of each of Figures  6-10  specifically product)  for each  foci  of evaluation.  development as producing  (context, Stufflebeam  input, process and described  such  a  a taxonomy of decision situations  which is exhaustive and mutually exclusive.  This study has  taken the f i r s t step toward this end, but much work remains to be done.  It is hoped that this methodology will be  69  pilot  tested  on  a  variety  of  ABE  programs  and  that  additional research w i l l result in strengthened ties between the best theories and models and the f i e l d of practice.  70 BIBLIOGRAPHY  Alkin, Marvin C. and Fitz-Gibbon, Carol. "Methods and Theories of Evaluating Programs." Journal of Research and Development, Spring 1975, 8 (3), 2-15. Bolton, Elizabeth B. "Towards a Practical Approach to Evaluating Adult Education Programs: An Application of Scriven's Methodology." Community/Junior College Research Quarterly, 1977, 1_ 409-420. Dickinson, Gary. Undereducated Adults in B r i t i s h Columbia1976. Vancouver: Department of Adult Education, The University of British Columbia, 1979. Dickinson, Gary. The Undereducated of B r i t i s h Columbia. Vancouver: Department of Adult Education, The University of British Columbia, 1978. Dickinson, Gary. "Judgmental Aspects of Program Evaluation". Adult Training, 1977, 2 (4), 26-29. Dickinson, Gary and Lamoureux, Educative Temporary Systems". XXV (2), 81-89.  Marvin. "Evaluating Adult Education, 1975,  Faris, Ron (Chairman). Discussion Paper 01/79: Report of the Committee on Adult Basic Education. Information Services, MinistTy of Education, Science and Technology, Province of British Columbia, 1979. Faris, Ron (Chairman). Report of the Committee on Continuing and Community Education in B r i t i s h Columbia Victoria: Ministry of Education, Province of British Columbia, 1976. Farmer, James. "Strengthening a Design for Evaluating a Broad-Aimed Functional Literacy and Family Life Education Program." Literacy Discussion. 5 (3), 421-40. Fitz-Gibbon, Carol T. and Morris Lynn Lyons. Evaluation Kit - t i t l e s : Evaluator'sHandbood ffow to Deal with Goals and Objectives How to Design A Program Evaluation fTow~to Measure Program Implementation How to Measure Achievement How to Measure Attitudes How to Cal cu 1 ate~~Stat i st ics How to Present An Evaluation Report Beverly H i l l s : Sage Publications, Inc., 1978.  Program  71 Forest, Laverne. "Program Evaluation For Reality". Education, Spring, 1976, 26_ (3 ), 167-1 77.  Adu1t  Grotelueschen, A.D., Gooler, D.D., Knox, A.B., Kemmis, S. , Dowdy, I. and Brophy, K. An Evaluation Planner. Urbana: University of I l l i n o i s , 1974. Groeleuschen, A.D., Gooler, D.D. and Knox, A. B. E yaluation in Adult Basic Education: How and Why. Danvi11e: The Interstate Printers and Publishers, Inc., 1976. Knox, A.B., Mezirow, J., Darkenwald, G.G., and Beder, H. An Evaluation Guide for Adult Basic Education Programs. New York: Center for Adult Education, Teachers College, Columbia University, 1972. Mezirow, Jack. Evaluating Statewide Programs in ABE: A Design with "Tnst rument at i on. iTew York : Center for Adult Education, Teachers College, Columbia University, 1975. Mezirow, J., Darkenwald, G., & Beder, H. An Evaluation of Adult Basic Education in the State of Iowa. New York: Columbia University, 1 975. Niemi  John A. and Anderson, Darrell V. "Remedial Adult Education. An Analytical Review of Evaluation Research." Continuous Learning, 8: 90-94.  Rossi, P.H. and Williams, Walter (Eds.) Evaluating Social Programs: Theory, Practice and P o l i t i c s . New York: Seminar Press, 1972. Saylor, J.G. and Alexander, W.M. Planning Curriculum for Schools. New York: Holt, Rinehart and Winston, Inc., 1966. Scriven, Michael. "The Evaluation of Educational Goals, Instructional Procedures, and Outcomes or the Iceman Cometh". ERIC Document Reproduction Service No. ED 079 394, 1972. Scriven, Michael. "The Methodology of Evaluation". In Worthen, B.R. and Sanders, J.R. (Eds.), E ducat ional Evaluation: Theory and Practice. Worthi ngton, Oh i o: Charles A. Jones Publishing Company, 1973. Scriven, Michael. "Prose and Cons About Goal-Free Evaluation". Evaluation Comment - III (4 ) 1 - 8. Shearon, Ronald. "Evaluating ABE Leadership 1_9 (1), 15-16; 20-21.  Programs".  AduIt  72 Stake, Robert. Evaluating the Arts in Education: Responsive Approach. Columbus Ohio: Charles M e r r i l l Publi shing Company, 1975.  A E.  Stake, Robert E. "The Countenance of Educational Evaluation". Teachers College Record, April 1967, 68, 623-40. Steele, Sara. Comtemporary Approaches to Program Evaluation: Implications for Evaluating Programs for Disadvantaged Adults. Washington: Capitol Pub., 1973. Steele, Sara. "Program Evaluation - A Broader Definition". Journal of Extension, Summer, 1970 pp. 5-17. Stufflebeam, Daniel L. "Educational Evaluation: Some Questions and Answers". A Paper Presented at the Annual Meeting of the American Association of School Administrators, Atlantic City, New Jersey. ERIC Document Reproduction Service No. 128 425. Stufflebeam, Daniel L. "Evaluation as a Community Education Process". Community Education Journal, March/April 1975, 5 (2), 7-12; 19"^ Stufflebeam, Daniel L. "Alternative Approaches to Educational Evaluation: A Self-Study Guide for Educators". In James W. Popam (Ed.), Evaluation in Educati on: Current Applications. Los Angeles: McCutchan Publishing Corporation, 1974. Stufflebeam, Daniel L. Educational Evaluation and Decision Making. Itasea, I l l i n o i s : F~.~E~. Peacock Publ ishers , Inc. 1971. Stufflebeam, Daniel L. "Evaluation as Enlightenment for Decision Making". In W.H. Beatty, (Ed.), Improvi ng Educational Assessment and an Inventory of Measures of Affective Behaviour. Wash i ngton, D.C.: Association for Supervision and Curriculum Development, NEA, 1969. Thomas, Audrey M. Adult Basic A c t i v i t i e s in Canada 1975-76. of Canada, 1976. Tyler, Ralph W. Instructi on. 1950.  Education and Literacy Toronto: World Literacy  Basic Principals of Curriculum and Chicago: University of Chicago Press,  Webster's Seventh New Col 1egiate Dictionary. Thomas Allan and Som Limited, 1972.  Toronto:  73 Wort hen, Blaine and Saunders, James. Educat i ona1 Evaluation: Theory and Practice. Worthington, Ohio: Charles A. Jones Pu b 1 i s h i ng~, 1973. Verner, C. and Booth, A. Adult Education. New York: Center for Applied Research in Education, 1964.  The  74  APPENDIX A  Appendix A consists of the three reports produced as a result of the evaluation study described in Chapter 4. Names and  locations  have  been  deleted  anonymity of individuals and i n s t i t u t i o n s .  to  preserve  the  75  FORMATIVE EVALUATION OF THE ADULT BASIC PROJECT  Report for the Quarter Ending September 30, 1979  EDUCATION  76 The  Department  of  Adult  Education  at the  University of British Columbia was requested by the Division of  Continuing  Education  of the Ministry of  Education,  Science and Technology to serve as external evaluator for a pilot  project in adult basic education  (  ) College.  being developed by  As specified in the agreement between the  Department and the Division, this document is the f i r s t of three  formative  evaluation reports to be submitted.  purpose of the report  is to review  the progress  The of the  project to September 30, 1979 and make recommendations for i t s improvement. In preparing this report, the evaluators  reviewed  documents provided by the Division and the College, met with the  project s t a f f ,  and consulted  Ministry o f f i c i a l s .  other  College and  Readers of this report should bear in  mind that the project is in i t s months having  with  elapsed  early stages, with six  in a proposed two year  span.  The  observations, judgments, and recommendations made herein are intended project's  to f a c i l i t a t e  the successful  objectives  than  rather  attainment  of the  to impede or halt i t s  progress. Official was  recognition of the need for the project  given by the College Council  in August, 1977 when a  77 motion was adult  passed to establish a special pilot project in  basic  education,  to  be  funding from the Ministry.  supported  by  supplementary  An early draft of the project  proposal was approved in principle by Council in September, 1978 pending a meeting with Ministry o f f i c i a l s which did not occur until  January 10, 1979.  College Council gave final  approval to the project in March, 1979 1979-80 fiscal  year was  and funding for the  included in the College  operating  budget. The Educational  intent of the project is in keeping with the Master  Plan  of the  College.  That document  recognized the fact that many of the region's residents have l i t t l e formal to  education.  serving the  The  educational  College has committed i t s e l f needs of  a l l area  residents  through part-time as well as full-time programs in smaller communities and  rural  areas as well as the major centres.  Non-traditional types of program delivery are acknowledged in the Educational Master Plan as being both desirable and necessary  given  the  geographic,  conditions of the region. in March, 1979  climatic,  and  population  In addition, the College Council  identified adult basic education  program p r i o r i t y for 1979-80.  as the top  It appears, then, that the  project has a firm grounding of support at the policy level within the College.  78 The programs  emphasis  appears  percentage of  given  justified  residents  to  in  who  light  have not  eight years of schooling.  As the  1,100  have  adults  (4  adult  percent)  basic  of  number  and  attained more than  1976 five  the  education  Census indicates, or  less  years  of  schooling and 6,295 (22 percent) have five to eight years of schooling.  thus, there are approximately 7,500 adults  are potential participants in the project.  who  Rural and native  Indian adults were identified as the principal target groups in the  project proposal,  and  this  is consistent  with  the  College policies noted above. The prior  to  the  preparatory. education  College  had  current  engaged  project  in  which  several  activities  can  viewed  be  as  During 1978-79, the College ran adult basic  programs at eight locations.  With the  exception  of two programs conducted on Indian reserves, the remainder were located  in College  facilities.  A total  programs were conducted involving 254 students.  of  sixteen  The range  of programs included BTSD levels I I , I I I , and IV, BJRT, GED, BEST, EOW,  REPLACE, Pre-technical , and Tutor Training.  of the students attended full-time.  Most  79 The stage play, (  ), was written and produced  under a separate agreement prior to the commencement of the project,  although  i t became  a major  project's public awareness campaign.  component  of the  The play was sponsored  by the Ministry and developed j o i n t l y by the College and (  ).  The play was designed  to introduce audiences to  the concept and magnitude of adult i l l i t e r a c y in society. Through a series of scenes emphasizing personal  accounts,  the play i l l u s t r a t e s the dilemma faced by the non-reader in a print-oriented society. awareness  of adult  It attempts to increase public  illiteracy  and to stimulate  a more  positive attitude toward the person who wants and needs to learn to read.  The tour consisted of 23 performances in the  period April through June, 1979. students (  and 450 adults.  It was seen by some 2,000  The reviews of and reactions to  ) were generally very favourable. A third  preparatory a c t i v i t y was the training of  29 tutors, of whom 19 completed training, in ( separate  contract  with  the Ministry.  That  ) under a program was  completed in A p r i l , 1979 so that the prospective tutors were available for matching with students at the time the project c ommenced.  80  The  goals  major categories:  of the project are divided  in three  public awareness, curriculum development,  and outreach development and delivery. Each of these goals, together with their derivative objectives and a c t i v i t i e s , is discussed below in terms of progress to date. The  goal  of  increasing the level  public  awareness  of public support  was  aimed  at  for the proposed  programs and encouraging adults in need of basic  education  to come forth and participate in project a c t i v i t i e s .  The  major vehicle for achieving this goal was the play (  ).  Public awareness was also increased by radio and newspaper advertising, development of a logo, posters, and speeches made to fraternal and community organizations.  A proposed  survey of rural areas and communities has been deferred to a later date.  The goal  of curriculum development involved the  rationalization of existing curricula being used throughout the College region, the development of materials in the area of social orientation s k i l l s , and the development of a tutor training curriculum. curricula  used  To date, the adult basic  in various  College  centres  education have  been  81 reviewed and plans have been made to integrate them.  The  social orientation s k i l l s materials await the appointment of a  new  project  staff  member.  are being  developing  one that is appropriate for the region. goal  with  tutor training  curricula  The  studied  Several  of outreach  a view to adapting  development  or  and delivery  involves the design and implementation of a program delivery system to f a c i l i t a t e areas  of the region  component of this program. trained  participation by adults in the rural on a part-time  delivery system  basis.  is a volunteer  As was noted e a r l i e r , the College 19 tutors,  and those  The main  tutors  were  tutor  had already paired  with  registered part-time students in A p r i l , 1979. Following the summer period,  16 tutors  tutoring  activities.  scheduled  to begin  were ready to recommence their  Six tutor  training  courses are  in September and October, and i t  expected that 75 tutors w i l l  be trained and working  is  with  adult learners by the end of 1979. Other proposed Learning  activities  included  Centre and an Oilpatch Camp Class.  a Laundromat There has been  no v i s i b l e progress to date in the establishment  of either  82 of those a c t i v i t i e s .  In May, 1979 a joint Project-Community  Education Services program was proposed for ( The  intent  was  to supplement  Services  program  with  project  Request  for Additional  ) Lake.  the Community Education funding  Course  and support.  proposal  has  The  not been  approved, however, and i t is unclear whether that a c t i v i t y w i l l be pursued. Since the project commenced in A p r i l , 1979, the major  substantive  establishment  accomplishments  of an advisory  have  committee,  been  the  the design of  administrative procedures such as a tutor registry, student performance measures, and tutorial centres, and the hiring of a second developer.  staff  member, an instructor  and curriculum  In addition, 19 tutor-student pairings were made  and operated from April through June, 1979. A comparison of the proposed and actual timetables shows  clearly  schedule.  that  the project  is considerably  behind  Two project staff were scheduled to be hired in  A p r i l , 1979. One was hired in June, but the other position was s t i l l  vacant as of September  21.  As a result of the  delays in staffing the project, some eight potential personmonths of productivity have been lost.  Given that much of  83  the period from April through August could have been used for planning  and  curriculum  development, those  activities  must now be undertaken between September and November which would  normally  be  peak  periods  for  the  delivery  of  i nst ruct i on. The  immediate reason for f a l l i n g behind schedule  was the lack of staff available in the early months of the project.  That staffing was  not completed earlier seems to  be a result of a number of events within the College which led to uncertainty on the part of the project coordinator. Those  events,  including the  Principal and the withdrawal  resignation  of  of sponsorship  the  College  of other adult  basic education programs by CEIC, apparently left the status of the project in an ambiguous position.  If  the project is  to approach i t s original objectives and timetable, i t w i l l require  strong  leadership  and  support  from  within  the  College. Based on the  observations  e a r l i e r , the evaluators recommend:  and  conclusions  noted  84  1.  That  the  assume  College  appoint  responsibility  activities  within  difficulties  for  the  a  senior  administrator  a l l adult  basic , education  institution.  encountered by the  to  Some  project  of  the  seem to have  stemmed from the lack of a clear line of authority and a confusion  of roles and  responsibilities.  The  a c t i v i t y and range of programs in adult basic  level  of  education  may be substantial enough to call for the sole attention of a senior o f f i c i a l within the College. 2.  That the staffing of the project be completed as rapidly as  possible  qualified  consistent  personnel.  education  has  without i t s f u l l complement of 3.  of  the keystone to the success of an adult basic project  and  availability  staff is  this  able  the  dedicated  program, and  An  with  operated  for too  long  personnel.  That the objectives and a c t i v i t i e s be reduced to those regarded as essential to the success of the  project.  This could result, for example, in the indefinite of  such  activities  Bookmobile, and could  reduce  the the  as  the  Oil Patch overall  community Class.  potential  survey, Although  impact  of  delay the this the  85 project, i t would enable the staff to focus on a smaller number  of  concerns  which  would  have  a  greater  probability of success. That the project staff concentrate i t s attention on the third goal, outreach development and delivery. time,  public  awareness  activities  such  By this  as the play  should be essentially completed, and the attention of the staff should focus on delivery instruction to the target populations. That the enrolment current projected  year  target of 400 participants  be reviewed.  enrolment  Only  5 percent  has been attained  in the of the  halfway through  the f i r s t year, and i t may be unrealistic to expect that goal to be reached by March 31, 1980 and s t i l l maintain a high quality of instruction. That the Project Advisory Committee be reactivated and assisted in defining i t s functions and responsibilities more clearly.  The Committee could probably be used more  effectively than i t has been to date, and could serve such  purposes  as  increasing  the v i s i b i l i t y  of and  «6  support  for the project  in the community, recruiting  participants, and improving the services provided by the project staff through the members' fami 1 i a r i t y with the target populations. 7.  That o f f i c i a l s of the Division of Continuing Education and  the  College  meet  to  review  and  clarify  objectives and attainments of the project to date.  the As  this is a pilot project with implications for extension to  other  regions, i t is important  that  continuous  program monitoring be conducted so that a l l parties can learn from the experiences of this i n i t i a l program.  87  FORMATIVE EVALUATION OF THE ADULT BASIC PROJECT  #2  Report for the Quarter Ending December 31, 1979  EDUCATION  88  INTRODUCTION The  Department  of  Adult  Education  at the  University of British Columbia was requested by the Division of  Continuing  Education  of the Ministry  of Education,  Science and Technology to serve as external evaluator for a pilot project in adult basic education being developed and implemented by ( agreement  ) College.  between the Department  As specified  in the  and the Division,  this  document is the second of three formative evaluation reports to be submitted. , The purpose of these reports is to review the progress of the project and to make recommendations for i t s improvement. In preparing this report, the evaluators reviewed documents  provided  by  the  College  and  the  Division,  interviewed project s t a f f , volunteer tutors and a student, observed tutor training classes in operation, and consulted with other College and Ministry o f f i c i a l s .  Readers of this  report should be aware that the project is approaching i t s mid-point  with nine months having  twenty-four month span.  elapsed  in a proposed  89 PROGRESS: The major  OCTOBER 1 - DECEMBER 31, 1979  goals for the project are divided into three  categories:  curriculum  outreach  development  and  development and public awareness.  goals is discussed  in the following  delivery,  Each of these  section  in terms of  progress during the period October 1 - December 31, 1979. The  goal  of outreach  development  and delivery  involves the design and implementation of a delivery system to  provide  educational  service  identified in the project  to the target  groups  proposal.  On November 26 a contract person was hired for two weeks to contact  those tutors  trained  in April  and to  attempt to pair them with c l i e n t s . Seven  tutor  training  classes,  seventy-four participants, were started covered by this report. ,  , and  during  involving the period  The classes are located in (  ,  ). Graduates of these classes w i l l be  ready to commence tutoring  activities  in late  January.  Plans have been made to train approximately twenty students  90  in regular adult basic education college programs to act as tutors. and  Full-time college ABE instructors from (  (  ) Lake  ) have received six days of orientation to tutor  training  and are prepared  to offer  a "Tutor  Training"  component in the ABE curriculum. An Outreach Tutorial Centre has been established in a t r a i l e r located on the (  ) campus.  This  facility  is intended to provide a location for tutor-client meetings when  other  arrangements  cannot  be  made,  for learning  materials and professional reference materials which cannot be  distributed to a l l .tutors, for independent  study by  clients and for meetings between a professional tutor and volunteer tutors as well as clients. used also  as the Tutor Training classroom.  underway between project (  Tutorial  Centre  ) centre i s Discussions are  staff and representatives  ) Public Library regarding  Outreach  The (  of the  the establishment of an  in the library.  Two  library  employees are currently enrolled in a Tutor Training class and are willing and qualified to work with clients who come to the l i b r a r y . considered  Outreach Tutorial Centres are also being  for other communities.  91 In the area of curriculum development, the project is  committed  to  the  development  Resource Manual, Social  of  Orientation  Foundation Literacy curriculum.  a  Skills  Tutor  Training  modules and  The lesson plans, notes and  handouts used for Tutor Training  have been compiled as a  f i r s t draft of the Resource Manual.  This  157  page draft  w i l l be revised in light of the post-program evaluations edited to produce the final Social  Orientation  Skills  document. modules  The  have  and  plans for the evolved  into  proposed series of audio tapes and worksheets entitled In Charge Here" (INCH).  a  a  "I'm  A contract worker was hired for the  period November 1 - December 21 to design and produce these materials. not  At the time of this report the materials were  available for review.  reported  No concrete progress has  development  of  the  Foundation  been  on  the  Literacy  The  public awareness component of the project i s  Curriculum.  an ongoing process. aims  and  activities  Project staff describe the rationale, of  individuals and groups.  the  project  to  all  interested  Posters, bookmarkers, tutor signs,  radio and television are being used to inform the public of the scope of undereducation among the adult population of the services which are available at (  ) College.  and  92 OBSERVATIONS During project  the period  has operated  with  covered i t s full  full-time professional staff. of  two short-term  increased  training,  contract  complement  report the of three  This combined with the hiring people  was  reflected  in an  level of a c t i v i t y and greater progress toward the  articulated represents  by this  goals  of the project.  the actual tutors  through the f i r s t  and projected  trained  The following  table  numbers of tutors-in-  and active  tutor-client pairs  year of the project.  The projections  indicate the maximum potential number as no allowance has been made for those who do not complete tutor-training or complete  but do  not commence  nineteen tutors trained during the following figures.  tutoring  activity.  The  A p r i l , 1979 are included in  93 TABLE 1 Actual and Projected Tutors In-Training, Tutors Trained, and Tutor-Client Pairs M onth  Tutors In-Training  Tutors Trained  0 0 0 0 0 0 0 74 74 74 20 20  19 19 19 19 19 19 19 19 19 19 93 93  April May June July August September October November December January February March  Tutor-Cl Pai rs 9 9 4 4 4 4 9 9 10 10 90* 90  * 3 of the original 19 trained tutors are not available for turoring a c t i v i t y .  intended  The (  ) Lake a c t i v i t i e s parallel closely the  activities  of the project  volunteer  tutor  educational region  succeeds  services to a part  that  services.  concept  has t r a d i t i o n a l l y Further,  (  proposal.  i t will  be  If the providing  of the population  of the  not had access to college ) Lake  is a Native  Indian  community and thus represents one of the main target groups identified in the project proposal. The impaired  operation  to some extent  of  the  project  appears  to be  by an apparent lack of structures  94 for cooperative planning and information sharing among staff members. not  The fact that two of the professional staff had  received  copies  of the f i r s t  evaluation  report  two  months after i t s submission, l e t alone met to discuss i t s implications, suggests a need for such structures. Preliminary  examination  of the tutor training  classes suggests that the participants possess the necessary prerequisite s k i l l s and attitudes.  This, combined with the  high quality of instruction provided should  by the project s t a f f ,  produce a corps of trained and motivated  volunteer  tutors. RECOMMENDATIONS Based observation  upon  the  progress  to  date  and  the  noted above, the evaluators make the following  recommendations for the consideration of the project staff. 1.  That a l l members of the project team meet regularly to review project.  the goals,  activities  There should  and p r i o r i t i e s  be a review  of the  of the project  proposal, the progress to date and the two evaluation reports.  Figure 1 (see p. 98) may be of assistance in  conceptualizing the overall project.  95  2.  That a revised  "program statement" be written.  document could be an abbreviated project proposal. establishing  priorities  informing  but updated form of the  This could assist the project team in and  specifying  the remainder of the project. for  This  other  a c t i v i t i e s for  It would also be of value  interested  groups  of the  pilot  proj ect. 3.  That the number one p r i o r i t y for the remainder of the project  be  c l e a r l y established  maintenance  and  support  Tutor-Client pairs. tutors  trained  of  as  the  the  activation,  maximum  number  of  The fact that the majority of the  in A p r i l ,  1979  were not  paired  with  clients until late November suggests that the rationale for  the  project,  undereducated attention  increased  adults,  because  of  educational  is not other  receiving college  services  for  the necessary and  project  activities. 4.  That the project staff reconsider their assumption that the majority of tutoring a c t i v i t y w i l l cease during the period  April  to  September.  If this  assumption is  96 correct, the f e a s i b i l i t y of the entire volunteer tutor concept  must  be questioned  as i t would  potential impact of the project.  halve the  Other programs of this  nature, including those in areas similar to the ( region,  have  maintained  a  high  level  )  of tutoring  a c t i v i t y on nearly a year round basis. 5.  That an additional staff member be hired on contract to complete  the development  Curriculum. was  of the Foundation  Literacy  If one of the reasons for this development  to provide guidelines  and materials  for use by  tutors with their c l i e n t s , then this should be completed as soon as possible so as to be available to those tutors already working in the f i e l d . 6. That special effort be made to promote and support the programs in (  ) Lake and (  a c t i v i t i e s in ( programs w i l l  ), (  ) and the proposed  ) Lake and (  ). These  serve those groups which were' identified  as target groups in the project proposal. Consideration should  be  given  to  employing  part-time  para-professional s as community contacts to support the tutors  and to staff  these are established.  Outreach  Tutorial  Centres where  97  7.  That the project staff consider reducing the length of the tutor training program in light of the success of shorter programs elsewhere.  In-service training could  be used to make up for the shorter training period would provide  and  an opportunity to reunite the tutors to  discuss and learn from their experience as tutors.  Figure 12 CONCEPTUALIZING THE PROJECT PLANNING STAGE  IMPLEMENTATION STAGE  PAYOFF STAGE  CURRICULUM DEVELOPMENT Tutor Training Manual Foundation Literacy Social Orientation S k i l l s PUBLIC AWARENESS Marks on Paper Medi a Personal Contacts MANAGEMENT Proposal Design Job Descriptions Hiring Orientation Delegation Problem Solving Mutual Planning Evaluati on  TUTOR TRAINING OUTREACH TUTORIAL CENTRES  TUTORING ACTIVITY (increased educational opportunity for undereducated adults)  FORMATIVE EVALUATION OF THE ADULT BASIC  EDUCATION  PROJECT  nal Report on Project A c t i v i t i e s for the Year Endi March 31, 1980  100 INTR0DUCTI0N The  Department  of  Adult  Education  of the  University of British Columbia was requested by the Division of  Continuing  Education  of the Ministry of Education to  serve as external evaluator for a pilot basic education agreement  at (  between  ) College.  the Department  project in adult  As contracted in an  and the Ministry, this  report is the third of three formative evaluation reports to be submitted. progress  The purpose of this report is to review the  of the project during  the period April  1, 1979  through March 31, 1980 and to make recommendations for i t s improvement. previous  Readers of this  report  reports supplement this  are reminded that  document and establish a  context for i t . In preparing this report, the evaluators  reviewed  documents provided by the Division and the College, met with project  staff,  personnel, tutors  consulted  visited  and c l i e n t s ,  with  tutor  other  training  and reviewed  emanating from the project.  College classes, documents  and Ministry interviewed and reports  As in the previous reports, the  observations, judgments and recommendations made herein are intended  to f a c i l i t a t e  project's objectives.  the successful  attainment  of the  101 NEED There is considerable data to support the need for programs for undereducated adults in the ( region.  The 1976 Census indicated that  ) College  1,100 adults (4  percent) have five or less years of schooling and 6,295 (22 percent) ahve five to eight years of schooling. are  approximately  7,500  adults  Thus there  in the region  who are  potential participants in the project. During education  1978-79 the College  programs at eight locations.  conducted  on Indian  reserves,  located in College f a c i l i t i e s .  while  offered  adult  basic  Two programs were the remainder were  A total of sixteen programs  were conducted involving 254 students, an average of sixteen students  per program.  Since most of those programs were  available only on a full-time basis in larger communities, the present project should assist the College to achieve i t s stated  goal  of serving the educational  needs of a l l area  residents through part-time as well as full-time programs in smaller communities and rural centres.  areas as well as the major  Non-traditional types  acknowledged in the Educational  of program  delivery are  Master Plan as being both  102 desirable and necessary given the geographic, climactic, and demographic conditions of the region.  In response to these  and other data, the College Board in March, 1979 identified adult  basic  education  as the top  program  p r i o r i t y for  1979-80. GOALS The development public  goals  and  of the project  awareness,  objectives  which  were in three  curriculum  development,  guided  general and  the  areas: outreach  development and delivery. Public Awareness:  1.  to identify the undereducated adults;  2.  to recruit volunteer tutors;  3.  to recruit clients (learners);  4.  to increase public awareness of the need for basic education programs for adults;  needs  of  to increase public awareness of programs available to undereducated adults. Curriculum Development  to develop basic l i t e r a c y materi als;  learning  to develop "social orientation s k i l l s " learning materials;  103  Outreach Development and Deli very:  3.  to develop a Tutor Training Manual.  1.  to develop and implement an effective Volunteer Tutor Program;  2.  to establish tutoring/1earning centres throughout the college region (e.g. o i l patch, Indian reserve, major centres).  SUMMARY OF PROJECT ACCOMPLISHMENTS A preliminary draft of the project proposal was approved in principle by the College Board in September, 1978.  Following  a meeting  with  Ministry of  Education  o f f i c i a l s in January, 1979, the Board gave final approval to the  project  in March, 1979 and funding  for the 1979-80  f i s c a l year was included in the College operating budget. Two  activities  which  occurred  as a result of  separate contracts between the College and the Ministry were completed prior to the project's o f f i c i a l start but prepared the way for i t . play ( financially  These were the development of the stage  ) and one tutor training separate,  related to the project.  0  those  activities  course.  Although  were educationally  104 The public  play was  awareness  introduce  a major component of the  campaign.  The  play  audiences to the concept and  illiteracy  in  society.  emphasizing  personal  illustrates  the  print-oriented  Through  accounts  dilemma  faced  by  society.  The  series  of  experiences the  tour  to  scenes  the  play  non-reader  in  consisted  performances between April and June, 1979. some 2,000 students and 450 adults.  designed  magnitude of adult  a  and  was  project's  of  It was  a 23  seen by  The reviews of (  )  were generally very favourable. A tutor training course for 29 held in March and A p r i l , 1979.  registrants  was  Nineteen of the registrants  successfully completed the course and 16 were matched with clients. Throughout the  first  year of the  were many a c t i v i t i e s , in addition to ( goal of increased too  numerous to  impossible  public awareness. describe  in  to assess their  staff described  impact.  signs,  These a c t i v i t i e s were and  i t is  all  B r i e f l y , the  but  project  a c t i v i t i e s of the  individuals and groups. radio,  there  ), related to the  the rationale, aims, and  project to a l l interested bookmarks, tutor  detail  project  television,  and  Posters, newspaper  105 were  used  to  inform  undereducation  among  the  the  services available at (  public  adult  of  the  population  scope and  of  of  the  ) College.  As of March 31, 1980, three curriculum development projects were underway and none of them had been completed. Of a proposed eight-chapter Tutor Training Resources Manual, two  chapters  were completed in draft form, four  were nearing completion were  in progress.  Social  chapters  of a f i r s t draft, and two  A seven-module  Orientation S k i l l s  and  program  Interpersonal  chapters  dealing  with  Communication  S k i l l s was under development, but only one module was ready for  production.  Four  sample  modules  in  a  Foundation  Literacy curriculum were being pilot tested. The involved  goal  of  outreach  development  and  delivery  designing  and  implementing a delivery system to  provide educational services to the target groups identified in the project proposal.  The main component of this system  was a volunteer tutor program. were  conducted  in  3 courses), ( (  ) and  six  Nine tutor training  locations  including  ) (2 courses), ( (  ), and  99  courses  (  ), (  ) ) Lake,  tutors were trained.  Ninety-three clients were recruited and matched with tutors, and 31 tutors were engaged in upgrading programs.  The  93  106 clients served  included 56 residents of c i t i e s , 27 of rural  areas, and 14 of native Indian communities. Two other a c t i v i t i e s of the project contributed to increased  public  tutor-client  pairs.  established (  awareness  and  Outreach  in ten locations  (  ) Lake, (  those centres  the  Tutorial  including  ) and one each i n (  Lake,  to  support Centres  three  ),(  were  places in  ),(  ), and (  ) Lake.  have at least some materials  of  ),( A l l of  for tutor use.  In addition, five issues of a newsletter t i t l e d (  ) were  produced and distributed to tutors, c l i e n t s , and interested citizens. OBSERVATIONS The intended one-fourth  project  fell  far short  of reaching i t s  target of 400 c l i e n t s , and in fact reached that  number.  In the year end report  only  of the  project s t a f f , three problems were l i s t e d as contributing to that disappointing outcome. 1.  The late  hiring  of staff.  The delay  in hiring the  second and third staff members meant a loss of eight person-months of productivity from the potential of the proj ect.  )  107 2.  Inadequate job descriptions. Provision was not made for adequate coverage of such vital areas as tutor training and tutor-client matching.  3.  Unrealistic  goals.  The  project  appeared  to  underestimate  the time and work required to identify and  t r a i n tutors, match them with c l i e n t s , and monitor the outcomes geographic  for 400 client-tutor area  covering  pairs  one-third  spread  over a  of the province.  Coupled with the goal of preparing curriculum materials, the goals overall  seemed too much to accomplish  given  the finance and staffing patterns used. In addition to the problems cited above, i t should be noted that the timing of the various project a c t i v i t i e s was  not  properly  synchronized.  For example,  public  awareness was being raised at the time when the project had no curriculum and few tutors who could provide any service. Curriculum development, which requires special s k i l l s and is best accomplished through a team approach, did not get well underway until late in the year and is s t i l l  not completed.  Meanwhile, tutors are being trained and instruction is being given to c l i e n t s .  The period in the autumn of 1979, which  would normally be that of high instructional delivery, was used for curriculum development.  108  It should also be noted that the project seemed to lack  a clear focus and direction.  proposal serving  spells  out some  400 c l i e n t s ,  Although the project  specific  the staff  objectives,  such  of the project  as  seemed  uncertain as to how best to direct their energies in regard to the various components. As a result, the project's funds and  human resources were spread t h i n l y  different which  and often  suffered  establishment  competing  the most and support  over a number of  activities.  from  this  The component  situation  of the volunteer  was the  tutor-client  system which should have been the p r i o r i t y objective. To determine  the nature  of the service  being  provided to educationally disadvantaged adults, a telephone survey of clients was conducted during A p r i l , 1980. Twenty clients were contacted, of whom fifteen (75 percent) were s t i l l active in the program. with their tutors an average  The fifteen active clients met of six times per month for  s l i g h t l y less than two hours per meeting.  This would result  in an average of approximately twelve hours of instruction per client per month, although the range went from a low of two hours to a high of 80 hours per month. five hours of tutoring per month.  The median was  109 Almost a l l of the clients contacted indicated that they would be unable or unwilling to attend a regular class at a College centre.  The most frequently stated reasons for  this were that employment involved shift work or there were young children who required the client's attention. Most of the clients contacted were satisfied with the tutoring they had received, and only three indicated d i f f i c u l t i e s . The tutor training courses, which o r i g i n a l l y were 30 hours but were reduced to 24 hours,  provided  3,000 student-contact hours of instruction. those  tutors  had received  an estimated  roughly  The clients of 1,500 hours of  instruction by the end of the f i r s t project year. RECOMMENDATIONS Based observations  upon  the  made during  information  the period  April,  gathered  and  1979 through  March, 1980, the following recommendations are made for the consideration of the staff as the project enters the 1980-81 year. 1.  That a revised program goal statement be prepared.  This  statement could be an abbreviated, revised form of the project  proposal  and  would  assist  the staff  in  110 establishing  priorities  and specifying a c t i v i t i e s for  the duration of the project.  This statement should be  specific and detailed in terms of products, locations, and  schedules.  That consideration be given to decreasing  the emphasis  on f u l l - t i m e , professional staff and increasing the use of part-time areas.  programmers in smaller centres  and rural  This would provide for increased f l e x i b i l i t y and  allow more a c t i v i i t e s to occur outside the major College centres. college  It would also permit better coverage of the area  and reduce  the travelling  required  by  full-time staff. That the number one p r i o r i t y for the second year of the project be clearly established as training, activating, maintaining,  and  supporting  the maximum  number of  tutor-cl i ent pai rs . That the curriculum development projects which are now underway be brought quickly to a close.  It might be  helpful to establish a firm target date, such as July 1, 1980,  for completion  work.  Those projects not completed by the target date  should be abandoned.  of a l l curriculum  development  Ill  5.  That  public  awareness  activities  extent required to attract  continue  tutors  only to  and c l i e n t s ,  the  and then  only when the program is operating at a higher level  of  efficiency.  6.  That  the  project  gathering  staff  procedure to  develop  enable  kept of project a c t i v i t i e s interest  here is the  tutors, enable  clients, the  systematic  an accurate  Of particular  need to keep records  staff  to  data  record to be  and services.  and services  project  a  and l i s t s  of  provided.  Such data will  report  on  its  a plan  to  activities  during the second year.  7.  That  the  project  staff  develop  assess  the  quality of the education being provided, and follow up with steps to  ensure that  best quality of instruction  The significant education the  first  trained  of  are receiving  in the  the  College region. of  the  campaign, tutors.  and  stage the  the  possible.  project  produced  contributions to the advancement  production  awareness  year  clients  this  of adult  few basic  Notable among these were  play,  the  preparation  For a variety  a  of  extensive of  reason,  a  public  group  however,  of the  112  project did not second  year  accomplish a l l that  should  seek  accomplishments of the  to  first  build  i t might have. upon  the  The  positive  year to expand greatly the  opportunities for adults in the College region to attain or upgrade their basic l i t e r a c y s k i l l s .  


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items