Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Contextual influences and post-educational application of learning among health professionals Patterson, Ilse Magdalene 1998

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_1998-0170.pdf [ 6.55MB ]
Metadata
JSON: 831-1.0055445.json
JSON-LD: 831-1.0055445-ld.json
RDF/XML (Pretty): 831-1.0055445-rdf.xml
RDF/JSON: 831-1.0055445-rdf.json
Turtle: 831-1.0055445-turtle.txt
N-Triples: 831-1.0055445-rdf-ntriples.txt
Original Record: 831-1.0055445-source.json
Full Text
831-1.0055445-fulltext.txt
Citation
831-1.0055445.ris

Full Text

C O N T E X T U A L I N F L U E N C E S A N D P O S T - E D U C A T I O N A L A P P L I C A T I O N O F L E A R N I N G A M O N G H E A L T H P R O F E S S I O N A L S by ILSE MAGDALENE PATTERSON B.Sc, University of Toronto, 1972 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARTS THE FACULTY OF GRADUATE STUDIES (Department of Educational Studies) We accept this thesis as conforming to the required standard THE UHHVERSITY OF BRITISH COLUMBIA April 1998 ® Use Magdalene Patterson, 1998 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. Department The University of British Columbia Vancouver, Canada DE-6 (2/88) A B S T R A C T The purpose of this study was to determine whether contextual influences of resources, encouragement from others, organizational support, opportunity to apply learning, and authority to act or apply learning were associated with post-educational application of learning in the work environment. Data were drawn from a national U.S. evaluation study of the Centre for Substance Abuse Prevention (CSAP) Training Systems (CTS). Four health professional groups — physicians, nurses, mental health counsellors and rehabilitation specialists, were included in this study. Chi-square analysis was the method of choice for within group comparisons. Statistically significant associations between contextual influences and post-educational application of learning were found for various health professional groups as follows: (1) organizational support was significantly associated with application for nurses, mental health counsellors and rehabilitation specialists; (2) encouragement from others was significantly associated with application for nurses and rehabilitation specialists; and (3) opportunity to apply learning was significantly associated with application for nurses and mental health counsellors. Authority to act and apply learning, and resources were not significantly associated with application for any of the four health professional groups. This study adds to our understanding of the role of the human element of support for post-educational application of learning. Planners can incorporate features which provide organizational support, encouragement from others, and opportunity to apply learning to increase the likelihood of post-educational application. The results confirm the relevance of ii organizational support, opportunity to apply learning and encouragement from others as valid indicators for evaluation of the post-educational environment. The variables of resources and authority require further clarification for future use in outcome evaluations. Future research should explore the role of age, gender, educational level, work focus and organizational work setting of participants on these associations between contextual influences and post-educational application of learning. iii T A B L E O F C O N T E N T S Page ABSTRACT ii TABLE OF CONTENTS iv LIST OF TABLES vii LIST OF FIGURES viii ACKNOWLEDGEMENTS ix CHAPTER ONE: INTRODUCTION 1 Problem Statement 1 Purpose Statement 2 Research Question 3 Definition of Terms 3 Training and Education 3 Continuing Professional Education 4 Contextual Influences 5 Post-Educational Application of Learning 5 Significance of this Study 6 Conceptual Framework for the Study 7 Summary 10 CHAPTER TWO: LITERATURE REVIEW 12 Goals of Adult Education 13 Goals of Continuing Professional Education 14 Evaluation of Continuing Professional Education 15 Continuing Professional Education Evaluation Models 17 Cervero's Model 18 Precede/Proceed Model 20 Evaluation Studies of Continuing Education for Health Professionals . . . . 23 Continuing Medical Education 24 Continuing Nursing Education 28 Continuing Education for Mental Health Counsellors 32 Continuing Education for Rehabilitation Specialists 33 Application of Learning and Related Concepts 35 Diffusion of Innovations 35 Implementation 36 Transfer of Training 37 Application of Learning 38 Summary 39 iv T A B L E O F C O N T E N T S , C O N T I N U E D Page CHAPTER THREE: RESEARCH DESIGN 42 Description of the Centre for Substance Abuse Prevention Training System 43 Health Professional Training 44 Application Focus 45 Evaluation and Evaluation Research 47 Standards for Program Evaluation Research 48 Implications for this Study 49 Instrumentation of the CTS Evaluation 50 Advantages and Disadvantages of Self-Report Survey Questionnaires in Evaluation Research 50 Strengths and Weaknesses of the CTS Evaluation Survey 51 CTS Instrumentation 52 The Variables in this Study 53 Independent Variables 54 Dependent Variable 55 Demographic Variables 56 Instrument Reliability 57 Instrument Validity 58 Construct Validity 59 Face and Content Validity 59 Concurrent Validity 60 Predictive Validity 60 Validity in Evaluation Research 61 Data Collection 62 CTS Instrument Administration 62 Anonymity and Confidentiality 62 Response Rates 64 Sample Selection 65 Data Analysis 67 Descriptive Analysis 67 Sample Representativeness 68 Inferential Analysis 71 Rationale for the use of Chi-Square 73 Significance Level 74 Methodological Limitations of this Study 74 CHAPTER FOUR: RESULTS 77 Demographic Characteristics 78 Physicians 78 Nurses 79 v TABLE OF CONTENTS, CONTINUED Page Mental Health Counsellors 79 Rehabilitation Specialists 80 Sample Representativeness 81 Descriptive Statistics of Contextual Influences 84 Physicians 84 Nurses 85 Mental Health Counsellors 86 Rehabilitation Specialists 86 Descriptive Statistics of Post-Educational Application of Learning 87 Associations between Contextual Influences and Post-Educational Application of Learning 88 Nurses 89 Mental Health Counsellors 90 Rehabilitation Specialists 90 Effects of Demographic Variables on Associations between Contextual Influences and Application of Learning 91 Summary of Results 94 CHAPTER FIVE: DISCUSSION 97 Post-Educational Application of Learning 97 Contextual Influences and Post-Educational Application of Learning 98 Organizational Support and Encouragement from Others 99 Resources 101 Opportunity to Apply Learning 102 Authority to Apply Learning 103 Representativeness 104 Summary 105 CHAPTER SIX: IMPLICATIONS FOR PLANNING, EVALUATION, THEORY AND RESEARCH 107 Implications for Planning 107 Implications for Evaluation 112 Implications for Theory and Future Research 114 The Conceptual Framework 114 Future Research 118 Conclusion 119 REFERENCES 121 APPENDICES 128 vi LIST O F T A B L E S Table Page 1. CTS Training Features Designed to Facilitate Post-Educational Application of Learning 46 2. Independent and Dependent Variables from the Follow-up Questionnaire 53 3. Sample and Response Rates Based on Matched Pre, Post and Followup Questionnaires 66 4. Summary of Chi-Square Tests of Association for Sample Representativeness by Health Professional Group 83 5. Median Scores of Reported Availability of Contextual Influences by Health Professional Group 85 6. Median Scores on Self-Reported Post-Educational Application Measure by Health Professional Group 87 7. Summary of Chi-square Tests of Association between Contextual Influences and Post-Educational Application of Learning by Health Professional Group 89 8. Effects of Demographic Variables on Associations between Contextual Influences and Application in Health Professionals 93 vii LIST O F F I G U R E S Figure Page 1. Conceptual Framework for this Study based on the Modified Precede/ Proceed Model viii A C K N O W L E D G E M E N T S I would like to thank Judith Ottoson for allowing me to use existing data, and for her ongoing spiritual and technical guidance. My thanks also to Frank Echols and Tom Sork for the generous donation of their time and energy. I would also like to extend my gratitude to my family who have supported me throughout these years as a student. ix CHAPTER ONE: INTRODUCTION 1 The application of learning should be of concern in the outcome of any educational experience, be it in adult education, continuing professional education or continuing education for health professionals. In the health science disciplines in particular, post-educational application of learning should produce outcomes of higher standards of patient care, accountability to the sponsor, and economic return on invested dollars. Each of these outcomes implies a change in practice behaviour by the learner. Determining whether the educational experience is facilitative to behaviour change in the practitioner can be established through evaluation. This study analyses evaluation data from health promotion training workshops for health professionals. The specific focus of the analysis is the relationship between contextual variables — resources, encouragement from others, opportunity to apply learning, authority to apply learning and organizational support for making changes — in the post-educational work environment and the post-educational application of learning among health professionals. Problem Statement The factors which influence the application of learning are complex. In the profession of adult education, the emphasis has traditionally been on the characteristics of the learner, participation and learning theory, rather than on the application of learning (Ottoson, 1994). In continuing professional education (CPE) specifically for health professionals, millions of dollars and hours of work are expended annually on the provision of continuing education workshops. There has 2 been growing concern among the health professional groups about the impact of this training on the professional, whether this training translates into more effective patient care, and whether the continuing professional education dollar is well spent (Davis, Thomson, Oxman & Haynes, 1995; Kiener & Hentschel, 1989). Workshop effectiveness has been documented in various forms, including participant satisfaction, increase in knowledge, attitude change and behaviour change. Of the above outcomes, behaviour change is the best indicator of application of learning because it demonstrates learning put into practice. It is also complex and difficult to measure, therefore the approach least often taken in evaluation. However, continuing professional education will continue to be called into question if it cannot demonstrate effectiveness. In order to assess workshop effectiveness, it is important to further our understanding of the process of application. Many factors influence application of learning. One of the influences on application which we know little about is the contextual, which can either facilitate or inhibit application of learning after an educational workshop. Without a better understanding of the influences of contextual factors in the work environment to which the learner returns, planners will be unable to modify their workshops to facilitate application of learning. Purpose Statement The purpose of this study is to determine which contextual variables are significantly associated with post-educational application of learning by physicians, nurses, mental health counsellors and rehabilitation specialists. Data are drawn from a survey which was part of a national evaluation study of the U.S. Centre for 3 Substance Abuse Prevention (CSAP) Training Systems (CTS). The overall mandate of the CTS evaluation study was to investigate the post-training application of learning of participants in the CTS workshops. The independent variables of concern in this study are: 'encouragement from others', 'support for making changes suggested in the workshop', 'sufficient resources', 'opportunity to apply learning' and 'authority to act or apply learning'. The dependent or outcome variable of post-educational application of learning is operationalized as 'made changes in how you do your work or volunteer activities'. Research Question The primary research question guiding this study is: For each health professional group, what is the nature of the association between each of the five contextual variables in the post-educational environment and the post-educational application of learning? The secondary research question is: How do demographic variables influence the associations between the five contextual influences and the application of learning? Definition of Terms Training and Education It is important to clarify the concepts of training and education. These two terms are often used interchangeably in the literature and in this study. As highlighted by the following definitions, the boundaries between training and education are vague. According to Lawson (1979), training has implications of narrowing, confining, and the development of standard performances designed to improve performance on 4 the job the employee is presently doing. More recently, the definition of training, arising from research in organizational effectiveness, includes the outcome of behaviour change of participants (Analoui, 1993). Education is defined as the teaching of cognitive and evaluative skills to enable the learner to move beyond the acquisition of specific goals (training) to the evaluation and analysis of the concepts underlying the goals (Lawson, 1979). In practice and in the references cited in this study, advanced learning for professionals is termed 'continuing professional education'. The data for this study, however, are drawn from an evaluation of training workshops. The goal of these workshops is to provide individual participants with skills to plan and carry out prevention programs. Therefore, although the workshops are called 'training', in reality the goals are closer to the above definition of education. The participants are expected to evaluate their specific situations and apply the learning in their context. Continuing Professional Education Continuing professional education (CPE) is designed to provide ongoing learning opportunities for professionals after graduation into a profession in order to keep up to date with advances and issues specific to their field of expertise. According to Houle (1980), the aim of continuing education is "to convey a complex attitude made up of a readiness to use the best ideas and techniques of the moment, but also to expect that they will soon be modified or replaced" (p. 75). Every professional must expect constant change and be continuously provided an opportunity to adapt their practice to those changes. Such adaptation implies 5 application of learning, demonstrated through behaviour change, and a work environment which supports those changes. Within the medical services disciplines, an additional purpose of CPE is to maintain a high standard of patient care, also implying application of learning to the practice setting. The focus of this thesis will be the effects of contextual influences on the application of learning of health professionals following a learning experience. The evaluation of the impact of CPE on patient care takes this issue one step further, and is beyond the scope of this project. Contextual Influences Contextual influences are either facilitators or barriers to application of learning in the work environment to which the learner returns. In this study, influences hypothesized to facilitate application of learning are broken down into variables of sufficient resources, opportunity to apply learning, authority to act or apply learning, encouragement from others, and support for making changes suggested during the workshop. Barriers to the application of learning can be considered as the lack of facilitative influences. Post-Educational Application of Learning Application of learning has been defined by Ottoson (1995a) as "the putting of knowledge into practical contact" or as "the use of principles learned for resolving problems of practice in the work setting" (p. 19). The application of learning in the context of this study, termed post-educational application of learning, also implies outcome variables of observable and measurable changes in behaviour. This definition encompasses various concepts from related disciplines. The industrial 6 psychological literature uses the term transfer of training. The political science and sociology literature use the term implementation, and the concept of diffusion is explored in the educational, sociological, industrial-agricultural, communications and anthropological literatures. Both psychologists and educationalists have been preoccupied with the theoretical rather than the practical aspects of application or transfer of learning, but are more recently appreciating the impact of the social forces of the workplace on post-training or post-educational application of learning (Analoui, 1993). Significance of this Study This study adds to our understanding of the factors that contribute to the application of learning, specifically the link between contextual influences and post-educational application of learning. Much research has been conducted to clarify the impact of workshop design and the characteristics of the learner on the transfer of training and subsequent application. However, there remains a deficit of empirical knowledge concerning the impact of work environment characteristics on the transfer and application of learning (Baldwin & Ford, 1988). Results of this study will have implications for planners, educators and evaluators. Significant associations between the contextual variables and the application of learning following a continuing education experience will provide useful knowledge and prospective tools for the planner, educator and evaluator. The incorporation into the learning experience of facilitative contextual influences to post-educational application can also assist planners in justifying expenditures. 7 Conceptual Framework for the Study The conceptual framework for this study is based on Green and Kreuter's (1991) Precede/Proceed Model, the relevant excerpt of which is presented in Appendix A. This model was originally designed to explain predisposing, reinforcing and enabling factors considered as multiple forces that shape health behaviours. In terms of planning, they are considered within the educational and organizational diagnostic phase as constructs which must be considered if health education is to be successful. In the Precede/Proceed Model, predisposing factors include any characteristics of a person or population that motivate behaviour prior to the occurrence of the behaviour, including knowledge, attitudes, beliefs, values and perceptions. An example of a predisposing factor is a religious belief that influences a person's reaction to western medical care. Reinforcing factors in this model include rewards or punishments following or anticipated as a consequence of a behaviour, serving to strengthen or lessen the motivation for the behaviour after it occurs. These could appear as attitudes or behaviours of health or other personnel, peers, parents or employers. An example of a rewarding reinforcing factor is the peer support offered to teenage girls for trying cigarettes. Enabling factors in this model are characteristics of the environment that facilitate health behaviours, including availability of resources, accessibility, referrals, rules or laws, and skills. The provision of low fat foods on a menu to assist the population in choosing healthy meals is an example of availability of resources 8 supporting healthy behaviours. The predisposing, reinforcing and enabling factors of the educational phase of the model also assist the planner to focus on targets for intervention strategies as well as to develop objectives for evaluation. Since health behaviours can be considered an example of application of learning, this model has also recently been adapted by Ottoson (1992; 1995c) to investigate the factors which influence the application of learning in adult education (Appendix B). In this model, predisposing factors have been broken down into variables of 'knowledge', 'feel able to apply', and 'likely to apply' learning. Enabling factors have been broken down into variables of 'capacity', 'resources', 'position', 'opportunity', 'organizational culture' and 'planning time'. Reinforcing factors have been broken down into variables of 'supervisor attitudes' and 'peer attitudes'. Ottoson's model organizes enabling and reinforcing factors as the contextual influences on the post-educational application of learning. A further adaptation — see Figure 1 — provides the framework for this study. This model illustrates that the surveyed health professionals who participated in the CSAP educational program represent the sample. Through quantitative data analysis, the relationships between the independent variables of contextual influences — here represented as reinforcing and enabling factors — and the dependent variable of post-educational application of learning are investigated. For the purposes of this study, the reinforcing and enabling factors, which have been drawn from the follow-up survey (see Appendix Q , include 'encouragement from others' and 'support for making changes suggested during the workshop', 'sufficient resources', 'opportunity to apply learning' and 'authority to act Figure 1. Conceptual Framework for this Study based on the Modified Precede/Proceed Model 9 EDUCATIONAL W O R K S H O P CSAP SAMPLE HEALTH PROFESSIONALS Physicians Nurses Mental Health Counsellors Rehabilitation Specialists PREDISPOSING FACTORS knowledge, attitude, reasons for participation C O N T E X T U A L INFLUENCES REINFORCING FACTORS encouragement from others organizational support ENABLING FACTORS resources, opportunity, authority BEHAVIOUR C H A N G E H POST-EDUCATIONAL APPLICATION OF LEARNING made changes in how you do your work or volunteer activities and apply learning'. Justification for the use of these variables will be presented in the literature review, which will include an examination of their analysis in other evaluation studies of continuing education for health professionals. For the purposes of this study, the reinforcing and enabling factors, which have been drawn from the follow-up survey (see Appendix Q , include 'encouragement from others' and 'support for making changes suggested during the workshop', 'sufficient resources', 'opportunity to apply learning' and 'authority to act and apply learning'. Justification for the use of these variables will be presented in the literature review, which will include an examination of their analysis in other evaluation studies of continuing education for health professionals. 10 The dependent variable, post-educational application of learning, will be examined through the item of 'made changes in how you do your work or volunteer activities'. This item, also drawn from the follow-up survey, is designed to capture self-reported behaviour change as a result of the educational experience. The arrows between the contextual influences and application of learning are two-headed, indicating that the relationship under investigation in this study is one of association between variables rather than one of causation. The arrow leading back to the CSAP educational workshop refers to the feedback of implications of this study to the planner, educator and evaluator. Predisposing factors of knowledge, attitude and reasons for participation, and aspects of the educational workshop, describe conditions relevant to pre-education analysis. It is beyond the scope of this paper to address these factors since this thesis is examining post-educational influences on application of learning. Summary This chapter has provided an introduction, rationale and overview for this study. The application of learning after continuing professional education for health professionals has far-reaching implications for practice, patient care, accountability and evaluation. Determining the influences of contextual factors in the work environment is critical to the improvement of CPE. The goal of this study is to determine, for each health professional group, what the nature of the association is between each of these contextual influences and post-educational application of learning. The following chapter will provide an indepth 11 literature review to support the investigation of these contextual influences and their role in post-educational application of learning among health professionals. CHAPTER TWO: LITERATURE REVIEW 12 The purpose of the following review is to provide an integrative overview of literature relating to goals and evaluation of adult and continuing professional education, and to critically review evaluation research of continuing education programs for health professionals. Specifically, the review will present the goals of adult education and continuing professional education. The evaluation of continuing professional education will be described next, followed by a discussion of two evaluation models used to evaluate continuing education in the health professions. A rationale for the model chosen to frame this analysis will be provided. A critical review of evaluation studies of continuing education programs relevant to this study will be presented next. A case will be presented to illustrate how these studies support the examination of the independent variables of contextual influence and the dependent variable of application. As well, gaps relating these variables in the literature will be brought to the forefront to further justify this analysis. As much of the literature in this area concerns programs rather than workshops, the term program will be used unless a workshop is specifically implied. Following this, the relevant literature on application of learning and related concepts of transfer of training, diffusion, implementation and transfer of learning will be discussed in order to illustrate how these various approaches provide support for this study's focus on the relationship of the post-educational environment to the application of learning. 13 Goals of Adult Education In 1976, the General Conference of UNESCO adopted the following official definition of adult education: The term adult education denotes the entire body of organized educational processes, whatever the content, level and method, whether formal or otherwise, whether they prolong or replace initial education in schools, colleges, and universities as well as in apprenticeship, whereby persons regarded as adult by the society to which they belong develop their abilities, enrich their knowledge, improve their technical or professional qualifications, or turn them in a new direction and bring about changes in their attitudes or behaviour in the two-fold perspective of full personal development and participation in balanced and independent social, economic and cultural development...1 (author's underline) This definition acknowledges a resultant change in behaviour as one of the goals of adult education. However, in practice, the field of adult education has not addressed the application of learning. Darkenwald and Merriam (1982) describe the goals of adult education as cultivation of intellect, facilitation of individual self-fulfillment, promotion of personal and societal improvement, social transformation, and organizational effectiveness. They do not include application of learning in this definition. Also, as a result of a long-standing process of establishing adult education as a profession by focusing research on participation, learning theory and program planning, and of structuring programs to reflect particular philosophical premises, the assessment of the application of learning as reflected in behaviour change has lagged behind (Scanlan, 1985). 1 UNESCO, The General Conference adopts a Recommendation on Adult Education (Adult Education Information Notes, no.1), p.2. 14 Goals of Continuing Professional Education Continuing professional education (CPE) has its roots in post Renaissance Europe, at a time when education was viewed as cultivation of the intellect, a search for basic truths, and life long learning was cherished by the newly developing professions, in particular, medicine. This intrinsic view of continuing education underwent change during the later 1800's and into the 1900's as professions began to specialize and technical knowledge advanced rapidly. These changes brought about a more instrumental approach to education, which came to be seen as a means to an end, a remediation for gaps in knowledge, and a way to cope with rapid growth and change (Scanlan, 1985). The 1960's and 1970's witnessed a decline in the public esteem of many professions, and an increased concern for improving both the substance and image of practice. Continuing professional education became a vehicle for professionals to disarm society's criticisms and build defenses against malpractice suits, loss of certification, and the requirements of compulsory periodic relicensure or recredential ing (Houle, 1980). Continuing professional education has changed significantly in the last ten years. Earlier, to meet the instrumental needs of professionals, emphasis was placed on skill development. Then, educators were able to demonstrate program effectiveness through knowledge increases and participant satisfaction. Today, the goals of continuing professional education include: (1) assistance in maintaining professional knowledge and skills, (2) preparation for assuming new responsibilities or changing career paths, and (3) a broadening of the practitioner's range of 15 knowledge and skills in order to cope with new developments in the field and to achieve flexibility in complex work environments (Azzaretto, 1992). Also, the demands for accountability have changed. When asked if continuing professional education makes a difference, educators realize that their understanding of and commitment to evaluation of continuing professional education programs is an underdeveloped skill, and is rendering far too little valuable information regarding the worth of the investment (Stufflebeam, 1985). Economic influences such as increasing national debt, rapid knowledge and technological change and social influences of health care reform have raised calls for evidence of effectiveness of continuing professional education (Azzaretto, 1992; Cox & Baker, 1981; Davis, Thomson, Oxman & Haynes, 1995; Garza & Evans, 1983). Effective continuing professional education has moved beyond participant satisfaction. Azzaretto (1992) states that the continuing educator must be prepared for a new breed of learner, the self-directed adult whose primary purpose is to effectively apply new learning to the workplace. Continuing professional education should "equip the professional to develop linkages between thought and action and knowledge and behaviour" (Azzaretto, 1992, p. 86). The focus of this study is this link between knowledge and behaviour, and the contextual influences of the workplace which either support this link or act as barriers to it. Evaluation of Continuing Professional Education The literature cites many purposes of evaluation. Often it is difficult to discern between a goal and a role of evaluation, and many definitions imply both. Smith (1994) lists varied purposes or goals of evaluation including: (a) making government 16 more effective, accountable, responsive; (b) making causal inference as to impact; (c) determining an object's value, worth, merit or quality (see also Knox, 1985; Martinoff, 1985); (d) making recommendations for program improvement; and (e) improving program implementation. Evaluation of continuing professional education has been classified into three main types, depending on the purpose, significance and methods. The first type of evaluation, 'instructional studies', defines evaluation as the process of determining whether or not educational objectives have been reached. Such evaluations generally address new or enhanced personal knowledge, participant satisfaction, content appropriateness, speaker or presenter adequacy, facility and scheduling matters, organizational and content aspects of the program during its presentation, and behavioural change within the protective environment of the training process. Such evaluations do not address longer term behavioural change, or application of learning to the worksite. The second type, 'inquiry studies' or formative evaluations, provide information for decision making during the process of the program. The third type, 'impact' evaluations, is designed to determine the extent and type of impact CPE has on subsequent performance, on the clientele they serve, or on the organizations with which the professional is associated (Knox, 1985). Impact evaluations must address contextual factors within the worksite if program effectiveness within the arena of subsequent performance is to be established. Impact evaluations designed to assess application of learning to practice are the most difficult type to carry out (Hast, 1989). Surveys are commonly used, but their administration is often costly and complicated, involving follow-up studies of former participants and 17 reviews of time-series performance data. Chart audits2 are another form of impact evaluation. The educational intervention, however, must be powerful enough to produce a change in the professional's performance which can be linked to the program. It is often difficult to objectify program outcomes in behavioural terms which can be picked up in chart audits. Contextual factors of the work environment and the personality influences of the participants also play a complex role in the outcome effects of an educational program over time (Cervero, 1988), and are difficult to account for in self-report surveys as well as chart audits. Although evaluation of program impact in all three of the arenas described by Knox (1985) is vital to achieve a total picture of program effectiveness, this thesis will concern itself with the first of these, the effect of the educational program on subsequent performance or the application of learning in the work environment. Contemporary demands of evaluation of continuing education include long-term behaviour change — application of learning — as evidence of program effectiveness (Turnbull & Holt, 1993). Continuing Professional Education Evaluation Models In response to demands for accountability of continuing professional education, evaluators and educators have developed models which attempt to 2Chart audits, which involve review of practitioner records, are an unobtrusive method of impact evaluation designed to measure behaviour change (Madaus, 1985).Problems with this method include high cost and time commitment, insensitivity to subtle changes in patient management, restriction in range of behaviours that can be recorded, practitioner inaccuracies and inconsistencies of data recording, as well as the difficulties related to reaching participants spread over large geographical areas (Wergin et al, 1988). 18 encompass the wide range of influences on program outcome. Two of these, Cervero's model (1988) and the Precede/Proceed model (Green & Kreuter, 1991), have been used extensively to evaluate continuing education for health professionals. Cervero's Model In order to understand why application of learning does or does not occur after an educational program, the evaluation must consider the organizational context to which the learner returns. Cervero (1988) has developed a conceptual framework from which to consider evaluation of continuing education programs. This framework includes the organizational context, which he terms the social system within which the professional works, as well as the characteristics of the continuing education program, the individual professional, and the nature of the proposed change (Appendix D). According to Cervero and Rottet (1984), Cervero, Rottet and Dimmock (1986), and Cervero (1988), evaluation must address the contextual factors in the workplace to better understand the influences on application of learning. Cervero et al. (1986) used his model to assess continuing professional education and subsequent performance among nurses. His framework, which used a post-test design, showed that even attitude and intent to change, which had been significant indicators of performance at one week, lost their strength at six months follow-up. The recommendation stemming from this study is that the variable of social context needs to be better operationalized by breaking it down into administrative controls, power and roles, authority, and representation of members of administration in the program planning process. 19 Cervero's model has also been used by Waddell (1992) to conduct a meta-analysis of 34 studies of continuing education research in nursing. This researcher found that although continuing nursing education appeared to positively affect practice, there was little consistency in the results linking environmental characteristics to application. The author recommended further analysis of the contextual factors in the environment and their role in supporting behaviour change. Peden, Rose and Smith (1992) tested Cervero's model in measuring changes in practice by nurses after continuing education, nurses' and supervisors' views of change, and the receptivity of the social system to change. These researchers viewed the social system as the key element in the transfer of learning. This pre, post and 60 day follow-up design involved twenty female nurses who complied 100% with the written self-report questionnaires. Although the respondents rated the social system as highly receptive to change and there was a significant increase in change in practice, this study did not break the social system down into different variables. This review of Cervero's model highlights some of the research in evaluation of continuing education which looks at the application of learning in the workplace from the perspective of his framework. Results of these studies point out the need to examine the social context where learners are to apply their learning. A further breakdown of this social context is required, however, to fully understand the facilitative or inhibiting aspects of the work environment which influence application of learning. 20 Precede/Proceed Model The Precede/Proceed model was originally developed to examine broad influences on health behaviour (Appendix A). These include: (a) the educational program (its structural and process characteristics); (b) predisposing factors such as learner knowledge, attitudes, values and beliefs; (c) enabling factors including learner skills and resources; and (d) reinforcing factors including attitudes and behaviours of significant others. The original framework encompasses phases which form the planning and implementation process — including evaluation — of the educational program (Green & Kreuter, 1991). The Precede/Proceed model has been used to guide research addressing application of learning in health professionals. Tamblyn and Battista (1993) used the model to conduct a meta-analysis of studies investigating interventions designed to change clinical practice among physicians. These authors describe determinants of practice patterns as predisposing (practitioner's knowledge and skills), enabling (factors which facilitate the initiation of a new behaviour), and reinforcing (factors which sustain behaviour over time). Characteristics of the practice setting which are considered to be enabling and reinforcing include resources or policy affecting remuneration, support in the form of consultation with peers, and opportunity to practice new behaviours. This research lends further support to the use of this model as a foundation for the framework of this study, as well as to the examination of the contextual influences of resources, support and opportunity. 21 The Precede/Proceed model has also been adapted by Ottoson (1993, 1995c, 1997) to assess the contextual influences of application of learning. Initially, Ottoson (1993) administered self-rating questionnaires and open-ended telephone interviews to two groups of participants (n=>27, n» 14) in a continuing education program in order to discern which variables are relevant to application. Building upon the models of Cervero and Green and Kreuter, Ottoson determined that predisposing, enabling and reinforcing factors are all relevant to the understanding of the context of post-educational application for professionals. This study also highlighted lack of resources, lack of opportunity to apply, and lack of support from supervisors and peers as inhibitors to application. Multiple influences on application were quantitatively assessed by Ottoson (1995c) by a self-administered pre, post and 3 month follow-up questionnaire of health professionals (n = 34 matched sets) following a continuing education workshop. Findings include support for the multiple influences on application, and in particular for the variables of opportunity to apply, authority (position in organization), and support/encouragement (supervisor's attitude). A similar study using qualitative data, again based on the adaptation of the Precede/Proceed model, provides support for the variables of opportunity, encouragement/support, as well as resources, in particular, time (Ottoson, 1997). In response to the lack of studies which address the contextual factors of the worksite in the application of learning, McDonald (1991), using an ex post facto design, assessed the relative contribution of organizational support on a quantifiable behavioural outcome of an accident response team training program. A post training 22 follow-up survey (4-15 months) was mailed to 391 participants, 297 of whom responded with usable questionnaires. This study divided the organizational context into variables of administrative commitment, perceived external support, perceived work environment motivators/incentives, personal attitude and personal competency. Factor analysis was used to identify the key organizational contextual problems and conditions perceived to inhibit training use. The variable of administrative commitment, made up of administrative expectation, supportive communication and supervisory follow-through, accounted for 32% of the variance in the data, suggesting a strong relationship of these factors to application of learning. Perceived external agency support and perceived work environment motivators/incentives also accounted for 12.5% and 9.5% respectively of the variance in the data. Multiple regression was then used to determine the relative contribution made by each factor in explaining the variances in training use. Results indicated that personal attitude, external agency approval/support, personal competency and administrative commitment/follow-through all proved to be significant predictors of training use. Although this exploratory study did not use one of the models, it supports the breakdown of contextual influences into more descriptive variables, and the evaluation of these contextual variables in terms of their impact on the dependent variable of behaviour change. The above discussion highlights existing research of two extensively used evaluation models of continuing education for health professionals. Studies using these models emphasize the need to address contextual influences on post-educational application of learning. 23 The Precede/Proceed model is the model of choice upon which to base the conceptual framework of this study for the following reasons. Firstly, it allows the independent variable to be considered in terms of predisposing, enabling and reinforcing factors, supporting the further breakdown of the contextual influences affecting application in the work environment. Secondly, this model addresses performance change. Cervero et al. (1986) acknowledge that this model fails to adequately address this aspect of application. Thirdly, the Precede/Proceed model allows for a feedback loop to the educational program implying implications for practice in program planning (Ottoson, 1995c, 1997). The McDonald (1991) study, although not concerned with health professionals, is particularily useful. It has a large sample size, a return rate of 75%, and the methodology is designed to measure concepts very similar to those in this study. In particular, McDonald points out the lack of assessment in the research literature of the contextual factors which impact on application of learning, and of the interaction of these factors on each other. Evaluation Studies of Continuing Education for Health Professionals The following literature review will explore evaluation studies of continuing professional education within the disciplines of medicine, nursing, mental health counselling and rehabilitation (occupational and physical) therapy. The variables that have been used in these studies to determine effectiveness of the educational program will be addressed in order to support the exploration of the contextual influences proposed in this study. 24 Continuing Medical Education Of all the health professions, physicians have the largest body of literature in continuing professional education, known as Continuing Medical Education (CME). The purpose of CME is to enable practicing physicians to achieve and maintain the knowledge, skills and attitudes they need to carry out their patient care responsibilities at an optimum level of competence (Crandall, 1990). CME is expensive. During the 1970's, $500 million was spent annually in the United States on the development and implementation of CME programs, and the cost of physicians taking leave to attend these programs during this decade was $1.4 billion (Haynes, Davis, McKibbon & Tugwell, 1984). Continuous developments in medical knowledge are changing health care systems, maintaining the demand for ongoing continuing education for physicians. At the same time, however, available funds are diminishing, creating a demand for accountability and proof of effectiveness of these expensive educational programs. Agreement as to what constitutes evidence of effectivess of CME varies. According to Abrahamson (1984), the measurement of effectiveness variables of CME has evolved from attendance, to happiness, knowledge, competence, through to performance and finally to patient care outcomes. As measurement moved from the first four measures to performance and patient care outcomes, the focus of assessment has moved from the academic to the work setting. Physicians are the most advanced group in evaluation research of CME (Abrahamson, 1984). Many studies to determine CME effectiveness have been carried out (Beaudry, 1989; Crandall, 1990; Davis et al. 1984; Davis, Thomson, Oxman & 25 Haynes, 1992; Davis, Thomson, Oxman & Haynes, 1995; Haynes et al. 1984; Wergin, Mazmanian, Miller, Papp & Williams, 1988). The earlier studies by Haynes et al. (1984), Davis et al. (1984) and Beaudry (1989) were large meta-analyses which produced inconclusive results on patient care outcome measures and did not address contextual influences in the workplace. The effects of CME on physician practice behaviour was addressed by Wergin et al. (1988). Using structured telephone interviews before and after a CME course, and examining patient charts (n = 53), these researchers found that changes in practice tended to be: (a) slight modifications of current practice, (b) more accurately expressed as intent rather than actual change, and (c) attributable to other influences other than CME, including research literature and discussions with colleagues. These researchers confirmed strong inter-rater reliability of their data collectors, and strong face validity of their interview schedule. It is recognized that face validity is a casual assessment by untrained individuals, and is therefore a questionable form of validity (Litwin, 1995). However, it is interesting to note the recommendation stemming from this study was to further address the influence of the social context of practice on behaviour change after a CME program. The influence of contextual factors was addressed in literature review studies by Davis et al. (1992, 1995). Both of these studies used the Precede/Proceed model to frame their analysis. In the 1992 study, 50 randomized controlled trials of CME interventions were categorized according to predisposing influences (computer generated information, presentations), predisposing and enabling influences (clinical policy and practice guidelines, patient education materials, practice protocols), 26 predisposing and reinforcing influences (feedback and reminders) and a combination of all three categories. Davis et al. found that the best physician performance and behaviour change occurred when enabling and reinforcing influences were built into the program. The 1995 study by Davis et al. examined 99 trials, and found that the most effective interventions producing change in behaviours were practice enabling strategies — office facilitators, patient educational methods — or reinforcing strategies, such as feedback or reminders. These studies demonstrate the positive effect of contextual influences on behaviour change as well as providing support for the use of the Precede/Proceed model of Green and Kreuter (1991) to assess the role of enabling and reinforcing influences on behaviour. Contextual influences on the application of learning can be facilitative or act as barriers. Crandall (1990), using a qualitative case study design, administered pre, post and follow-up open-ended style questionnaires and follow-up taped interviews to five physicians. This researcher found that behaviour change by these physicians six months after a CME conference was constrained by barriers of lack of resources — financial — and by resistance of clinic staff to change. Although the sample size is very small, limiting the strength of the findings, this qualitative study provides further support for the examination of the facilitative or supportive role of contextual influences on application of learning in the workplace. Martin and Mazmanian (1991) explored the relationship between anticipated barriers to change, planned change and encountered barriers reported by physicians after participation in a CME program. They organized their results around Cervero and Rottet's (1984) framework of four variables that affect the extent of performance 27 change resulting from a continuing professional education program. The four variables include the program itself, the individual professional, the nature of the professed change and the social system in which the professional works (Appendix D). An end-of-program survey and a matched follow-up survey questionnaire were administered to determine intent to change and actual change respectively. The survey questionnaires were tested for construct validity and test-retest reliability. A 94% response rate of matched questionnaires produced 51 subjects. Martin and Mazmanian reported 86% of participating physicians indicated intent to change, and 87% made the intended changes. Reasons given for not planning a change of behaviour or carrying out a planned change included lack of opportunity, influence of co-workers' different work patterns and lack of time. Although these barriers were not described further, they were considered by the researchers as aspects of the social system or context in which the professional works, and as a main force in the process of change in behaviour of the learner. The high cost of CME for physicians, demands for accountability to the paying public and sponsors, and an ever increasing knowledge load have forced the medical establishment to ascertain whether CME is effective in producing behaviour change in the practice setting and in turn improving patient care. Each of the quantitative studies discussed above is methodologically sound, with high response rates, reasonable sample sizes and reported instrument reliability and validity. The qualitative study by Crandall (1990) relates many of the findings to a theoretical model of changing and learning, and supports the results of many quantitative studies. These evaluations of CME have looked at the influence of contextual factors on behaviour change after an educational program and have confirmed the need to further address this variable. They have also provided support for the examination of specific variables of resources, support from peers and colleges, and opportunity to apply learning. Continuing Nursing Education The American Nursing Association defines continuing nursing education (CNE) as planned learning experiences designed to promote the development of knowledge, skills and attitudes for the enhancement of nursing practice, thus improving health care to the public (Gosnell, 1984). Nurses face the same economic constraints as physicians and recognize the need for accountability of continuing education (Alexander, 1990; Donovan, Wolpert & Yasko, 1981). According to Hefferin (1987), evaluation of continuing education for nurses needs to demonstrate accountability, effectiveness and efficiency of the program. Many evaluations of training programs for nurses have been carried out. Hefferin (1987) conducted a meta-analysis of nursing education programs between 1980 and 1985. Outcomes included increases in knowledge. Chart audits were used to determine behaviour change. None of the studies assessed contextual support or its impact on behaviour change. Waddell (1992) conducted a more recent meta-analysis of 34 studies which included research on outcome effects of CNE on nursing practice. The overall mean effect size supported the hypothesis that CNE positively effects practice. The actual analysis of the data, however, revealed so many types of reported practice change 29 that the researchers ended up examining the method of measurements used in the studies. No consistency of results regarding environmental characteristics was established, and the recommendation arising from this study was a further analysis of the ability of the environment to support change. The potential influence of contextual support on application of learning has been investigated by Ramprogus (1989). Using an experimental design, this researcher conducted a formative and summative evaluation of a nurse development course using a post program self-report and a two month follow-up survey with 22 subjects. She noted poor results in change of behaviour over time due to lack of support and interest from management and other staff, and concluded that for CNE to be effective, the working conditions of the nurses in terms of resources and support must be improved. Similar results were found by Bolte and Presler (1983) who evaluated a continuing education program of 3152 maternal child health nurses using pre, post and three, six and nine month follow-up mailed questionnaires. No return rates of any of the questionnaires were reported. The researchers confirmed instrument reliability and validity. This study asked questions designed to measure barriers to instituting change. Findings included declines experienced by participants over time in support by supervisors and colleagues, inadequate resources — such as time — and a link between declining support and declining motivation and practice change. This study was unable to confirm a causitive link between the educational project and enhanced nursing skills. 30 The positive effect of supervisor support on behaviour change is discussed in the evaluation study by Donovan et al. (1981) of eleven continuing education workshops for 199 nurses. In this case, gaps in patient care were identified by participating nurses and a contract was drawn up and presented to the supervisor as to how these gaps could be eliminated. Ongoing support for the participants was provided by workshop facilitators during the writing of the contract and its presentation to the supervisors, through regular follow-up visits to the participants in the work setting, and with the supervisor during implementation. The written contracts served as measures of behaviour change. This approach gave the participants permission, opportunity to apply learning, and support, both from the workshop facilitators and by the supervisors. Identified barriers to implementation included lack of time, resistance from peers and other care givers, and lack of time on the part of the supervisor. Although the sample size is adequate, this study is descriptive in nature only. The only indication of methodology are the written plans and contracts. Campbell and Gammache (1980) describe a nurse executive development program which demonstrates successful carryover from a continuing education program to the workplace and practice. In this eighteen month program, residential sessions, seminars and independent study modules are combined with close contact with a preceptor and faculty for consultation. Time series surveys, telephone interviews and follow-up surveys collected both quantitative and qualitative data from 25 participants, reflecting a 68% response rate. Multiple data sources included self-report, reports from participant's immediate superior, colleagues, preceptor and 31 subordinates, as well as selected documents. The surveys were pretested for validation. Weaknesses of the study included limited baseline data and lack of a control group. Preliminary findings of the first cycle of the program were positive. The authors postulate that the program demonstrated high success rates of application of learning in the workplace due to the length of time of the learning experience, opportunity to practice and continue learning back in the work environment, as well as support received from the study modules and preceptors. At the time of printing, end of program data were not yet available. Kiener and Hentschel (1989) conducted a qualitative exploratory study of 443 nurses after a continuing education workshop. The open-ended three item questionnaire was administered post-program. Participants were asked to identify one new idea or practice that they hoped to use, one factor that would facilitate this process, and one factor that would hinder this process. Each question was presented as a frequency distribution. The majority of participants described a new practice they intended to carry out. Facilitative factors included peer support, administrative support and resources. Hindrance factors included lack of time/workload and lack of administrative support/resources. Limitations of the study included the exploratory nature of the methodology, thereby limiting direct implications for practice, and the lack of a link between intent and actual behaviour change. The researchers recommend a longitudinal study to measure application of learning over time. Next to physicians, nurses have the second most developed body of literature on evaluation of continuing professional education. As a professional body, they are also aware of the potential role of continuing professional education on behaviour 32 change and patient care. The above evaluation studies of continuing education for nurses demonstrate the desire of this professional group to determine the influence of contextual factors on application of learning. Unfortunately, methodological limitations in all but the Campbell and Gammache (1980) study limit the validity of the results. However, the studies do lend support to the examination in this analysis of the contextual influences of encouragement, support, resources and opportunity. Continuing Education for Mental Health Counsellors The available literature on evaluation of continuing education of mental health counsellors is limited. In a meta-analysis of evaluation studies of continuing education for allied health professionals within the past twenty years, Turnbull and Holt (1993) found that this term produced citations including disciplines of nursing, medicine, dentistry, pharmacy, nursing assistants, licensed practical nurses, laboratory workers, dental hygienists, medical residents as well as allied health professionals, who include physical and occupational therapists. After eliminating medicine and nursing, they were left with only twenty-two studies. Of these, the nine which measured application of learning did so in terms of changes in behaviour and patient outcomes. The most relevant and useful of these will be presented below. Richardson (1981) conducted an impact evaluation of a sexual health workshop for health care professionals (clergy, nurses, social workers, psychologists, physicians and vocational rehabilitation therapists) which looked at behavioural change as improvement in quality of counselling. The study involved 126 participants in a pre, post and follow-up self-report design. Although the results indicated significant post-educational behaviour change which participants related to the 33 workshop, there was no attempt to determine which factors supported the behaviour change. In the field of marital and family therapy, as of 1990, there has been little rigorous evaluation of continuing education (Cook, Heath & Martin, 1990). Although these authors give lip service to the necessity of evaluation to ensure effectiveness of continuing education programs, their own evaluation did not concern itself with outcome effectiveness. Rather, it remained with the more tangible items of effects of the presenter, participant characteristics, and participant ratings of the educational experience. Although this group of professionals is aware of the need to determine effectiveness of continuing education, evaluation studies of continuing education for mental health counsellors are few. The data resulting from this study may begin to address this need. Continuing Education for Rehabilitation Specialists Evaluation studies of the effectiveness of continuing professional education within the fields of physical and occupational therapy are also few. Turnbull and Holt's (1993) search included only the following evaluation study of continuing education for physical therapists. Mays (1984) conducted a 2 week to 6 month follow-up impact study of 30 participants which focused on differences of change of practice between an experimental and control group. Methods used to document the change included self-report, record audit and direct observation. Self-report indicated an increase in use of newly acquired techniques by the experimental group over the control group even at the 6 month follow-up time, although no link was made to 34 contextual factors influencing this application. Neither the record audit nor the direct observations yielded significant results. Woog and Hyman's (1980) study of 96 mental health professionals (nursing supervisors, nurses, aides, technicians, social workers, psychologists, psychiatrists, occupational and physical therapists, a dance therapist and licensed practical nurses) evaluated application of learning after the program as well as the impact of application of learning on the clientele. Although the researchers attempted a controlled experimental design, with a pre, post and follow-up format, they were faced with problems which limited their ability to randomly assign participants and retain true experimental control. Application of learning was assessed by self-report, and only one of the four groups to go through the program showed a statistically significant change in application of learning. The variable of 'support' in the ward atmosphere was significantly rated by staff as evident, however no clear link to application was established. Results of this study are limited by low numbers in some of the groups, compromise to the design and questions as to the validity of the instruments. Even more problematic is the fact that participant behaviour was measured indirectly through ward atmosphere. Glenn's (1994) analysis of pre and post-education questionnaires of rehabilitation specialists who participated in the CSAP training program indicated that participants were planning on applying their learning to the post-educational work setting. This study did not include follow-up data from this evaluation, however Glenn recommends the analysis of follow-up data to determine the actual application 35 of learning, which she cites as a vital component of the CTS training program. This study will analyse aspects of the follow-up data to help fill this gap. Evaluation studies of continuing education rehabilitation specialists are scarce. Of the few which do address application of learning following continuing education and contextual influences on behaviour change, only one cites support as an important factor, none link contextual support to application, and the methodology of the studies reviewed is weak. This indicates a gap in the literature for this group of professionals which this research study may help to address. Application of Learning and Related Concepts The research and literature of concern to this thesis address attempts to understand how ideas are transformed into actions. Various disciplines have investigated this process, including psychology, political science, social policy, and communications. These will be outlined below and their contribution to our understanding of application of learning and its use as a variable for this study supported. Diffusion of Innovations The concept of diffusion of innovations is developed through the sociological/psychosociological research and literature, and has been used in such varied fields as education, industry, medicine, agriculture, public health, rural sociology and anthropology. The underlying premise concerns the process of implementation or diffusion of an idea or innovation into widespread daily use or acceptance by the intended audience (Rogers, 1995). Critical elements in the process include the innovation (the idea perceived as new), its communication or diffusion 36 from its source of invention through the social system over time, to its ultimate users or adopters. The above definition recognizes the role of the social system in the adoption of a new idea. This concept of putting an idea into action, however, concentrates more on the movement of the idea through a social system than on the contextual aspects of the social system that support the adoption of the idea and its maintenance of use over time. Implementation Implementation concerns the manifestation of ideas in behavioural terms. The research supporting this concept arises from policy analysis. The concept of implementation is relevant to this study in that it addresses process, and attempts to describe the relationship of the policy, program, or innovation in terms of social, economic, organizational and interpersonal factors (Ottoson, 1993). Hord, Rutherford, Huling-Austin and Hall (1987) elaborate on the process of implementation in their Concerns Based Adoption Model, which emphasizes that change does not occur in a vacuum, but rather is influenced by peoples' feelings about the innovation, their perception about the ability to use it, and important to this study, by the setting in which change occurs and the support and assistance received as the individual attempts to implement the change. These last two variables of the organizational context highlight their potential importance in supporting behaviour change after an educational program. The implementation lens, therefore, also addresses the organizational factors upon which successful behaviour change partly depends. This concept of implementation takes us one step closer to a framework whereby we can 37 begin to address the link between behaviour change following an educational program and contextual variables in the work environment. Transfer of Training Transfer of training research and literature stems from industrial psychology, and concerns the degree to which trainees effectively apply the knowledge, skills and attitudes gained in a training context to the job and maintain them over time (Baldwin & Ford, 1988; Ford, 1994; Georgenson, 1982). According to these authors, the transfer process includes factors of trainee characteristics (addressed in this study as predisposing factors), training design (the workshop), and the work environment (contextual factors in this study). Baldwin and Ford (1988) conducted a meta-analysis of major works in the training literature. According to these authors, there is a scarcity of empirical evidence between factors in the work environment and positive transfer. Highlighted as significant work environment factors, however, are 'support' and 'opportunity to use', two of the contextual variables under analysis in this study. Berger (1977), in a research project designed to examine the relationship between aspects of organizational context and transfer of training, used the variables of support from colleagues/boss and opportunity to try newly learned behaviours in his pre-course, end of course, and follow-up questionnaires of the managers who participated in the program. His results indicated that the strongest relationship to transfer of learning was in motivation, relevance of the learning as well as job autonomy or freedom to innovate, which is closely linked to opportunity. 38 Fox (1984), whose work is based on case examples, also provides support for the investigation of contextual influences in the work environment and their role in the application of learning. As pointed out by Ottoson (1994), the research base of the transfer of training literature was conducted on college students. It generally concerned straight forward memory and psychomotor skills, and did not examine more varied clientele or contexts. It did, however, begin to address the contextual influences of support and opportunity which are included in the framework for this study. Application of Learning The term 'transfer of learning' and 'application of learning' are used interchangeably in various sources (Berger, 1977; Fox, 1984). The confusion in the literature regarding these terms and the resultant difficulty in researching application has been succinctly explained by Ottoson (1995a). As with many aspects of practical and applied skills the focus over recent time by science and academia has been ruled by positivism and technical rationality. The concept of application is defined by Ottoson (1995a) as "to put a thing into practical contact with another" (p. 18). This concept has been lost in the worlds of research and practice under a myriad of other terms and within other disciplines, including those of transfer (psychology), diffusion (communications) and implementation (political science) as described above. Ottoson (1995a), however, suggests that the post-education experience is far too complicated to be understood from any one of these above mentioned lenses alone. Through the process of deconstruction and close examination of the definition 39 of application, this author attempts to revive the term and return it to its rightful place in the understanding of the post-educational process of applying learning. Of particular interest to this study is Ottoson's (1995a) explanation of the word 'into' taken from the definition "to put a thing into practical contact with another" (p. 18). According to this author, the very process of 'into' implies a contextual description of the thing and the another. Contextual influences affect application, and can vary from values, priorities, resources, power, opportunity, authority, relationships and communication channels. Each of these factors will make up a part of the post-educational work environment, and their presence or absence in this environment will either support or present a barrier to the application of the learned behaviours initially and over time. Summary As the above literature review demonstrates, the contextual influences which enable application of learning are multiple and complex. Many factors are at work, including the predisposing influences the learner brings with them, the program itself, and the social or contextual influences of the work environment to which the learner returns. In review, the following references from the transfer of training literature provide support for the investigation of the five contextual influences. Support from colleagues, supervisors, and family is cited as facilitative of transfer by Baldwin and Ford (1988), Berger (1977), Ford (1994) and Fox (1984). Resources are cited as facilitative of transfer by Fox (1984). Opportunity is cited as facilitative of transfer by Baldwin and Ford (1988), Berger (1977), Ford (1994) and Fox (1984). 40 In addition to the references above from the transfer of training literature, the following is a summary of support for these contextual influences from the literatures of continuing professional education and from continuing education for health professionals. Enabling influences considered facilitative of application of learning are authority to act and opportunity to apply learning. The variable of authority is cited as facilitative of application by Ottoson (1995c) in continuing education, and by Donovan et al. (1981) in nursing. The variable of opportunity is cited as facilitative of application of learning by Ottoson (1993, 1995c, 1997) in continuing education and continuing education for health professionals respectively; in the medical literature by Tamblyn and Battista (1993) and Martin and Mazmanian (1991); and in the nursing literature by Campbell and Gammache (1980) and Donovan et al. (1981). The three contextual influences considered reinforcing of application of learning include resources, encouragement from others and organizational support. Resources are cited as facilitative of application of learning in the continuing education literature by Ottoson (1993, 1997); in the medical literature by Crandall (1990), Davis et al. (1992, 1995), Martin and Mazmanian (1991) and Tamblyn and Battista (1993); and in the nursing literature by Bolte and Presler (1983), Donovan et al. (1981), Kiener and Hentschel (1989) and Ramprogus (1989). Encouragement from others is often implied along with organizational support, but is cited separately as facilitative of application by Ottoson (1995c, 1997). Organizational support is cited in the continuing education literature as facilitative of application by Ottoson (1993, 1995b, 1997) and McDonald (1991); in the medical literature by Davis et al. (1992, 1995), Martin and Mazmanian (1991) and Tamblyn 41 and Battista (1993); in the nursing literature by Bolte and Presler (1983), Donovan et al. (1981), Kiener and Hentschel (1989) and Ramprogus (1989); and in the mental health counselling literature by Woog and Hyman (1980). The use of the Precede/Proceed model of Green and Kreuter (1991) as a framework for this analysis is supported by strong research studies of Davis et al. (1992, 1995), Ottoson (1995c, 1997) and Tamblyn and Battista (1993). The existing gap in the literature linking the association of contextual variables in the post-educational work environment to the application of learning has been well established in the previous literature review. The strongest studies in this regard are those of Campbell and Gammache (1980), Crandall (1990), Donovan et al. (1981), Kiener and Hentschel (1989), Martin and Mazmanian (1991), McDonald (1991), and Wergin et al. (1988). Each of these, in their respective disciplines, points to the need to develop a better understanding of the influences of contextual variables on application of learning. Continuing professional education, CME and CNE have made great strides in developing this body of literature. However, there remains a scarcity of literature from the disciplines of mental health and rehabilitation. A need for this information exists to further our understanding of the different ways that contextual factors influence the health professionals' post-educational application of learning in the worksite. C H A P T E R T H R E E : R E S E A R C H D E S I G N 42 This chapter will describe the research design of this study. The Centre for Substance Abuse Prevention (CSAP) Training System (CTS) will be described first with emphasis on aspects designed to enable post-educational application. This will be followed by a discussion of evaluation research and instrumentation procedures, including variable selection. Data collection, sample selection and response rates will be followed by data analysis. Presentation of these steps of the research design process will be followed by a discussion of the limitations of this study. The primary research question gujding this study is: For each health professional group, what is the nature of the association between each of the five contextual influences in the post-educational environment and the application of learning? Also, the role of demographic variables on the associations between the five contextual influences and the application of learning is under investigation. The conceptual framework presented in Chapter One (see page 9) continues to guide this study. The research question is symbolized by the arrows between the contextual influences and post-educational application of learning. These arrows are double-ended to describe the relationships between the variables as associations rather than as causitive. The independent variables are represented by the five contextual influences of organizational support, encouragement from others, opportunity to apply learning, authority to act or apply learning, and sufficient resources. The dependent variable is represented by post-educational application of learning. 43 Description of the Centre for Substance Abuse Prevention Training System The data for this study were drawn from an evaluation of the Centre for Substance Abuse Prevention (CSAP) Training System (CTS)3. The CTS, provided by the U.S. Federal government, offered training and related services to national, state and local organizations to assist them in substance abuse prevention. The training sessions were directed at both professionals and volunteers working in varied professional and community settings across the United States, including religious, government, law enforcement, hospital/health care, private practice, education, volunteer and business organizations (CSAP, 1995). It was the intent of all CTS workshops that practice changes related to substance abuse prevention be sustained over time. To this end, the broad goals of the CTS were to strengthen individual, organizational and community capacity to plan and carry out prevention programs by: (a) encouraging community and organizational development; (b) exploring healthy alternatives to substance abuse for youth and adults; (c) providing information about alcohol, tobacco and other drug abuse; and (d) addressing environmental change and policy development (CSAP, 1995). Of particular relevance to this thesis is the CTS goal that "people who participate in training and technical assistance will apply what they learn, on the job or in their community" (CSAP, 1995). Glenn (1994) confirms, "It is important that training be transferred to real-life situations" (p. 166). 3The terms 'training' and 'education' have been defined in the literature review. Both are applicable in this discussion as the boundaries between them are vague. The CTS workshops from which data are drawn were defined as training sessions by the workshop developers. 44 Health Professional Training Within the broadly stated goals of the CTS, specific curricula were created for various health professions. Of interest here are workshops for physicians, nurses, mental health counsellors and rehabilitation specialists. Curricula for these groups were developed by trainers who represented each group and worked in consultation with members of the target audience. The curriculum for physicians, for example, included pre-event materials and a one-day workshop including modules on prevention of alcohol, tobacco and other drug abuse, prevention in the clinical setting and prevention in the community. The nurses' curriculum included pre-event workshop materials, as well as a one-day workshop with modules on self-exploration of drug abuse issues, biopsychosocial perspectives on drug abuse, drug abuse screening and a full module on development of a preventive action plan. The curriculum for rehabilitation specialists was designed to assist therapists in developing a plan to prevent problems related to alcohol, tobacco and other drugs from interfering with rehabilitation. Participants of this group received pre-workshop information, a one-day workshop intended to develop and practice new skills, and a specific application phase, which included guidance, assistance and support to the agency in the development of a practical, effective and integrated prevention program. The curriculum for the mental health counsellors involved modules designed to introduce them to their role in primary prevention through reinforcement of healthy lifestyles, early identification of drug abuse problems, provision of information on screening techniques, and awareness of the 45 importance of collaborative efforts with diverse organizational groups (Ottoson, 1995b). Application Focus According to Cervero (1984), workshops must possess characteristics conducive to transfer of learning if post-educational application is to occur. Aspects of the CTS designed to support application for the four groups are presented in Table 1. One main intention of the CTS was to enable participants to apply their learning in the workplace. Activities of the CTS designed to enable post-educational application varied from group to group, as each curriculum was determined by the curriculum developers and trainers for each group. All four sample groups were encouraged to prepare for application during the workshop by considering potential barriers in the workplace. Three groups, the nurses, mental health counsellors and the rehabilitation specialists, had a full module dedicated to the development of an action plan to devise specific implementation strategies within their personal lives, with their clients, in their work settings and in their communities. The rehabilitation specialists received specific post-educational application assistance in their workshop, with a final phase termed the Application Phase. Post-event features included funds (for mental health counsellors), contacts and resources, written information including mailings of prevention organizations, and follow-up meetings to confer on implementation of action plans (for nurses, mental health counsellors and rehabilitation specialists). All four groups received support for replication of the course (Ottoson, 1995b). Of particular interest to this study is the fact that post-educational application was an intent of all health professional workshops, however somewhat different 46 Table 1 CTS Training Features Designed to Facilitate Post-Educational Application of Learning Training Features Physicians Nurses Rehabilitation Specialists Mental Health Counsellors Pre-Event Reading Materials On-Site Consideration of Potential Barriers to Application Action Plans Post-Event Funds Application Assistance Course Replication Support Written Information Follow-up Meetings approaches to post-educational application were used to engage the different professional audiences. This supports in part the researcher's decision to analyse post-educational application of each group separately. 47 In order to relate the above features of the CTS training to this study, comparisons will be made between these features and the five contextual influences under investigation. The literature reviewed supports these comparisons. Pre-event reading materials, funds and post-event written information will be categorized as resources (Davis et al, 1992, 1995; Martin & Mazmanian, 1991; Tamblyn & Battista, 1993). The post-event follow-up meetings, as well as the application assistance received by the rehabilitation specialists, will be categorized as encouragement from others and organizational support (Campbell & Gammache, 1980; Donovan et al, 1991). The on-site features of consideration of potential barriers and the development of action plans will be categorized as providing opportunity to apply learning (Campbell & Gammache, 1980; Donovan etal, 1991; Kiener & Hentschel, 1989; Martin & Mazmanian, 1991). The post-event course replication support, although not documented in this literature review, will also be considered as providing permission or authority to apply learning. Evaluation and Evaluation Research Evaluation is defined by Sanders (1994) as the systematic investigation of the worth or merit of an object which can be a program, a curriculum, a policy or administrative regulation, a workshop, or the educational or training materials themselves. Evaluation provides information to aid in the decision making process about educational practices, as well as adding to the research-based knowledge about a specific practice. Initially, decisions regarding educational practices were informal. However, as society placed increasing emphasis on education, and allocated it increasing 48 resources, it also demanded that the knowledge derived from evaluation be more systematic and verifiable. Evaluation has become increasingly systematic in its application of research skills to determine merit or worth, and is now termed evaluation research. Evaluation research requires a formal evaluation design and procedures in order to systematically collect and analyse data (Schumacher & McMillan, 1993). Evaluators work in either quantitative or qualitative methodologies, or a combination of both. As discussed in the literature review, evaluation studies can take many forms. Some evaluations of CPE are impact studies which attempt to determine the extent and type of impact the educational process has on subsequent performance (Knox, 1985). The CTS evaluation is an impact study which attempts to assess the post-educational application of learning of participants. Standards for Program Evaluation Research The root of the term evaluate implies a determination of value of the object being evaluated. There has been considerable debate among evaluators about how to assign value. To address this in part, and to improve credibility of evaluation practices and cohesion between practitioners and researchers, standards for program evaluation have been developed (Sanders, 1994). These standards ensure that a good evaluation study satisfies utility, feasibility, propriety and accuracy criteria. Utility standards ensure that evaluations are informative, timely and influential. Feasibility standards ensure that evaluations are efficient, cost effective and practical. Accuracy standards ensure that evaluations produce comprehensive, sound and valid information. Propriety standards ensure protection of participants against unlawful, 4 9 unethical or inept actions of the evaluators by providing for confidentiality and anonymity. According to Sanders (1994), these standards are equally applicable to longer programs and shorter workshops. Implications for this Study The above standards impacted on the data available for this thesis in several ways. The feasibility standards of economics and practicality are evident in the use of self-report survey questionnaires as the main data collection tool. This thesis is therefore subject to the advantages and disadvantages of this methodology. Propriety standards, which are designed to ensure anonymity, also affected this study. The evaluators assigned participants codes, then asked for signed consent to link names to codes for mail-out of follow-up questionnaires. As this threatened anonymity, many participants did not sign consent forms and were not included in the mail-out, greatly reducing sample size. The CTS Evaluation provided the data for this thesis. This study is therefore shaped and guided by this larger work in the following ways. The questions were preset by the evaluating team. The sample was determined by the kinds of participants who attended the workshops. Options for data analysis were constrained by the design of the questions and the measurement scales. This thesis is not an evaluation assigning worth or merit to these workshops, but rather is an investigation into how contextual influences in the workplace are associated with application of learning. This investigation is using data from this larger evaluation study. Results may assist adult educators and health professionals in several ways. Firstly, they may assist evaluators and workshop planners in 50 understanding the factors in the post-educational work environment that facilitate application of learning. Secondly, they may provide evaluators with potential variables to consider when determining workshop merit, worth and outcomes. Instrumentation of the CTS Evaluation This thesis draws its data from the CTS evaluation study which began in 1991 and took four years to complete. The purpose of this national U.S. evaluation study was to investigate the post-educational application of learning of workshop participants in the Centre for Substance Abuse Prevention (CSAP) Training Systems (CTS). Data for the CTS evaluation study was collected through multiple methods, including questionnaires, observation, curriculum review and on-site and telephone interviews with participants and staff (CSAP, 1995). Data of interest to this study were derived from self-report survey questionnaires. Advantages and Disadvantages of Self-Report Survey Questionnaires in Evaluation Research Self-report survey questionnaires are the most commonly used instrument in continuing education evaluation (Nyquist, 1985); they have the following strengths and weaknesses. Survey questionnaires are a cost-effective and efficient means of collecting original data from a population too large or too spread out to observe directly (Fowler, 1993). They are convenient to administer and provide ease of scoring. They are also unobtrusive, allow for assessment across time, and may elicit more honest answers than face-to-face interviews. Self-report surveys are a reasonable 51 approach when more direct measures such as participant observation or chart audits are not feasible due to cost, time, or distance. Despite the above advantages, survey questionnaires have weaknesses which can counter the strength of the results. Disadvantages of this method include low rates of return when mailed, the inability to ensure understanding of questions (Nyquist, 1985), and the difficulty of measuring behaviour without an understanding of the social context within which the behaviour occurs (Babbie, 1992). Additional weaknesses of survey questionnaires include inability to probe, imposition of reality through options to the questions, and no recourse to blank or strange answers (Fowler, 1993). Strengths and Weaknesses of the CTS Evaluation Survey In addition to these general advantages and disadvantages of survey questionnaires, there are also issues specific to this evaluation context. One strength of this survey is that it does attempt to assess the social context within which the application of learning is to occur. This instrument has several weaknesses. The first concerns question design. A problem specific to the CTS evaluation questionnaires is their use in all CTS workshops, including volunteers, health professionals and community representatives. This attempt to collect data across all CTS workshops meant that questionnaire items had to be relevant to all groups. For example, the question from which the dependent variable is drawn refers to both 'work and volunteer activities' (Appendix C). Even though attempts were made to ensure separation in analysis of data relating 52 to the different groups, wording in the question referring to all groups could create confusion for the respondents. Another weakness specific to this survey relates to question wording. An example is the term 'resources' in the question regarding available contextual influences in the post-educational work environment (Appendix Q . The literature tells us that resources can imply various concepts, including time, financial or economic support, or written materials (Bolte & Presler, 1983; Kiener & Hentschel, 1989; Martin & Mazmanian, 1991; Ramprogus, 1989). This term may, therefore, not accurately reflect the varied post-educational circumstances for participants from different backgrounds. Another key weakness of survey questionnaires and a problem in this study is non-response. Failure to collect responses from a large number of individuals in a sample is an important source of error in survey research (Fowler, 1993). Low response rates, and therefore small samples, impair statistical power of the analysis. In this study specifically, the small sample sizes led to often inadequate cell counts in the crosstabulations. CTS Instrumentation The survey instruments used in this study were designed by a team of evaluators hired by CSAP, training contractors in the CSAP Training System, and federal project officers in the U.S. Department of Health and Human Services who sponsored the CSAP investigation and CTS workshops. The questionnaires were based in part on a conceptual framework of the application process developed by the CSAP investigators (Ottoson, 1995c) (Appendix B). 53 The Variables in this Study The variables in this study represent contextual influences on post-educational application of learning. They were originally conceived through the combined effort of the stakeholders involved in the CTS evaluation study. These stakeholders included policy makers, program developers, the trainers involved in presenting the workshops, and the evaluators hired to conduct the evaluation. The variables used in the evaluation study are based on the conceptual framework discussed above (Appendix B). They are supported by extensive literature reviews carried out by the team of evaluators of evaluation studies of continuing professional education for health professionals, as well as by the literature reviewed in Chapter Two. The independent and dependent variables drawn from the follow-up questionnaire are presented in Table 2. Table 2 Independent and Dependent Variables from the Follow-up Questionnaire Independent Variables Dependent Variable Support for making changes suggested during the workshop Encouragement from others Opportunity to apply learning Authority to act or apply learning Sufficient resources These variables are based on ordinal units of measurement. An ordinal measurement is only possible when different amounts of an attribute can be discerned (Glass & Hopkins, 1996). In this study, this measurement scale ranks the Made changes in how you do your work or volunteer activities 54 reported availability of the contextual influence from 'not at all' to 'substantial' and the application of learning from 'not at all' to 'extremely'. Ongoing debate exists about the treatment of data derived from ordinal scales as interval data, and the appropriateness of parametric analytic techniques to such converted data (Glass & Hopkins, 1996). Traditionally, an ordinal scale was considered limited in use as it can only describe data in terms of frequencies, percentages and medians or modes. Technically, an ordinal scale cannot be converted to an interval scale, nor can it provide means or standard deviations. Therefore, the complexity and strength of analytic techniques is limited by ordinal scales. However, some authors make exceptions to this ruling by allowing use of an ordinal scale to compute means and standard deviations (Schumacher & McMillan, 1993). The data in this study will be treated as ordinal because the variables are ranked without a standard unit of measurement indicating magnitude of difference. The relevant non-parametric statistics used will be less sensitive but will make fewer assumptions about the data. Independent Variables The independent variables, contextual influences on post-educational application of learning, refer to the situational aspects of the work environment to which the learner returns. These five contextual variables, presented in this study as facilitative of post-educational application of learning, are found in question 7a-e on the Follow-up Questionnaire: To what extent do the following exist in your organization or community to help you apply your learning? 55 (a) Sufficient resources, (b) Encouragement from others, (c) Opportunity to apply learning, (d) Authority to act or apply learning, (e) Support for making changes suggested during this workshop. The availability of each contextual influence in the post-educational organization or community was ranked by participants from 1 (not at all) to 5 (substantial). Only these end points on the 5-point scale were anchored with labels. Dependent Variable The dependent variable in this study is post-educational application of learning. As noted in the literature review, outcomes of CPE include new or enhanced personal knowledge, participant satisfaction, program evaluation, application of learning and impact on clientele (Knox, 1985; Martinoff, 1985). Application of learning, evidenced through behaviour change, is increasingly being viewed as an indicator of successful education, both from a practice and a financial point of view (Turnbull & Holt, 1993). Evaluations of CPE are consequently looking to establish application of learning as an impact of the educational process. The Follow-Up Questionnaire contains two indicators of application of learning in response to Question 5: To what extent did you do the following as a result of this workshop? (b) Made changes in how you do your work or volunteer activities, (g) Increased substance abuse prevention activities. 56 Question 5(g) was considered to imply activities involving policy and procedural changes, thereby taking the change in behaviour through and beyond the initial application concept. Since the workshop curricula included specific action plans, making changes in work activities which involved implementation of these plans was considered evidence of application of learning. For this reason, question 5(b) was chosen as the dependent variable. This indicator of application of learning was ranked by participants from 1 (not at all) to 5 (extremely). Again, only these end points on the 5-point scale were anchored with labels. Demographic Variables The demographic variables used to describe each health professional group include age, gender, educational level, primary work focus and organizational type represented. All of these variables are found in the pre-questionnaire (Appendix C), and only data from participants matched across time is used. The variable of age was coded into five categories as follows: 25 years or less; 26-35 years; 36-45 years; 46-55 years; and over 55 years. This variable is treated as a categorical variable. Gender was a categorical or nominal variable of male or female.4 Educational level, measured on an ordinal scale, was assessed as follows: grade school or some high school; completed high school; some college or trade/ vocational school; college degree; and graduate school, graduate degree. 4 A nominal scale describes characteristics that have no numerical values. This most rudimentary scale uses numbers to distinguish among the categories and assigns the same value to each individual in that category (Glass & Hopkins, 1996). 57 The categorical or nominal variable of primary work focus asked participants which of the following best described the primary focus of their substance abuse activities: prevention/education; treatment/intervention; both prevention and treatment; or other. The categorical or nominal variable of organizational type represented asked participants which type of organization they primarily represented: government; law enforcement/legal; hospital/health care; private practice; education; religious; volunteer; business; not representing an organization; or other (Appendix Q . The demographic variables also act as control variables in the inferential analysis. Although not 'controls' in the true experimental sense, these variables are relevant in the context of this study. Gender has been investigated in terms of women's learning in the feminist learning literature (Hayes, 1989). Educational level has been linked to motivation and response rate (Fowler, 1993). Age has been addressed by Cox & Baker (1981) as relevant to application. The focus of this study on the worksite adds relevance to the investigation of variables of organizational type represented and primary work focus. The investigation into the role of these demographic variables may provide a better understanding of the associations and indicate directions for further research (Eichelberger, 1989). Instrument Reliability Reliability of an instrument is a statistical measure of the reproducibility of the survey instrument's data (Litwin, 1995). Test-retest reliability, which requires the same 58 set of respondents to complete the same survey at two different points in time, was not possible in this situation, as the subjects were only available for questionnaire distribution at one time, namely at the CTS workshops. Also, since the questionnaires were administered to many different groups, test-retest reliability measures would have been necessary for each group. This would have been time consuming and expensive for such a large study. An alternative option, that of running a test-retest or equivalency trial using only part of the study sample would have resulted in a smaller sample, weakening the potential power of the statistical analysis. Reliability of an instrument is related to question design. Good question design ensures an accurate measure of reality. Reliability of the instrument in this study is questionable because the same questionnaire was designed for and administered to all the different volunteer and professional groups. Instrument reliability is therefore impaired because responses will reflect the different realities of the varied post-educational work environments. Reliability of data can also be impaired by other aspects of self-report questionnaires, such as the inability to probe, the imposition of reality through options to questions and the lack of recourse to blank answers (Fowler, 1993). These are also issues of concern to this study. For the above reasons the reliability of this survey questionnaire was not established. Instrument Validity Instrument validity refers to how well the instrument measures what it is designed to measure (Litwin, 1995). Important measures of instrument validity are 59 construct, content and face, concurrent and predictive validity. There are also validity issues specific to evaluation research. Construct Validity Construct validity refers to a theoretical measure of how meaningful the survey instrument is (Litwin, 1995). In this case, the survey was based on the conceptual framework developed by Ottoson (1995b) and presented in Appendix B. The variables within this contextual framework were grounded in the following theoretical models of adult education, health education/promotion and decision making, each of which have developed their own construct validity through years of experience and application in their respective fields. Cervero's Model, developed by Cervero and Rottet (1984), attempts to determine the effectiveness of continuing professional education through an analysis of multiple factors, including the individual professional, the proposed change, the CPE program and the social system to which the learner returns. The Precede/Proceed Model, developed by Green and Kreuter (1991) was designed to provide a framework to understand the multitude of factors influencing health behaviour change through health education. Models supporting the decision making process of the framework are based on works by Hall and Hord (1979) and Rogers (1995). The CTS evaluation questionnaires can be described as having strong construct validity. Face and Content Validity Face and content validity refer, respectively, to a casual and formal expert review of how reasonably an item measures a variable (Litwin, 1995). In this case, face and content validity were established through an examination of the literature, consultation with an expert community panel, technical advisory panel, CTS contractors and trainees, use of focus groups, and pilot tests with health care professionals and community groups. The consensus of these groups was that the survey questionnaire was an appropriate, accurate and representative measure of the application process, and therefore is strong in face and content validity. Concurrent Validity Concurrent validity is a measure of survey accuracy in which the results of a new survey are compared with the results from a generally accepted 'gold standard' test after both tests are administered to the same group of respondents (Litwin, 1995). This was not possible in this evaluation study due to the expansive geographical distribution of participants. Also, the cost of administering two sets of pre, post and follow-up questionnaires to such a large group of individuals would have been prohibitive. Predictive Validity Predictive validity refers to how well the scale or item predicts expected future observations or behaviours (Litwin, 1995). Establishing predictive validity requires the administration of a second test later in the study, and correlating the results of an initial test with the second. Although predictive validity is one of the most important ways to measure item accuracy it is seldom used in studies with longer time frames because of the possibility of interference from outside influences. For this reason it was not practical to determine predictive validity of this survey. 61 Validity in Evaluation Research According to Abrahamson (1984), there are conflicting opinions as to the validity of self-report survey questionnaires. The Joint Committee on Standards for Educational Evaluation describes validity in terms of the soundness or trustworthiness of the inferences made based on the results of the data gathering process, rather than of the specific instruments themselves. Methods recommended to ensure valid information include the measurement of multiple outcomes, multiple information gathering procedures and wording of questions to maximize understanding and minimize bias or socially acceptable responses (Sanders, 1994). The follow-up survey from which the data for this study is drawn does measure multiple outcomes, although only one will be used for this thesis. The evaluators have also drawn on multiple information gathering procedures, although in this thesis, only the survey questionnaires will be used. There remain, however, several concerns. One regards question design to ensure understanding, as in the multiple possible meanings of the term 'resources'. Also, there is the problem of response bias inherent in survey research. Response bias refers to the bias resulting from the likelihood that respondents who answer all three questionnaires are different in several ways from those who only answered the first questionnaire. By essentially volunteering their time to answer all three questionnaires, they demonstrate more motivation, and could also be better educated and of a higher social class (Schumacher & McMillan, 1993). Determining representativeness of the sample will help to address this issue. 62 Data Collection The workshops and data collection began in 1994 and extended through 1996. The instruments and workshops were pilot tested in 1993. CTS Instrument Administration The national CTS evaluation team collected data using three self-report survey questionnaires (see Appendix C). The first, the participant profile questionnaire, coded as 'PRE', was distributed and introduced by the trainer to all participants at the beginning of each training session using a standardized presentation. The purpose of this questionnaire was to obtain demographic and background information about the participants and to assess their reasons and preparation for participation. The second questionnaire, coded 'POST', was administered to all participants at the end of the training sessions. This questionnaire was designed to obtain feedback about the training sessions, participant self-assessment of skill and intent, and anticipated support for application of learning in the post-educational context. Participants were asked to complete this questionnaire before leaving the session. The third questionnaire was a follow-up, coded 'FU', and was mailed 2 to 4 months after the training session had been completed. This questionnaire was designed to obtain information concerning the application of learning following the training and contextual influences on that process. For those who did not respond to the first mailing, a reminder notice was sent. Anonymity and Confidentiality Maintaining anonymity and confidentiality was a concern of the researchers, trainers and policy makers. Of relevance specifically to evaluation research is the 63 propriety standard proposed by the Joint Committee on Standards for Educational Evaluation (Sanders, 1994). The 'Rights of Human Subjects' standard ensures that the evaluation is designed and conducted in such a way as to protect the rights and welfare of human subjects. Procedures which ensure anonymity and confidentiality can ensure that this standard is upheld. Cervero (1984) states that anonymity is important in evaluations, as it can help reduce the likelihood of bias toward socially acceptable responses. A great deal of effort was expended to ensure anonymity in this evaluation. All questionnaires were coded to ensure participant confidentiality, and only matched coded questionnaires, tracked over time, were used for the data base of this thesis. Matched responses were those of participants who completed all three questionnaires. Three different kinds of coding systems were established in order to facilitate the matching of respondent data while preserving anonymity and confidentiality. The first system, designed to protect anonymity, involved participant generated codes. Participants were to remember their code and put it on all three forms. Problems arose when people forgot their codes, resulting in lost data. Eventually, home phone numbers were used. Although this improved recall and ensured valid forms, the use of home phone numbers can compromise anonymity. A better method involves the use of one's mother's maiden name. The second system, designed to maintain confidentiality, involved trainer generated codes. Trainers for the dentists and physicians coded the questionnaires and maintained a master list. The evaluators who collected the data from the questionnaires never had access to this list. 64 The third system involved a consent form. Participants were assigned a code by the researchers, then asked to sign a consent form that linked their name with the code to allow for mailing of follow-up questionnaires. This system was only used in a few nurse, counsellor and rehabilitation training workshops as it was seen as a threat to confidentiality. Physicians did not use this system at all. The data log (Appendix E) indicates that only 146 nurses, 117 mental health counsellors and 98 rehabilitation specialists signed consent forms. Substantial numbers of participants did not sign consent forms, and therefore were not included in the mail out of the follow-up questionnaires. An ethics review process was submitted by the researchers which was approved by the University of British Columbia Behavioural Sciences Screening Committee for Research and Other Studies Involving Human Subjects (Appendix F). Confidentiality was maintained by limiting access to respondent codes and contact information to the principle researchers. Response Rates Failure to collect responses from a large number of individuals in a sample is a major source of error in survey research. The effect of non-response on survey results depends both on the percentage not responding and the extent to which those not responding are biased, or systematically different, from the whole population (Fowler, 1993). Non-response bias is an important weakness of mail surveys. According to Fowler (1993), mail surveys are consistently biased towards more educated, and therefore more motivated, participants. This is especially critical in this case since this 65 thesis is concerned with education and the application of learning. A sample biased towards high levels of education will determine to whom the results can be generalized. Opinions about acceptable response rates vary. According to Babbie (1992), a response rate of 50% is considered adequate in social research. Reported response rates in the reviewed literature to follow-up questionnaires of health professionals in CPE range from 94% (Martin & Mazmanian, 1991) to 42% (M. Grosser, personal communication, September 30, 1997) for physicians, and a reported 100% by Peden et al. (1992) for nurses. Many of the studies which report high response rates also had a single administration of the questionnaire to smaller samples. This study involved multiple administrations — pre, post and follow-up questionnaires — as well as distribution over vast geographical distances. Additional issues of using only matched questionnaires and the problems with the consent forms led to these lower response rates. Taking these aspects of the administration of the survey into consideration, it is suggested that these response rates, although not optimal, are adequate for this study. Response rates to the follow-up questionnaires of participants from each health professional group are reported as percentages in Table 3. The participants included in this study are those who responded to all three questionnaires and who are matched anonymously by participant code. Sample Selection During 1994 and 1995, 9500 participants took part in 31 different types of CTS training sessions conducted across the United States in 249 separate events. Table 3 Sample and Response Rates Based on Matched Pre. Post and Followup Questionnaires Sample # Pre # Matched Pre/Post/Fol low-up % Matched Physicians 110 38 34 Nurses 468 122 26 Mental Health Counsellors 468 119 25 Rehabilitation Specialists 242 62 26 Note. Pre = number of respondents who answered the Participant Profile Questionnaire; Matched Pre/Post and Follow-up = number of respondents who answered all three questionnaires, matched anonymously by participant code; Percent = percent of matched respondents who answered all three questionnaires, matched anonymously by participant code. Participants represented different multi-cultural, community and volunteer groups, as well as health care services. Included in the total of all CTS participants were 1771 health care professionals, such as representatives from health care delivery, HIV & alcohol, tobacco and other drug (ATOD) groups, dentistry, physicians, physician's assistants, counsellors, social workers, nurses and rehabilitation therapists. Participants from these groups attended 54 training events (Ottoson, 1995 b, 1996). The sample for this study is presented in Table 3. The unit of analysis is the individual participant. The table illustrates the number of participants from each health professional group who completed the participant profile questionnaire. It also shows the number who completed all three questionnaires: the participant profile 67 (pre), the participant feedback (post), and the 2 month follow-up (FU) questionnaire. The participants of interest to this study are those who completed all three questionnaires and who are matched across time anonymously by participant codes. This thesis is concerned with the training of four health professional groups. The choice to specifically address physicians, nurses, mental health counsellors and rehabilitation specialists is based on the following reasons. Firstly, some groups were not included because they had few participants and only met once, generating insufficient data for analysis. Secondly, the low response rates of the other groups of health professionals made the data from those groups too weak. Lastly, the researcher's personal work experiences with individuals from these groups provided additional incentive to chose these four groups for the final sample. This is a nonprobability sample of participants who attended all the training sessions. Nonprobability sampling involves using the subjects who are available to the researcher. Ease of obtaining the sample by this method is countered by the disadvantage of restricted generalizability of results to the larger population (Schumacher & McMillan, 1993). Data Analysis SPSS (Statistical Program for the Social Sciences) version 7.5 is used to conduct descriptive and inferential analysis of the relevant data from each health professional group. Descriptive Analysis Descriptive statistics are used to provide information about the distribution of attributes of each of the four health professional groups who represented the final 68 sample. Included in this analysis are the demographic variables, the five contextual influences representing the independent variables, and application of learning representing the dependent variable. Since the demographic variables are of varying measurement scales they are described by frequencies and percentages. The independent and dependent variables are ordinal in measurement, indicating rank without a measurement unit. The central tendency — the average or typical measure — of these variables is described by the median. The median is a measurement of central tendency for variables which are rank-ordered (Schumacher & McMillan, 1993). The shape of the distribution of the independent and dependent variables is described by their skew, which, if not symmetrical, is either negative or positive. A negative skew has a higher percentage of scores in the lower values of a distribution, while a positive skew has a higher percentage of scores in the higher values. The median is more stable in skewed distributions than the mean as it is not as subject to the effects of extreme scores. However, the median is not as reliable a measure of central tendency as the mean, and not as adaptive to statistical techniques (Hopkins, Glass & Hopkins, 1987). Sample Representativeness Establishing representativeness is a method of determining that the respondents look essentially like a random sample of the larger population. The greater the non-response, the more important the degree of representativeness of those responding (Fowler, 1993). Some experts state that establishing how well those responding 69 represent those in the larger population is even more important than the return rate (B. Perrin, personal communication, February 27, 1997). The nine situational variables used to determine representativeness of the participants matched at pre, post and follow-up with those who completed the pre-questionnaire only are summarized by descriptive statistics. This involves frequency counts of unmatched and matched respondents from each health professional group. Frequency counts are the method of choice because all the variables except age are nominal or ordinal in measurement and means are therefore not appropriate (Schumacher & McMillan, 1993). Thirty-six crosstabulations were made as follows. On the pre-questionnaire and for each of the four health professional groups, comparisons were made between matched across time and unmatched respondents on the following variables: 1. Reason for participation, operationalized in 2(b) as 'need to do your job or volunteer activities differently'; 2. Previous training in substance abuse prevention, operationalized in 4(a) as 'to what extent have you had previous training or education in substance abuse prevention'; 3. Expectation of others, operationalized in 4(f) as 'does anyone, other than yourself, expect something from you as a result of attending this workshop'; 4. Organization represented; 5. Gender; 6. Age; 70 7. Education. The latter four variables were described fully on page 55 and 56. On the follow-up questionnaire and for each of the four health professional groups, comparisons were made for those matched across time and unmatched respondents on the following variables: 8. Relevance of workshop to your job, question 3(a); 9. Preparation for application, question 4(f). Crosstabulations testing for representativeness of matched to unmatched respondents on variables of age, organizational type represented, relevance to your job, and preparation for application, resulted in cell counts of unacceptable levels. According to SPSS, version 7.5 (1997), in crosstabulations no more than 20% of categories should have expected frequencies less than 5. In order to achieve acceptable cell counts with these variables, they were recoded as follows: (a) age categories of 25 years or less, 26-35 years and 36-45 years were recoded into one group, and age categories of 46-55 years and 55 years and older were recoded into a second group; (b) organizational type represented, originally nine categories, was recoded into three as follows: (1) law/legal and government were combined into one category; (2) hospital became the second category; (3) education became the third category. The remainder of the categories, which included low counts in religious, volunteer, business, private practice and other, were omitted; 71 (c) the variable of relevance to your job, and (d) the variable of preparation for application, both of which were on a five point scale, were recoded from five categories to two as follows: value 1, 2 and 3 were recoded as low, and values 4 and 5 were recoded as high. This resulted in adequate cell counts for the crosstabulations between unmatched and matched nurses and mental health counsellors. The crosstabulations for these two variables for the rehabilitation specialists remained inadequate. The variables of age, gender, organization represented and education were chosen to address the question of generalizability of the matched sample to the original larger population of participants. The remaining five variables were chosen because they were felt to be relevant indicators of the outcome behaviour of application of learning. Their selection was informed by the literature (Berger, 1977; Campbell & Gammache, 1980; Davis et al, 1995; Donovan et al, 1981; Green & Kreuter, 1991; McDonald, 1991; Ottoson, 1992, 1995c; Ramprogus, 1991). Inferential Analysis Inferential analysis refers to the use of statistical procedures to make inferences about the similarity of the sample to the larger population. Inferential analysis uses the results of the analysis of descriptive statistics (Schumacher & McMillan, 1993). For this study, two stages of inferential analysis are used to answer the research question. The first stage treats each group separately for the following reasons. Each group had: (a) a different educational background, (b) varied preparation for application within the CTS (see Table 1), (c) different working 72 conditions, and (d) a different professional philosophy. For example, in the CTS workshops, the nurses, mental health counsellors and rehabilitation specialists received specific action plans focused on application while the physicians were only encouraged to consider barriers to application. In the workplace, nurses and rehabilitation therapists tend to work in teams while physicians and mental health counsellors tend to work alone on a case by case basis or in solo practice. The first stage of analysis included twenty crosstabulations. For each health professional group, each of the five contextual variables was crosstabulated with the dependent variable. This produced high percentages of cells with counts less than five. As noted above, according to SPSS, version 7.5, in crosstabulations no more than 20% of categories should have expected frequencies less than 5. Therefore, in order to ensure adequate cell counts, the independent and dependent variables were recoded from five values to three. One and 2 were recoded as low, 3 was left as moderate, and 4 and 5 were recoded as high. The analysis was then rerun with three value categories instead of five. Once again all the crosstabulations for the physicians and three of five crosstabulations of the rehabilitation specialists had insufficient cell counts. The crosstabulations of the nurses and mental health counsellors had adequate cell counts. To attempt to ameliorate the low cell counts of the physicians and rehabilitation specialists, the three value categories were again recoded into high and low. Values 1, 2 and 3 were recoded as low, and values 4 and 5 were recoded as high. The physician group still had inadequate cell counts in all crosstabulations. The rehabilitation specialists had adequate cell counts in all cases. 73 The first stage results of the crosstabulations of the physicians are invalid due to the inadequate cell counts and are not presented. All other results are presented at three levels of value categories, with the exclusion of the three crosstabulations of the rehabilitation specialists which are invalid. The rationale to report the results at this level is based on the fact that with each compression of values, the power of the statistical test weakens. The second stage, which addresses the role of demographic variables — age, gender, primary work focus, organizational type represented and education — on the associations between contextual influences and application of learning, combined the four groups, including the physicians, into one sample. The rationale for this is based on the descriptive analysis which resulted in the same median scores on three of the four reported contextual influences and on the reported application of learning. In addition to these similarities, although not demonstrated statistically, the majority of respondents were of a similar age, reported similarly high levels of education, represented hospital settings and had both treatment and prevention as their primary work focus. Also, the investigation of the role of gender would be irrelevant without the physicians, as this group is predominantly male. Both independent and dependent variables remained recoded as above from five value categories into three. Rationale for the use of Chi-Square Chi-square is an inferential non-parametric statistical procedure which can be used when data is nominal or ordinal and when distributions are not normal (Schumacher & McMillan, 1993). Non-parametric statistics require only the assumptions that scores are independent observations, and that a research question is 74 being tested (Eichelberger, 1989). Since the independent and dependent variables in this study are based on ordinal units of measurement, and data are non-normal in distribution, chi-square is the method of choice for this analysis. Chi-square is also felt to be appropriate since the research question implies an association between rank-ordered variables. Chi-square answers questions about association based on frequencies in categories, however it does not indicate the strength or direction of the relationship (SPSS, 1993). The chi-square test of association is used in this study to determine whether an association exists between the variables within each health professional group. Significance Level The significance level is set at .05. Babbie (1992) states that in social science research, decreasing the significance level to .01 requires sample sizes of 400 or more. Since the sample sizes of two of the four health professional groups in this study are just over 100, and in the other two groups, less than 100 (see Table 2), .05 is felt to be a reasonable level. The Pearson chi-square, a statistic calculated by SPSS, produces a significance level which is compared to alpha of .05 to test for an association. Methodological Limitations of this Study This study demonstrates several methodological limitations which must be recognized. The weaknesses inherent to the survey questionnaire can impact on the strength of results of this thesis. The survey instrument in this study has not been 75 demonstrated to be reliable. Also, question design can result in misunderstanding of the meaning of concepts. Resources, for example, were described in the literature by different health professionals as representing either time or funding. Similar confusion could result in understanding any of the other contextual variables, impairing validity. Another issue of question design concerns the use of labels only to anchor the end points of the 5-point measurement scales of the independent and dependent variables. A lack of concise anchors at the other points of the scale can cause indecision or ambiguity over meaning for respondents. There exists the potential for response bias inherent in individuals who voluntarily dedicate time and energy to responding to follow-up questionnaires. The low return rates of the follow-up questionnaires also undermine the analysis through cell counts of less than five in a high percentage of crosstabulations. Although recoding and regrouping the samples relieved this problem to some degree, there were still cells with counts less than five in the final analysis, weakening the validity of the results. The use of the self-report survey as the only method of data collection to understand the post-educational setting, although convenient and cost-effective, may not be able to assess the nuances of such a complex social environment. These methodological limitations must be recognized in order to determine whether the results of this study are generalizable to the larger population of health 76 professionals who are re-entering the work environment after a continuing education experience. CHAPTER FOUR: RESULTS 77 This chapter will present the results of the data analysis. The format will follow the conceptual framework presented in Figure 1 (see page 9). Specifically, a descriptive analysis of demographic variables of each health professional group will be presented in order to describe each group. The variables used to determine representativeness will then be summarized, followed by the results of the within group tests of association to determine sample representativeness. Following the direction described by the conceptual framework in Figure 1, descriptive statistics will be presented to summarize the five reported contextual influences and the reported post-educational application of learning by each health professional group. These results, as well as the demographic analysis, will provide support for combining the four groups into one group in the second stage of the inferential analysis. The results of both stages of the inferential analysis will follow. The level of significance is set at .05. The primary research question guiding this study is: For each health professional group, what is the nature of the association between each of the five contextual influences in the post-educational environment and the application of learning? An additional area under investigation is the role of demographic variables on these associations among health professionals. This question is symbolized in the conceptual framework by double-ended arrows between the contextual influences and post-educational application of 78 learning. The arrows are double to indicate that this is a question of association rather than causation. Demographic Characteristics The demographic variables used to describe each health professional group, summarized above in Chapter Three, include age, gender, educational level, organizational type represented and primary work focus. Demographic statistics for each of the four health professional groups are described below as frequencies and percentages. In each case, the sample group represents respondents who answered all three questionnaires, the pre-questionnaire, the post-questionnaire and the follow-up questionnaire, and who were matched anonymously by participant code. Physicians The matched response rate of physicians was 34%, representing a sample group of 38. The majority (60.5%) of physicians were males, 37.8% of whom were between the ages of 36 and 45 years. Seven of this group were under 35 years, and 16 respondents were over 45 years in age. Two physicians reported having a college degree, and the remaining 94.7% reported having a graduate degree. Fourteen (37.8%) physicians reported hospitals as the organization they represented. Eight represented government, 4 were in private practice, 3 represented education and 1 represented a volunteer organization. The majority of physicians (63.2%) reported having both prevention and treatment as their primary work focus, 18.4% reported having a prevention focus, and 5.3% a treatment focus. 79 Nurses The matched response rate for nurses was 26%, representing a sample group of 122. In this sample group, the majority (97.4%) of participants were female, with only 3 male respondents in the group. The largest group of nurses (39.3%) were between the age of 36-45 years. The next largest group (26.2%) were between 46 and 55 years. Twenty-one respondents were over 55, and 18 were under 36 years of age. Nurses also represented a high level of education, with 41 % of those responding reporting a college degree and 29.9% a graduate degree. Thirty-one nurses had some college. Three remaining nurses reported high school and grade school as their level of education. This result is questionable as either a coding error, or the respondent could have been a nurse's aide or licensed practical nurse rather than a nurse. Almost half (45.3%) of the nurses represented hospitals, 13% represented government, and 23% education. One nurse represented private practice, 3 the law/legal system, and 3 volunteer organizations. The largest group (42%) of nurses had both prevention and treatment as their primary work focus, 31.9% had prevention as a primary work focus, and 19.3% had a treatment only focus. Mental Health Counsellors This sample group was represented by a matched response rate of 25%, representing a sample group of 119 respondents. Mental health counsellors were predominantly female (77.8%). As with the above groups, the largest number (35%) were in the 36-45 age range. Eight respondents (6.8%) were under 25 years old, 80 17.9% between 26 and 35 years. The second largest group (29.1 %) were between 46-55 years old, and 11.1% were 55 years or older. Of the respondents who reported their educational level, 60.7% had a graduate degree, 27.4% a college degree, 9.4% had some college education, and 2.6% reported a high school education. As with the nurses, this latter percentage is questionable as a valid response. As with the above groups, the largest percentage (24.3%) represented a hospital. Government organizations were represented by 17.4%, educational organizations by 13.9%, private practice by 5.2%, and law/legal organizations by 6.1%. The remaining respondents represented volunteer organizations (2.6%), business (3.5%) and other (4.3%). Prevention as a primary work focus was reported by 38.5% of respondents, both prevention and treatment by 35.9% of respondents, and treatment only by 22.2% of respondents. Rehabilitation Specialists This sample group provided a matched response rate of 26%, which represented 62 participants. The majority (61.3%) of this group were female. The largest group (37.1%) were between 36-45 years old. The second largest group (33.9%) were between 26-35 years old. One participant was under 26, 17.7% were between 46-55, and 9.7% were over 55 years of age. The majority of rehabilitation specialists (55.7%) reported having a graduate degree. A college degree was reported by 36.1%, some college education by 6.6%. One respondent reported having only a high school degree. This result, as with the nurses and mental health counsellors, is questionable, and could be a coding error. 81 Most rehabilitation specialists within this group represented either the government (40.3%) or worked in hospitals (35.5%). A small number (9.7%) represented the educational system, 1 respondent represented a volunteer organization, and 1 a business organization. The majority (44.3%) had both prevention and treatment as their primary work focus, 39.3% just had prevention, and 9.7% had treatment as their primary work focus. As a summary of the demographic variables, rehabilitation specialists tended to be the youngest group. However, the majority of respondents from each group were between 36-45 years of age. The majority of physicians were males. The other three groups were predominantly female. Mental health counsellors had the largest number of respondents (22.2%) reporting treatment as the primary work focus. Physicians, nurses and rehabilitation specialists had the highest percentage of respondents reporting both treatment/prevention as the primary work focus, while mental health counsellors had the highest percentage of respondents reporting prevention as a primary work focus. All respondents were highly educated, with the largest percentage of physicians, mental health counsellors and rehabilitation specialists reporting a graduate degree. The majority of physicians, nurses and mental health counsellors represented a hospital setting. The largest group of rehabilitation specialists represented a government organization. A small percentage of respondents reported private practice and education as organizations represented. Sample Representativeness Determining sample representativeness is critical to this study for two reasons. First, it adds to the external validity of this research. External validity refers to the 82 generalizabiIity of the results to the larger population (Schumacher & McMillan, 1993). In this case, generalizability of the results is from each health professional group to their larger professional group who were participants from the outset of the national training effort. Secondly, establishing sample representativeness will help make up for the low response rate to the follow-up questionnaires (Fowler, 1993). A descriptive analysis was carried out on the variables used to determine representativeness (Appendix G). Data were taken from unmatched — those who either answered only the pre-questionnaire or who did not provide a participant code to enable them to be matched across time — and matched across time respondents. These variables, described in detail in Chapter Three, included age; gender; education; previous training in substance abuse prevention; organizational type represented; expectation of others; reasons for participation; relevance of workshop to one's job; and preparation for application (Appendix C). Results of the chi-square within groups tests of association are presented in Table 4. Chi-square analysis revealed significant differences between matched and unmatched mental health counsellors on two variables: (a) 'reasons for participation', /2(4, N = 436) = 13.990, p_ = .007, and (b) 'relevance to your job', *2(1, N = 167) = 5.95, fj = .02. All other tests for representativeness indicated no significant differences between the matched and unmatched respondents on any of the variables used in this study. However, as indicated by dashes in Table 4, there were five crosstabulations that cannot be considered valid because of unacceptably low counts in more that 20% of cells. 83 Table 4 Professional Group Physicians Nurses Mental Health Counsellors Rehabilitation Specialists Age NS NS NS NS Gender NS NS NS NS Education — NS NS NS Previous Training NS NS NS NS Organizational Type Represented NS NS NS NS Expectation of Others NS NS NS NS Reasons for Participation NS NS S NS Relevance of Workshop — NS S — Preparation for Application — NS NS — Note. A dash indicates chi-square tests with more than 20% of cells below the expected frequency count. NS = p. > .05; S = p. < .05 In summary, only the matched nurses are representative of the unmatched nurses on all nine variables. The results for this group can be generalized to the larger group of nurses who were in the program from the outset. Matched participants of the other three groups are representative of their unmatched counterparts only on certain variables, limiting the generalizability of their results. 84 Descriptive Statistics of Contextual Influences Descriptive statistics of the contextual influences of the health professional groups include frequency distributions, skew and median scores. The raw data of the frequency distributions are presented in Appendix H and are discussed below. The descriptive results for each health professional group from the follow-up questionnaire, of each contextual variable, are summarized in Table 5. The question was: 'To what extent do the following exist in your organization or community to help you apply your learning?' Respondents rated their answers on the extent of available contextual support on a scale of 1 (not at all) to 5 (substantial). A median score of 3 on this scale will be referred to as a moderate score. Of note in this table is the similarity of median scores of reported availability of contextual influences by all groups. Physicians Among physicians, frequency distributions for the variable 'sufficient resources' were normally distributed, with 26.3% reporting a value of 3. Reported scores for the variables 'organizational support' and 'encouragement from others' were positively skewed (.5 and .212 respectively), with the highest percentage of respondents reporting in the low to mid range. The remaining two variables, 'authority to act and apply learning' and 'opportunity to apply learning' demonstrated negative skews (-.2 and -.43 respectively). The largest percentage of physicians reported opportunity as the most available contextual influence, with the values of 4 and 5 (substantial) receiving the most scores (28.9% for each value). This variable demonstrated the strongest negative skew. 85 Table 5 Median Scores of Reported Availability of Contextual Influences by Health Professional Group Physicians Nurses Mental Health Counsellors Rehabilitation Specialists Contextual Influence Mdn n Mdn n Mdn n Mdn n Encouragement From Others 3.00 38 3.00 118 3.00 118 3.00 61 Organizational Support 3.00 38 3.00 117 3.00 114 3.00 61 Resources 3.00 38 3.00 122 3.00 113 3.00 61 Opportunity to Apply Learning 4.00 38 4.00 120 3.00 116 4.00 61 Authority to Act 3.00 38 3.00 119 3.00 114 3.00 60 or Apply Learning Note. Mdn = median score; n = number of matched respondents in sample group. The median score of available contextual influences reported by physicians was a 3 or moderate for each contextual influence except opportunity to apply, which physicians reported as being available at a score of 4 out of 5 (substantial). Nurses Among nurses, frequency distributions of the variable 'sufficient resources' demonstrated a positive skew of .071. This is a distribution very close to normal. All the other variables demonstrated negative skews as follows: organizational support (-.167); encouragement from others (-.209); authority to act or apply learning (-.236); and opportunity to apply learning (-.270). For each of these four variables, the highest 86 percent of respondents reported the contextual influence in the range of 3 or 4 out of 5 (substantial). The median available contextual influences reported by nurses was 3 or moderate for all influences except opportunity, the median of which nurses reported as a 4 on the scale of 1 (not at all) to 5 (substantial). Mental Health Counsellors Among mental health counsellors, the frequency distribution of the variable 'sufficient resources' was, as with the above groups, very close to normal, with a positive skew of .113. The other four variables had frequency distributions with negative skews as follows: organizational support (-.164); encouragement from others (-.307); authority to act or apply learning (-.415); and opportunity to apply learning (-.259). The highest percentages of scores were consistently in the value range of 3 to 4 out of 5 (substantial). The median of available contextual influences reported by this group was 3 or moderate for each of the five influences. Rehabilitation Specialists Among rehabilitation specialists, the frequency distribution of the variable 'sufficient resources' was, as with the other groups, very close to normal, with a weak positive skew of .137. The four remaining contextual influences all demonstrated negative skews in their frequency distributions as follows: organizational support (-.272); encouragement from others (-.405); authority to act or apply learning (-.231); and opportunity to apply learning (-.673). The latter is the highest negative skew of all the distributions of any of the four groups. 87 The median score of the available contextual influences reported by this group was again moderate or 3 out of a scale of 1 (not at all) to 5 (substantial) for each contextual influence except opportunity to apply learning, which this group rated as being available at a median score of 4 out of 5 (substantial). Descriptive Statistics of Post-Educational Application of Learning The frequency distributions of the dependent variable of post-educational application of learning for each health professional group are presented in Appendix H and are discussed below. These scores were in response to the question on the follow-up questionnaire: 'To what extent did you do the following as a result of this workshop?' Respondents rated the indicator of: 'Made changes in how you do your work or volunteer activities' on a 5-point scale of 1 (not at all) to 5 (extremely). Table 6 summarizes the median scores and frequencies, for each health professional group, for the reported post-educational application of learning. Table 6 Median Scores on Self-Reported Post-Educational Application Measure by Health Professional Group Physicians Nurses Mental Health Counsellors Rehabilitation Specialists Post-educational Application Measure Mdn n Mdn n Mdn n Mdn n Made Changes in How You do Your Work or Volunteer Activities 3.00 38 3.00 118 3.00 118 3.00 62 Note. Mdn = median score; n =» number of matched respondents in sample group. 88 In all cases, data are not normally distributed, all having negative skews as follows: physicians (-.168); nurses (-.308); mental health counsellors (-.281); and rehabilitation specialists (-.091). In each group, the highest percent of participants reported scores of the value 3. The nurses reported the highest degree of application of learning, with 10.2% of this group reporting change in the substantial range. The application scores from this group also demonstrated the strongest negative skew (-.308). The rehabilitation specialists had the highest percentage (19.4%) of participants reporting no application. As Table 6 illustrates, all groups reported the same median of post-educational application of learning, with a score of 3 or moderate out of 5 (extremely). Associations between Contextual Influences and Post-Educational Application of Learning The associations between contextual influences and post-educational application of learning represent the first stage of the inferential analysis. As outlined in Chapter Three, this analysis involved a recoding of values of the independent variables of contextual influences and the dependent variable of post-educational application of learning from five categories to three. Due to inadequate cell counts of crosstabulations relating to physicians in the first stage, results of this group are not reported. Chi-square analysis was carried out between each contextual variable and the dependent variable of behaviour change for the nurses, mental health counsellors and rehabilitation specialists. This involved a total of fifteen crosstabulations using 3 x 3 89 contingency tables (see Appendix I). The results of the crosstabulations and the chi-square tests of association are discussed below, and are summarized in Table 7. Although not significant, the crosstabulations of resources x post-educational application of learning had the highest count in the moderate x moderate cell, indicating a moderate importance of resources to application for this group. Moderate organizational support is associated with moderate application. Nurses responded strongly to the facilitative role of encouragement, authority and opportunity, with high counts in the 5 (substantial) by 5 (substantial) cells. In the first level of analysis of chi-square tests, nurses demonstrated significant associations between organizational support and application of learning, Table 7 Summary of Chi-square Tests of Association between Contextual Influences and Post-Educational Application of Learning by Health Professional Group Contextual Influence Nurses Mental Rehabilitation Nurses Health Counsellors Specialists Encouragement From Others Organizational Support S S NS S S S Resources NS NS Opportunity to Apply Learning Authority to Apply Learning NS S NS S Note. Dash indicates chi-square tests with more than 20% of cells below the expected frequency count. NS = p. > .05; S = p_ < .05 90 ^2(1, N = 114) - 5.206, E = .02; encouragement and application of learning, jf2(2, N = 116)= 10.605, p. = .01; and opportunity and application of learning, X2{2, N = 117) = 8.637, p. = .01. Contextual influences of resources and authority to apply were not significantly associated with post-educational application of learning in this group. Each of these associations had adequate cell counts, adding to the validity of these results. Mental Health Counsellors The crosstabulations for this group also add support to the facilitative role of contextual influences on post-educational application of learning. High counts in the 3 x 3 cells were recorded for each of the five influences except encouragement from others. Cells of 5 (substantial) encouragement from others and 5 (substantial) authority to apply were linked to moderate post-educational application, and 5 (substantial) authority was also linked to 5 (substantial) post-educational application. In the first level of analysis of chi-square tests, mental health counsellors demonstrated two significant associations with application of learning: organizational s u p p o r t e d , N = 110) = 5.525, fj = .02; and opportunity to apply learning, x2(2, N = 111) = 6.036, fj = .05. Contextual influences of encouragement from others, resources and authority to apply learning were not significantly associated with post-educational application of learning for this group. Each of these associations had adequate cell counts, adding to the validity of these results. Rehabilitation Specialists The crosstabulations of this group had high counts in moderate x moderate cells of organizational support and encouragement from others by post-educational 91 application of learning. Even 5 (substantial) reported availability of authority and opportunity are linked to moderate post-educational application of learning. The crosstabulation of available resources had the highest count in the 1 (not at all) x moderate application of learning, possibly indicating that this group is motivated towards moderate application even without available resources. The only results which had adequate cell counts and will therefore be reported for rehabilitation specialists were: encouragement from others, ;r2 (2, N = 61) = 8.198, p. - .02; and organizational support, ^2(1, N = 61) - 6.222, p. - .01. Crosstabulations of opportunity to apply learning, resources and authority to apply learning had inadequate cell counts, making these results invalid. In answer to the question of association between contextual influences and application of learning, neither resources nor authority were significantly associated with post-educational application of learning by any group. Organizational support was significantly associated with application of learning by all three groups, encouragement was significantly associated with application by nurses and rehabilitation specialists; opportunity to apply by nurses and mental health counsellors. Effects of Demographic Variables on Associations between Contextual Influences and Application of Learning This second stage of analysis involved combining the four groups into one sample group (N = 351). The crosstabulations between each contextual influence and post-educational application of learning were then rerun, 'controlling' for variables of age, gender, primary focus of work setting and organizational type 92 represented. The variable of education was not used because the majority of respondents of physicians, mental health counsellors and rehabilitation specialists reported having a graduate degree, and among nurses, 70.9% reported having a college degree or higher. These high percentages in higher education would result in inadequate cell counts in the other categories. The independent and dependent variables remained recoded as per the first level of analysis. This level of analysis produced 45 contingency tables summarized in Table 8. The emphasis of this stage of analysis is on the influences of demographic variables on the associations between contextual influences and post-educational application of learning. Having encouragement from others in the post-educational work environment was a significant factor for application for respondents of both age groups, for female respondents, and for individuals working in hospitals who had both treatment and treatment/prevention work foci. Encouragement from others was not a significant factor for respondents representing law/legal organizations. The crosstabulations of encouragement from others in application controlling for males and for respondents representing educational organizations had inadequate cell counts and are not valid. Having organizational support in the post-educational environment was a significant factor in application for all respondents, except those representing educational organizations. Having opportunity to apply learning was a significant factor in application for respondents of all ages, females, those representing both treatment and 93 Table 8 Effects of Demographic Variables on Associations between Contextual Influence and Application in Health Professionals Demographic Variable Encouragement From Others Organizational Support Opportunity to Apply Authority to Apply Resources Age 36-45 S S S S NS 55 + S S S S NS Cender Male — S — — NS Female S S S S S Primary Work Focus Treatment S S S s NS Treatment and Prevention s S s s NS Organization Represented Law/Legal NS S — S NS Hospital s S s NS NS Educational — NS — — NS Note. Dash indicates chi-square tests with more than 20% of cells below the expected frequency count. NS = p. > .05; S - e < .05 94 treatment/prevention, and those representing hospitals. The crosstabulations controlling for males, law/legal and educational organizations had inadequate cell counts and are invalid. Having authority to act or apply learning in the post-educational work environment was a significant factor in application for respondents of both age groups, for females, for respondents with both treatment and treatment/prevention foci and those who represent law/legal organizations. This contextual influence was not a significant factor in application for respondents from hospitals. The crosstabulations controlling for males and educational organizations had inadequate cell counts and are invalid. Having sufficient resources in the post-educational work environment was not a significant factor in post-educational application for any of the controls except for female respondents. Summary of Results Only nurses were representative of the original group of participants on all nine variables used to determine representativeness. These variables included age, gender, education, previous training in substance abuse prevention, organizational type represented, expectation of others, reasons for participation, and preparation for application. Generalizability of the results of the nurses to the original group is possible. Inadequate cell counts invalidated three crosstabulations of the physicians, and two crosstabulations of the rehabilitation specialists, limiting generalizability of the results on those variables. Mental health counsellors were representative of the 95 original group on seven of the nine variables, again limiting the generalizability of their results to these variables. The demographic results indicate that the matched respondents are a highly educated and likely motivated group who have both prevention and treatment as their primary work focus. The majority of respondents represent hospitals. All groups reported availability of encouragement, organizational support, resources, and authority to apply learning at same median score of 3. Physicians, nurses and rehabilitation specialists reported availability of opportunity to apply learning at a median score of 4. Mental health counsellors reported availability of this contextual influence at a 3. All groups reported the same median score of 3 of post-educational application of learning. However, the skew of the scores indicated nurses reported the most application of learning, and rehabilitation specialists the least. In the first stage of the inferential analysis, inadequate cell counts invalidated all the results of the physicians, and three crosstabulations of the rehabilitation specialists. Neither resources nor authority to apply learning were significantly associated with post-educational application by any group. Organizational support was significantly associated with application of learning by all three groups, encouragement from others with application of learning by nurses and rehabilitation specialists, and opportunity to apply learning with application of learning by nurses and mental health counsellors. Combining the four groups into one, and utilizing the variables of gender, primary work focus and organizational type represented as 'controls' in the second stage of the analysis, produced interesting data. Resources were not associated with 96 application for any group except females. All other associations between contextual influences and application of learning were significant for female participants. The association between organizational support and application of learning was significant for male participants. The influence of male gender on associations between encouragement from others, opportunity to apply learning, and authority to apply learning was invalidated by low cell counts. The influence of the workplace in post-educational application of learning is highlighted by these results. Individuals, particularly those representing hospitals, rated encouragement from others, organizational support, opportunity to apply learning, and authority to apply learning as significantly associated with post-educational application of learning. Resources, on the other hand, were consistently not significantly associated with post-educational application of learning. CHAPTER FIVE: DISCUSSION 97 In this chapter the results of the research will be discussed and interpreted in light of the literature reviewed. The descriptive results concerning post-educational application of learning will be addressed first. The discussion will continue with an analysis and interpretation of the descriptive and inferential results of the contextual influences and the associations between these contextual influences and post-educational application of learning. Lastly, the issue of generalizabiIity of these results will be addressed. Post-Educational Application of Learning Although the participants of this study were highly motivated, received carefully planned in-workshop application practice and some post-workshop support, all four respondent groups reported only moderate behaviour change in their post-educational environments. It appears that factors additional to predisposing influences and the educational program also influence post-educational application of learning. One of these potential factors is the nature of the planned change. In this case, the planned change of the CTS trainings involved developing the capacity of individuals, organizations and communities to plan and carry out substance abuse prevention programs. This far-reaching and complex outcome may have been an unrealistic expectation for this single episode workshop. A longer educational session which extends into the workplace, such as that demonstrated in the study of Campbell & Gammache (1980), may be a better approach for such an extensive proposed change. 98 The expansive coast to coast implementation of these workshops made it economically and logistically impossible to provide participants with facilitative contextual supports in their post-educational environments. The most feasible supports, in terms of reaching participants across the continent, were financial and written resources (Table 1). However, these do not appear to have been successful at facilitating application. Features which would have provided encouragement, such as application assistance, course replication support and follow-up meetings, were in reality impossible to provide to all participants due to enormous geographical barriers. Even more problematic would have been the integration of organizational support and opportunity to apply learning into the work environment through prior negotiation with administration and management. In summary, the moderate reported application of learning by respondents in this study reflects, in part, goals which were not compatible with workshop training, and the limited available contextual supports. These factors add support to the theoretical models upon which this study is based, which recognize the complexity of application of learning. Contextual Influences and Post-Educational Application of Learning The purpose of this study was to determine whether an association exists between the contextual influences in the work environment and the post-educational application of learning of nurses, mental health counsellors and rehabilitation specialists. 99 This section will discuss and analyse the results of the descriptive and inferential analysis in terms of the contextual influences. The five contextual influences under investigation in this study are categorized as either reinforcing or enabling influences to post-educational application of learning. Contextual influences considered reinforcing are organizational support and encouragement from others. These two influences are often considered together in the literature and will be addressed together here as well. Contextual influences considered enabling are resources, authority to act and apply learning, and opportunity to apply learning. Organizational Support and Encouragement from Others The significant association of organizational support with post-educational application of learning is one of the major findings of this study. This is the only contextual influence in this study that was significantly associated with application in each of the three groups, supporting previous literature in nursing (Bolte & Presler, 1983; Campbell & Gammache, 1980; Donovan et al., 1981; Kiener & Hentschel, 1989; Ramprogus, 1989), and adding to the sparse mental health (Woog & Hyman, 1980) and rehabilitation literature. The moderate reported level of organizational support and encouragement from others experienced by all respondents can be accounted for, in part, by the fact that the majority of participants represented hospitals. Hospitals are large, often impersonal organizations with treatment orientations, not likely to be supportive of individuals in their attempts to apply newly learned preventive measures in work settings already stretched for economic and personnel resources. 100 The distribution of the scores of these two influences is interesting as well. Physicians reported the highest frequency of scores in the lower range. Nurses and mental health counsellors rated their available encouragement and organizational support marginally higher. Rehabilitation specialists indicated the highest rating of available encouragement and organizational support. It is possible that since physicians tend to work individually, they may not need or recognize organizational support and encouragement from others to the same degree as the other groups. Rehabilitation specialists, on the opposite extreme, work within departments which tend to be smaller than those of nursing, and possibly more supportive of its members. Encouragement from others was significantly associated with post-educational application of learning for nurses and rehabilitation specialists. This finding both augments previous research in the nursing literature which treated encouragement and organizational support as one contextual influence (Bolte & Presler, 1983; Donovan et al., 1981; Kiener & Hentschel, 1989; Ramprogus, 1989), and adds to the very limited research in the rehabilitation literature. The lack of a significant association between encouragement and post-educational application among mental health counsellors is noteworthy. It could be that practitioners in this group, although hospital based, tend to work individually with clients and are not part of a larger team as are nurses and rehabilitation specialists. Mental health counsellors, therefore, would not experience the same atmosphere of encouragement. Encouragement may be a contextual influence that colleagues lend to each other, as opposed to organizational support that might be available through the larger administration. 101 Resources Although resources are cited as facilitative of application in the literature (Bolte & Presler, 1983; Donovan etal., 1981; Kiener & Hentschel, 1989; Ramprogus, 1989), they were not significantly associated with application of learning in this study. Several factors could explain this. Firstly, the moderate reported level and almost normal distribution of resources by all four groups could be accounted for by the reality of present day economics which influence the availability of financial resources in the health care system. In the literature, resources are described as economic support (insurance rates, reimbursement policy, overhead costs) (Crandall, 1990; Ramprogus, 1989; Tamblyn & Battista, 1993) or time (Bolte & Presler, 1983; Donovan et al., 1981; Martin & Mazmanian, 1991). Similar understanding of this term could have been a factor in this study. If resources were interpreted by participants as financial or economic, rating them as only moderately available could represent the reality of present-day cutbacks in medical services. This could also apply if resources were interpreted as time, as a moderate availability of time could reflect similar cutbacks of personnel, leaving fewer people to do more work. The normal or close to normal distribution of scores of this contextual influence implies consistency of agreement on the degree of availability of this variable, reflecting perhaps the similarity of the economic reality facing these professionals in their work settings. Secondly, the lack of an association of resources with post-educational application of learning could be a methodological issue of question design, causing a lack of understanding of the meaning of resources. Thirdly, this result could simply 102 be evidence that impersonal influences such as resources do not have the same facilitative capacity as personal influences of encouragement from others and organizational support. Opportunity to Apply Learning Despite the lower negative skews of the reported availability of opportunity among nurses and mental health counsellors, opportunity was significantly associated with post-educational application of learning in both of these groups. This supports previous research in the nursing literature (Campbell & Gammache, 1980; Donovan et al., 1981) and adds significantly to the sparse counselling literature. The higher rating of available opportunity to apply learning by physicians, nurses and rehabilitation specialists could be due to two factors. The first factor is the practice situation. The medical literature describes opportunity as closely linked to available cases (Martin & Mazmanian, 1991; Tamblyn & Battista, 1993). Physicians and nurses generally carry extensive case loads which provide potential opportunity to apply learning. The work of rehabilitation specialists, which tends to emphasize regaining health and well-being, would be especially well suited to practitioners coming from a health promotion workshop. The lowest negative skew and reported level of available opportunity was reported by the mental health counsellors, which could reflect the lower case load of mental health practitioners who often spend up to an hour with a client. Opportunity to apply substance abuse prevention concepts may not present themselves as readily in this field due to the complex nature of mental illness. 103 The second factor potentially related to available opportunity is position within a system. Opportunity has been linked to power and authority in the literature (Ottoson, 1995c). Being in a position of power within the health care system could allow practitioners to create opportunity for themselves. The highest rating of opportunity by physicians in this study could reflect their position within the health care system. Authority to Apply Learning Although each of the groups reported moderate availability of authority, this influence was not significantly associated to post-educational application by any group. The role of authority to application is not addressed in the medical literature, and only one study in the nursing literature reported authority as facilitative to application (Donovan at al., 1981). Where does authority to act or apply learning come from? The literature describes authority as related to one's position in the system (Ottoson, 1995c), being given autonomy (Berger, 1977), responsibility (Analoui, 1993), and permission to apply (Donovan et al., 1981). In this study, it can be speculated that nurses have a relatively strong, albeit hardwon, position of authority in the health system. However, the majority of nurses in this sample represented hospitals which are traditionally very hierarchical institutions within which gaining permission to apply new learning could involve much time and effort. Autonomy could be the stronger factor for mental health counsellors (who had the highest negative skew) as they work alone. However, neither of these groups indicated that authority was facilitative to application of learning. These findings imply a need to further understand the meaning of authority, particularly as it relates 104 to the social context (Cervero et al., 1986), and to operationalize this influence differently on future evaluation surveys. Rep resentativen ess Overall, the results of this study may be cautiously generalized to male and female health professionals of all age groups represented, who have some previous training in substance abuse prevention, and who work in hospitals. Due in part to their larger sample size, the results of this study for nurses can be generalized to the original group of nurse participants of the CTS. There are two cautions with representativeness in this study. The first involves response bias. Individuals who answered all three questionnaires of a set of three, as these matched respondents did, demonstrate a higher level of commitment or motivation than those who answer only one or two of a set of three questionnaires (Schumacher & McMillan, 1993). The matched group could be biased towards higher motivation which could account for their willingness to complete all three questionnaires. It is unclear whether this higher motivation influenced the reported application of learning by the participants in this study, as all respondents reported only moderate application of learning. This adds support to the conceptual foundation of this study, that multiple factors such as contextual ones, in addition to predisposing ones of motivation, influence application. The second issue concerns the low response rate, an unusual occurrence with such a highly motivated and educated group (Fowler, 1993), and for continuing education programs in general. The propriety standards adapted for this evaluation survey to ensure anonymity and confidentiality resulted in coding restrictions on 105 consent forms, a large percentage of which were not signed by participants and therefore not linked to follow-up addresses (Appendix E). The resulting small samples and subsequent inadequate cell counts further limit generalizabiIity. Summary Post-educational application of learning remains a complex process. This study indicates that application of learning integrates not only factors of motivation, workshop design and desired outcomes, but also facilitative contextual influences in the post-educational environment. Organizational support is the contextual influence most consistently rated as facilitative of post-educational application in this study, a finding consistent with the available literature. The results of this analysis indicate that even a moderate amount of this contextual influence can play a facilitative role in post-educational learning among nurses, mental health counsellors and rehabilitation specialists. The implications of this significant association to post-educational application of learning are far-reaching to the practice of program planning and evaluation. Organizational support and encouragement from others, with their personal implications, can be considered more facilitative of application of learning than impersonal resources. Resources and authority were not considered facilitative of post-educational application of learning by these study participants. This finding regarding resources is not consistent with the literature, which indicates a need for further investigation of this variable. Authority appears to have multiple understandings in the literature and is difficult to separate conceptually from power, position, responsibility and autonomy. It is unclear whether the association between 106 opportunity and post-educational application of learning is due to the position of the practitioner in the system, to the availability of cases, or to a combination of these factors. These findings of significant and non-significant associations add important knowledge to the continuing education literature of these three health professional groups. The low response rate and the bias towards high motivation are the major methodological weaknesses of this study, limiting generalizability of the results. 107 CHAPTER SIX: IMPLICATIONS FOR PLANNING, EVALUATION, THEORY AND RESEARCH In this final chapter, implications of the results of this study will be presented. The discussion will follow the arrows on the conceptual framework (page 9) back to the educational program, and explore implications for workshop planning, evaluation and research. Implications for Planning Implications for planning will include relevant data generated through the analysis with controls as well as the potential role of the CTS training features designed to enable application of learning (see Table 1). Even though this study focuses on workshops, the results and discussion are relevant to the planning of other programs as well. If post-educational application of learning remains the outcome goal of financially viable continuing education for health professionals, the knowledge generated in this study can make a contribution to improving the effectiveness of workshop planning. According to Caffarella (1994), preparing participants for the transfer of learning to the workplace is an integral part of planning. Choosing strategies that incorporate the contextual influences found to be associated with application in this study, as well as consideration of the additional results generated by the analysis involving demographic variables, can add to the practice base of workshop planners. 108 This study highlights the association between organizational support and post-educational application of learning. Establishing organizational support from all levels of administration and services within the work environment early in the planning process can help to facilitate the transfer of skills to the post-educational work environment. This will require negotiation skills on the part of the planner, as competing interests and agendas of the various stakeholders, including the learners, evaluators and administration, can interfere with the achievement of fiscally and morally responsible planning (Cervero & Wilson, 1994). Organizational support can have different meanings for different professional groups. An example of the facilitative role of organizational support in the nursing literature can be found in the study of Campbell & Gammache (1980) who successfully integrated study modules, preceptors and close contact with supervisors and colleagues into the program design. Follow-up visits were also a key factor in the successful program described by Donovan et al. (1981), which highlighted support and encouragement as key contextual influences. Potentially effective tools used in the CTS trainings included follow-up meetings and guidance, assistance and support for implementation. Encouragement from others can be integrated into workshop design by teaching a group of individuals together, who work together, and who will then be in a position to provide encouragement for application for each other in the post-educational environment (Kemerer, 1991). The results of gender as a factor in the associations indicated that organizational support and encouragement were highly relevant for women and that organizational support was also a significant contextual influence for men. 109 Organizational support for male and female learners could include more communication and interaction. The relevance of encouragement for female learners has been addressed in the feminist learning literature. According to Hayes (1989), women learners tend to exhibit cooperative patterns of interaction, such as including group members in discussion, active listening, mutual elaboration, and supportive interaction. These can all be considered forms of encouragement. Workshop planning for women could include an environment which encourages sharing and interaction. Additional methods of ensuring encouragement between learners include establishment of buddies or teams, and the identification of potential on-site assistance to enable the change process not only to begin but to ensure completion. The powerful influence of organizational support on application of learning for men is supported in the literature. Davis et al. (1995) reported concepts of peer discussion, audit with feedback and reminders as significantly associated with application of learning among physicians. McDonald (1991), whose study included 96% males, found administrative commitment to be the second best predictor of training use after personal attitude. Although these examples define organizational support differently, they support the important role this influence can play in application of learning. Opportunity was rated as facilitative of application by nurses and mental health counsellors. CTS workshop design features that provide opportunity to apply learning are the consideration of potential barriers, the development of action plans and the practicing of implementation techniques in the safe environment of the 110 workshop. Support for such techniques is found in the literature (Davis et al., 1995; Donovan et al., 1981; Martin & Mazmanian, 1991; Tamblyn & Battista, 1993). Taking the working conditions of the learners into consideration in the workshop planning process is also supported by the results of the analysis. It may be that encouragement from others, opportunity to apply learning and organizational support are more available in hospital settings, while authority to apply learning, particularly relevant to women learners, may be more difficult to gain in this traditional, hierarchical setting. Extra negotiation may be required on the part of the planners with the administration in the work site to ensure that the learners, especially if they are women, have permission or authority to apply learning. It may be that authority, which did not show as significant to post-educational application in the first analysis, could be embedded, explicitly and implicitly, in organizational support. The literature supports this by describing authority as giving the learner responsibility, autonomy and freedom to innovate (Analoui, 1993; Berger, 1977). Another aspect of the working conditions which may be relevant in this context is whether the learners work in teams or not. Nurses, who often work in teams, indicated an association between encouragement and post-educational application. Mental health counsellors, who generally work individually with clients, did not report encouragement as associated with post-educational application. All groups were provided with resources, including pre-event reading materials, post-event written information, and funds for mental health counsellors (see Table 1). Despite receiving these resources, this contextual influence was not significantly associated with post-educational application by any of the groups. The 111 most commonly cited resource in the literature, although not statistically shown to be associated with application, is time. This study could not adequately assess the role of resources due to ambiguity of the term on the survey questionnaire. A further implication of this study for planning concerns the expected outcomes. The CTS workshops involved short educational sessions with extensive outcome goals of establishing substance abuse prevention programs. It could be that the moderate reported application of learning scores reflect this unrealistic outcome expectation. Incremental learning application, the style most commonly reported by professionals (Davis et al., 1995; Wergin et al., 1988), may be a more realistic outcome of shorter workshop formats. An important planning implication of this study is the appropriate match between workshop length and intensity, and outcome expectation. Finally, it may be that the shorter nature of workshops requires more concentration of contextual influences in the post-educational work environment to enable application of learning than longer programs would. Several of the examples of successful transfer of learning from the literature are based on longer programs (Campbell & Gammache, 1980; Donovan et al., 1981). It is often difficult for shorter educational interventions such as workshops to be powerful enough to produce a change in the professional's performance (Cervero et al., 1986), making the role of facilitative contextual influences even more important. In order to successfully implement any of these design features of workshops, the planner will have to negotiate with the administration of the organization in question. Understanding the work environment as a highly complex system made up 112 of power and control issues, time and financial constraints and unpredictable circumstances is vital to realistic planning for application of learning. Extra effort will be required on the part of the planners, facilitators, administrators and supervisors, not only to plan for, but also to integrate these features into the workshop design and the post-educational environment. The contextual influences of encouragement, organizational support and opportunity to apply learning, indicated in this study as significantly associated with application of learning, each represent a human or personal element. Implications for Evaluation This study highlights several important considerations for evaluators of workshops. Firstly, the limitations of surveys to measure behaviour change must be recognized. According to Martin & Mazmanian (1991), self-report surveys have been demonstrated to be more reliable in measuring intent to change behaviour than actual behaviour change itself. Evaluators may find that designing a tool which measures intent as well as outcome behaviour, and using a pre/follow-up design, may provide a better indicator of change. Also, behaviour change is difficult to measure. Change may be incremental or all at once. In addition to self-reports, the use of multiple methods such as interviews, observation and chart audits (Cervero, 1984; Turnbull & Holt, 1993), and/or several outcome indicators, may improve validity of the results and provide a clearer picture of workshop outcome. In extensive evaluation studies such as this one, the use of multiple methods is often too costly or simply impossible, however it may be realistic to have several outcome indicators on a questionnaire. 113 Secondly, other design features in the evaluation can be used to add validity to the data. Examples include: (a) ensuring an adequate sample size and improving the response rate through coding procedures which both address confidentiality as well as allow for distribution of the survey instrument; and (b) improving question design to adequately reflect the working environments of each target group. This study validates the evaluation of three of the contextual variables-encouragement from others, organizational support, opportunity to apply learning-as relevant facilitators to post-educational application of learning among health professionals. The variables of resources and authority require further clarification through question design and assessment of the work environment. Although resources are cited as facilitative of application of learning in the literature, their non-significant association with application in this study warrants further investigation. It is unclear whether resources were actually not as available in the post-educational environment of the health professional groups, or whether this variable was inadequately operationalized. Also, evaluation of the variable of authority may prove more realistic if authority is framed as having permission to apply, responsibility or autonomy. This study has identified the potential role of the work setting and gender of participants to application of learning. These factors have not been examined to any depth in the evaluation literature. The variable of age as a factor in post-educational application has been alluded to in the research. Cox & Baker (1991) found a negative correlation between years of work experience and application of learning, although the link between years of experience and age was not clarified. In this study, age was not investigated in terms of its role in application. However, participants of both age 114 categories significantly associated all contextual variables except resources with post-educational application of learning. Implications for Theory and Future Research In addition to the issues raised regarding the evaluation of the contextual influences and application of learning, broader implications for theory and future research are also implied from this study. The Conceptual Framework A conceptual framework is a useful tool to the researcher. Not only does it provide a link to the literature and the theoretical foundations of the research, it offers a guide to the reader to follow the research story. In the following discussion, the strengths and weaknesses of the framework used in this study will be analysed and recommendations made to improve the fit of the framework to the evaluation of post-educational application of learning. One strength of this framework is that it acknowledges the importance and actively seeks evidence of application of learning following a continuing education experience. This is an important variation of Green and Kreuter's (1991) model which addresses behaviour, but not specifically behaviour change or application of learning. Cervero (1988) comes closer to the issue of behaviour change. The model in this study, as does Ottoson's (1995c) model, specifically addresses behaviour change as application of learning in the post-educational environment. Attempting to determine this outcome measure is a vital component of the evaluation process in today's economic climate of limited funds and cutbacks. 115 Secondly, this framework is similar to the models of Cervero and Ottoson, as it also differentiates between the training or educational environment and the work environment. Participants in this study received practice in the educational environment through role plays, action plans and discussion of anticipated barriers to application (see Table 1; Ottoson, 1995b). All these are considered in the literature as valid techniques to encourage opportunity to apply learning (Campbell & Gammache, 1980; Donovan et al., 1991; Kiener & Hentschel, 1989; Martin & Mazmanian, 1991). However, as evidenced by the moderate behaviour change of participants in the post-educational environment, practice in the educational environment does not necessarily mean transfer of skills to the work environment. Thirdly, and building again on the concepts of models by Green & Kreuter and Ottoson, this conceptual framework addresses contextual factors from a positive perspective, as reinforcing or enabling influences, rather than negatively as barriers. Establishing and assessing positive supportive influences in the work environment through the negotiation and evaluation process may facilitate the planning process. These three aspects of the framework support its use as a guide for evaluating application of learning among health professionals, as well as validating the conceptual foundations of the Precede/ Proceed Model (Green & Kreuter, 1991) on which it is based. Several revisions of the conceptual framework are recommended for future research. Firstly, increased emphasis should be placed on how predisposing variables influence application of learning. This study addressed only how contextual influences relate to application. However, the literature strongly suggests that predisposing influences alone, and in combination with contextual influences, play a 116 critical role in application of learning (Berger, 1977; Davis et al., 1992; Martin & Mazmanian, 1991; McDonald, 1991; Ottoson, 1993, 1995c). Future research could analyse if and how predisposing and contextual variables interrelate in their influence on post-educational application of learning. A second recommended revision involves regrouping the contextual variables from two categories as enabling and reinforcing influences to one category as 'facilitative' influences. In the Precede/Proceed Model, reinforcing influences are defined as rewards or punishments following or anticipated as a consequence of a behaviour (Green & Kreuter, 1991). However, in the context of this study, encouragement from others and organizational support are not rewards or punishments. Rather, they are facilitative influences which should be in place and ongoing in the post-educational environment for application to occur and be maintained. Also, enabling influences of resources, authority and opportunity are defined in the Precede/Proceed Model as characteristics which facilitate the initiation of the desired behaviour. Here again, though, the availability of these contextual influences in the worksite should ideally be in place and ongoing if they are to be supportive of application. It may be more useful to move beyond contextual influences as reinforcing and/or enabling to a conception of these influences as sustaining behaviour over time. The term facilitative influences, that encompasses all five contextual variables as initiating and maintaining application of learning over time, may provide a more realistic representation of these factors in the post-educational environment. 117 Thirdly, the conceptual framework in this study addresses but does not analyse the role of the nature of the educational intervention in post-educational application of learning. Cervero (1988) includes the proposed change as a key component in his framework. Ottoson's (1995c) study further supports this interrelated role of the educational intervention as one of many factors in application. Additional research indicates that single episode educational activities have been shown to be more effective than large complex programs (Campbell & Gammache, 1980; Davis et al., 1995; Martin & Mazmanian, 1991). A conceptual framework which accounted for and analysed the nature of the planned change would add significantly to understanding the complexity of post-educational application of learning. Lastly, the conceptual framework in this study concentrated on influences on application, however it did not address the type of behaviour change that occurred. There is evidence to suggest that incremental change over time may be a more realistic outcome than major structural changes in how one does one's work, and that these different types of change could be related to the type of intervention (Campbell & Gammache, 1980; Davis et al., 1995; Martin & Mazmanian, 1991; Wergin et al., 1988). The medical literature indicates that professionals are often already performing at or near optimum levels, and changes tend to be adjustments or modifications of current practice and occur incrementally (Davis et al., 1995; Wergin et al., 1988). Future research could investigate the type of change reported by participants, and try to determine what role the style of intervention has in this change. In summary, this conceptual framework is a reasonable starting point for adult educators looking to evaluate behaviour change. It may, however, be more relevant if 118 the contextual variables were all considered as facilitative of initiation and maintenance of application of learning, and if predisposing factors, the nature of the educational intervention, and the type of the behaviour change were also included in the analysis. Future Research Despite the varied training, traditions and professional philosophies of the four groups, learning was experienced more similarly than expected. This researcher's original plan to treat the health professional groups separately was unsupported by the unexpected similarities arising from the descriptive analysis of the contextual influences and the outcome behaviour. Future analysis of this data could combine the groups from the outset, placing greater emphasis on the varied application contexts and less on the diversity of the groups. Analysis of open-ended questions on the survey could enhance and support the knowledge generated by the closed format of this instrument, as well as allow for clarification of concepts such as resources and authority. According to Abrahamson (1984), support for qualitative evaluation surveys is becoming stronger because they take place in the worksite and can provide context-rich information to augment statistical analysis. The majority of participants in the groups were female. The small number of males in the groups made analysis difficult due to low cell counts. Future research could address the role of these contextual influences on male learners, as well as on the role of age in post-educational application. A clearer conceptualization of resources and authority as contextual influences needs to be developed. Authority in 119 particular, linked as it is to power, responsibility, opportunity, and position in a system, is very complex. This study described the GTS training features designed to facilitate post-educational application of learning. Future research could attempt to determine whether they were (a) considered as facilitative contextual influences by the respondents, and (b) significantly related to the reported post-educational application of learning. This would provide planners with even more direction in the actual planning process. Conclusion Effectiveness of continuing education for health professionals is under scrutiny (Azzaretto, 1992; Cox & Baker, 1981; Davis et al., 1995; Kiener & Hentschel, 1989; Stufflebeam, 1985). The goal of continuing professional education for adult learners has moved beyond participant satisfaction. Evidence of effectiveness in the form of post-educational application of learning is being sought by sponsors, planners, evaluators and learners. In addition to the implications for planning, evaluation, theory and research, several conceptual realizations have been elucidated by this study. The first concerns the research process. The use of data generated by others has had both advantages and disadvantages. Advantages include time and effort saved initially where survey design, administration and data collection would normally have occurred. However, this resulted in several disadvantages. The flaws and weakness of the original evaluation study, resulting from the coding, data collection, and survey design, could not be altered or improved upon. Also, without the involvement normally associated 120 with this process from the outset, gaining a clear understanding of the sample groups, the training system being evaluated, the data, the conceptual foundations of the subject matter and the research tools was more difficult and led to a delay in ownership of the project. The second is highlighted by the results. This study reminds us that post-educational application of learning remains a highly complex process. As the foundational models of this study indicate, many factors interplay to determine outcome behaviours. Contextual influences in the worksite, particularly organizational support, encouragement from others, and opportunity to apply learning can facilitate post-educational application. However, this study has indicated that the work environment, the hierarchy of power, the duration of the workshop, and the gender of the participant also play a role in this process. Of most relevance, however, is the consistently stronger association of organizational support, opportunity to apply learning, and encouragement from others, all representing human or personal contact, over resources. These associations between human or personal support in the post-educational work environment and application of learning are important knowledge for adult educators who design, implement and evaluate workshops. REFERENCES 121 Abrahamson, S. (1984). Evaluation of continuing education in the health professions: State of the art. Evaluation and the Health Professions. 7(1). 3-23. Alexander, M. A. (1990). Evaluation of a training program in breast cancer nursing. lournal of Continuing Education in Nursing. 21(6), 260-266. Analoui, F. (1993). Training and transfer of learning. Great Britain: Avebury. Azzaretto, J. F. (1992). Quality control in continuing professional education: Accountability, effectiveness and regulation. In E. S. Hunt (Ed.), Professional workers as learners. Office of Educational Research and Improvement: U.S. Department of Education Babbie, E. R. (1992). The practice of social research (6th ed.). Belmont, California: Wadsworth. Baldwin, T. T. & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology. 41. 63-105. Beaudry, J. S. (1989). The effectiveness of CME: A quantitative synthesis, lournal of Continuing Education in the Health Professions. 9. 285-307. Berger, M. (1977). Training and the organizational context, lournal of European Industrial Training. 1(2). 7-12. Bolte, I. M. & Presler, E. P. (1983). Continuing education for maternal child health nurses: A means to improve the health care of mothers and children. Public Health Reports. 98(6). 572-584. Caffarella, R. S. (1994). Planning programs for adult learners. San Francisco: Jossey-Bass. Campbell, M. D. & Gammache, R. J. (1980, October). Detecting the links from a continuing professional education program to improved professional practice. Paper presented at the National University Continuing Education Association, Region IV Annual Conference, Ames, Iowa. Cervero, R. M. (1984). Evaluating workshop implementation and outcomes. In T. Sork (Ed.), Designing and implementing effective workshops: Vol. 22. New Directions of Continuing Education (pp. 55-67). 122 Cervero, R. M. & Rottet, S. (1984). Analyzing the effectiveness of continuing professional education: An exploratory study. Adult Education Quarterly. 34(3). 135-146. Cervero, R. M., Rottet, S. & Dimmock, K. H. (1986). Analyzing the effectivness of continuing professional education at the workplace. Adult Education Quarterly. 36(2), 78-85. Cervero, R. M. (1988). Effective continuing education for professionals. San Francisco: Jossey-Bass. Cervero, R. M. & Wilson, A. L. (1994). Planning responsibly for adult education: A guide to negotiating power and interests. San Francisco: Jossey-Bass. Cook, D. P., Heath, A. W. & Martin, M. J. (1990). Evalution of continuing education in family therapy: A preliminary report. American lournal of Family Therapy. 18(1), 106-113. Cox, C. L. & Baker, M. G. (1981). Evaluation: The key to accountability in continuing education, lournal of Continuing Education in Nursing. 12(1). 11-19. Crandall, S. J. S. (1990). The role of CME in changing and learning, lournal of Continuing Education in the Health Professions. 10. 339-348. CSAP. (1995). The CSAP training system: Making prevention work in communities. Prevention Works: CSAP training system. [Briefing packet]. Rockville, MD: Centre for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration, Public Health Service, U.S. Department of Health and Human Services. Darkenwald, G. G. & Merriam, S. B. (1982). Adult education: Foundations of practice. New York: Harper & Row. Davis, D. A., Haynes, R. B., Chambers, L, Neufield, V.R., McKibbon, A. & Tugwell, P. (1984). The impact of CME: A methodological review of the continuing medical education literature. Evaluation and the Health Professions. 7(3). 251-283. Davis, D. A., Thomson, M. A., Oxman, A. D. & Haynes, R. B. (1992). Evidence for the effectiveness of CME: A review of 50 randomized controlled trials, lournal of the American Medical Association. 268(9). 1111-1117. Davis, D. A., Thomson, M. A., Oxman, A. D., & Haynes, R. B. (1995). Changing physician performance: A systematic review of the effect of continuing medical 123 education strategies, lournal of the American Medical Association. 274(9). 700-705. Donovan, M., Wolpert, P., & Yasko, J. (1981). Gaps and contracts. Nursing Outlook. 29(8), 467-471. Eichelberger, R. T. (1989). Disciplined Inquiry: Understanding and Doing Educational Research. New York: Longman Inc. Ford, J. K. (1994). Defining transfer of learning: The meaning is in the answers. Adult Learning. 5(4). 22-30. Fowler, F. J. (1993). Survey research methods. Newbury Park: Sage Publications. Fox, R. D. (1984). Fostering transfer of learning to work environments. In T. Sork (Ed.), Designing and implementing effective workshops: Vol. 22. New Directions for Continuing Education (pp.25-38). Fox, R. D. (1994). Planning continuing education to foster the transfer of learning. Adult Learning. 5(4). 24-25. Garza, D. & Evans, D. W. (1983). Boom or bust: Future of health profession education, lournal of School Health. 53(8). 494-498. Georgenson, D. L. (1982). The problem of transfer calls for partnership. Training and Development lournal. 36(10), 75-78. Gingiss, P. L., Smith, D. W. & Buckner, W. P. (1991). After the conference is over: Future directions for school health programs, lournal of Health Education. 22(5), 325-327. Glass, G. V. & Hopkins, K. D. (1996). Statistical methods in education and psychology (3rd ed.). Needham Heights, MA: Simon & Schuster Co. Glenn, M. K. (1994). Preparing rehabilitation specialists to address the prevention of substance abuse problems. Rehabilitation Counselling. 38(2). 164-179. Gosnell, D. J. (1984). Evaluating continuing nursing education, lournal of Continuing Education in Nursing. 15(1), 9-11. Green, L. W. & Kreuter, M. (1991). Health promotion planning: An educational and environmental approach (2nd ed.). Toronto: Mayfield. Hast, A. S. (1989). Evaluating the impact of education in critical care nursing. Critical Care Nurse. 9(4). 12-15. 124 Hayes, E. (1989). Insights from women's experiences for teaching and learning. In E. Hayes (Ed.), Effective teaching styles: Vol; 43. New Directions for Continuing Education (pp.55-65). Haynes, R. B., Davis, D. A., McKibbon, A. & Tugwell, P. (1984). A critical appraisal of the efficacy of continuing medical education, lournal of the American Medical Association. 251(1). 61-64. Hefferin, E. A. (1987). Trends in the evaluation of nursing inservice education programs, lournal of Nursing Staff Development. 3(1), 28-40. Hopkins, K. D., Glass, G. V. & Hopkins, B. R. (1987). Basic statistics for the behavioural sciences (2nd ed.). Needham Heights, Massachusetts: Allyn and Bacon. Hord, S. M., Rutherford, W. L, Huling-Austin, L, & Hall, G. E. (1987). Taking Charge of Change. Alexandria, VA: Association for Supervision and Curriculum Development. Houle, C. O. (1980). Continuing Learning in the Professions. San Francisco: Jossey-Bass. Kemerer, R. W. (1991). Understanding the application of learning. In T. J. Sork (Ed.), Mistakes made and lessons learned: Overcoming obstacles to successful program planning: Vol. 49. New Directions for Adult and Continuing Education. San Francisco: Jossey-Bass. Kiener, M. E. (1994). After educational programs. Adult Learning. 5(4). 28. Kiener, M. E. & Hentschel, D. (1989). What happens to learning when the workshop is over? lournal of Continuing Education in Nursing, 20(5). 206-211. Knox, A. B. (1985). Evaluating continuing professional education. In R. M. Cervero & C. L. Scanlan (Eds.), Problems and prospects in continuing professional education: Vol. 27. New directions for continuing education. San Francisco: Jossey-Bass. Lawson, K. H. (1979). Philosophical concepts and values in adult education (Rev. ed., pp. 105-107). Milton Keynes: The Open University Press. Litwin, M. S. (1995). How to measure survey reliability and validity. Thousand Oaks: Sage. 125 Madaus, G. F. (1985). Data collection problems in evaluation. In S. Abrahamson (Ed.), Evaluation of continuing education in the health professions. Boston/Dordrecht/Lancaster: Kluwer Nijhoff. Martin, K. O. & Mazmanian, P. (1991). Anticipated and encountered barriers to change in CME: Tools for planning and evaluation, lournal of Continuing Education in the Health Professions. 11. 301-318. Martinoff, J. T. (1985). Contemporary evaluation designs in continuing education. In S. Abrahamson (Ed.)f Evaluation of continuing education in the health professions. Boston/Dordrecht/Lancaster: Kluwer Nijhoff. Mays, M. J. (1984). Assessing the change of practice by physical therapists after a continuing education program. Physical Therapy. 64(1). 50-54. McDonald, R. M. (1991). Assessment of organization context: A missing component in evaluations of training programs. Evaluation and Program Planning. 14. 273-279. Nyquist, J. G. (1985). Data collection techniques in evaluation. In S. Abrahamson (Ed.), Evaluation of continuing education in the health professions. Boston/Dordrecht/Lancaster: Kluwer Nijhoff. Ottoson, J. (1992). Developing a conceptual framework to analyze application of learning from continuing professional education. Proceedings of the 11th Annual Conference of the Canadian Association for the Study of Adult Education. 262-266. Ottoson, J. (1993). Implementation research: Understanding the application of learning following continuing professional education. 34th Annual Adult Education Research Conference AERC. 215-220. Ottoson, J. (1994). The post adult education experience: Multiple understandings of how adults bring back learning to work, home or community settings. Canadian Association for the Study of Adult Education: Theory and Practice 1994 Conference. 314-319. Ottoson, J. (1995a). Reclaiming the concept of application: From social to technological process and back again. Adult Education Quarterly. 46(1). 17-30. Ottoson, J. (1995b). Training evaluation report of 1994 profile, feedback and follow-up data. (Contract #277-91-2004), Pacific Institute for Research and Evaluation and the University of British Columbia, Faculty of Education. Rockville, MD: USHHS, SAMHASA, Centre for Substance Abuse Prevention, Division of Community Prevention and Training, Training and Evaluating Branch. 126 Ottoson, J. (1995c). Use of a conceptual framework to explore multiple influences on the application of learning following a continuing education program. The Canadian lournal for the Study of Education. 9(2). 1-18. Ottoson, J. (1996). Training evaluation report of 1994-5 profile, feedback and follow-up data. (Contract #277-91-2004), Pacific Institute for Research and Evaluation and the University of British Columbia, Faculty of Education. Rockville, MD: USHHS, SAMHASA, Centre for Substance Abuse Prevention, Division of Community Prevention and Training, Training and Evaluating Branch. Ottoson, J. (1997). After the applause: Exploring multiple influences on application following adult education programs. Adult Education Quarterly. 2(47). 92-107. Peden, A. R., Rose, H. & Smith, M. (1992). Transfer of continuing education to practice: Testing an evaluation model, lournal of Continuing Education in Nursing. 23(4). 152-155. Ramprogus, V. K. (1989). Evaluation of a sister/charge nurse development course. Nursing Times. 85(46). 34-37. Richardson, G. E. (1981). A sexual health workshop for health care professionals. Evaluation and the Health Professions. 4(3). 259-274. Rogers, E. M. (1995). Diffusion of innovations (3rd ed.). New York: The Free Press. Sanders, J. R. (1994). The program evaluation standards. 2nd ed. The loint Committee on Standards for Educational Evaluation. Thousand Oaks: Sage. Scanlan, C. L. (1985). Practicing with purpose: Goals of continuing professional education. In R. M. Cervero & C.L Scanlan (Eds.), Problems and Prospects in Continuing Professional Education: Vol. 27. New Directions for Continuing Education. San Francisco: Jossey-Bass. Schumacher, S. & McMillan, J.H. (1993). Research in education: A conceptual introducation.(3rd ed.). United States: Harper Collins College Publishers. Smith, M. F. (1994). Evaluation: Review of the past, preview of the future. Evaluation Practice. 15(3). 215-227. SPSS for Windows: Base System User's Guide, Release 6.0. [Computer program manual]. (1993). Chicago, IL: SPSS Incorporated. SPSS for Windows: Release 7.5. [Computer Program]. 1997. 127 Stufflebeam, D. L. (1985). The relevance of the joint committee Standards for Improving Evaluations in continuing education in the health professions. In S. Abrahamson (Ed.), Evaluation of Continuing Education in the Health Professions. Boston/Dordrecht/Lancaster: Kluwer Nijhoff. Tamblyn, R. & Battista, R. (1993). Changing clinical practice: Which interventions work? lournal of Continuing Education in the Health Professions. 13. 273-288. Turnbull, D. C. & Holt, M. E. (1993). Conceptual frameworks for evaluating continuing education in allied health, lournal of Continuing Education in the Health Professions. 13. 177-186. Waddell, D. L. (1992). The effects of continuing education on nursing practice: A meta-analysis, lournal of Continuing Education in Nursing. 23(4). 164-168. Wergin, J. F., Mazmanian, P. E., Miller, W. W., Papp, K. K. & Williams, W. L. (1988). Assessing the impact of continuing medical education through structured physician dialogue, lournal of Continuing Education in the Health Professions. 8, 147-159. Woog, P. & Hyman, R. B. (1980). Evaluating continuing education: A focus on the client. Evaluation an the Health Professions. 3(2), 171-190. 128 APPENDICES APPENDIX A EXCERPT FROM THE PRECEDE/PROCEED MODEL Phase 5 Phase 4 Administrative Educational and and policy organizational . diagnosis diagnosis Health education components of health promotion program i Policy Regulation Organization Direct com-munication to public, patients, students, employees Indirect communication through su(T training. supervision, consultation, feedback Training; community organization Predisposing factors: Knowledge Attitudes Beliefs Values Perceptions Reinforcing factors: Attitudes and behavior of health and other personnel, peers, parents employers, etc Enabling factors: Availability of resources Accessibility Referrals Rules or laws Skills Behavior (actions) of individuals, groups, or communities Environmental factors Green and Kreuter, 1991 APPENDIX B OTTOSON'S (1995c) ADAPTATION OF THE PRECEDE/PROCEED MODEL 130 c ~ — > P r e d i s p o s i n g fNt sots to Apply fikety to «ppty 131 APPENDIX C CSAP SURVEYS: PARTICIPANT PROFILE, PARTICIPANT FEEDBACK AND PARTICIPANT FOLLOW-UP QUESTIONNAIRES —Evem Code Pafoapam Code C E N T E R FOR S U B S T A N C E A B U S E PREVENTION TRAINING S Y S T E M S (CTS) (INSERT N A M E O F TRAINING HERE) PARTICIPANT P R O F I L E QUESTIONNAIRE 1. To what extent w r a you irrfarmmd about ttie purpoeei of thta workshop? Circle one number. 1 2 3 4 5 8 7 8 9 10 Not at all Comptetery 2. To what extent did each of the following influence you to coma to thia workshop? Circle one number for each item. No QlfllMnCB Extremely trtfluontiai a. Genera) interest 1 2 3 4 5 b. Need to do your job or volunteer activities differently 1 2 3 4 5 c Required by organtYation to participate 1 2 3 4 S d. Confirm wnat you are already doing is Okay 1 2 3 4 5 e. Chance to network with others 1 2 3 4 S f. Other 1 please specify 2 3 4 s 3* To what extent do you expect that this workshop will make a oWwrence in the wv activities? you do your Job or volunteer 1 2 3 4 s a 7 8 9 10 _No dUTefancs Substantial drfferenca — ™ — 1'"""'"—™——™——g flHr Tlim afrHnytntrtTIF a. To what extant have you had pravioua training or aducaflon in aubstanceabua i prevention? • Not at afl • Vary Ode • S a m Q A lot • Extent* • b. To wrat extern have you wor t t^ or votartaered In the field of aubatanoa abuaa pre»anUun? • Not at afl • VeryOUa • S o m a • A lot • Extensive e* Which o^tho foliowinc; i s the pifniaiV'focus of yots*s^bstar>cs aouss acttvtttea? Q Pnvveno^n/education Q Both pfevention and traePTient O Treatn^entAritsiveiitkMi Q Othen _ pleaae specify d. Are you p i t of an otganteed prevemion coalition orpartneranip In your community? • No D Yes 132 e. Which type of organization are you primarily representing at this workshop? Please check the _QS best answer • Government • Education • Not representing an organization O Law enforcement/legal • Religious • Other _ _ _ _ _ _ _ _ O Hospital/health care • Volunteer organization p»a*_«p-eify • Private practice • Business 1 Does anyone, other than yourself, expect something from you as a result of attending this workshop? a NO • Yes: Please explain g. Did anyone accompany you to this workshop? • No, came alone • Yes, as part of a team > • Yes, with others, but not as a team • Other please specify tv. How are your expenses for this workshop being paid? • By you • Shared between you and organization • By supporting organization, • Other _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ government etc. please specify L Your sex: • Male • Female j. Your age: • 25 or less • 26-35 • 36-45 • 48 - 55 • over 55 Jc The highest level of formal education you have reached: • Grade school or some high school • College degree • Completed high school • Graduate school/graduate degree • Some college or a trade/vocational school L Based on the categories below, how would you identify your race or ethnicity? • African Amencan •Caucasian • American Indian or Alaskan Native • Asian Amerkan or PactBc Islander • Hispank^ Latino •Other ~___^p0_y^^ m. Based on your answer to question "4-T above, how do you further define your race or ethnicity? 133 . -Ysnt Ca.< Participant Code CENTER FOR SUBSTANCE ABUSE PREVENTION TRAINING SYSTEMS (CTS) (INSERT NAME OF TRAINING HERE) PARTICIPANT FEEDBACK QUESTIONNAIRE 1. What are likely changes related to substance abuse prevention you may make in your job, organization or community, as a result of this workshop? 2. Overall, how would you rate the usefulness of this workshop? Circle _ _ number. 1 2 3 4 5 6 7 8 9 10 Not at all Completely 3. Overall, how would you rate the following about this workshop? Circle one number for each item. Extremely poor Excellent a. Learning methods or processes 1 2 3 4 5 b. Orgariization, schedule, or flow of activities 1 2 3 4 5 c. Cuiturai competence* of wwfcshop content and process 1 2 3 4 5 d. Cultural competence* of trainers 1 2 3 4 5 e. Other. ' • 1 2 3 please speaty 4 5 « A set of academic and interpersonal skills that allow individuals to increase their understanding and appreciation of cultural differences and similarities within, among, and between groups. 4. To what extent dia tnis worxsnop proviae tne loiiowmg t u i i o e one nuiituei IUI j _ _ item. 134 Not at aU Substantial a. Practical examples 1 2 3 4 5 b. Time for discussion 1 2 3 4 5 c . Practice time 1 2 3 4 5 d. Opportunity to consider what will help or stop you from 1 2 applying your learning 3 4 5 e. Help in preparing to apply teaming 1 2 3 4 5 f. Other 1 2 please specify 3 4 5 5. To what extent would you describe your experiences in this workshop as each item. follows? Circle one number for Not at all - Completely a. Relevant to your job i 2 3 4 5 b. Possible to apply 1 2 3 4 5 C. ' Better than what you were doing 1 2 3 4 5 d. Met your needs 1 2 3 4 5 e. Met the needs of your organizatiori/comrnuniry 1 2 3 4 5 f. Other 1 2 ptease specify 3 4 5 6. Overall, to what extent did the workshop lead to the following? Circle one number for fia e_item. Not at all Substantial a. increased knowledge or skills 1 2 3 4 5 b. Encouraged links with others for support « 1 2 3 4 5 c Changed attitudes or feelings 1 2 3 4 5 d. Confirmed what you were already doing is okay 1 2 3 4 5 .**• - Offered insights into doing your Job or 1 2 volunteer activities differently 3 4 5 t Prepared you to apply toanvng 1 2 3 4 5 g- Other. • 1 2 please specify 3 4 5 7. To what extent are you likely to do the following as a result of this workshop? Circle one number for each item. Not at all Extremely a. SMIBIV Eironrsnon wnn outers I * 3 4 5 b. Make changes in how you cto your work or vottinteer 1 2 activities 3 4 5 c. Use materials from the workshop 1 2 3 4 5 d. Contact others for support to apply learning ' 1 2 3 4 5 e. Actively encourage your org-rrtteation to appiy ideas 1 2 from the workshop 3 4 5 f. Get more training or information on topic 1 2 3 4 S 0- Increase substance abuse prevention activities 1 2 3 5 h. Other 1 2 oiease soecrrv 3 4 5 135 8. To what extent do you expect this workshop will make a drrforence in the way y ou do y our job or volunteer activities? 1 2 . 3 4 5 6 7 8 9 10 No difference Substantial difference 9. To what extent do the following exist in your organization or community to help you apply your learning? Circle one number for each item. Not at all Substantial a. Sufficient resources 1 2 3 4 5 b. Encouragement from others 1 2 3 4 5 c Opportunity to appty teaming 1 2 3 4 5 d. Authority to act or apply learning 1 2 3 4 5 e. Support for making changes suggested during this 1 2 workshop 3 4 5 f. Other * '1 2 please specify 3 4 5 10. To what extent do you feel able to apply your learning from this workshop to your job or volunteer activities? Circle one number. 1 2 3 4 5 6 7 8 9 10 Not at all Completely fmal:6727/94 136 CENTER FOR SUBSTANCE ABUSE PREVENTION TRAININC SYSTEMS TRAINING COURSE ON PREVENTION OF A T O D PROBLEMS - FOLLOW-UP QUESTIONNAIRE MENTAL HEALTH SPECIALISTS - AUGUSTA - OCTOBER 2,1995 Please take a few moments to complete this form and return it as soon as possible. This form is a self-mailer, so enve-lopes or postage are not needed. You r cooperation is sincerely appreciated. Please fill i n the last four digits o f y o n r home phone number at the t ime of t ra in ing ____ T h i s code is very impor tan t It a l lows us to treat part ic ipant responces anonymously wh i le matching answers across t ime. 1 . "What three changes related to substance abuse prevention have you made in your job, organization or community as a result o f the workshop? 2. Overall, how would you rate the usefulness o f the workshop? Circle one number. 1 2 3 4 5 6 7 Not at all 10 Completely • 3 . To what extent would you describe your experiences in the workshop as follows? Circ le one number per item. a. Relevant to your job Not at all 1 2 3 4 Completely 5 b. Possible to apply 1 2 3 4 5 c. Better man pre-workshop prevention practice 1 2 3 4 5 d. Met your needs 1 2 3 4 5 e. Met the needs of your organization/community 1 2 3 4 5 X Other please specify 1 2 3 4 5 4 . Overall, to what extent d id the workshop lead to the following? Circle one number for each item. Not at all Substantial a . Increased knowledge or sk i l ls 1 2 3 4 5 b. Encouraged l inks w i th others for support 1 2 3 4 5 c. Modi f ied attitudes or feelings 1 2 3 4 5 d . Confirmed what you were already doing 1 2 3 4 5 e. Offered insight into doing your job or volunteer activities differently 1 2 3 4 5 t Prepared you to apply learning 1 2 3 4 5 a. Other 1 2 3 4 5 please specify 137 5 . To "what extent did you do the fol lowing as a result o f this workshop? Circle one number for each item. Not at all Extremely a. Shared information with others 1 2 3 4 5 b. Made changes in how you do your work or volunteer activities 1 2 3 4 5 c. Used materials from the workshop 1 • 2 3 4 5 d. Contacted others for support to apply learning 1 2 3 4 5 e. Act ively encouraged your organization to apply ideas from the workshop 1 2 3 4 5 f. Got more training or information on topic 1 2 3 4 5 g. Increased substance abuse prevention activities 1 2 3 4 5 h. Other: 1 2 3 4 5 please specify 6 . To what extent did the workshop make a difference i n the way you do your job or volunteer activities? 1 2 3 4 5 6 7 8 9 10 No Substantial Difference Difference 7 - To what extent do the fol lowing exist in your organization or community to help you apply your learning? Circle one number for each item. Not at all Substantial a. Sufficient resources 1 2 3 4 5 b. Encouragement f rom others 1 2 3 4 5 c. Opportunity to apply learning 1 2 3 4 5 d. Authority to act or apply learning 1 2 3 4 5 e. Support for making changes suggested during mis workshop 1 2 3 4 5 £ Other: 1 2 3 4 5 please specify 8 . To what extent do you feel able to apply your learning from the workshop to your job or volunteer activities? Circle one number. 1 2 3 4 5 6 7 8 9 10 Not at all Completely 9 . Please circle the number that most closely reflects your opinion: Strongly Disagree Strongly Agree a. The pre-work materials were valuable 1 2 3 4 5 b. 1 think health professionals w i l l actually get involved in ATOD prevention efforts and activities 1 2 3 4 5 c. I attended this course because C E U s were offered 1 2 3 4 5 Please fold this sheet i n half two times, so that the shaded portion al the top of this page is showing. Tape (dont staple) the sheet at the top o f the return address section and mai l , f=fr Thank you for taking the l ime to complete this survey. 138 APPENDIX D CERVERO'S (1988) MODEL Proposed Change Individual Professional Social Behaviour System Change CPE Program 139 APPENDIX E DATA LOG OF PHYSICIANS, NURSES, MENTAL HEALTH COUNSELLORS, AND REHABILITATION SPECIALISTS *CSA»y^ / .LOG 1 Date Location EC p PRE Dl POST rot2 FU D T O i:'*iV';irarer^_.r',.— c • EC Evmntooda P PRE Prt-OitTfTKimthTf*** D1 Pra Qutsuui instras with vaUd ID nuimMi 1 1 POST P-rt-Hin TtkJlWislS s— | | »12 Matcnad Pra- and PosMkMiUoniialrsB" 1 ' FU Foflow-«a> Ojiasuuniiaaaa*** I I B I Z * Matchad Pca^ Poat^ and roftow up Qiiasoonnahaa"** 1 1 1 sum .'Ir'i.'lt MstctMd Pre* And Poet*43UMUOIUUIres/ P n Q i m H u m n l i - i wttti v*—kl K) nun_Mr i ' i r t i i ' i l Mttctwd Pres Post-^ aiidFo<km-upCHi**tk>nntJra(/Pr*-Cu«sUOT i C COftMnt fofiM i 1 i i 1 I I I I 140 CSAi J LocaMoa j EC .riWwsPECaUttlrs-4*29/94 Poebte 4 20 20 20 It 11 4 4 90.0 20.0 20.0 16 5/B/B4 KJamathFIa 5 43 31 3S 31 34 t t 89.5 15.8 14.0 23 5/17/94 Phoanot a 21 It 10 It 14 2 0 73.7 0.0 98 0 6/3/94 Ram 7 52 13 83 to 41 1T 11 00.8 30.2 32.7 46 7/2&V4 S(. TnomM 8 27 27 21 2t It 7 3 90.5 14.3 25.9 7 6/11/94 6/1/94 SI Croix 9 45 41 48 43 40 13 10 87.0 21.7 26.9 19 6/8/94 OUK* 10 28 26 21 23 11 3 8 71.4 0.0 10.7 6 9/9/94 Tampa 11 35 33 33 34 27 7 4 81.8 111 20.0 ••• 9/12/94 Taooma 12 45 4* 43 41 31 It t 88.4 18.6 33.3 *~ 10/21/94 13 35 38 31 34 31 11 7 68.8 20.0 31.4 *** 11/9/94 Provftdtno§ 14 82 M 88 12 41 23 14 62.8 24.1 37.1 mm 11/9/94 Greensboro 15 28 M 38 21 24 12 10 92.3 38.6 46.2 mm 4/24/95 R Leonard Woo 18 28 26 20 33 22 ' 4 0 84.6 0.0 14.3 57205 ShrtapM 17 22 22 22 It M 2 2 77.3 9.1 9.1 6/9/95 Concord 18 33 33 33 It 21 t 0 64.6 0.0 0.0 7/17/95 Sicnvntnto 1ft 68 It It M 84 t 0 91.6 0.0 0.0 7/21/95 Sim Arrfonto 20 32 32 32 27 27 0 0 64.4 0.0 0.0 7/26*6 21 38 38 34 33 31 0 0 91.2 0.0 0.0 6/4/06 Orond Junction 22 23 21 It It 11 0 0 7Z0 0.0 0.0 1tt3 Subtotal 146 71 71 M 84 0 0 60.1 68 64 IBM Subtotal 431 428 413 3t7 3tt 120 12 MJ 274 1IM Subtotal 236 M 231 211 itr t 2 •6.3 •J It LUliJ MENTAL HEALTH 6PECIAU8T - tOC 1AL WORKERS 1 1 IIS Oat* J 4/25/94 Location 1 EC P 55 PRE mm •i Bl •1 POST #Iawlg(vlW8Jw 3t 1012 IPS 31 f.u ii 8)123 S§' t mug D12/B1 76.5] D123/B1 17.6 20.0 c 42 6/1304 Washington 3 23 21 21 11 It 4 4 76.2 1 19.0 17.4 17 •••1 27 • 8 0 0 t 0 MM tntv/oi 64 1N4 Subtotal n 72 72 18 et 11 13 764 "T8.1" 164 143 I I 3 s s 23 S 3 MS i i a a a I I 8 3 8 8 B 8 5J 2 8 8 8 8 a 8 * S • 8 3 3 888 88 3 88 38828 8 3S 88882S8S8S 88 e e 2 88 1 2 8 i i 8 S8S8S-2 8828 23 888 88 8 3 8 8 ii • — ! I I I ! -ft 3 b ^ • 5 »5 83$8$B2ft 88282886288 88882888288 88882882828 53* I 2888 8 2 2 1 1 1 "as ui 4V, OE ro-1 jog 0-B " f t 3 StyS.i w*te~ 145. — . — — —, -P«q« 2 j 12 Summary or rnathodoiogy and procadura. (Must b« typawrman in tfaa spaca). NOTE If your study invofraa dacasUan, you must atao cwnptoa paoa 7, tha *0«oapttan •arm*. UBC w i l l receive processed, matched, anonymous, raw quantitative and qualitative data front PPF questionnaires collected at a l l CTS trainings. The data (not questionnaires) w i l l be transferred to UBC via the electronic Management Information System (MIS) supported by CSAP. A data base w i l l be established at UBC with CTS data. Aggregate data w i l l be used to answer research questions about training effectiveness and influences on outcomes of training including educational, demographic, and contextual. Comparisons w i l l be made within types of training, e.g., health professional or state agency, and across multiple types of training. The data coming to UBC w i l l come front PPF questionnaires collected at a l l CTS trainings by CTS contractors. The pre and post questionnaires and consent forms w i l l be sent to PIRE's offices i n Bethesda, Maryland. The data w i l l be entered, primary analysis w i l l be conducted, and data w i l l be stored by PIRE. PIHE w i l l send follow-up questionnaires to participants who have signed consent forms. DESCHIPTION OF POPULATION 1U How tniny subjects,w*8 bo uootf? There may be up to 1000 sub j ects, depending on the number of CTS training events held'and the attendance at those trainings. There is no control. A l l CTS contractors who provide training are required by CSAP to participate in the CIS evaluation aa a condition of their contract. Everyone who attends a CTS training event will be recruited by contractors to complete pre, post, and follow-up questionnaires. 146 ; r r " 13 What suotacu m l be aadudad from parnooabon? [ 16 How an the subjects bang recruit aer? (If irabal comrnct is by laOar or V a lauurtmant notice is to bs posted, attach a copy.) NOTE Oast UBC pafcy otecouraoa* MW contact by haaphana. Howavar, surveys wMeh usa random cSgrt dbing may ba alowad. V your study twotoae such contact, you must also cornpiate paga a. tba Tataphrtna Coatest form*. Participants w i l l be recruited to the PPF questionnaires by a standardized ' presentation made by a designated trainer at the beginning of each training event. At the end of the event, participants w i l l be invited to complete the consent form for participation i n the follow-up questionnaire ! i j 17 a a control oroup is ifflpofved. and V tharr satacaoti and/or raouitmartt cflfats from the above, provide dstais. C/Ar ~OJECT DETAILS . 16 Wham wS the p-ujait be cortductsdT (room or area) j The research w i l l be conducted at the Adult Education Research Centre at ! UBC. • . 19 who wa irtwty conduct the study awe -mm am their TuaMretienr; The secondary data analysis w i l l be conducted by Judith M Ottoson, Bd.O., Assistant Professor in Adult Education. A doctoral and master's student w i l l be supervised in assisting j— with data analysis. I 20 wa the group ot subjects have any problem gmrg inlormad consent on their own behalf? Considar prryncai or mental | oondttort, age, avtguaoe, or othar bsiiaas. i i ! 21 V the aubjecte are not corapetant to gne fasy Wormed consant. who wfl oonaant on thajr behalf? 22 What la leowa about the data and banafct of the proposed raiaaiUr? Oo you haw aptaaanat esaraara rn rha issue? Data w i l l be useful to CSAP and other agencies i n decision making about training as a means of achieving policy ends. 147 . BW.» 23 What discomfort or incapacity are the subjects Bcdy to widura as t result of the eajwwnentaJ procedures? t)o«ie. 24 U menatary compensation ia to be offered the subjects, provide detafls of amounta and payment schedule*. i 23 Hew much ame wl a subject have to dedicate ta the project? It w i l l take the subjects approximately a total of 30-40 minutes total to complete a l l three questionnaires. 2ft How much ttme wil a member of the control group (S any) have to dedteate to the project? tiff* DATA j 27 Who w» have irraai to the data? At UBC, those with access to the data include j Judith Ottoson and two supervised students. At PIRE, the PPF coordinator. Dr. Dionne Jones, a data entry person, and research assistant w i l l have access to the data. At training events, contractors collecting questionnaires w i l l have access to the data. at How wl eonadertnaaty of the data be iiadraetwdT UBC w i l l only have access to matched, coded data and w i l l not have access to participant contact information. Participants w i l l complete pre-coded pre and post questionnaires anoymously. Contact information and codes w i l l only be linked those j participants who sign consent forms. i \ 29 Wheat «OT tfM pSaVM faf aaSaaaffW U N Of tfW dbaaal (bflyORd «tMt OjaaMfeaMd Itl ttatal pfOUMMlift HflMf 4nt)d WtWft ttrfl ths) datine eM j daaaoyed? The matched anonymous data held by UBC w i l l not be destroyed. Questionnaires w i l l be kept i n a locked storage room at PIRE's. offices i n Bethesda, Maryland. The questionnaires w i l l be destroyed within fiv e years after the end of the CTS, according to U.S. ethic review guidelines. ! 30 W any data which Jdondtoa inolviduala be avaiabte to paraona or aqandai outtido the University? N ^ , <^a. * a i 149 APPENDIX G Frequencies of Variables Used to Determine Sample Representativeness by Health Professional Group Physicians Nurses Mental Health Counsellors Rehabilitation Specialists UM n = M n = UM n = M n = UM n = M n = UM n = M n = Age 36-45 49 20 189 63 174 62 108 44 55 + 23 16 170 53 142 47 64 17 Gender Male 39 23 26 3 102 26 53 24 Female 35 15 348 114 244 91 125 38 Education Some college or less 2 0 107 34 58 14 22 5 College degree 3 2 181 48 116 32 58 22 Graduate Degree 70 36 86 35 170 71 100 34 Previous Training in Substance Abuse Prevention 1 - not at all 7 3 34 9 23 3 13 5 2 15 6 78 30 55 19 34 12 3 33 18 172 45 136 51 92 33 4 14 7 67 33 90 31 25 7 5 - extensive 6 4 25 5 50 13 16 5 Organization Represented Law/Legal 12 8 58 19 94 27 61 25 Hospital 26 14 159 53 63 28 55 22 Educational 7 3 62 23 39 16 19 6 Appendix G, Continued: 150 Physicians Nurses Mental Health Counsellors Rehabilitation Specialists UM M UM M UM M UM M n = n = n = n = n = n = n = n = Expectation of Others Yes 23 14 135 40 132 50 101 41 No 52 24 227 72 205 65 74 19 Reasons for Participation 1 - no influence 6 2 30 11 34 4 12 5 2 14 6 31 9 38 7 17 6 3 21 18 91 30 80 20 32 19 4 24 6 110 38 93 48 60 16 5 - extremely influential 7 4 78 24 83 29 45 13 Relevant to your Job Low 2 14 19 50 14 54 4 28 High 5 24 34 70 38 61 4 33 Preparation for Application Low 3 23 16 50 18 53 5 32 High 4 15 38 68 32 63 3 27 Note. UM =• number of unmatched respondents; M = number of matched respondents APPENDIX H FREQUENCY DISTRIBUTIONS OF INDEPENDENT AND DEPENDENT VARIABLES BY HEALTH PROFESSIONAL GROUP 152 APPENDIX H: PHYSICIANS Table H1 Frequency Distribution of Variable 'Sufficient Resources' Value Frequency Percent 1 - not at all 5 13.2 2 9 23.7 3 10 26.3 4 9 23.7 5 - substantial 5 13.2 TOTAL 38 100 Table H2 Frequency Distribution of Variable 'Organizational Support for Changes Implied by Workshop' Value Frequency Percent 1 - not at all - — 2 9 23.7 3 17 44.7 4 7 18.4 5 - substantial 5 13.2 TOTAL 38 100 Physicians, Continued: Table H3 Frequency Distribution of Variable 'Encouragement from Others' 153 Value Frequency Percent 1 - not at all — — 2 9 23.7 3 13 34.2 4 10 26.3 5 - substantial 6 15.8 TOTAL 38 100 Table H4 Frequency Distribution of Variable 'Authority to Act and Apply Learning' Value Frequency Percent 1 - not at all 2 5.3 2 7 18.4 3 11 28.9 4 9 23.7 5 - substantial 9 23.7 TOTAL 38 100 Physicians, Continued: Table H5 Frequency Distribution of Variable 'Opportunity to Act and Apply Learning' 154 Value Frequency Percent 1 - not at all 1 2.6 2 5 13.2 3 10 26.3 4 11 28.9 5 - substantial 11 28.9 TOTAL 38 100 Table H6 Frequency Distribution of Variable 'Changed how you do your Work or Volunteer Activities' Value Frequency Percent 1 - not at all 4 10.8 2 8 21.6 3 13 35.1 4 10 27 5 - extremely 2 5.4 TOTAL 38 100 155 APPENDIX H: NURSES Table H7 Frequency Distribution of Variable 'Sufficient Resources' Value Frequency Percent 1 - not at all 5 4.3 2 26 22.2 3 41 35.0 4 24 20.5 5 - substantial 21 17.9 TOTAL 122 100 Table H8 Frequency Distribution of Variable 'Organizational Support for Changes Implied by the Workshop' Value Frequency Percent 1 - not at all 8 6.8 2 25 21.4 3 36 30.8 4 36 30.8 5 - substantial 12 10.3 TOTAL 122 100 Nurses, Continued: Table H9 Frequency Distribution of Variable 'Encouragement from Others' 156 Value Frequency Percent 1 - not at all 10 8.5 2 20 16.9 3 37 31.4 4 31 26.3 5 - substantial 20 16.9 TOTAL 122 100 Table H10 Frequency Distribution of Variable 'Authority to Act and Apply Learning' Value Frequency Percent 1 - not at all 3 2.5 2 18 15.1 3 42 35.3 4 43 36.1 5 - substantial 13 10.9 TOTAL 122 100 Nurses, Continued: Table H11 Frequency Distribution of Variable 'Opportunity to Apply Learning' 157 Value Frequency Percent 1 - not at all 3 2.5 2 14 11.7 3 41 34.2 4 40 33.3 5 - substantial 22 18.3 TOTAL 122 100 Table H12 Frequency Distribution of Variable 'Changed how you do your Work or Volunteer Activities' Value Frequency Percent 1 - not at all 12 10.2 2 16 13.6 3 44 37.3 4 34 28.8 5 - extremely 12 10.2 TOTAL 122 100 158 APPENDIX H: MENTAL HEALTH COUNSELLORS Table HI3 Frequency Distribution of Variable 'Sufficient Resources' Value Frequency Percent 1 - not at all 4 3.5 2 22 19.5 3 49 43.4 4 25 22.1 5 - substantial 13 11.5 TOTAL 118 100 Table HI4 Frequency Distribution of Variable 'Organizational Support for Changes Implied by the Workshop' Value Frequency Percent 1 - not at all 11 9.6 2 22 19.3 3 40 35.1 4 31 27.2 5 - substantial 10 8.8 TOTAL 118 100 Mental Health Counsellors, Continued: Table HI5 Frequency Distribution of Variable 'Encouragement from Others' 159 Value Frequency Percent 1 - not at all 8 7.1 2 19 16.8 3 34 30.1 4 37 32.7 5 - substantial 15 13.3 TOTAL 118 100 Table H16 Frequency Distribution of Variable 'Authority to Act and Apply Learning' Value Frequency Percent 1 - not at all 9 7.9 2 12 10.5 3 40 35.1 4 36 31.6 5 - substantial 17 14.9 TOTAL 118 100 Mental Health Counsellors, Continued: Table H17 Frequency Distribution of Variable 'Opportunity to Apply Learning' 160 Value Frequency Percent 1 - not at all 4 3.4 2 13 11.2 3 45 38.8 4 38 32.8 5 - substantial 16 13.8 TOTAL 118 100 Table H18 Frequency Distribution of Variable 'ChanRed how you do your Work or Volunteer Activities' Value Frequency Percent 1 - not at all 16 14.2 2 19 16.8 3 35 31.0 4 35 31.0 5 - extremely 8 7.1 TOTAL 11.8 100 APPENDIX H: REHABILITATION SPECIALISTS 161 Table H19 Frequency Distribution of Variable 'Sufficient Resources' Value Frequency Percent 1 - not at all 2 3.3 2 17 27.9 3 20 32.8 4 16 26.2 5 - substantial 6 9.8 TOTAL 62 100 Table H20 Frequency Distribution of Variable 'Organizational Support for Changes Implied by the Workshop' Value Frequency Percent 1 - not at all 4 6.6 2 7 11.5 3 26 42.6 4 17 27.9 5 - substantial 7 11.5 TOTAL 62 100 Rehabilitation Specialists, Continued: Table H21 Frequency Distribution of Variable 'Encouragement from Others' 162 Value Frequency Percent 1 - not at all 4 6.6 2 8 13.1 3 22 36.1 4 21 34.4 5 - substantial 6 9.8 TOTAL 62 100 Table H22 Frequency Distribution of Variable 'Authority to Act and Apply Learning' Value Frequency Percent 1 - not at all 3 5.0 2 13 21.7 3 16 26.7 4 21 35.0 5 - substantial 7 11.7 TOTAL 62 100 Rehabilitation Specialists, Continued: Table H23 Frequency Distribution of Variable 'Opportunity to Apply Learning' 163 Value Frequency Percent 1 - not at all 4 6.6 2 7 11.5 3 14 23.0 4 26 42.6 5 - substantial 10 16.4 TOTAL 62 100 Table H24 Frequency Distribution of Variable 'Changed how you do your work or Volunteer Activities' Value Frequency Percent 1 - not at all 12 19.4 2 17 27.4 3 18 29.0 4 15 24.2 5 - extremely — — TOTAL 62 100 164 APPENDIX I CROSSTABULATIONS OF CONTEXTUAL INFLUENCES BY POST-EDUCATIONAL APPLICATION OF LEARNING BY NURSES, MENTAL HEALTH COUNSELLORS AND REHABILITATION SPECIALISTS 165 APPENDIX I: NURSES Crosstabulation — Sufficient Resources X Changed How You Do Things CHANGE SUFFICIENT LOW MODERATE HIGH TOTAL RESOURCES LOW — 19 12 31 MODERATE - 27 12 39 HIGH — 24 20 44 X2 = 1.882 a = .39 TOTAL 114 Crosstabulation — Organizational Support X Changed How You Do Things CHANGE ORGANIZATIONAL LOW MODERATE HIGH TOTAL SUPPORT LOW — — — — MODERATE - 47 21 68 HIGH — 22 24 46 X2 = 5.206 a = -02 TOTAL 114 Crosstabulation — Encouragement From Others X Changed How You Do Things CHANGE ENCOURAGEMENT LOW MODERATE HIGH TOTAL FROM OTHERS LOW — 23 6 29 MODERATE - 25 11 36 HIGH — 23 28 5 1 _ X2 = 10.605 a = .005 TOTAL 116 Note: Dash indicates no data in this cell. 166 APPENDIX I: NURSES (Continued) Crosstabulation — Authority to Act and Apply Learning X Changed How You Do Things CHANGE AUTHORITY TO ACT AND APPLY LEARNING LOW MODERATE HIGH TOTAL LOW — 16 5 21 MODERATE — 27 14 41 HIGH — . 28 27 55 X2 - 4.778 a = .09 TOTAL 117 Crosstabulation — ODDortunitv to ADDIV Learning X Changed How You Do Things CHANGE OPPORTUNITY TO APPLY LEARNING LOW MODERATE HIGH TOTAL LOW — 14 3 17 MODERATE — 28 12 40 HIGH — 29 31 60 X2 = 8.637 a - .01 TOTAL 117 Note: Dash indicates no data in this cell. 167 APPENDIX I: MENTAL HEALTH COUNSELLORS Crosstabulation — Sufficient Resources X Changed How You Do Things CHANGE SUFFICIENT LOW MODERATE HIGH TOTAL RESOURCES LOW — 17 8 25 MODERATE — 29 17 46 HIGH - 20 18 3 8 _ X2 - 1.698 a = .43 TOTAL 109 Crosstabulation — Organizational Support X Changed How you Do Things CHANGE ORGANIZATIONAL LOW MODERATE HIGH TOTAL SUPPORT LOW MODERATE HIGH 49 22 71 18 2J 39 TOTAL 110 X2 - 5.525 a - .02 Crosstabulation — Encouragement From Others X Changed How You Do Things CHANGE ENCOURAGEMENT LOW MODERATE HIGH TOTAL FROM OTHERS LOW - 20 6 26 MODERATE - 20 14 34 HIGH — 27 23 50 X2 = 3.865 a - .15 TOTAL 110 Note: Dash indicates no data in this cell. 168 APPENDIX I: MENTAL HEALTH COUNSELLORS (Continued) Crosstabulation — Authority to Act and Apply Learning X Changed How You Do Things CHANGE AUTHORITY TO ACT AND APPLY LEARNING LOW MODERATE HIGH TOTAL LOW — 15 5 20 MODERATE — 27 13 40 HIGH — 25 25 50 X2 = 4.897 a - -09 TOTAL 110 Crosstabulation — ODDortunitv to ADDIV Learning X Changed How You Do Things CHANGE OPPORTUNITY TO APPLY LEARNING LOW MODERATE HIGH TOTAL LOW — 11 5 16 MODERATE — 32 12 44 HIGH — 25 26 51 X2 - 6.036 a - -05 TOTAL 111 Note: Dash indicates no data in this cell. 169 APPENDIX I: REHABILITATION SPECIALISTS Crosstabulation — Sufficient Resources X Change How You Do Things CHANGE SUFFICIENT LOW MODERATE HIGH TOTAL RESOURCES LOW — 19 0 19 MODERATE — 13 7 20 HIGH - 14 8 22 X2 = 9.009 a = .01 TOTAL 61 Crosstabulation — Organizational SuDDort X Changed How You Do Things CHANGE ORGANIZATIONAL LOW SUPPORT MODERATE HIGH TOTAL LOW — — — — MODERATE - 32 5 37 HIGH - 14 10 24 X2 = 6.222 a - .01 TOTAL 61 Crosstabulation — Encouragement from Others X Changed How vou Do Things CHANGE ENCOURAGEMENT LOW FROM OTHERS MODERATE HIGH TOTAL LOW - 12 0 12 MODERATE - 18 4 22 HIGH - 16 11 27 X2 = 8.198 a = .02 TOTAL 61 Note: Dash indicates no data in this cell. 170 APPENDIX I: REHABILITATION SPECIALISTS (Continued) Crosstabulation — Authority to Act or ADDIV X Changed How You Do Things CHANGE AUTHORITY TO ACT OR APPLY LEARNING LOW MODERATE HIGH TOTAL LOW — 15 1 16 MODERATE — 12 4 16 HIGH — 18 10 28 X2 =4.714 a - - io TOTAL 60 Crosstabulation — ODDortunitv to ADDIV Learning X Changed How You Do Things CHANGE OPPORTUNITY TO APPLY LEARNING LOW MODERATE HIGH TOTAL LOW — 11 0 11 MODERATE — 13 1 14 HIGH — 22 14 36 / = 9.854 a - .007 TOTAL 61 Note: Dash indicates no data in this cell. 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0055445/manifest

Comment

Related Items